Method and apparatus for restoring image

Abstract

The present disclosure relates to a method and apparatus for restoring an image. An image restoration method performed by a computing device according to one embodiment may include the steps of acquiring an incomplete image, estimating the size of a hole included in the incomplete image, comparing the size of the hole with a predetermined reference value, determining a restoration method for the incomplete image based on the comparison result, and restoring the incomplete image by applying the restoration method. According to the embodiments, an optimal image restoration method is applied according to the state of the incomplete image, thereby enabling the incomplete image to be restored quickly and precisely.

Description

Technology Field

[0001] The present disclosure relates to a method and apparatus for restoring an image. Background Technology

[0003] Image restoration technology is a technique that restores and enhances damaged or lost images. It is primarily utilized in the fields of computer vision and image processing, and can be usefully applied in various other applications.

[0004] Recently, technologies for acquiring images of patient lesions or pathologies and restoring these images to high quality are being widely researched in the medical field. For example, when collecting pathological images of body tissues using scanning methods in a medical setting, it is necessary to restore the acquired images to higher resolution for more precise tissue examination and subsequent accurate diagnosis.

[0005] Conventionally, scanning was performed over long periods at low frame rates to acquire high-resolution images. However, prolonged scanning can hinder physicians' rapid diagnosis and surgical decisions in medical settings. Therefore, technology is required to restore images acquired through scanning more quickly and precisely. The problem to be solved

[0007] The purpose of this specification is to provide an image restoration method and apparatus capable of obtaining a high-resolution image by rapidly and precisely restoring an incomplete image.

[0008] The purpose of this specification is not limited to the purposes mentioned above, and other purposes and advantages of this specification not mentioned will be more clearly understood by the embodiments of this specification described below. Furthermore, the purposes and advantages of this specification may be realized by the components and combinations thereof described in the claims. means of solving the problem

[0010] An image restoration method performed by a computing device according to one embodiment may include the steps of acquiring an incomplete image, estimating the size of a hole included in the incomplete image, comparing the size of the hole with a predetermined reference value, determining a restoration method for the incomplete image based on the comparison result, and restoring the incomplete image by applying the restoration method.

[0011] In one embodiment, the step of estimating the size of a hole included in the incomplete image may include the step of performing interpolation on the incomplete image, the step of counting the number of pixel value restorations when the interpolation is performed, and the step of estimating the size of the hole based on the number of pixel value restorations.

[0012] In one embodiment, the interpolation may be the nearest neighbor interpolation method.

[0013] In one embodiment, the step of determining a restoration method for the incomplete image according to the comparison result may include determining a machine learning-based inpainting technique as the restoration method if the size of the hole is greater than or equal to the reference value, and determining a pixel value-based image interpolation method as the restoration method if the size of the hole is less than the reference value.

[0014] In one embodiment, the pixel value-based image interpolation method may be any one of nearest neighbor interpolation, bilinear interpolation, trilinear interpolation, bicubic interpolation, and spline interpolation.

[0015] A computing device for image restoration according to another embodiment may include at least one processor and memory. The at least one processor may acquire an incomplete image, estimate the size of a hole included in the incomplete image, compare the size of the hole with a predetermined reference value, determine a restoration method for the incomplete image based on the comparison result, and restore the incomplete image by applying the restoration method.

[0016] In another embodiment, the at least one processor performs interpolation on the incomplete image, counts the number of pixel value restorations when the interpolation is performed, and can estimate the size of the hole based on the number of pixel value restorations.

[0017] In another embodiment, the interpolation may be the nearest neighbor interpolation method.

[0018] In another embodiment, the at least one processor may determine a machine learning-based inpainting technique as the restoration method if the size of the hole is greater than or equal to the reference value, and determine a pixel value-based image interpolation method as the restoration method if the size of the hole is less than the reference value.

[0019] In another embodiment, the pixel value-based image interpolation method may be any one of nearest neighbor interpolation, bilinear interpolation, trilinear interpolation, bicubic interpolation, and spline interpolation.

[0020] A computer program stored on a computer-readable storage medium according to another embodiment, wherein the computer program, when executed on one or more processors, may perform the following steps for restoring an image. The operations may include acquiring an incomplete image, estimating the size of a hole included in the incomplete image, comparing the size of the hole with a predetermined reference value, determining a method for restoring the incomplete image based on the comparison result, and restoring the incomplete image by applying the restoration method. Effects of the invention

[0022] According to the embodiments, an incomplete image can be restored quickly and precisely by applying an optimal image restoration method depending on the state of the incomplete image.

[0023] In particular, when the embodiments according to the present disclosure are applied in a medical setting, rapid and precise reconstruction of images acquired through high-frame-rate scanning based on a micro-probe becomes possible. Consequently, faster and more accurate diagnosis becomes possible in the medical setting, the time required for surgery is reduced, and the success rate of surgery can be increased. Brief explanation of the drawing

[0025] FIG. 1 is a block diagram of a computing device for image restoration according to one embodiment. FIG. 2 is a schematic diagram showing a network function according to one embodiment. FIG. 3 is a flowchart of an image restoration method according to one embodiment. Figure 4 shows an incomplete image containing a hole having a size smaller than a predetermined threshold value and an image restored by applying different restoration methods to the incomplete image. Figure 5 shows an incomplete image containing a hole having a size greater than a predetermined threshold value and a restored image when different restoration methods are applied to the incomplete image. FIG. 6 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented. Specific details for implementing the invention

[0026] Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to provide an understanding of the present disclosure. However, it is evident that these embodiments can be practiced without such specific descriptions.

[0027] As used in this disclosure, terms such as “component,” “module,” “system,” etc. refer to computer-related entities, hardware, firmware, software, combinations of software and hardware, or executions of software. For example, a component may be, but is not limited to, a procedure executed on a processor, a processor, an object, an execution thread, a program, and / or a computer. For example, both an application executed on a computing device and the computing device itself may be a component. One or more components may reside within a processor and / or an execution thread. A component may be localized within a single computer. A component may be distributed among two or more computers. Additionally, these components may be executed from various computer-readable media having various data structures stored therein. Components may communicate through local and / or remote processes, for example, according to signals having one or more data packets (e.g., data from a component interacting with another component in a local system or distributed system, and / or data transmitted through signals to other systems and networks such as the Internet).

[0028] Furthermore, the term "or" is intended to mean an implicit "or" rather than an exclusive "or." That is, unless otherwise specified or evident from the context, "X uses A or B" is intended to mean one of the natural implicit substitutions. In other words, if X uses A; if X uses B; or if X uses both A and B, "X uses A or B" may apply to any of these cases. Additionally, the term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the enumerated related items.

[0029] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that such features and / or components are present. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, components and / or groups thereof. Furthermore, unless otherwise specified or clearly evident from the context to indicate a singular form, the singular in this specification and claims should generally be interpreted to mean “one or more.”

[0030] And, the term "at least one of A or B" should be interpreted to mean "a case including only A," "a case including only B," or "a combination of A and B."

[0031] Those skilled in the art should recognize that the various exemplary logical blocks, configurations, modules, circuits, means, logics, and algorithmic steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. To clearly exemplify the interchangeability of hardware and software, various exemplary components, blocks, configurations, means, logics, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented in hardware or software depends on the specific application and design constraints imposed on the overall system. Skilled technicians may implement the described functionality in various ways for each specific application. However, such decisions regarding implementation should not be construed as moving out of the scope of this disclosure.

[0032] The description of the presented embodiments is provided so that those skilled in the art may use or practice the presented embodiments. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the scope of the invention according to the present disclosure is not limited to the embodiments presented herein. The scope of the invention according to the present disclosure should be interpreted in the broadest possible sense consistent with the principles and novel features presented herein.

[0033] In the present disclosure, network functions, artificial neural networks, and neural networks may be used interchangeably.

[0034] FIG. 1 is a block diagram of a computing device for image restoration according to one embodiment.

[0035] The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

[0036] The computing device (100) may include a processor (110), memory (130), and a network unit (150).

[0037] The processor (110) may be composed of one or more cores and may include processors for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in memory (130) and perform data processing for machine learning according to one embodiment of the present disclosure. According to one embodiment of the present disclosure, the processor (110) may perform calculations for learning a neural network. The processor (110) may perform calculations for learning a neural network, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process the learning of a network function. For example, a CPU and a GPGPU can work together to process the learning of a network function and data classification using the network function. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices can be used together to process the learning of a network function and data classification using the network function. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

[0038] In one embodiment, the processor (110) can acquire an incomplete image. The processor (110) can estimate the size of a hole included in the incomplete image. The processor (110) can compare the size of the hole with a predetermined reference value. The processor (110) can determine a method for restoring the incomplete image based on the comparison result. The processor (110) can restore the incomplete image by applying the determined restoration method.

[0039] According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

[0040] According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may operate in conjunction with web storage that performs the storage function of the memory (130) on the internet. The description of the memory described above is merely an example and the present disclosure is not limited thereto.

[0041] A network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed ​​DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

[0042] In addition, the network unit (150) presented in this specification may use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division MultiAccess), SC-FDMA (Single Carrier-FDMA), and other systems.

[0043] In the present disclosure, the network unit (150) can be configured regardless of the mode of communication, such as wired and wireless, and can be configured as various communication networks, such as a Local Area Network (LAN), a Personal Area Network (PAN), and a Wide Area Network (WAN). Additionally, the communication network may be the known World Wide Web (WWW). The network unit (150) may also utilize wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.

[0044] The technologies described in this specification can be used not only in the networks mentioned above but also in other networks.

[0045] FIG. 2 is a schematic diagram showing a network function according to one embodiment of the present disclosure.

[0046] Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node. The nodes (or neurons) constituting the neural networks may be interconnected by one or more links.

[0047] In a neural network, one or more nodes connected via links can form relative input and output node relationships. The concepts of input and output nodes are relative; any node in an output node relationship with respect to one node may be in an input node relationship with respect to another node, and vice versa. As described above, the input node versus output node relationship can be generated based on links. One or more output nodes may be connected to a single input node via links, and vice versa.

[0048] In a relationship between an input node and an output node connected through a single link, the value of the output node's data can be determined based on the data input to the input node. Here, the link interconnecting the input node and the output node may have a weight. The weight may be variable and may be varied by a user or an algorithm to enable the neural network to perform the desired function. For example, if one or more input nodes are interconnected to a single output node by respective links, the output node's value can be determined based on the values ​​input to the input nodes connected to the output node and the weights set on the links corresponding to each input node.

[0049] As described above, a neural network consists of one or more nodes interconnected through one or more links, forming input-output node relationships within the network. The characteristics of a neural network can be determined by the number of nodes and links within the network, the relationships between the nodes and links, and the weight values ​​assigned to each link. For example, if two neural networks exist with the same number of nodes and links but different weight values ​​for the links, the two neural networks can be recognized as being different from each other.

[0050] A neural network can be composed of a set of one or more nodes. A subset of the nodes constituting a neural network can form a layer. Some of the nodes constituting a neural network can form a layer based on their distances from an initial input node. For example, a set of nodes with a distance of n from an initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links that must be traversed to reach that node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network may be defined in a way different from that described above. For example, a layer of nodes may be defined by its distance from a final output node.

[0051] Initial input nodes may refer to one or more nodes within a neural network to which data is directly input without passing through links in their relationships with other nodes. Alternatively, in terms of link-based relationships between nodes within the neural network, they may refer to nodes that do not have other input nodes connected by links. Similarly, final output nodes may refer to one or more nodes within a neural network that do not have output nodes in their relationships with other nodes. Furthermore, hidden nodes may refer to nodes constituting the neural network that are neither initial input nodes nor final output nodes.

[0052] A neural network according to one embodiment of the present disclosure may have the number of nodes in the input layer equal to the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. Additionally, a neural network according to another embodiment of the present disclosure may have the number of nodes in the input layer less than the number of nodes in the output layer, and may be a neural network in which the number of nodes decreases as it progresses from the input layer to the hidden layer. Additionally, a neural network according to yet another embodiment of the present disclosure may have the number of nodes in the input layer greater than the number of nodes in the output layer, and may be a neural network in which the number of nodes increases as it progresses from the input layer to the hidden layer. A neural network according to yet another embodiment of the present disclosure may be a neural network in which the above-described neural networks are combined.

[0053] A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a deep neural network allows for the identification of the latent structures of data. That is, it is possible to identify the latent structures of photos, text, videos, voice, and music (e.g., what objects are present in a photo, what the content and emotions of a text are, what the content and emotions of a voice are, etc.). Deep neural networks may include convolutional neural networks (CNN), recurrent neural networks (RNN), autoencoders, Generative Adversarial Networks (GAN), restricted Boltzmann machines (RBM), deep belief networks (DBN), Q networks, U networks, Siamese networks, Generative Adversarial Networks (GAN), etc. The description of deep neural networks described above is merely illustrative and the present disclosure is not limited thereto.

[0054] In one embodiment of the present disclosure, the network function may include an autoencoder. The autoencoder may be a type of artificial neural network for outputting output data similar to the input data. The autoencoder may include at least one hidden layer, and an odd number of hidden layers may be placed between the input and output layers. The number of nodes in each layer may be reduced from the number of nodes in the input layer to an intermediate layer called a bottleneck layer (encoding), and then expanded symmetrically from the bottleneck layer to the output layer (symmetrically with respect to the input layer). The autoencoder may perform non-linear dimensionality reduction. The number of input and output layers may correspond to the dimension after preprocessing of the input data. In the autoencoder structure, the number of nodes in the hidden layer included in the encoder may have a structure in which the number of nodes decreases as it moves away from the input layer. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be transmitted, so it may be maintained at a certain number or more (e.g., more than half of the input layer).

[0055] Neural networks can be trained in at least one of supervised learning, unsupervised learning, semisupervised learning, or reinforcement learning. The training of a neural network may be the process of applying knowledge to the neural network to perform a specific action.

[0056] Neural networks can be trained to minimize the error in their output. The training process involves repeatedly inputting training data into the network, calculating the error between the network's output and the target for the training data, and updating the weights of each node by backpropagating the error from the output layer to the input layer in a direction that reduces the error. In the case of supervised learning, training data is used where the correct answer is labeled for each individual training data point (i.e., labeled training data), whereas in the case of unsupervised learning, the correct answer may not be labeled for each training data point. For instance, in the case of supervised learning for data classification, the training data may consist of data where each training data point is labeled with a category. The labeled training data is input into the neural network, and the error can be calculated by comparing the network's output (category) with the labels of the training data. As another example, in the case of unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network's output. The calculated error is backpropagated in the neural network (i.e., from the output layer to the input layer), and through backpropagation, the connection weights of each node in each layer of the neural network can be updated. The amount of change in the connection weights of each node being updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of training to quickly achieve a certain level of performance and increase efficiency, while a low learning rate can be used in the later stages to improve accuracy.

[0057] In the training of neural networks, the training data is generally a subset of the real-world data (i.e., the data intended to be processed by the trained neural network). Consequently, a training cycle may exist where errors decrease on the training data but increase on the real-world data. Overfitting is a phenomenon where the network learns excessively on the training data, leading to increased errors on the real-world data. For example, a neural network trained on yellow cats might fail to recognize cats when seeing anything other than yellow, which can be considered a form of overfitting. Overfitting can act as a cause for increased errors in machine learning algorithms. Various optimization methods can be used to prevent this overfitting. To prevent overfitting, methods such as increasing the training data, regularization, dropout (which disables some nodes in the network during training), and the use of batch normalization layers can be applied.

[0058] In the present disclosure, the “reinforcement learning control model” is an artificial neural network model having the configuration of the neural network described above with reference to FIG. 2, and may be an artificial neural network model trained by a reinforcement learning method.

[0059] Reinforcement learning is a type of learning method in which an artificial neural network model is trained based on the rewards generated for actions selected by the model, so that the model can determine better actions based on input states. Reinforcement learning can be understood as a "learning method through trial and error" in that rewards are given for decisions (i.e., actions). In the reinforcement learning process, the reward given to the artificial neural network model may be a reward accumulated from the results of multiple actions. Through learning, reinforcement learning generates an artificial neural network model that considers various states and rewards associated with actions to maximize the reward itself or the return (the sum of rewards). In this disclosure, the "reinforcement learning control model" is the entity that determines actions and may be used interchangeably with the term "Agent." In the technical field related to reinforcement learning, the terms "environment (Env)" or "environment model" are used to refer to a "model that returns results considering the agent's actions." An environment model may be a model that returns output data (e.g., state information) for given input data (e.g., control information). An environment model may be a model where the model structure for reaching the output from the input or the causal relationship between the input and output data is unknown. The agent and the environment can operate by exchanging data.

[0060] FIG. 3 is a flowchart of an image restoration method according to one embodiment.

[0061] The image restoration method illustrated in Fig. 3 can be performed by the computing device (100) described above.

[0062] The computing device (100) can acquire an incomplete image (301). The incomplete image may be an image generated by an image generating device (e.g., a medical scanner).

[0063] A computing device (100) can estimate the size of a hole included in an incomplete image (302). In the present disclosure, a hole may refer to a pixel or a set of pixels in which a pixel value is missing among the pixels included in an incomplete image. In the present disclosure, an incomplete image is an image containing a hole, and restoring an incomplete image may mean restoring the pixel value of each pixel included in the hole included in the incomplete image.

[0064] In one embodiment, the computing device (100) may perform interpolation on the incomplete image to estimate the size of a hole included in the incomplete image. In one embodiment, the computing device (100) may apply nearest neighbor interpolation to the incomplete image to estimate the size of a hole included in the incomplete image.

[0065] According to the nearest neighbor interpolation method, if there is a non-missing pixel among the adjacent pixels of each pixel whose pixel value is missing, the pixel value of that pixel is interpolated to restore the pixel value of the pixel whose pixel value is missing. If all the pixel values ​​of adjacent pixels of a specific pixel whose pixel value is missing are missing, the pixel value is not restored, and an attempt is made to restore the pixel value of another pixel. For example, if the pixel values ​​of the 2nd, 3rd, and 4th pixels among the 1st, 2nd, 3rd, 4th, and 5th pixels that are adjacent to each other are missing, in the first restoration, the pixel values ​​of the 2nd and 4th pixels are restored to be the same as the pixel values ​​of the 1st and 5th pixels, respectively, and in the second restoration, the pixel value of the 3rd pixel is restored as the median value of the 2nd and 4th pixels.

[0066] Ultimately, according to nearest neighbor interpolation, pixel value restoration is performed simultaneously for all holes contained within the image, and the process continues until the pixel values ​​of the pixels contained in the hole with the most missing pixels—that is, the largest hole—are restored. Therefore, the number of pixel value restorations counted when nearest neighbor interpolation is performed can be proportional to the size of the largest hole contained within the image.

[0067] The computing device (100) can count the number of pixel value restorations performed when applying the nearest neighbor interpolation method to an incomplete image. The computing device (100) can estimate the size of a hole included in the incomplete image based on the number of pixel value restorations performed when applying the nearest neighbor interpolation method to the incomplete image. For example, the computing device (100) can convert the number of pixel value restorations into the size of a hole included in the incomplete image by referring to a pre-stored table in which the number of pixel value restorations and the corresponding hole sizes are recorded. In another embodiment, the computing device (100) may use the counted number of pixel value restorations as is as the size of the hole.

[0068] As such, when estimating the size of holes included in incomplete images, applying a fast nearest-neighbor interpolation method can speed up both the hole size estimation and image restoration speeds.

[0069] The computing device (100) can compare the size of the estimated hole with a predetermined reference value (303). Here, the reference value is a value that can be set differently depending on the embodiment. The computing device (100) can determine a method for restoring the incomplete image based on the comparison result (304).

[0070] In one embodiment, the computing device (100) may determine a machine learning-based inpainting technique as a restoration method if the size of the estimated hole is greater than or equal to a predetermined threshold value. The computing device (100) may determine any one of a number of known machine learning-based inpainting techniques (e.g., DeepFillv2, LaMa(Large Mask), GIN, AOT-GAN, Palette, etc.) as a restoration method for an incomplete image.

[0071] In one embodiment, the computing device (100) may include the step of determining a pixel value-based image interpolation method as a restoration method if the size of the estimated hole is less than a predetermined reference value. In one embodiment, the pixel value-based image interpolation method may be any one of nearest neighbor interpolation, bilinear interpolation, trilinear interpolation, bicubic interpolation, and spline interpolation, but the types of pixel value-based image interpolation methods are not limited thereto.

[0072] When a restoration method is determined, the computing device (100) can restore the incomplete image by applying the determined restoration method (305).

[0073] The steps mentioned in the foregoing description may be further subdivided into additional steps or combined into fewer steps, depending on the embodiment of the present disclosure. Additionally, some steps may be omitted as necessary, and the order of the steps may be changed.

[0074] FIG. 4 shows an incomplete image containing a hole having a size smaller than a predetermined threshold value and an image restored by applying different restoration methods to the incomplete image. FIG. 5 also shows an incomplete image containing a hole having a size larger than a predetermined threshold value and an image restored when different restoration methods are applied to the incomplete image.

[0075] First, FIG. 4 illustrates an incomplete image (20) generated by synthesizing a pattern image containing holes having a size smaller than a predetermined threshold value onto a high-resolution original image. FIG. 4 also illustrates an image (21) restored by applying a machine learning-based inpainting technique (LaMa) to the incomplete image (20) and an image (22) restored by applying nearest neighbor interpolation to the incomplete image (20). As shown in FIG. 4, when the holes included in the incomplete image (20) are smaller than a predetermined threshold value, the image (21) restored by applying the machine learning-based inpainting technique (LaMa) is blurry and lacks sharpness, whereas the image (22) restored by applying nearest neighbor interpolation is confirmed to have relatively higher quality compared to the image (21) restored by applying the machine learning-based inpainting technique (LaMa).

[0076] Next, FIG. 5 illustrates an incomplete image (30) generated by synthesizing a pattern image containing holes having a size greater than or equal to a predetermined threshold value onto a high-resolution original image. FIG. 5 also illustrates an image (31) restored by applying a machine learning-based inpainting technique (LaMa) to the incomplete image (30) and an image (32) restored by applying nearest neighbor interpolation to the incomplete image (30). As shown in FIG. 5, when the holes included in the incomplete image (30) are greater than or equal to a predetermined threshold value, the image (32) restored by applying nearest neighbor interpolation has a lower image quality because the area near the boundary where pixels meet is sharp, whereas the image (31) restored by applying a machine learning-based inpainting technique (LaMa) is confirmed to have relatively higher quality compared to the image (32) restored by applying nearest neighbor interpolation.

[0077] As such, it is confirmed that the restoration method suitable for image restoration varies depending on the size of the hole included in the incomplete image. According to an embodiment of the present disclosure, the size of the hole included in the incomplete image is rapidly estimated, and accordingly, an image restoration method suitable for restoring the incomplete image is determined and image restoration is performed, so that a high-quality restored image can be obtained quickly.

[0078] According to one embodiment of the present disclosure, a computer-readable medium storing a data structure is disclosed.

[0079] A data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. A data structure can refer to the organization of data for solving specific problems (e.g., data retrieval, data storage, data modification in the shortest possible time). A data structure may also be defined by physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements may include connections between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., a permanent storage device). Specifically, a data structure may include sets of data, relationships between data, and functions or instructions applicable to the data. Through an effectively designed data structure, a computing device can perform operations while minimizing the use of the computing device's resources. Specifically, through an effectively designed data structure, a computing device can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and retrieval.

[0080] Data structures can be classified into linear and non-linear data structures based on their form. A linear data structure is one where only one piece of data is connected to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a set of data that maintains an internal order. Lists can include linked lists. A linked list is a data structure where data is connected in a line, with each piece of data possessing a pointer. In a linked list, the pointer can contain information regarding the connection to the next or previous data. Depending on its form, a linked list can be represented as a singly linked list, a doubly linked list, or a circular linked list. A stack is a data arrangement structure that allows for restricted access to data. A stack can be a linear data structure where data can be processed (e.g., insertion or deletion) only at one end of the structure. Data stored in a stack can be a Last-In, First-Out (LIFO) data structure, meaning that the later an item is entered, the sooner it is retrieved. A queue is a data sequence structure that allows for limited access to data; unlike a stack, it can be a FIFO (First in First Out) data structure where data stored later is retrieved later. A deque is a data structure that can process data at both ends.

[0081] Non-linear data structures can be structures where multiple data are connected after a single piece of data. Non-linear data structures can include graph data structures. Graph data structures can be defined by vertices and edges, and edges can include lines connecting two different vertices. Graph data structures can include tree data structures. A tree data structure can be a data structure where there is only one path connecting two different vertices among the multiple vertices included in the tree. In other words, it can be a data structure that does not form loops in a graph data structure.

[0082] FIG. 6 is a brief and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

[0083] Although the present disclosure has been described as generally being implementable by a computing device, a person skilled in the art will be well aware that the present disclosure may be implemented in combination with computer-executable instructions and / or other program modules that can be executed on one or more computers and / or as a combination of hardware and software.

[0084] Generally, a program module includes routines, programs, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Additionally, a person skilled in the art will be well aware that the method of the present disclosure may be implemented in other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. (each of which may be connected to and operated with one or more associated devices).

[0085] The embodiments described in this disclosure may also be implemented in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

[0086] Computers typically include various computer-readable media. Any medium accessible by a computer may be a computer-readable medium, and such computer-readable media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media. By example, not a limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media include volatile and non-volatile media, transitory and non-transitory media, and removable and non-removable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD (digital video disk) or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or any other media that can be accessed by a computer and used to store desired information.

[0087] Computer-readable transmission media typically include all information transmission media that implement computer-readable instructions, data structures, program modules, or other data, etc., on a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal in which one or more of the characteristics of the signal are set or modified to encode information within the signal. By example, not limiting, computer-readable transmission media include wired media, such as wired networks or direct-wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the media described above is also considered to be within the scope of computer-readable transmission media.

[0088] An exemplary environment (1100) for implementing various aspects of the present disclosure, including a computer (1102), is shown, wherein the computer (1102) includes a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including system memory (1106) (but not limited thereto), to the processing unit (1104). The processing unit (1104) may be any processor among various commercial processors. Dual processor and other multiprocessor architectures may also be used as the processing unit (1104).

[0089] The system bus (1108) may be any of several types of bus structures that can be additionally interconnected to a local bus using any of the memory bus, peripheral bus, and various commercial bus architectures. System memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). The basic input / output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, EEPROM, etc., and this BIOS includes basic routines that help transfer information between components within the computer (1102) at times such as during startup. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

[0090] The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA)—this internal hard disk drive (1114) may also be configured for external use within a suitable chassis (not shown)—a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from a CD-ROM disk (1122) or reading from or writing to other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may each be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128). The interface (1124) for implementing an external drive includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

[0091] These drives and associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, etc. In the case of a computer (1102), the drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will know that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc., may also be used in exemplary operating environments and that any of these media may contain computer-executable instructions for performing the methods of the present disclosure.

[0092] A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or part of the operating system, application, module and / or data may also be cached in RAM (1112). It will be well known that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

[0093] The user can input commands and information into the computer (1102) through one or more wired / wireless input devices, such as a pointing device like a keyboard (1138) and a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, etc. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) connected to the system bus (1108), but may also be connected via other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, etc.

[0094] A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface such as a video adapter (1146). In addition to the monitor (1144), the computer generally includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

[0095] The computer (1102) may operate in a networked environment using a logical connection to one or more remote computers, such as remote computer(s) (1148), via wired and / or wireless communication. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), but for brevity, only the memory storage device (1150) is illustrated. The illustrated logical connection includes a wired / wireless connection to a local area network (LAN) (1152) and / or a larger network, e.g., a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a global computer network, e.g., the Internet.

[0096] When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and / or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communication to the LAN (1152), and the LAN (1152) may also include a wireless access point installed therein to communicate with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communication computing device on the WAN (1154), or have other means to establish communication through the WAN (1154), such as through the Internet. The modem (1158), which may be an internal or external and a wired or wireless device, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, the program modules described for the computer (1102) or parts thereof may be stored in a remote memory / storage device (1150). It will be well known that the illustrated network connection is exemplary and that other means of establishing a communication link between computers may be used.

[0097] The computer (1102) operates to communicate with any wireless device or object that is deployed and operated via wireless communication, for example, a printer, scanner, desktop and / or portable computer, PDA (portable data assistant), communication satellite, any equipment or place associated with a wireless detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or simply ad hoc communication between at least two devices.

[0098] Wi-Fi (Wireless Fidelity) enables connectivity to the Internet and other sources without wires. Wi-Fi is a wireless technology, similar to a cell phone, that allows devices, such as computers, to transmit and receive data indoors and outdoors—that is, anywhere within the coverage area of ​​a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band).

[0099] Those skilled in the art of the present disclosure will understand that information and signals may be represented using any various different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0100] Those skilled in the art will understand that the various exemplary logic blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented by electronic hardware, various forms of programs or design code (referred to herein as software for convenience), or a combination of all such. To clearly illustrate this interoperability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in relation to their functions. Whether such functions are implemented as hardware or software depends on the design constraints imposed on the specific application and the overall system. Those skilled in the art may implement the functions described in various ways for each specific application, but such implementation decisions should not be interpreted as being outside the scope of the disclosure.

[0101] The various embodiments presented herein may be implemented as methods, devices, or articles using standard programming and / or engineering techniques. The term "article" includes a computer program, a carrier, or a medium accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical discs (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Additionally, the various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

[0102] It should be understood that the specific order or hierarchy of steps in the presented processes is an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but do not imply being limited to the specific order or hierarchy presented.

[0103] Description of the presented embodiments is provided so that a person skilled in the art may use or practice the present disclosure. Various modifications to these embodiments will be apparent to a person skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure is not limited to the embodiments presented herein, but should be interpreted in the broadest possible scope consistent with the principles and novel features presented herein.

[0104] Although embodiments of the present disclosure have been described with reference to the drawings exemplified above, the present disclosure is not limited by the embodiments and drawings described herein, and various modifications may be made by those skilled in the art. Furthermore, even if the effects according to the configuration of the present disclosure were not explicitly described while explaining the embodiments of the present disclosure above, the effects predictable by said configuration should also be recognized.

Claims

Claim 1 An image restoration method performed by a computing device, comprising: a step of acquiring an incomplete image; a step of repeatedly performing pixel value restoration according to nearest neighbor interpolation on the incomplete image to estimate the size of a hole included in the incomplete image; a step of counting the number of times the pixel value restoration is repeated as the number of pixel value restorations; a step of estimating the size of a hole included in the incomplete image based on the number of pixel value restorations; a step of comparing the size of the hole with a predetermined reference value; and a step of determining a restoration method for the incomplete image according to the comparison result. The method includes the step of restoring the incomplete image by applying the restoration method described above, wherein when pixel value restoration according to the nearest neighbor interpolation method is performed repeatedly, if there is a pixel among the adjacent pixels of the pixel with missing pixel value that does not have a missing pixel value, the pixel value of the pixel with missing pixel value is restored based on the pixel value of the pixel without missing pixel value, and if all of the adjacent pixels have missing pixel values, the pixel value of the pixel with missing pixel value is not restored, and the step of estimating the size of the hole includes the step of converting the number of pixel value restorations into the size of the hole by referring to a pre-stored table in which the number of pixel value restorations and the size of the hole corresponding to the number of pixel value restorations are recorded, and the step of determining the restoration method of the incomplete image according to the comparison result includes the step of determining a machine learning-based inpainting technique as the restoration method if the size of the hole is greater than or equal to the reference value; and the step of determining a pixel value-based image interpolation method as the restoration method if the size of the hole is less than the reference value. Claim 2 delete Claim 3 delete Claim 4 In claim 1, the machine learning-based in-painting technique is one of DeepFillv2, LaMa(Large Mask), GIN, AOT-GAN, and Palette. Claim 5 In claim 1, the pixel value-based image interpolation method is any one of nearest neighbor interpolation, bilinear interpolation, trilinear interpolation, bicubic interpolation, and spline interpolation. Claim 6 As a computing device for image restoration, at least one processor; The apparatus includes memory, wherein at least one processor acquires an incomplete image and, in order to estimate the size of a hole included in the incomplete image, repeatedly performs pixel value restoration according to nearest neighbor interpolation on the incomplete image, counts the number of times the pixel value restoration is repeated as the number of times the pixel value restoration is repeated, estimates the size of the hole included in the incomplete image based on the number of times the pixel value restoration is repeated, compares the size of the hole with a predetermined reference value, determines a restoration method for the incomplete image according to the comparison result, applies the restoration method to restore the incomplete image, and when the pixel value restoration according to nearest neighbor interpolation is repeatedly performed, if there is a pixel among the adjacent pixels of the pixel with missing pixel value that does not have missing pixel value, the pixel value of the pixel with missing pixel value is restored based on the pixel value of the pixel without missing pixel value, and if the pixel values ​​of all the adjacent pixels are missing, the pixel value of the pixel with missing pixel value is not restored, and the at least one processor has a pre-stored table in which the number of times the pixel value restoration is repeated and the size of the hole corresponding to the number of times the pixel value restoration is recorded. A device that converts the number of pixel value restorations into the size of the hole by reference, determines a machine learning-based inpainting technique as the restoration method if the size of the hole is greater than or equal to the reference value, and determines a pixel value-based image interpolation method as the restoration method if the size of the hole is less than the reference value. Claim 7 delete Claim 8 delete Claim 9 In claim 6, the machine learning-based in-painting technique is one of DeepFillv2, LaMa(Large Mask), GIN, AOT-GAN, and Palette. Claim 10 In claim 6, the above pixel value-based image interpolation method is any one of nearest neighbor interpolation, bilinear interpolation, trilinear interpolation, bicubic interpolation, and spline interpolation. Claim 11 A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs the following steps for restoring an image, the steps comprising: acquiring an incomplete image; repeatedly performing pixel value restoration according to nearest neighbor interpolation on the incomplete image to estimate the size of a hole included in the incomplete image; counting the number of times the pixel value restoration is repeated as the number of pixel value restorations; estimating the size of a hole included in the incomplete image based on the number of pixel value restorations; comparing the size of the hole with a predetermined reference value; and determining a method for restoring the incomplete image according to the comparison result. A computer program stored on a computer-readable storage medium, comprising: an operation of restoring the incomplete image by applying the above restoration method; wherein, when pixel value restoration according to the nearest neighbor interpolation method is performed repeatedly, if there is a pixel among the adjacent pixels of the pixel with missing pixel value that does not have a missing pixel value, the pixel value of the pixel with missing pixel value is restored based on the pixel value of the pixel that does not have a missing pixel value, and if all of the adjacent pixels have missing pixel values, the pixel value of the pixel with missing pixel value is not restored; wherein the operation of estimating the size of the hole includes an operation of converting the number of pixel value restorations into the size of the hole by referring to a pre-stored table in which the number of pixel value restorations and the size of the hole corresponding to the number of pixel value restorations are recorded; and the operation of determining the restoration method of the incomplete image according to the comparison result includes: an operation of determining a machine learning-based inpainting technique as the restoration method if the size of the hole is greater than or equal to the reference value; and an operation of determining a pixel value-based image interpolation method as the restoration method if the size of the hole is less than the reference value.

Images