Image restoration method and apparatus based on a series of images

Abstract

An image restoration method and apparatus based on a burst image are provided. The image restoration method includes determining an anchor image based on individual images of a burst image set, using anchor information of the anchor image when performing a feature extraction network based on the burst image set, and generating a restored image based on a feature map corresponding to an output of the feature extraction network.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2021-0017439, filed on February 8, 2021, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to methods and apparatus for recovering images based on burst-shot images. Background Technology

[0003] Image restoration is a technique used to restore images of degraded quality to images of enhanced quality. For image restoration, deep learning-based neural networks can be used. A neural network can be trained based on deep learning and then performs inference by mapping input and output data in a non-linear relationship to achieve its purpose. The ability to generate such mappings can be represented as the learning ability of the neural network. Furthermore, neural networks trained for a specific purpose (e.g., image restoration) can have generalization capabilities to generate relatively accurate outputs for unlearned input patterns. Summary of the Invention

[0004] This summary is provided to introduce, in a simplified form, the selection of concepts that will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, the image restoration method includes: determining an anchor image based on multiple individual images from a burst image set; performing a feature extraction network based on the burst image set while using anchoring information from the anchor image; and generating a restored image based on a feature map corresponding to the output of the feature extraction network.

[0006] The step of determining the anchor image may include selecting the anchor image from the plurality of individual images based on any one or any combination of two or more of the following: quality-based selection, time-based selection, and random selection.

[0007] The step of determining the anchor image may include generating the anchor image by applying weights to the plurality of individual images.

[0008] Anchoring information may include one or both of the image information of the anchoring image and the feature information of the anchoring image.

[0009] The method may further include generating an input image for a feature extraction network by fusing anchoring information with each of the plurality of individual images.

[0010] The steps of performing the feature extraction network may include: extracting anchored local features from the anchored image; extracting local features from another image among the plurality of individual images; extracting global features from the anchored local features and the local features; and fusing the anchored local features with the global features.

[0011] The steps of performing a feature extraction network may include: extracting anchored local features from the anchored image; extracting local features from another image among the plurality of individual images; and extracting global features from the anchored local features and local features when using the anchored local features.

[0012] The steps of performing global feature extraction may include: extracting global features from anchored local features and local features of the other image by assigning greater weights to anchored local features than to local features of the other image.

[0013] The steps of performing a feature extraction network may include: extracting local features while using anchoring information for each of the multiple layers in the feature extraction network.

[0014] The steps of performing local feature extraction may include: extracting first-level local features from a first individual image among the plurality of individual images using a first layer group among the plurality of layer groups; transforming the first-level local features by fusing anchoring information with the first-level local features; extracting second-level local features from the transformed first-level local features using a second layer group among the plurality of layer groups; and determining global features based on the second-level local features.

[0015] The steps for generating a restored image may include: performing an image restoration network based on the feature map.

[0016] A non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the method described above.

[0017] In another general aspect, the image restoration device includes one or more processors; and a memory including instructions executable by the one or more processors. In response to the instructions being executed by the one or more processors, the one or more processors are configured to: determine an anchor image based on a plurality of individual images from a burst image set; execute a feature extraction network based on the burst image set when using anchoring information from the anchor image; and generate a restored image based on a feature map corresponding to the output of the feature extraction network.

[0018] The one or more processors may also be configured to: select an anchor image from the plurality of individual images based on any one or any combination of two or more of quality-based selection, time-based selection, and random selection, or generate an anchor image by applying weights to the plurality of individual images.

[0019] The one or more processors may also be configured to: extract anchored local features from the anchored image; extract local features from another image among the plurality of individual images; and extract global features from the anchored local features and the local features of the other image when using the anchored local features.

[0020] The one or more processors may also be configured to extract local features when using anchoring information for each of the multiple layers of the feature extraction network.

[0021] In another general aspect, an electronic device includes: a camera configured to generate a burst set of images, and one or more processors configured to: determine an anchor image based on a plurality of individual images of the burst set; perform a feature extraction network based on the burst set of images when using anchoring information of the anchor image; and generate a restored image based on a feature map corresponding to the output of the feature extraction network.

[0022] The one or more processors may also be configured to: select an anchor image from the plurality of individual images based on any one or any combination of two or more of quality-based selection, time-based selection, and random selection, or generate an anchor image by applying weights to the plurality of individual images.

[0023] The one or more processors may also be configured to: extract anchored local features from the anchored image; extract local features from another image among the plurality of individual images; and extract global features from the anchored local features and the local features of the other image when using the anchored local features.

[0024] The one or more processors may also be configured to extract local features when using anchoring information for each of the multiple layers of the feature extraction network.

[0025] In another general aspect, an electronic device includes: an image sensor configured to capture a plurality of images; and one or more processors. The one or more processors are configured to: determine an anchor image from the plurality of images; perform a feature extraction network based on the plurality of images when using anchoring information from the anchor image; and generate a restored image based on a feature map corresponding to the output of the feature extraction network.

[0026] The one or more processors may also be configured to: extract anchored local features from the anchored image; extract local features from another image among the plurality of individual images; and extract global features from the anchored local features and local features by assigning a greater weight to the anchored local features than to the local features.

[0027] The electronic device can be a camera or a smartphone.

[0028] The multiple images can be captured sequentially.

[0029] The electronic device may further include a memory configured to store instructions. The one or more processors are further configured to execute the instructions to configure the one or more processors to: determine an anchor image from the plurality of images; perform a feature extraction network based on the plurality of images when using anchoring information from the anchor image; and generate a restored image based on a feature map corresponding to the output of the feature extraction network.

[0030] Other features and aspects will become apparent from the following detailed description, drawings, and claims. Attached Figure Description

[0031] Figure 1 An example of the operation of an image restoration device is illustrated schematically.

[0032] Figures 2 to 4 An example of the operation of selecting an anchor image is shown.

[0033] Figure 5 An example of generating a restored image based on anchoring information is shown.

[0034] Figure 6 Examples of configurations and operations related to neural network models are shown.

[0035] Figure 7 An example of using anchoring information in the processing of an input image is shown.

[0036] Figure 8 An example of using anchoring information in the processing of the output feature map is shown.

[0037] Figure 9 An example of using anchoring information in the process of extracting global features is shown.

[0038] Figure 10 It shows Figure 9 Examples of operations.

[0039] Figure 11 An example of using anchoring information in the process of extracting local features is shown.

[0040] Figure 12 An example of an image restoration method is shown.

[0041] Figure 13 An example configuration of an image restoration device is shown.

[0042] Figure 14An example of the configuration of an electronic device is shown.

[0043] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be drawn to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0044] The following detailed description is provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will be apparent upon understanding the disclosure of this application. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein, but may be obviously changed upon understanding the disclosure of this application, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0045] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways in which the methods, apparatus, and / or systems described herein will become apparent upon understanding the disclosure of this application.

[0046] The following structural or functional descriptions of the examples disclosed in this disclosure are intended to describe the purpose of the examples only, and the examples may be implemented in various forms. The examples are not intended to be limiting, but rather to imply that various modifications, equivalents, and substitutions are also covered within the scope of the claims.

[0047] Although the terms “first” or “second” are used to describe various components, the components are not limited to these terms. These terms should only be used to distinguish one component from another. For example, within the scope of the conception of this disclosure, a “first” component may be referred to as a “second” component, or similarly, a “second” component may be referred to as a “first” component.

[0048] It will be understood that when a component is referred to as being "connected to" another component, the component can be directly connected to or combined with the other component, or there can be an intermediate component.

[0049] As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. As used herein, the term “and / or” includes any one and any combination of any two or more of the associated listed items. It should also be understood that when the terms “comprising” and / or “including” are used in this specification, they specify the presence of the stated features, integrals, steps, operations, elements, components, or combinations thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0050] Unless otherwise defined herein, all terms used herein (including technical or scientific terms) shall have the same meaning as commonly understood. Unless otherwise defined herein, terms defined in commonly used dictionaries shall be interpreted as having a meaning matching the contextual meaning in the relevant field and shall not be interpreted as having an ideal or overly formal meaning.

[0051] In the following description, examples will be described in detail with reference to the accompanying drawings, and the same reference numerals in the drawings always denote the same elements.

[0052] Figure 1 An example of the operation of an image restoration device is illustrated schematically. (Refer to...) Figure 1 The image restoration device 100 can receive a burst image set 101, generate a restored image 102 based on the burst image set 101, and output the restored image 102. The burst image set 101 can be generated by a camera. The burst image set 101 can include multiple images captured in succession. The burst image set 101 can be, for example, a video generated by a video capture function, or a series of still images generated by a continuous capture function. Each of the multiple images in the burst image set 101 can refer to a "single image". In the example of video, each image frame can correspond to a single image. In the example of burst images, each still image can correspond to a single image. The camera can be an image sensor.

[0053] For example, when it is assumed that a burst image set 101 is generated by capturing a target object using a camera, the individual images of the burst image set 101 may have different characteristics due to movement of the camera and / or the target object and / or changes in ambient light (e.g., illuminance or color). In one example, when the burst image set 101 is acquired in a weak environment (e.g., a low-light environment), and / or when each individual image has degraded quality, a restored image 102 with improved quality can be obtained by appropriately combining the various characteristics of the individual images. Therefore, a restored image 102 with high quality can be obtained by processing the restoration of individual images with low quality.

[0054] Since the position of objects in each individual image changes due to camera and / or target object movement, a pre-task of matching objects in each individual image by aligning the centerline of each individual image may be necessary. The centerline does not have to be an actual line represented in each individual image; a virtual line can be used as a standard for aligning individual images. Without such a pre-task, severe blurring may occur. However, performing this pre-task can significantly increase computational cost and time. Furthermore, since the task of aligning the centerline requires iterative processing, computational cost and time can increase significantly with the number of individual images.

[0055] Image restoration device 100 can determine anchor images based on individual images from a burst image set 101, use the anchoring information of the anchor images when executing a neural network model (e.g., by executing a neural network model using the anchoring information of the anchor images), and generate a restored image 102. For example, the operation of generating the restored image 102 using anchoring information can include generating the restored image 102 based on the anchoring information by iteratively using (e.g., emphasizing) the anchoring information in the image restoration process. This image restoration operation of image restoration device 100 can be provided based on the anchoring information of the anchor images instead of the center line, thus obtaining a restored image 102 with improved quality without the task of aligning the center line. Since the task of aligning the center line is not required, the computational cost and computation time for image restoration can be reduced, and the trend of computational cost and computation time increasing significantly based on the number of individual images can be addressed.

[0056] Image restoration device 100 can select anchor images from individual images in a burst image set 101, or generate anchor images based on image information of individual images. In one example, image restoration device 100 can select anchor images from individual images based on any one or any combination of two or more of quality-based selection, time-time-based selection, and random selection. In another example, image restoration device 100 can assign weights to each individual image based on criteria (such as image quality), and can generate anchor images by applying weights to each individual image.

[0057] Image restoration device 100 can execute a neural network model based on a set of burst images 101 and generate a restored image 102. For example, the neural network model may include a feature extraction network configured to extract features from individual images of the burst image set 101, and an image restoration network configured to convert the extracted features into the restored image 102. At least a portion of each of the feature extraction network and the image restoration network may correspond to a deep neural network (DNN) comprising multiple layers. The multiple layers may include an input layer, at least one hidden layer, and an output layer.

[0058] A DNN can include any one or any combination of a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, at least some of the layers in a neural network may correspond to a CNN, and another part may correspond to an FCN. In this example, a CNN may be referred to as a convolutional layer, and an FCN may be referred to as a fully connected layer.

[0059] In a CNN, the data input to each layer can be called an "input feature map," and the data output from each layer can be called an "output feature map." Input and output feature maps can also be referred to as activation data. When a convolutional layer corresponds to an input layer, the input feature map of that input layer can correspond to the input image.

[0060] Neural networks can be trained based on deep learning and can perform inferences tailored to the training process by mapping input and output data in a non-linear relationship. Deep learning can be a machine learning approach used to solve problems such as image or speech recognition from large datasets. Deep learning can be understood as solving an optimization problem—finding the point where energy is minimized—while simultaneously training a neural network based on prepared training data.

[0061] Through supervised or unsupervised learning in deep learning, the structure of a neural network or the weights corresponding to the model can be obtained, and the input and output data can be mapped to each other through the weights. For example, if the width and depth of the neural network are large enough, the neural network can have a capacity large enough to implement arbitrary functions. Optimal performance can be achieved when the neural network is trained on a sufficiently large amount of training data with appropriate training processing.

[0062] In the following description, a neural network may be referred to as "pre-trained," where "pre" may indicate the state of the neural network before it is "started." A "started" neural network may indicate that the neural network is ready to perform inference. For example, "starting" a neural network may include loading the neural network into memory, or, after the neural network is loaded into memory, feeding input data for inference into the neural network.

[0063] Image restoration device 100 can use anchoring information of an anchored image when executing a neural network model. For example, image restoration device 100 can emphasize (e.g., use) anchoring information when performing any one or any combination of operations such as inputting an input image into a neural network model, extracting features from the input image using the neural network model, and outputting the extracted features. Anchoring information may include, for example, image information of the anchored image and / or feature information extracted from the anchored image. Anchoring information can provide a geometric standard for image restoration. Therefore, image information at corresponding locations can be combined without needing to align center lines, thus improving image quality while preventing blurring.

[0064] Figures 2 to 4 An example of selecting an anchor image is shown. (See reference...) Figure 2 The image restoration device can select an anchor image 220 from multiple individual images 211 to 216 of the burst image set 210. In one example, the image restoration device can select the anchor image 220 based on image quality. In this example, the image restoration device can determine the quality of each of the individual images 211 to 216 based on any one or any combination of noise, blur, signal-to-noise ratio (SNR), and sharpness, and can select the image with the highest quality as the anchor image 220. The image restoration device can use a deep learning network and / or computational modules to calculate the aforementioned quality.

[0065] In another example, the image restoration device can select anchor image 220 based on image order. In this example, individual images 211 to 216 can be captured sequentially. The image restoration device can select the first individual image (individual image 211) as anchor image 220. In another example, the image restoration device can select any image among individual images 211 to 216 as anchor image 220. This is because even if the anchor image 220 used to provide the standard for image restoration has relatively low quality, the image quality can be improved based on the image information of individual images 211 to 216.

[0066] Reference Figure 3 The image restoration device can select an anchor image 320 from individual images corresponding to a predetermined time interval. For example, a first time interval 331 can cover a period of time from the start of shooting, and the image restoration device can select an anchor image 320 within the first time interval 331. In another example, multiple time intervals can be used. For example, a second time interval 332 and a third time interval 333 can cover different shooting times, and the image restoration device can select an anchor image 320 within the second time interval 332 and the third time interval 333.

[0067] refer to Figure 4The image restoration device can determine a weight set 420 for the burst image set 410, and can restore the image by adjusting the weights W. 41 To W 46 Anchor images 430 are generated by applying individual images 411 to 416 to the burst image set 410. For example, an image restoration device can determine weight W based on the image quality of individual images 411 to 416. 41 To W 46 And it can be based on weight W 41 To W 46 Anchor image 430 is generated by reflecting image information from individual images 411 to 416. In this example, the amount of image information to be provided to anchor image 430 can increase as the weights applied to the images increase.

[0068] Figure 5 An example of generating a restored image based on anchoring information is shown. (Reference) Figure 5 The image restoration device can determine the anchor image based on individual images 521 to 524 of a burst image set 520, can emphasize the anchor information 530 when executing the neural network model 510, and can generate a restored image 540. The burst image set 520 may include individual images 521 to 524, and the image restoration device can determine the anchor image based on the individual images 521 to 524 according to various criteria. Figure 5 In the example, a single image 521 is selected as the anchor image. Although examples of four single images (e.g., single images 521 to 524) are described below, the number of single images may be greater than or less than "4".

[0069] The image restoration device can sequentially input individual images 521 to 524 into a neural network model 510, and emphasize anchoring information 530 while executing the neural network model 510. For example, the image restoration device can emphasize anchoring information 530 when performing any one or any combination of operations such as inputting individual images 521 to 524 into the neural network model 510, extracting features from individual images 521 to 524 using the neural network model 510, and outputting the extracted features. Anchoring information 530 may include, for example, image information of the anchor image and / or feature information extracted from the anchor image.

[0070] The neural network model 510 may include a feature extraction network 511 and an image restoration network 512. The feature extraction network 511 can extract features from individual images 521 to 524 in response to input. For example, the feature extraction network 511 can extract local features from individual images 521 to 524, and can extract global features from local features. The image restoration network 512 can convert the features into a restored image 540. The feature extraction network 511 may correspond to, for example, an encoder configured to convert image information into feature information, and the image restoration network 512 may correspond to, for example, a decoder configured to convert feature information into image information.

[0071] Figure 6 Examples of construction and operations related to neural network models are shown. (See reference...) Figure 6 The feature extraction network 610 may include a local feature extractor 611 and a global feature extractor 612. The local feature extractor 611 can extract local features from each individual image of the burst image set 620, and the global feature extractor 612 can extract global features from the local features. The image restoration network 640 can convert the global features into a restored image 650. The feature extraction network 610 and the image restoration network 640 may each include a neural network and may be pre-trained to perform extraction and transformation operations, respectively.

[0072] The image restoration device may iteratively use and / or emphasize the anchoring information 630 while executing the feature extraction network 610. For example, the image restoration device may emphasize the anchoring information 630 when performing any one or any combination of operations such as inputting a single image into the feature extraction network 610, extracting features from a single image using the feature extraction network 610, and outputting the extracted features. Examples of operations related to the use of the anchoring information 630 will be further described below.

[0073] Figure 7 An example of using anchoring information in the processing of an input image is shown. (See reference...) Figure 7 The image restoration device can utilize anchoring information from the anchored images during processing, where individual images 721 to 724 are input into a feature extraction network. The feature extraction network can correspond to, for example, a local feature extractor. Figure 7 In the example, individual image 721 can be assumed to be the anchor image, and the image restoration device can fuse the image information of individual image 721, which serves as anchor information, with individual images 721 to 724.

[0074] For example, fusion can include concatenation and / or addition. Concatenation can be linking elements together, and addition can be summing the elements. Therefore, concatenation may affect dimensions, while addition may not. Concatenation can be performed in the channel direction. For example, when each of the individual images 721 to 724 has dimensions "W×H×C", the concatenation result can have dimensions "W×H×2C", and the addition result can have dimensions "W×H×C".

[0075] The image restoration device can input the fusion result as an input image into a feature extraction network, and can extract local features in operations 711 to 714. For example, in operation 711, the image restoration device can extract local features by inputting the fusion result of individual images 721 and anchor image information into the feature extraction network, and as a result, local feature map 731 can be obtained. Similarly, in operations 712 to 714, the image restoration device can extract local features by sequentially inputting the fusion result of other individual images 722 to 724 and anchor image information into the feature extraction network, and as a result, local feature maps 732 to 734 can be obtained.

[0076] Figure 8 An example of using anchoring information in the processing of the output feature map is shown. (Refer to...) Figure 8 The image restoration device can extract a global feature map 850 from local feature maps 831 to 834 in the global feature extraction operation 840, and can use anchoring information from the anchored image in the processing of the output global feature map 850. The image restoration device can use a feature extraction network to perform operation 840. The feature extraction network can correspond to, for example, a global feature extractor. Figure 8 In this example, the local feature map 831 can be assumed to be extracted from the anchor image, and the image restoration device can fuse the feature information of the local feature map 831, which serves as anchor information, with the global feature map 850. In this example, the fusion can include cascading and / or addition. The result of the fusion can correspond to the output feature map of the feature extraction network, and the image restoration device can use the image restoration network to convert the output feature map into a restored image.

[0077] Figure 9 An example of using anchoring information in the process of extracting global features is shown. (See reference...) Figure 9 The image restoration device can extract a global feature map 950 from local feature maps 931 to 934 in the global feature extraction operation 940, and the anchoring information of the anchored image can be used as guiding information. Figure 9In the example, local feature map 931 can be assumed to be extracted from the anchor image, and the image restoration device can use the feature information of local feature map 931 as anchor information (i.e., guidance information). For example, when local feature map 931 is assumed to be the anchor local feature, and when local feature maps 932 to 934 are assumed to be adjacent local features, the image restoration device can assign a larger weight to the anchor local feature than to the adjacent local features, and can perform operation 940. Therefore, the information of the anchor local feature may have a larger impact on the global feature map 950 compared to the adjacent local features.

[0078] Figure 10 It shows Figure 9 Examples of operations. For example... Figure 9 As shown, various weighting schemes can be used to utilize anchor information as guiding information. Figure 10 A scheme for emphasizing anchoring information through weight allocation operation 1040 and weighted fusion operation 1060 is illustrated. Since the anchoring information corresponding to the geometric criteria used for image restoration is emphasized in the above scheme, it is more efficient than schemes that extract global features from local features in combining corresponding image information through pooling operations (e.g., max pooling or average pooling). (Refer to...) Figure 10 The image restoration device can assign different weights to local feature maps 1031 to 1034. In operation 1060, the image restoration device can consider the weight set 1050 when fusing the local feature maps 1031 to 1034 with each other, and as a result, a global feature map 1070 can be generated. For example, the image restoration device can use softmax to assign weights W... 101 To W 104 Assigned to local feature maps 1031 to 1034, and can be based on weight W 101 To W 104 The local feature maps 1031 to 1034 are summed (e.g., weighted fusion) to generate the global feature map 1070.

[0079] For example, it can be assumed that local feature map 1031 is extracted from the anchored image, and local feature maps 1032 to 1034 are extracted from other individual images. In this example, the image restoration device can assign weights W to the local feature map 1031. 101 Set the weight W to be greater than the local feature map 1032 to 1034. 102 To W 104Therefore, anchoring information can be emphasized through the feature information of local feature map 1031. In another example, the image restoration device can determine the similarity between local feature map 1031 and each of local feature maps 1032 to 1034, and can assign relatively high weights to feature maps similar to local feature map 1031 as well as local feature map 1031. In this example, when the similarity between local feature map 1032 and local feature map 1031 is high and the similarity between each of local feature maps 1033 and 1034 and local feature map 1031 is low, the image restoration device can assign relatively high weights W to local feature maps 1031 and 1032. 101 and W 102 Set the weight W to be greater than that of local feature maps 1033 and 1034. 103 and W 104 Therefore, the anchoring information can be enhanced by the feature information of local feature maps 1031 and 1032.

[0080] Figure 11 An example of using anchoring information in the process of extracting local features is shown. (See reference...) Figure 11 The image restoration device can extract local feature maps 1131 to 1134 from individual images 1151 to 1154 of the burst image set 1150 through operations 1110 to 1140 for extracting local features. The image restoration device can emphasize anchoring information during the extraction of local feature maps 1131 to 1134. For example, the image restoration device can select individual image 1151 as the anchor image and can generate anchoring information 1101 based on individual image 1151. The anchoring information 1101 may include image information and / or feature information of individual image 1151.

[0081] A feature extraction network (e.g., a local feature extractor) may include multiple layers. These multiple layers can be classified into layer groups, each layer group comprising a portion of multiple layers. For example, each layer group may include convolutional layers and / or pooling layers. The image restoration device may use anchoring information from each of the multiple layer groups of the feature extraction network when extracting local features. The image restoration device can extract local features through each layer group and can fuse the anchoring information 1101 with the extracted local features. The image restoration device may iteratively perform the above processing on all layer groups to generate local feature maps 1131 to 1134.

[0082] For example, an image restoration device can extract first-level local features from a single image 1151 using the first-level group feature extraction operation 1111, and can transform the first-level local features by fusing anchoring information 1101 with the first-level local features. The image restoration device can extract second-level local features from the transformed first-level local features using the second-level group feature extraction operation 1112, and can transform the second-level local features by fusing anchoring information 1101 with the second-level local features. Furthermore, the image restoration device can extract tertiary local features from the transformed second-level local features using the third-level group feature extraction operation 1113, and can transform the tertiary local features by fusing anchoring information 1101 with the tertiary local features. When the final layer group feature extraction operation 1115 is completed, a local feature map 1131 can be generated as a result of operation 1115. Operations 1120 to 1140 associated with other single images 1152 to 1154 can also correspond to operation 1110 for the single image 1151, and as a result, local feature maps 1132 to 1134 can be generated.

[0083] In this example, the image restoration device can fuse the same anchoring information 1101 with the output of each layer group (e.g., each layer group except the last one), or it can fuse anchoring information 1101 specific to each layer group. The fusion via common anchoring information 1101 is described below. Common anchoring information 1101 can be image information and / or feature information of the anchored image. To obtain feature information, an operation to extract features from the anchored image can be performed beforehand. For example, a feature extraction network used for operations 1110 to 1140 (e.g., ...) can be used. Figure 5 Feature extraction network 511 or Figure 6 The aforementioned preliminary operations can be performed using a local feature extractor 611 or a different feature extraction network. When common image information and / or common feature information is provided, the common image information and / or common feature information can be fused with the output (e.g., local features) of each layer group according to operations 1110 to 1140.

[0084] The fusion via dedicated anchoring information 1101 is described below. Unlike common anchoring information 1101, dedicated anchoring information 1101 can be information processed to suit each layer group. Dedicated anchoring information 1101 can include hierarchical local features of the anchored image extracted through each layer group. For example, when the first to third local features (e.g., first-level to third-level local features) of the anchored image are extracted through the first to third layer groups respectively, the first to third local features can be used as anchoring information 1101 dedicated to the first to third layer groups respectively. Therefore, the first local feature can be fused with each local feature extracted through operations 1111 and 1121, the second local feature can be fused with each local feature extracted through operations 1112 and 1122, and the third local feature can be fused with each local feature extracted through operations 1113 and 1123.

[0085] Figure 12 An example of an image restoration method is shown. (See reference...) Figure 12 In operation 1210, the image restoration device can determine the anchor image based on individual images from a burst of images. The image restoration device can also select the anchor image from among the individual images based on their quality. Furthermore, the image restoration device can select any image from among the individual images as the anchor image.

[0086] In operation 1220, the image restoration device can use anchoring information of anchored images when performing a feature extraction network based on a set of burst-shot images. The image restoration device can extract first-level local features from a first individual image within the individual images using the first layer group of the feature extraction network. The first-level local features can be transformed by fusing the anchoring information with the first-level local features. Furthermore, the second layer group of the feature extraction network can extract second-level local features from the transformed first-level local features. Additionally, the image restoration device can transform second-level local features by fusing the anchoring information with the second-level local features. The third layer group of the feature extraction network can extract tertiary local features from the transformed second-level local features, and global features can be determined based on the tertiary local features.

[0087] In one example, the image restoration device can extract anchored local features from an anchored image, extract local features (e.g., adjacent local features) from images other than the anchored image within a single image, and use the anchored local features when extracting global features from the anchored local features and the local features of the image. In this example, the image restoration device can extract global features from the anchored local features and the local features of the image by assigning a larger weight to the anchored local features than the weight assigned to the local features of the image.

[0088] In another example, the image restoration device can extract anchored local features from an anchored image, extract local features from images other than the anchored image within individual images, extract global features from both anchored local features and local features of the images, and fuse the anchored local features with the global features. Furthermore, the image restoration device can generate the input image for a neural network model by fusing anchoring information with each individual image.

[0089] In operation 1230, the image restoration device can generate a restored image based on a feature map corresponding to the output of the feature extraction network. The image restoration device can execute the image restoration network based on the feature map. Figures 1 to 11 The description can be applied to image restoration methods.

[0090] Figure 13 An example of the construction of an image restoration device is shown. (Refer to...) Figure 13 The image restoration device 1300 may include a processor 1310 and a memory 1320. The memory 1320 may be connected to the processor 1310 and may store instructions executable by the processor 1310, data to be computed by the processor 1310, or data processed by the processor 1310. The memory 1320 may include, for example, a non-transitory computer-readable storage medium, such as high-speed random access memory (RAM) and / or a non-volatile computer-readable storage medium (e.g., at least one disk storage device, flash memory device, or other non-volatile solid-state memory device).

[0091] Processor 1310 can execute the above-mentioned reference. Figures 1 to 12 The described operations. For example, processor 1310 can determine anchor images based on individual images from a burst image set, use anchoring information of the anchor images when performing a feature extraction network based on the burst image set, and generate a restored image based on a feature map corresponding to the output of the feature extraction network. Additionally, Figures 1 to 12 The description also applies to the image restoration device 1300.

[0092] Figure 14 An example of the construction of an electronic device is shown. (Refer to...) Figure 14Electronic device 1400 may include a processor 1410, a memory 1420, an image sensor 1430, a storage device 1440, an input device 1450, an output device 1460, and a network interface 1470. The processor 1410, memory 1420, image sensor 1430, storage device 1440, input device 1450, output device 1460, and network interface 1470 may communicate with each other via a communication bus 1480. For example, electronic device 1400 may be implemented as at least a part of, for example, a mobile device (such as a camera, mobile phone, smartphone, personal digital assistant (PDA), netbook, tablet computer, or laptop computer), a wearable device (such as a smartwatch, smart bracelet, or smart glasses), a computing device (such as a desktop computer or server), a home appliance (such as a television (TV), smart TV, or refrigerator), a security device (such as a door lock), and a vehicle (such as an autonomous vehicle or intelligent vehicle). Electronic device 1400 may structurally and / or functionally include... Figure 1 Image restoration device 100 and / or Figure 13 Image restoration device 1300.

[0093] Processor 1410 can execute instructions and functions within electronic device 1400. For example, processor 1410 can process instructions stored in memory 1420 or storage device 1440. Processor 1410 can execute the instructions described above. Figures 1 to 13 At least one of the described operations. Memory 1420 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. Memory 1420 may store instructions to be executed by processor 1410, and may also store information associated with software and / or applications while the software and / or applications are being executed by electronic device 1400.

[0094] Image sensor 1430 can capture photographs and / or videos. For example, image sensor 1430 can continuously capture photographs or videos to generate a burst image set. When the burst image set includes consecutive photographs, each individual image in the burst image set can correspond to each photograph. When the burst image set is a video, each individual image in the burst image set can correspond to each frame of the video. Storage device 1440 may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. In one example, storage device 1440 may store a larger amount of information than the amount of information in memory 1420 over a relatively long period of time. For example, storage device 1440 may include a magnetic hard disk, optical disk, flash memory, floppy disk, or other forms of non-volatile memory known in the art.

[0095] Input device 1450 can receive input from a user using conventional input methods such as a keyboard and mouse, as well as new input methods such as touch input, voice input, and image input. Input device 1450 may include, for example, a keyboard, mouse, touchscreen, microphone, or other devices configured to detect input from the user and send the detected input to electronic device 1400. Output device 1460 can provide output from electronic device 1400 to a user through a visual channel, auditory channel, or tactile channel. Output device 1460 may include, for example, a display, touchscreen, speaker, vibration generator, or any other device configured to provide output to a user. Network interface 1470 can communicate with external devices via wired or wireless networks.

[0096] Perform the actions described in this application Figures 1 to 13The image restoration device 100, image restoration device 1300, processor 1310, and memory 1320 are implemented by hardware components configured to perform the operations described herein. Where appropriate, examples of hardware components that can be used to perform the operations described herein include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described herein. In other examples, one or more of the hardware components performing the operations described herein are implemented by computing hardware (e.g., one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. The hardware components can be any one or more with different processing configurations. Examples of different processing configurations include a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0097] Perform the operations described in this application Figures 1 to 13The illustrated methods are executed by computing hardware (e.g., one or more processors or a computer) that implements the execution instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0098] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring one or more processors or computers to operate as machines or special-purpose computers to perform operations performed by the hardware components and the methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, which disclose algorithms for performing operations performed by the hardware components and the methods described above.

[0099] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide instructions or software and any associated data, data files, and data structures to one or more processors or computers such that one or more processors or computers can execute the instructions. In one example, instructions or software, along with any associated data, data files, and data structures, are distributed across a networked computer system, such that the instructions or software, along with any associated data, data files, and data structures, are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0100] While this disclosure includes specific examples, it will be apparent upon understanding the disclosure of this application that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is to be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if the components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in the disclosure.

Claims

1. An image restoration method, comprising: Anchor images are determined based on multiple individual images from a burst-shot image set; When performing feature extraction networks based on burst-shot image sets, anchoring information from anchored images is used; as well as The restored image is generated based on the feature map corresponding to the output of the feature extraction network, and The steps of executing the feature extraction network include: using anchoring information for each of the multiple layers in the feature extraction network when extracting local features, and The step of extracting local features includes: extracting primary local features from a first individual image among the plurality of individual images using a first layer group among the plurality of layer groups; transforming the primary local features by fusing anchoring information for the first layer group with the primary local features; extracting secondary local features from the transformed primary local features using a second layer group among the plurality of layer groups; and determining global features based on the secondary local features.

2. The image restoration method according to claim 1, wherein, The steps of determining the anchor image include: selecting the anchor image from the plurality of individual images based on any one or any combination of two or more of the following: quality-based selection, time-based selection, and random selection.

3. The image restoration method according to claim 1, wherein, The steps for determining the anchor image include generating the anchor image by applying weights to the plurality of individual images.

4. The image restoration method according to claim 1, wherein, Anchoring information includes one or both of the image information of the anchored image and the feature information of the anchored image.

5. The image restoration method according to claim 1, further comprising: An input image for the feature extraction network is generated by fusing anchoring information with each of the plurality of individual images.

6. The image restoration method according to claim 1, wherein, The steps involved in executing a feature extraction network include: Extract anchoring local features from the anchoring image; Extract local features from another image among the plurality of individual images; Extract global features from anchored local features and local features of the other images; and The anchored local features are fused with global features.

7. The image restoration method according to claim 1, wherein, The steps involved in executing a feature extraction network include: Extract anchoring local features from the anchoring image; Extract local features from another image among the plurality of individual images; and Anchored local features are used when extracting global features from anchored local features and local features of the other image.

8. The image restoration method according to claim 7, wherein, The steps for extracting global features include: extracting global features from anchored local features and local features of the other image by assigning weights greater than those assigned to anchored local features.

9. The image restoration method according to any one of claims 1 to 8, wherein, The steps for generating the restored image include: executing an image restoration network based on the feature map.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the image restoration method according to any one of claims 1 to 9.

11. An image restoration device, comprising: One or more processors; as well as The memory includes instructions executable by the one or more processors. Wherein, in response to the instruction being executed by the one or more processors, the one or more processors are configured to: Anchor images are determined based on multiple individual images from a burst-shot image set; When performing feature extraction on a burst-image set, anchoring information from the anchored images is used; and The restored image is generated based on the feature map corresponding to the output of the feature extraction network, and The one or more processors are further configured to: use anchoring information for each of the multiple layers of the feature extraction network when extracting local features, and The one or more processors are further configured to: extract first-level local features from a first individual image among the plurality of individual images using a first layer group among the plurality of layer groups; transform the first-level local features by fusing anchoring information for the first layer group with the first-level local features; extract second-level local features from the transformed first-level local features using a second layer group among the plurality of layer groups; and determine global features based on the second-level local features.

12. The image restoration device as claimed in claim 11, wherein, The one or more processors are further configured to: An anchor image is selected from the plurality of individual images based on any one or any combination of two or more of the following: quality-based selection, time-interval-based selection, and random selection. An anchored image is generated by applying weights to the multiple individual images.

13. The image restoration device as claimed in claim 11, wherein, The one or more processors are further configured to: Extract anchoring local features from the anchoring image; Extract local features from another image among the plurality of individual images; and Anchored local features are used when extracting global features from anchored local features and local features of the other image.

14. An electronic device comprising: The camera is configured to generate a burst of images; as well as One or more processors are configured as follows: Anchor images are determined based on multiple individual images from a burst-shot image set; When performing feature extraction networks based on burst-image sets, anchoring information from anchored images is used; and The restored image is generated based on the feature map corresponding to the output of the feature extraction network, and The one or more processors are further configured to: use anchoring information for each of the multiple layers of the feature extraction network when extracting local features, and The one or more processors are further configured to: extract first-level local features from a first individual image among the plurality of individual images using a first layer group among the plurality of layer groups; transform the first-level local features by fusing anchoring information for the first layer group with the first-level local features; extract second-level local features from the transformed first-level local features using a second layer group among the plurality of layer groups; and determine global features based on the second-level local features.

15. The electronic device of claim 14, wherein, The one or more processors are further configured to: An anchor image is selected from the plurality of individual images based on any one or any combination of two or more of the following: quality-based selection, time-interval-based selection, and random selection. Anchored images are generated by applying weights to the multiple individual images.

16. The electronic device of claim 14, wherein, The one or more processors are further configured to: Extract anchoring local features from the anchoring image; Extract local features from another image among the plurality of individual images; and Anchored local features are used when extracting global features from anchored local features and local features of the other image.

Images