Intelligent cloud-based image enhancement system, method of generating an enhanced image

Abstract

An image enhancement system based on intelligent cloud and a method of generating an enhanced image to improve the quality and aesthetics of images captured via electronic devices are disclosed. The image enhancement system based on intelligent cloud includes a cloud database containing a collection of high-quality photos taken at different locations and different environmental settings around the world. Information from the photos stored in the cloud database (photos taken using cameras with better photography capabilities) is used to improve details that new photos cannot capture. The system not only relies on machine learning models but also utilizes data associated with new images collected from the hardware of the electronic devices (e.g., camera settings, GPS, and phone sensors) to obtain more reliable information.

Description

Technical Field

[0001] This invention relates to systems and methods for providing high-quality images. More specifically, this invention relates to improving the aesthetics of images by using a cloud database composed of high-quality images. Background Technology

[0002] Traditionally, much work has been done on processing images to improve their quality or aesthetics. One common enhancement is super-resolution, which is a process of magnifying and improving image details. There are many existing proposed methods in this field, including traditional methods and deep learning methods. Regardless of whether these methods are based on single-frame or multi-frame input, the amount of information that can be enhanced is limited by the amount of detail available within the input frame.

[0003] Some deep learning models using GANs claim to introduce additional detail, but they risk unrealistic and unnatural artifacts. Another proposed deep learning approach, more closely related to our system, is sample-based super-resolution, involving the transfer of relevant textures from a high-resolution reference image to the query image.

[0004] Currently, enhancements in image processing applications include object removal through inpainting, which refers to filling in missing parts of a photograph. Some common techniques for performing inpainting include traditional methods and deep learning methods. Traditional methods use pixel information from other unmissable image parts, so their performance can be unstable, especially when the details within the gaps are not structurally repetitive. Deep learning, similar to super-resolution, has the main drawback of introducing unrealistic details.

[0005] Besides improving and adding details to a photo, another way to enhance its aesthetic value is to improve its composition. A common technique is to crop to remove unnecessary content and adjust the aspect ratio, thereby improving the composition.

[0006] Different methods were used to determine the area to be cropped: attention-driven, aesthetic-driven, and data-driven methods. Cropping alone is insufficient to alter the relative proportions and positions of elements in a photograph. Therefore, some papers have proposed automatically repositioning the image and moving these elements to their respective optimal locations. A major drawback of using these post-processing methods is that the amount of information processed is limited to what is available in the photograph.

[0007] Finally, another useful application is panoramic photography, a technique that captures an image with a wider field of view than a camera can capture. Because of the wider field of view, the resulting photographs are more like what the human eye is accustomed to seeing, and therefore have greater aesthetic value. The most common method for producing such images is to take a series of photographs from different angles from a stationary position and then stitch them together.

[0008] However, these results still depend on the quality of the entire series of photographs. In other words, these techniques are only applicable to any given input. If some of the input images are of poor quality, such as inconsistent brightness between images or insufficient overlap, the results will be unsatisfactory. Furthermore, the final field of view is limited by the angles covered when these photographs were taken; therefore, if the user prefers a wider field of view, there is no alternative but to retake the photographs.

[0009] Many existing techniques focus on improving image quality through better photographic composition. A research paper by Ligang Liu, titled "Optimizing Photographic Composition," reveals a method for optimizing image aesthetics. Based on an analysis of the spatial structure of an image and the distribution of prominent regions and lines, aesthetic scores are assigned to the image. Furthermore, cropping techniques are used to alter the relative positions of salient areas within the image, thus modifying the image's compositional aesthetics. A drawback associated with the disclosed method is that it only uses pixel information within the image, therefore, it does not introduce additional details into the enhanced image.

[0010] U.S. Patent Application 20150363922, assigned to Samsung Electronics, Inc., relates to an apparatus and method for increasing image resolution. The invention includes capturing multiple frames of an image, a control unit determining a reference frame from the multiple frames, iteratively determining the offset of each of the multiple frames to the reference frame until unity scaling is achieved, and determining pixel values ​​to be inserted between pixels in the reference frame. While the prior art provides high-resolution images, it relies solely on conventional image transformation methods without incorporating the use of a cloud database to compare a single image with multiple high-quality images and utilizing hardware data associated with the single image, such as time, temperature, and location information. This limits the amount of enhancement in the final image.

[0011] Another U.S. patent, 7,889,948, assigned to Microsoft Technology Licensing Ltd., and U.S. application 20070237421, assigned to Monument Peak Ventures Ltd., relate to methods for replacing faces in a first digital image with another image. This invention provides for evaluating rendered images to identify the optimal seams along which various images should be blended. The result is a mosaic image constructed by distorting an input image with corrected source regions and blending the corrected source regions into the digital image.

[0012] Another U.S. application, 20180075581, assigned to Twitter Inc., relates to machine learning techniques for processing visual data using multiple datasets. One prior art method discloses training an algorithm to process at least a portion of received low-resolution visual data to estimate a high-resolution version of the low-resolution visual data using a training dataset and a reference dataset. While the present invention overcomes the shortcomings of the prior art described above by introducing a reference dataset, it focuses only on improving image resolution, thus falling short in many other aspects. Furthermore, the reference dataset is only a small set comprising a limited number of images and may not provide enhanced quality to meet user needs.

[0013] Therefore, there is a need for an intelligent cloud-based image enhancement system that improves the quality and aesthetics of new images by utilizing a large number of images stored in a cloud database. This intelligent cloud-based image enhancement system not only relies on traditional machine learning techniques but also leverages hardware data associated with the electronic devices used to capture the images.

[0014] It is now clear that numerous methods and systems have been developed in the prior art, suitable for a wide range of purposes. Furthermore, even if these inventions may be suitable for the specific purpose they address, they are correspondingly unsuitable for the purposes of this invention as described above. Therefore, there remains a need for an intelligent cloud-based image enhancement system that uses software and hardware tools to improve the quality and aesthetics of images. Summary of the Invention

[0015] With advancements in mobile hardware technology and artificial intelligence, the demands for image quality and aesthetics are constantly increasing. Both hardware and software tools have their advantages and disadvantages. Therefore, this invention discloses an intelligent cloud-based image enhancement system that integrates the advantages of both hardware and software tools. The proposed framework provides automatic photo enhancement by using a database containing high-quality images to overcome the low quality of images obtained due to hardware limitations.

[0016] The primary objective of this invention is to provide a cloud database containing a collection of high-quality photographs taken using different types of camera models in various locations and environmental settings around the world. Other components of the image enhancement system include a feature extraction module, a recommender model, and an image enhancement model. All of these components work together to enhance the quality of query images captured via electronic devices.

[0017] When the camera app on an electronic device is opened, the entire workflow is triggered, and the user places the camera in the shooting position to take a photo. Information such as location, temperature, and time when the image is clicked is collected by hardware installed in the camera and sent to a cloud database along with the captured image. Along with the photo, image information data is sent to the cloud database to search for images with similar content.

[0018] During the search process, multiple processes run concurrently. Feature extraction models, such as location classification, position projection, and environmental condition classification, are used to extract features from the image and its corresponding information, which are then compared with features in the database for similarity measurement. After a photo is taken, the enhancement process is activated. Depending on the enhancement objective, the captured photo is then passed to the appropriate model to improve its quality by referencing similar-looking photos retrieved from the database. The resulting enhanced photo is then sent back to the user.

[0019] In the proposed system, the photos stored in the cloud database were taken using cameras with superior photographic capabilities to extract details that cannot be captured in newly clicked photos. On the other hand, users sometimes wish to remove unwanted objects from photos. However, after object removal, additional information is needed to fill in the missing parts. This invention can utilize data from the cloud database to obtain such information.

[0020] Another potential limitation that this system can address is the constraints imposed by the surrounding environment. For example, a photograph might have higher aesthetic value when shot from a certain angle, but this might not be achievable because the photographer would need to be standing in water. This can be addressed using the proposed system, as a set of images taken from different but adjacent locations can be used to reconstruct a new three-dimensional image. The captured photograph can then be projected at the angle desired by the user.

[0021] In addition, the range of scenes that a camera can capture is also limited. By using existing mature photo stitching technology and a large number of photos of similar scenes taken from different angles, a cloud-based image enhancement system can automatically create a panoramic image from just one current photo.

[0022] Another objective of this invention is that, unlike image enhancement using purely machine learning models (which only have information from captured images), this system utilizes multiple photographs from a database to add actual details to the scene and further enhance the image. When extracting information to determine which similar images to retrieve from the database, the system relies not only on machine learning models to interpret information from the images, but also utilizes data collected from hardware such as camera settings, GPS, and phone sensors, thus obtaining more reliable information.

[0023] Another objective of this invention is that the proposed system directly outputs enhanced images, thus minimizing the requirements for the user's photography experience.

[0024] Other objects and aspects of the invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate features according to embodiments of the invention.

[0025] To achieve the above and related objectives, the present invention may be implemented in the form shown in the accompanying drawings. However, it should be noted that the drawings are merely illustrative and that changes may be made to the specific structures shown and described within the scope of the appended claims.

[0026] Although the invention has been described above with reference to various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functions described in one or more individual embodiments are not limited to those applicable to the specific embodiments in which they are described, whether or not those embodiments are described, and whether or not these features are presented as part of the described embodiments. Therefore, the breadth and scope of the invention should not be limited by any of the exemplary embodiments described above.

[0027] In some cases, the appearance of broadening words and phrases such as “one or more,” “at least,” “but not limited to,” or other similar phrases should not be interpreted as an intentional or necessary use of a narrower context where such broadening phrases may not exist. Attached Figure Description

[0028] The objects and features of the invention will become more apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. It is to be understood that these drawings depict only exemplary embodiments of the invention and are therefore not intended to limit its scope. The invention will be described and explained with additional specificity and detail using the drawings, wherein:

[0029] Figure 1 An image enhancement system according to the present invention is shown.

[0030] Figure 2 The structure of the image enhancement system according to the present invention is shown.

[0031] Figure 3 A feature extraction module of the image enhancement system according to the present invention is shown.

[0032] Figure 4 An image enhancement module of the image enhancement system according to the present invention is shown.

[0033] Figure 5 A method for generating enhanced images based on intelligent cloud, according to the present invention, is shown.

[0034] Figure 6A method for generating enhanced images based on intelligent cloud, according to the present invention, is shown.

[0035] Figure 7 A block diagram of an electronic device according to the present invention is shown. Detailed Implementation

[0036] Cameras have a long history as devices for recording visual data. Thanks to technological advancements, today's available electronic devices, such as smartphones, tablets, and digital cameras, can capture super-resolution images and videos. Features like autofocus and image quality correction are some of the options available to users to improve the quality of captured images.

[0037] Due to the recent surge in camera-equipped mobile devices (such as smartphones, tablets, smart glasses, etc.), individuals generate a vast number of images in their daily lives. With advancements in hardware and software technologies in the field of image processing, the demands for the quality and aesthetics of captured images are constantly increasing. This invention discloses an intelligent cloud-based image enhancement system that leverages hardware and artificial intelligence tools in the image processing industry to generate high-quality images.

[0038] Figure 1 An image enhancement system based on intelligent cloud technology is illustrated. The main component of the proposed intelligent cloud-based image enhancement system 100 is a cloud database 102, which contains a collection of photographs taken using different types of camera models in various locations and environmental settings around the world. For the initial setup of the cloud database 102, photographs can be obtained from online resources. To ensure the quality of the photographs, they need to undergo a data cleaning process performed by photography experts or through existing image quality and aesthetic evaluation methods. In other words, each photograph is rated according to its quality and aesthetic value.

[0039] Photos scoring below a predetermined threshold will be removed from cloud database 102. For photos collected through social media platforms such as Facebook and Instagram, another straightforward method is to determine their aesthetic value based on the number of likes they receive. If available, the corresponding photo's EXIF ​​data will also be stored. This provides information such as the date, time, and location when the photo was taken.

[0040] Another approach is to utilize the cloud storage installed on most smartphones. In other words, photos taken using these devices are automatically synced to the storage and, with the user's permission, can be used as part of the cloud database 102. Similarly, they also need to undergo a data selection process. The EXIF ​​data of these photos is likely available.

[0041] Once the system is online, photos taken using the platform can be continuously collected. In other words, whenever a photo is sent to the system for enhancement, if it meets quality and aesthetic criteria, it can also be stored as part of the cloud database 102. Similarly, existing image quality and aesthetic evaluation methods can be used to automatically evaluate the quality or aesthetic quality of photos.

[0042] The final practical consideration for Cloud Database 102 is its storage size. Perfectly reconstructing 3D space requires a large number of photos, and storing all original-quality photos in Cloud Database 102 would require an impractical storage size. Therefore, the amount of detail stored for each photo is compressed. One approach is to adopt the idea of ​​a visual feature encoding strategy. The feasibility of this idea is based on the assumption that features in photos of local regions are similar and can be shared across these images. Therefore, feature packets for each region and each photo within that region can be stored. This reduces the storage space required per photo and improves scalability.

[0043] Furthermore, the intelligent cloud-based image enhancement system 100 includes a feature extraction module 104, which comprises several feature extraction models. Simple perceptual metrics (such as mean squared error, peak signal-to-noise ratio, or structural similarity index) are not suitable for this invention because they do not measure the similarity between two identical images. Instead, the system maps images to features, where the distance between features directly corresponds to image similarity. In the primary embodiment, the extraction model within the feature extraction module 104 is a position projection model.

[0044] Location information is one of the features extracted from captured images. It is useful, especially when dealing with photos of tourist attractions, as the recommender model (described later) will be able to significantly narrow down to photos of scenes with geographically close proximity. Therefore, the goal of a location projection model is to obtain features about the geographical location of the photograph. The simplest approach is to extract latitude and longitude coordinates from the camera's GPS information and embed them into this feature. However, when considering image similarity, there are other important factors to consider, such as altitude, because photos of the same object taken from high altitudes and low altitudes still differ significantly.

[0045] In alternative embodiments, other feature extraction models can also be utilized. These models may include object detection models that accurately locate and classify objects of interest appearing in the image. This functionality is important because it provides information about the context of the photograph. It also helps in understanding what the focus of the photograph is, a key factor in ensuring good composition.

[0046] Another important factor to consider is the conditions under which the photos were taken. The purpose of environmental condition modeling is to extract information about the surrounding environmental conditions based on the information shown in the photos. Some relevant factors are lighting and weather. This is because images of the same scene taken under different environmental conditions may not be suitable for integration. Therefore, depending on the use case, it is important to use photos taken under similar conditions as references for the input photos.

[0047] Additional sensors will be installed on the camera to provide data such as temperature, fog, and humidity. Combined with the information gathered from the hardware, the model will be able to better identify the environmental conditions the user is currently experiencing.

[0048] The recommender module 106 is trained so that for each new query received, it returns the closest image from the cloud database 102 based on the image features extracted by the feature extraction module 104 described above. Factors indicating the similarity between images include, but are not limited to, the geographical location of the photo, surrounding conditions, and objects detected in the image. The goal of the recommender module 106 is to recommend several similar images as references to improve the quality and aesthetic value of new photos.

[0049] Only images with similarity scores exceeding a certain threshold are considered to ensure the quality of the resulting enhanced photos. Even if a single image with identical content does not exist in cloud database 102, the quality of the original photo will at least be maintained and will not be degraded by incorrectly suggested images. Another point to consider is that the weight of each feature when determining similarity between images may differ depending on the use case.

[0050] Using information from the image obtained in the previous step, an image enhancement module 108 is introduced to increase the value of the captured photograph. Instead of using super-resolution, a deblurring or denoising model is employed, where photos with the same content but better resolution and quality can be used to enhance the details of a given photograph. For example, at each tourist attraction, many photos are taken using various types of cameras. A cloud database 102 will contain high-quality photos of these locations (e.g., photos taken using a DSLR camera). When a new photo of the same scene is taken using a smartphone camera with lower photographic capabilities, the model can use high-frequency details from the aforementioned high-quality photos to enhance the same objects in the new photo.

[0051] First, align the photos, then merge them. An example method for merging details of different frequencies (i.e., low frequencies from the current photo and high frequencies from high-quality photos in cloud database 102) is to use Laplacian pyramid blending. Unlike existing super-resolution and deblurring deep learning models, these additional details are realistic. The similarity metric should consider location information and detected objects more, as it only requires high-frequency information. Features such as time and temperature have little impact, as they primarily affect low-frequency information in the image.

[0052] Sometimes, unwanted objects appear in a photograph, and people want to remove them for aesthetic reasons. A common use case is portraits taken at tourist attractions, where people want to remove crowds from the background. However, after removing objects from an image, the gaps created need to be filled with a visually realistic background. Therefore, by collecting aligned photos of the same scene, missing information from the background can be found and used to replace the gaps.

[0053] Unlike existing models that only use information from undisturbed image regions, the method disclosed in this invention leverages additional realistic details from those regions in other images, thus resulting in a more seamless reconstruction. For similarity metrics in this use case, environmental condition features are as important as location information, as the reference images should have similar contexts. Mixing day and night or summer and winter in the same image is unreasonable.

[0054] One factor influencing the aesthetic value of a photograph is its composition. There are methods to arrange various elements within a scene within a frame, resulting in a more visually appealing image. However, users may not know the optimal shooting angle. Furthermore, certain angles may be unattainable due to environmental limitations. This application utilizes a collection of images that may have been taken from different but adjacent locations and constructs a 3D virtual space for that area.

[0055] Simply put, these images contain some objects of interest, but are viewed from different angles. A 3D virtual space is constructed based on this set of images. The optimal viewing angle and distance are found so that the objects of interest in the images remain, but are reprojected to improve the composition of the photographs. Sometimes, camera lenses (especially smartphones) may not be wide enough to capture the entire scene. Users don't need to take multiple shots of the scene from different angles; the database will consist of multiple photographs that provide a better perspective for the images.

[0056] Figure 2The structure of the image enhancement system is illustrated. When the camera application is opened, the entire workflow is triggered, and the user places the camera of the electronic device at the shooting position to take a photo. The electronic device can be a smartphone, tablet, digital camera, or any other device equipped with a camera. In the primary embodiment, the device used is a smartphone. The captured photos and the hardware data 102b of the electronic device are collected by the image database 102a.

[0057] Image database 102a contains a collection of photographs taken using different types of camera models in various locations and environmental settings around the world. For the initial setup of image database 102a, photographs can be obtained from various online resources. The database can also acquire photographs from social networking sites or cloud storage on various electronic devices. Hardware data includes information such as location, temperature, and time. The hardware includes multiple sensors configured within electronic devices. Some examples of sensors include GPS, environmental sensors, compass sensors, barometer sensors, and light sensors.

[0058] Multiple machine learning models and feature extraction models configured within the feature extraction module 104 are known to extract features from captured images. These feature extraction models utilize information extracted from the hardware of the electronic device used to capture the images. Feature extraction models such as location classification, position projection, and environmental condition classification will be used to extract features from the images and their corresponding information.

[0059] Location information is useful, especially when it comes to photos of tourist attractions, because the recommender model (described later) is able to significantly narrow down to photos with geographically proximate scenes. Therefore, the goal of this model is to extract features about the geographical location where the photo was taken.

[0060] Another model, the object detection model, detects and classifies objects of interest appearing in an image. This function is important because it provides information about the context of the photo. It also helps in understanding what the focus of the photo is, which is a key factor in ensuring good composition.

[0061] Another important factor to consider is the shooting conditions. The purpose of environmental condition modeling is to extract information about the surrounding environmental conditions based on the information shown in the photograph. Some relevant factors are lighting and weather. This is because images of the same scene but shot under different environmental conditions may not be suitable for integration.

[0062] The feature extraction model then compares the extracted features with features from the image set stored in the image database 102a. Based on the comparison, the feature extraction model groups the image set, for example, grouping the photos based on the location where they were captured.

[0063] These images from the same locations can be used to construct a virtual 3D space. For each newly incoming photograph, matching points are found by comparing its detected keypoints and descriptors with those from other images. Therefore, to avoid redundant calculations, the calculated keypoint and descriptor information can also be stored along with the new image database 102a.

[0064] Similarly, for 3D space, it is not necessary to reconstruct the 3D space every time a new photo is obtained. Instead, the corresponding position of the new photo can be determined within the previously constructed 3D space. In this way, the search process can be accelerated when looking for other relevant photos in the image database 102a. Therefore, not only are features 110 extracted by the machine learning model running within the database collected, but also new features 112 corresponding to the hardware of the electronic device are collected.

[0065] The recommender module 106 is trained such that for each new query received, it returns the closest image from the image database 102a. Factors indicating the similarity between images include, but are not limited to, the geographical location where the photo was taken, surrounding conditions, and objects detected in the image. The goal of the recommender module 106 is to recommend several similar images 114 as references to improve the quality and aesthetic value of the new photo. Images with similarity scores exceeding a certain threshold are considered to ensure the quality of the resulting enhanced photo.

[0066] Image enhancement module 108 receives images recommended by recommender module 106. These images 114 are similar to the captured image in terms of previously extracted features. Upon receiving the recommended images 114, enhancement processing is activated. Depending on the purpose of enhancement, the captured photo is passed to the appropriate model to improve its quality by referencing similar-looking photos retrieved from image database 102a. Enhancement may include image deblurring or denoising, perspective transformation, finding a better viewpoint, or removing one or more unwanted objects from the image. The resulting enhanced photo 116 is then sent back to the user.

[0067] Figure 3 The feature extraction module of the image enhancement system is shown. The feature extraction module 104 of this invention includes two components: an extractor 118 and a comparator 120. Several processes are executed concurrently after the search process to extract features from the query image. Without a good reference photograph, the final enhanced image provided by the system is likely to be of low quality, regardless of the performance of the enhancement model.

[0068] Simple perceptual metrics (such as mean squared error, peak signal-to-noise ratio, or structural similarity index) are unsuitable for this system because they do not measure the similarity between two identical images. Instead, images are mapped to features, where the distance between features directly corresponds to image similarity. Some exemplary and useful feature extraction models include location projection, object detection, and environmental condition classification.

[0069] The extractor 118 extracts a large number of features from the captured image. These features are captured with the help of hardware associated with the electronic device used to capture the image. The electronic device can be a tablet, smartphone, or digital camera. The hardware associated with the electronic device includes multiple sensors that store image information. Examples of sensors include light sensors, ambient light sensors, barometer sensors, and compass sensors.

[0070] Location information is one of the features extracted by extractor 118. It is useful, especially when dealing with photos of tourist attractions, as the recommender module will be able to significantly narrow down to photos of scenes with geographically close proximity. Therefore, the purpose of this module is to obtain features about the geographical location of the photo. The simplest method is to extract the latitude and longitude coordinates from the camera's GPS information and embed them into this feature.

[0071] In object detection models, the goal is to accurately locate and classify objects of interest appearing in an image. This function is important because it provides information about the context of the photo. It also helps to understand what the focus of the photo is, which is a key factor in ensuring good composition. In short, the recommender module must take this information into account and find photos with similar backgrounds and focus points.

[0072] Another important factor to consider is the conditions under which the photos were taken. The purpose of module 104 is to extract information about the surrounding environmental conditions based on the information displayed in the photos. Some relevant factors are lighting and weather. This is because images of the same scene but taken under different environmental conditions may not be suitable for integration.

[0073] Therefore, depending on the use case, it is important to use photos taken under similar conditions as a reference for the input photos. The comparator in the feature extraction module uses this information to compare the features of the captured image with the features of images stored in the database.

[0074] Figure 4 An image enhancement module of an image enhancement system is shown. Using information from the system's recommender module, the image enhancement module 108 adds value to captured photographs. Some of the functions of the image enhancement module 108 include detail enhancement, restoration, perspective transformation, and wide-angle viewing, which are performed by dedicated units configured within the image enhancement module 108.

[0075] The detail enhancement unit 122 is responsible for deblurring and denoising the captured image. Photos with the same content but higher resolution and quality can be used to enhance the details of a given photo. For example, many photos are taken at each tourist attraction using various types of cameras. The database will contain high-quality photos of these locations (e.g., photos taken with a DSLR camera). When a new photo of the same scene is taken using a smartphone camera with lower photographic capabilities, model 104 can use the high-frequency details from the aforementioned high-quality photo to enhance the same object in the new photo.

[0076] First, the photos are aligned by the detail enhancement unit 122, and then merged. An example method for merging details of different frequencies (i.e., low frequencies from the current photo and high frequencies from photos in the high-quality database) is to use Laplacian pyramid blending. Unlike existing super-resolution and deblurring deep learning models, these additional details are realistic. The similarity metric should consider location information and detected objects more, as it only requires high-frequency information. Features such as time and temperature have little impact, as they primarily affect low-frequency information in the image.

[0077] In a photograph, there are sometimes unwanted objects that people want to remove for aesthetic reasons. A common use case is portraits taken at tourist attractions, where people want to remove crowds from the background. However, after removing objects from the image, these areas need to be filled with a visually realistic background. The repair unit 124 is responsible for removing unwanted objects and filling gaps. Therefore, by collecting aligned photos of the same scene, gaps can be replaced using actual information missing from the background.

[0078] Unlike existing models that only use information from undisturbed image regions, the method disclosed in this invention leverages additional realistic details from those regions in other images, thus resulting in a more seamless reconstruction. For similarity metrics in this use case, environmental condition features are as important as location information, as the reference images should have similar contexts. Mixing day and night or summer and winter in the same image is unreasonable.

[0079] One factor influencing the aesthetic value of a photograph is its composition. There are methods to arrange various elements within a scene within a frame, resulting in a more visually appealing photograph. However, users may not know the optimal shooting angle. Furthermore, certain angles may be unattainable due to environmental limitations. The perspective transformation unit 126 utilizes a collection of images that may have been taken from different but adjacent locations and can construct a 3D virtual space for that area.

[0080] Sometimes, camera lenses (especially smartphones) may not be wide enough to capture the entire scene. Again, an image database is used to utilize the reconstructed 3D space. The 3D image is projected onto the 2D space, making the original scene captured by the user the focal point, and cropping additional side views from the projection. This function is performed by the panorama generation unit 128. In other words, the user does not need to take multiple photos of the scene from different angles; this information can be compensated for using the reconstructed images described above. The user is no longer limited to the view captured in a series of photos and can flexibly choose the field of view to include because the images from the database extensively cover the scene.

[0081] Figure 5 A method for generating enhanced images based on intelligent cloud technology is illustrated. For example... Figure 5 As shown in flowchart 200a, a method is illustrated that includes the following.

[0082] Step 204: Receive an image from the electronic device.

[0083] Step 206: Store the image in the database.

[0084] Step 208: The feature extraction module extracts one or more features from the image.

[0085] In step 206, the image, along with some information associated with the image captured by the hardware of the electronic device, is sent to and stored in a cloud database. The database already stores multiple images obtained from multiple sources. Once the image is stored in the database, the feature extraction module extracts multiple features from the image. In step 208, feature extraction models such as location classification and object detection perform feature extraction with the help of extractors configured within the feature extraction module.

[0086] Step 210: Compare the image with multiple images stored in the cloud database based on one or more features. In step 210, the extracted features are compared with the features of multiple images stored in the cloud database, which is done by a comparator configured within the feature extraction module.

[0087] Step 212: Recommend one or more images from multiple images based on similarity scores.

[0088] In step 212, based on the comparison results generated by the comparator, the recommender module recommends one or more images based on the similarity score.

[0089] Step 214: Receive one or more images based on the recommendation.

[0090] In step 214, the recommended image is sent to the image enhancement module.

[0091] Step 216: Merge at least a portion of one or more images with the image to obtain an enhanced image via the image enhancement module.

[0092] Finally, in step 216, the image enhancement module merges at least a portion of one or more recommended images with the image to obtain an enhanced image.

[0093] Figure 6 A method for generating processed images is shown. For example... Figure 6 As shown in flowchart 200b, a method is illustrated, which includes steps 202 and 216. Specifically, step 202 includes the following.

[0094] Step 204: Receive an image from the electronic device.

[0095] Step 206: Store the image in the database.

[0096] Step 208: The feature extraction module extracts one or more features from the image.

[0097] In step 206, the image, along with some information associated with the image captured by the hardware of the electronic device, is sent to and stored in a cloud database. The database already stores multiple images obtained from multiple sources. Once the image is stored in the database, the feature extraction module extracts multiple features from the image. In step 208, feature extraction models such as location classification and object detection perform feature extraction with the help of extractors configured within the feature extraction module.

[0098] Step 210: Compare the image with multiple images stored in a cloud database based on one or more features.

[0099] In step 210, the extracted features are compared with features of multiple images stored in the cloud database, which is done by a comparator configured within the feature extraction module.

[0100] Step 212: Recommend one or more images from multiple images based on similarity scores.

[0101] In step 212, based on the comparison results generated by the comparator, the recommender module recommends one or more images based on the similarity score.

[0102] Step 214: Receive one or more images based on the recommendation.

[0103] In step 214, the recommended image is sent to the image enhancement module.

[0104] The image enhancement module includes multiple components for enhancing image quality by merging at least a portion of one or more images with the image shown in step 216, which also includes multiple steps listed below.

[0105] Step 218: Align multiple attributes of one or more images with the image and then merge them to generate an aligned image.

[0106] In step 218, the detail enhancement unit of the image enhancement module aligns and merges the details from one or more images with the captured image to generate an aligned image. The details include high frequencies and light intensity.

[0107] Step 220: Use multiple images to repair and align the images to generate a repaired image.

[0108] The image enhancement module's repair unit removes one or more unwanted objects from the aligned image and fills the gaps created using the recommended image to generate a repaired image.

[0109] Step 222: Transform the repaired image by a certain angle to generate a transformed image.

[0110] In step 222, a suitable viewpoint for the repaired image is found to generate the transformed image.

[0111] Step 224: Generate a wide field of view of the transformed image to generate an enhanced image.

[0112] Finally, in step 224, the panoramic generation unit of the image enhancement module covers a wider field of view of the transformed image to generate a processed image.

[0113] Figure 7 A block diagram of an electronic device according to the present invention is shown.

[0114] Reference Figure 7 The electronic device 700 includes a processing component 710, which further includes one or more processors, and memory resources represented by memory 720 for storing instructions, such as application programs, that can be executed by the processing component 710. The application programs stored in memory 720 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 710 is configured to execute instructions to perform the aforementioned method for generating enhanced images.

[0115] Electronic device 700 may also include a power supply component configured to perform power management of electronic device 700, a wired or wireless network interface configured to connect electronic device 700 to a network, and an input / output (I / O) interface. Electronic device 700 can operate on an operating system, such as Windows Server, stored in memory 1420. TMMac OSX TM Unix TM Linux TM FreeBSD TM Or similar.

[0116] A non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of the aforementioned electronic device 700, the electronic device 700 is able to perform the aforementioned method for generating an enhanced image.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0118] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0122] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program verification codes.

[0123] Furthermore, it should be noted that the combination of the various technical features in this case is not limited to the combination methods described in the claims of this case or the combination methods described in the specific embodiments. All technical features described in this case can be freely combined or combined in any way, unless they contradict each other.

[0124] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

[0125] While various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as a limitation. Similarly, the figures may depict exemplary architectures or other configurations of the invention to aid in understanding the features and functions that may be included in the invention. The invention is not limited to the exemplary architectures or configurations shown, but various alternative architectures and configurations may be used to achieve the desired features.

[0126] Although the invention has been described above with reference to various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functions described in one or more individual embodiments are not limited to those applicable to the specific embodiments in which they are described, whether or not those embodiments are described, and whether or not these features are presented as part of the described embodiments. Therefore, the breadth and scope of the invention should not be limited by any of the exemplary embodiments described above.

[0127] In some cases, the appearance of broadening words and phrases such as “one or more,” “at least,” “but not limited to,” or other similar phrases should not be interpreted as an intentional or necessary use of a narrower context where such broadening phrases may not exist.

Claims

1. An image enhancement system based on intelligent cloud, used for image enhancement, characterized in that, The image enhancement system includes: Cloud databases are used to receive images from electronic devices; The feature extraction module includes: Extractor for extracting one or more features from the image, the one or more features including GPS location; and A comparator is used to compare the image with a plurality of images stored in the cloud database based on one or more of the features; A recommender module is configured to recommend one or more images from the plurality of images based on a similarity score with the image; and An image enhancement module is configured to receive the one or more images from the recommender module, and merge at least a portion of the one or more images with the image to obtain an enhanced image. The feature extraction module groups the multiple images based on the GPS location of the image to obtain a group of images with the same or adjacent locations as the image, and the image enhancement module creates a three-dimensional virtual space for the group of images. The image enhancement module includes a panoramic generation unit, which uses the image as the focal center to project the three-dimensional virtual space onto a two-dimensional plane and crop to obtain a panoramic image corresponding to the user-selected field of view, which is then used as the enhanced image. The field of view of the enhanced image is wider than that of the original image.

2. The image enhancement system according to claim 1, characterized in that, The electronic device is any one of a smartphone, laptop, digital camera, or tablet computer.

3. The image enhancement system according to claim 1, characterized in that, The one or more features of the image are obtained from one or more sensors of the electronic device.

4. The image enhancement system according to claim 3, characterized in that, The one or more sensors include at least one of a Global Positioning System (GPS) sensor, an environmental sensor, a light sensor, a compass sensor, and a barometer sensor.

5. The image enhancement system according to claim 1, characterized in that, The similarity score is based on the weights assigned to the one or more features.

6. The image enhancement system according to claim 1, characterized in that, The image enhancement module further performs deblurring and noise reduction processing on the image to generate the enhanced image.

7. The image enhancement system according to claim 1, characterized in that, The image enhancement module includes an image alignment unit, which is used to align multiple attributes of the one or more images with the image and then merge them to generate an aligned image.

8. The image enhancement system according to claim 7, characterized in that, These properties are based on high frequency and light intensity.

9. The image enhancement system according to claim 7, characterized in that, The image enhancement module further includes a repair unit for repairing the aligned image with the plurality of images to generate a repaired image.

10. The image enhancement system according to claim 9, characterized in that, The image enhancement module also includes a perspective transformation unit, which is used to transform the repaired image at a certain angle to generate a transformed image.

11. The image enhancement system according to claim 10, characterized in that, The image enhancement module also includes a panoramic generation unit for generating a wide field of view of the transformed image to generate an enhanced image.

12. An image enhancement system based on intelligent cloud, used for enhancing images, characterized in that, The image enhancement system includes: Cloud databases are used to receive images from electronic devices; The feature extraction module includes: Extractor for extracting one or more features from the image, the one or more features including GPS location; and A comparator is used to compare the image with a plurality of images stored in the cloud database based on one or more of the features; A recommender module is configured to recommend one or more images from the plurality of images based on a similarity score with the image; and An image enhancement module is configured to receive the one or more images from the recommender module, wherein the image enhancement module includes: An image alignment unit is used to align multiple attributes of one or more images with the images and then merge them to generate an aligned image, wherein the multiple attributes are based on high frequency and light intensity; A repair unit is used to repair the aligned image using the plurality of images to generate a repaired image; A perspective transformation unit is used to transform the repaired image at a certain angle to generate a transformed image; and A panoramic generation unit is used to generate a wide field of view of the transformed image to generate an enhanced image. The feature extraction module groups the multiple images based on the GPS location of the image to obtain a group of images with the same or adjacent locations as the image, and the image enhancement module creates a three-dimensional virtual space for the group of images. The image enhancement module includes a panoramic generation unit, which uses the image as the focal center to project the three-dimensional virtual space onto a two-dimensional plane and crop to obtain a panoramic image corresponding to the user-selected field of view, which is then used as the enhanced image. The field of view of the enhanced image is wider than that of the original image.

13. A method for generating enhanced images, characterized in that, include: Receive images from electronic devices; The image is stored in a cloud database; One or more features are extracted from the image by a feature extraction module, the one or more features including GPS location; Based on one or more of the features, the image is compared with multiple images stored in the cloud database; Based on the similarity score with the image, one or more images are recommended from the plurality of images; Based on the recommendation, receive one or more of the images; as well as At least a portion of the one or more images is merged with the image to obtain an enhanced image. The method further includes: The multiple images are grouped based on their GPS locations to obtain a group of images with the same or adjacent locations as the images, and a three-dimensional virtual space of the group of images is created. Using the image as the focal center, the three-dimensional virtual space is projected onto a two-dimensional plane, and a panoramic image corresponding to the user-selected field of view is obtained by cropping, which is used as the enhanced image. The field of view of the enhanced image is wider than that of the image.

14. A method for generating enhanced images, characterized in that, include: Receive images from electronic devices; The image is stored in a cloud database; One or more features are extracted from the image by a feature extraction module, the one or more features including GPS location; Based on one or more of the features, the image is compared with multiple images stored in the cloud database; Based on the similarity score with the image, one or more images are recommended from the plurality of images; Based on the recommendation, receive one or more of the images; Multiple attributes of the one or more images are aligned with the image and then merged to generate an aligned image, wherein the multiple attributes are based on high frequency and light intensity; The aligned image is repaired using the multiple images to generate a repaired image; The repaired image is transformed by a certain angle to generate a transformed image; and Generate a wide field of view of the transformed image to generate an enhanced image. The method further includes: The multiple images are grouped based on their GPS locations to obtain a group of images with the same or adjacent locations as the images, and a three-dimensional virtual space of the group of images is created. Using the image as the focal center, the three-dimensional virtual space is projected onto a two-dimensional plane, and a panoramic image corresponding to the user-selected field of view is obtained by cropping, which is used as the enhanced image. The field of view of the enhanced image is wider than that of the image.

15. A computer-readable storage medium, characterized in that, It includes computer program logic for enabling at least one processor in a computer system to generate enhanced images based on a smart cloud via a software platform, said computer program logic including the method as described in claim 13 or 14.

16. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions. The processor is used to perform the method as described in claim 13 or 14.

Images