Photo feedback method
Through cloud database combining object detection and environmental condition classification, camera settings and shooting position are adjusted in real time, the problem of unrelated photography feedback in the existing technology is solved, executable guidance is provided, the composition and quality of photos is improved, and the improvement is achieved. Efficient shooting experience.
Patent Information
- Application Number
- CN202111081443.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-10-01
- Filing Date
- 2021-09-15
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-09-15
AI Technical Summary
Existing photo feedback technology ignores the scene content, resulting in the searched results being unrelated to the location and environment or being unrealistic. They only provide reference images and cannot provide executable guidance to users, and cannot improve the composition and quality of photos in real time.
Through a cloud database combining object detection, location projection, environmental condition classification and feature extraction, camera settings and shooting locations are adjusted in real time, and similar photos based on user history and preferences are provided as references to improve the aesthetic quality and composition of the query images.
It provides executable action guidance, improves the composition and quality of photos in real time, ensures that the photos taken more in line with the scene and environmental conditions expected by users, and improves the aesthetic quality and user experience of photos.
Smart Images

Figure CN113794837B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to convolutional neural networks, and in particular to providing picture taking feedback. Background Art
[0002] Currently, deep learning and artificial intelligence are becoming more and more popular in the mobile industry. In particular, various deep learning techniques (such as image enhancement, face detection, and facial recognition) can be found to have been implemented in mobile devices and combined with cameras. Summary of the invention
[0003] An exemplary photo feedback method includes: inputting a query image, camera settings, camera position, time stamp, and environmental conditions; detecting an object in the query image; projecting a position of the object; classifying the environmental conditions at the position of the object; querying features of the query image of the object under the environmental conditions of the projected position of the object; determining an image that is closest to the query image of the object under the environmental conditions of the projected position of the object and has a high rating value by means of a cloud database; and sending a position for an updated image and at least one camera setting for the updated image.
[0004] In one of the embodiments, the at least one updated camera setting includes at least one of ISO, aperture, and shutter speed.
[0005] In one of the embodiments, the detecting of the object in the query image is performed by means of the cloud database.
[0006] In one of the embodiments, projecting the position of the object is performed by means of the cloud database.
[0007] In one of the embodiments, classifying the environmental condition is performed by means of the cloud database.
[0008] In one of the embodiments, querying the features of the query image is performed by using the cloud database.
[0009] In one embodiment, the cloud database includes a collection of worldwide photos input under different environmental conditions.
[0010] In one of the embodiments, determining the closest image is based on one of a user's usage history and a user preference.
[0011] In one embodiment, the updated position includes a pose of the camera.
[0012] In one embodiment, the updated position includes an orientation of an element in the updated image.
[0013] In one embodiment, the updated position includes a relative scale of an element in the updated image.
[0014] In one of the embodiments, the updated position includes scene content of the updated image.
[0015] In one of the embodiments, the updated position includes one of an orientation offset and a global orientation change.
[0016] In one of the embodiments, sending the updated location and the at least one updated camera setting is performed in real time or near real time.
[0017] In one embodiment, the updated location includes navigation instructions.
[0018] In one of the embodiments, the at least one updated camera setting includes adjusting a camera setting via instructions sent from the cloud database.
[0019] In one of the embodiments, the closest image is selected based on geographic location.
[0020] In one of the embodiments, the closest image is selected based on features that map the query image to a cloud database image, and wherein a distance between the features indicates image similarity.
[0021] In one embodiment, the closest image is selected based on a focus point of the object in the query image, wherein the query image is similar to the photo in the cloud database in terms of scene content and focus of the picture.
[0022] In one of the embodiments, the closest image is selected based on the geographic location and the environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In the attached picture:
[0024] Figure 1 is a schematic diagram of a first exemplary system according to an embodiment of the present invention;
[0025] Figure 2 is a schematic diagram of a second exemplary system according to an embodiment of the present invention;
[0026] Figure 3 is an exemplary photo feedback architecture according to one embodiment of the present invention; and
[0027] Figure 4 is an exemplary method according to one embodiment of the present invention. DETAILED DESCRIPTION
[0028] The following examples are only used to illustrate the application of the device and method, rather than to limit the scope of the present invention. Modifications of equivalent forms of modifications to the device and method should fall within the scope of the claims.
[0029] Certain terms used throughout the following specification and claims are used to refer to specific system components. As will be appreciated by those skilled in the art, different companies may refer to components and / or methods by different names. This document is not intended to distinguish between components and / or methods that are different in name but not in function.
[0030] In the following discussion and in the claims, the terms "including" and "comprising" are used in an open-ended manner, and thus may be interpreted to mean "including, but not limited to...". Additionally, the terms "coupled" or "connected coupled (third person)" are intended to represent either an indirect or direct connection. Thus, if a first device couples with a second device, that connection may be achieved through a direct connection or through an indirect connection via other devices and connections.
[0031] Figure 1 An exemplary hybrid computing system 100 is depicted that can be used to implement a deep neural network associated with one or more portions or steps in process 600. In this example, processors associated with the hybrid system include a field programmable gate array (FPGA) 122, a graphics processing unit (GPU) 120, and a central processing unit (CPU) 118.
[0032] The CPU 118, GPU 120, and FPGA 122 have the ability to provide a neural network. The CPU is a general-purpose processor that can perform many different functions. Its versatility makes it capable of performing many different tasks, however, it is limited in its processing of multiple data streams, and its functionality with respect to neural networks is very limited. The GPU is a graphics processor that has many small processing cores that can sequentially process parallel tasks. The FPGA is a field programmable device that has the ability to be reconfigured and perform any function that can be programmed into the CPU or GPU in a hard-wired circuit. Since the FPGA is programmed in circuit form, its speed is several times faster than the CPU and significantly faster than the GPU.
[0033] The system may include other types of processors, such as accelerated processing units (APUs, which include CPUs with on-chip GPU elements) and digital signal processors (DSPs) that are specialized for performing high-speed digital data processing. Application-specific integrated circuits (ASICs) may also perform the hardwired functions of an FPGA, however, the lead time for designing and producing an ASIC is approximately three months, which is not a fast-turnaround implementation option available in programming an FPGA.
[0034] The GPU 120, CPU 118 and FPGA 122 are connected and connected to the memory interface and controller 112. The FPGA is connected to the memory interface through a programmable logic circuit (PLC) to memory interconnect 130. Due to the fact that the operating bandwidth of the FPGA is very large and in order to minimize the circuitry utilized by the FPGA to perform storage tasks, additional devices can be used. The memory interface and controller 112 is additionally connected to the permanent storage disk 110, the system memory 114 and the read-only memory (ROM) 116.
[0035] Figure 1 The system can be used to program and train the FPGA. The GPU works well with unstructured data and can be used for training. Once the data is trained, a deterministic inference model can be found, and the CPU can program the FPGA using the model data determined by the GPU.
[0036] The memory interface and controller are connected to a central interconnect 124, which is further connected to GPU 120, CPU 118, and FPGA 122. Central interconnect 124 is further connected to input and output interfaces 128 and network interface 126.
[0037] Figure 2 A second exemplary hybrid computing system 200 is depicted that may be used to implement a neural network associated with the operation of one or more portions or steps of process 1000. In this example, the processor associated with the hybrid system includes a field programmable gate array (FPGA) 210 and a central processing unit (CPU) 220.
[0038] The FPGA is electrically connected to an FPGA controller 212, which is connected to a direct memory access (DMA) 218 interface. The DMA 218 is connected to an input buffer 214 and an output buffer 216, both of which are connected to the FPGA to buffer data to and from the FPGA, respectively. The DMA 218 has two first-in-first-out (FIFO) buffers, one for the host CPU and the other for the FPGA, and the DMA allows data to be written to or read from the appropriate buffer.
[0039] On the CPU side of the DMA is a master switch 228 that transfers data and commands to and from the DMA. The DMA is also connected to a synchronous dynamic random access memory (SDRAM) controller 224 that allows data to be transferred between the FPGA and the CPU 220, which is also connected to an external SDRAM 226 and the CPU 220. The master switch 228 is connected to a peripheral device interface 230. A flash memory controller 222 controls permanent storage and is connected to the CPU 220.
[0040] Currently, there are a variety of post-processing techniques that can be applied to photos to provide enhanced resolution, brightness adjustment, etc. Post-processing can be performed on the image to adjust its composition within the photo, so that the aesthetic quality can be improved. In one operation, photo cropping can be performed to remove irrelevant content and modify the aspect ratio of the photo, which will produce an improved composition structure. In order to determine the area to be cropped, different methods are used, such as attention-driven methods, aesthetic-driven methods, and data-driven methods. The attention-driven method identifies the most informative areas of the image, the aesthetic-driven method focuses on the overall aesthetic quality of the image, and the data-driven method trains an end-to-end neural network cropping model. Cropping a photo is not enough to change the relative proportions and orientations of the elements in the photo.
[0041] Other post-processing operations may be performed, such as retargeting the image and moving elements toward updated positions within the photograph. This approach may include warping, patch rearrangement, and seam carving. Another post-processing method currently in use is image overlapping, which is the synthesis of multiple images in a single photograph. A potential disadvantage of using these post-processing methods is that the amount of information to be processed is limited to the amount of information available in the photograph, which limits the extent to which the quality of the photograph can potentially be improved. Furthermore, it is problematic to use these post-processing operations to modify the relative proportions of objects in a realistic manner.
[0042] The current solution modifies the position settings of the photo by preprocessing the query image and the camera hardware sensor data in part. The position settings can be modified over a small range (e.g., translating the photo or changing the angle of the photo) or over a large range (e.g., physically moving from a first photo position to an updated position), resulting in a better composite photo. In addition, updated camera settings (e.g., ISO (sensitivity), aperture, and shutter speed) based on the query image and the camera hardware sensor output can have a beneficial effect on the aesthetic quality of the photo.
[0043] Pre-processing based in part on query images and hardware sensor data to find updated positions and updated camera settings can help prevent photo errors, rather than trying to correct them later by relying on post-processing techniques to correct bad photos. The proposed solution can include improving the composition and quality of the final photo in real time before the photo is taken.
[0044] In one embodiment, based on the current location and surrounding conditions of the photo, the photographer is provided guidance on how to position the camera and what settings to use to achieve high quality results by suggesting or setting camera settings and sending direction on where to reposition the photographer or how to recast the composition of the photo.
[0045] One possible part of the proposed solution is to utilize cloud databases ( Figure 3 , 310), the cloud database contains a collection of photos taken at various locations and in different environmental settings around the world.
[0046] The cloud database can be cleaned by going through a data cleaning process guided by photography experts. Figure 3 , 310) to ensure that the database includes aesthetically pleasing and pleasing photos. Database photos may be rated based on their aesthetic quality, and those with scores below a predetermined threshold may be removed from the database. In one embodiment, exchangeable image file format (EXIF) data may be stored to provide information about the date and time of the photo, camera settings used, location, etc.
[0047] In one embodiment, the model is trained so that when a query is received, it can be retrieved from the cloud database ( Figure 3 , 310) returns the closest image ( Figure 3 , 322). The similarity between the proposed photo and the database images may include the geographic location where the photo was taken, the surrounding conditions, the objects detected in the image, etc.
[0048] An example output of this model would be to recommend similar photos ( Figure 3 , 332) as a reference to improve the query image ( Figure 3 , 314) quality and composition. The perceptual assessment of photo quality is subjective and varies between users due to personal preferences. Therefore, the model can take into account the user's history ( Figure 3 , 328), so that future recommended images will tend toward the user’s preferences.
[0049] Current real-time solutions ignore aspects of the photo such as scene content and may retrieve results that are irrelevant or unrealistic for the location and environment, and only provide example photos as reference feedback for the user. Providing reference photos may not be useful to the user because it does not provide actionable items for the photographer to achieve similar results.
[0050] The proposed method provides executable actions ( Figure 3 , 336, 338) instead of just providing a reference image, and using information from the entire image content to retrieve related images.
[0051] exist Figure 3 An example of the workflow of the proposed solution is shown in When the camera application is accessed and the user positions the camera to take a picture, an example of the proposed method is instantiated.
[0052] Figure 3 and Figure 4 An exemplary photo feedback method is described, comprising: inputting a query image ( Figure 3 , 314), Camera Settings ( Figure 3 , 312), location, time stamp, and environmental conditions ( Figure 4 , 410); detection( Figure 3 , 318) the object in the query image ( Figure 4 , 412) and the position of the object ( Figure 3 , 320) for projection ( Figure 4 , 414). The method further includes: classifying the environmental conditions at the location of the object ( Figure 3 ,322)( Figure 4 , 416) and query ( Figure 3 , 326) features of the query image of the object under the environmental conditions of the projected position of the object ( Figure 4 , 418). The method further includes: relying on a cloud database ( Figure 3 ,310) confirm( Figure 3 , 332) an image that is closest to the query image of the object under the environmental conditions of the projected position of the object and has a high rating value ( Figure 4 , 420), and sends the updated position for the updated image ( Figure 3 , 336) and at least one updated camera setting for the updated image ( Figure 3 , 338)( Figure 4 , 422).
[0053] The method may also include: utilizing a cloud database to perform at least one of: detecting an object in a query image, projecting a location of the object, classifying an environmental condition of the query image, and querying a feature of the query image. The cloud database may include a collection of world-wide photos input under different environmental conditions. Determining the closest image may be based on one of a user's usage history and preferences.
[0054] The updated position may include at least one of: a pose of the camera, an orientation of elements within the updated image, a relative scale of elements within the updated image, scene content of the updated image, and an orientation offset and a global orientation change. The updated position and at least one updated camera setting may be performed in real time or near real time. The updated position may include navigation instructions to move to the updated position.
[0055] The at least one updated camera setting may include adjusting a camera setting via instructions sent from a cloud database, and may include at least one of ISO, aperture, and shutter speed.
[0056] The closest image can be selected based on geographic location; the closest image can be selected based on features that map a query image to a cloud database image, wherein the distance between the features indicates image similarity; the closest image can be selected based on a focus point of the object in the query image, wherein the query image is similar to a photo in the cloud database in terms of scene content and focus point of the picture; and the closest image can be selected based on geographic location and environmental conditions.
[0057] A query image can be sent to a cloud database to search for similar photos. A query image is a sample of a scene captured by the system from the application when the camera is positioned.
[0058] In addition to images, data collected from the camera hardware can be provided to assist the search process. Several processes can be run simultaneously, such as object detection, location projection, environmental condition classification, and feature extraction for comparison with photos in the database for similarity determination.
[0059] A query image can be passed to a photo optimization model to improve the aesthetic quality of the query image by referencing similar-looking photos retrieved from a database. Based on the model, suggestions and / or guidance on camera pose changes and camera settings for obtaining better quality photos and camera settings will be sent directly to the user or the camera system.
[0060] Feature extraction model
[0061] Figure 3 A feature extraction model module 316 is depicted, where multiple processes can be performed simultaneously to extract features from the query image 314 and the cloud database 310. In one embodiment, the extracted features are utilized by a recommender model module (recommender model) to evaluate the similarity between the query image 314 and the images of the cloud database 310, and determine those images of the cloud database 310 that are closest to the query image 314. In one embodiment, the proposed solution maps images to features, where the distance between the features directly corresponds to image similarity. Note that perceptual metrics used in current systems (such as mean squared error, peak signal-to-noise ratio, or structural similarity index) are not applicable to the proposed solution because it does not measure the similarity between two identical images.
[0062] Object Detection
[0063] Figure 3 An object detection module 318 is depicted where objects of interest that appear in a query image are pinpointed and classified. This feature can provide information about the scene content of the frame of the photo. It can also indicate the focus of the photo, which can be a factor in ensuring good photo composition. The recommender model module can use this information to find photos with similar scene content and focus. This can provide additional utility when there are no photos in the database that match the location of the query photo.
[0064] Position Projection
[0065] Figure 3 A location projection module 320 is depicted, where location information may be particularly useful when it comes to photos of tourist attractions, as the recommender model module will be able to significantly narrow the search to photos with geographically close scenes. Therefore, the model can obtain features about the geographical location of the photos. The model can extract the latitude and longitude coordinates from the camera's GPS information and embed them into the features. The model can consider additional factors (such as latitude) when considering the similarity of images, as there are still significant differences in photos of the same subject taken from a high perspective and a low perspective.
[0066] Environmental Condition Classification
[0067] Figure 3 An environmental condition classification module 322 is depicted, which can influence the evaluation of the similarity between the query image and the cloud database image based on the conditions under which the photo was taken. The model can extract information related to the surrounding environmental conditions displayed in the photo. Some factors that have environmental influences are light and weather. Photos of the same scene under different lighting conditions can benefit from different shooting angles and are considered aesthetic photos. Therefore, it may be preferable to use photos taken under similar conditions as a reference for the query image. To ensure the reliability of feature extraction, camera information such as time and date can be utilized in combination with information visually obtained from the image. Additional sensors can also be provided on the camera to provide data such as temperature, fog, and humidity. Combined with the information collected from the camera hardware sensors, the model is able to better identify the environmental conditions currently experienced by the user.
[0068] Recommender Model
[0069] Figure 3 The recommender model module 330 of the proposed solution is depicted, wherein the cloud database 310 contains a collection of photos taken at various locations around the world and under different environmental settings. In an exemplary embodiment, the quality of the photos is ensured through a data cleaning process guided by a photography expert, wherein the photos are rated based on their aesthetic quality and photos with scores below a predetermined threshold are deleted from the database. For the photos in the database, Exchangeable Image File Format (EXIF) data may also be stored. The EXIF data may provide information about the date and time of the associated photo, the camera settings used, the location, etc. The recommender model module 330 may receive database features 324, query features 326, and the user's historical preferences 328 to help determine the closest image 332 from the database 310.
[0070] Given a cloud database, the model can be trained so that for a new query received, the closest image can be returned from the cloud database. Factors that take into account the similarity between images may include the geographical location where the photo was taken, the surrounding conditions, the objects detected in the image, etc. The model can recommend multiple similar photos as reference photos to improve the quality and composition of the query image. The perceptual assessment of photo quality is subjective and varies between users due to personal preferences. Therefore, the model can take into account the user's history so that the recommended images will be consistent with the user's preferences.
[0071] Photo Optimization Model
[0072] Figure 3A photo optimization model module 334 of the proposed solution is depicted, which may include changing the camera pose position for taking a photo and changing camera settings. The photo optimization model module 334 may receive input from the hardware sensor data 312 and the query image 314 from the camera 308 .
[0073] Camera pose and orientation guidance
[0074] Figure 3 The position suggestion module 336 of the photo optimization model module 334 of the proposed solution is depicted. On a smaller scale, the photo composition can be improved by some small changes, such as tilting the camera's posture up, down, or slightly to the left or right. This can be achieved by aligning the query image with a reference image with similar scene content. The position suggestion model can use the camera's GPS position or the camera's accelerometer sensor data to determine the orientation. The model can transform the query image through geometric displacement so that the object and the reference image are aligned. Through this alignment process, information about the difference in camera orientation between these images can be estimated. Based on this information, the camera orientation offset to achieve the same composition effect as the reference image can be sent to the photographer as a guide.
[0075] On a larger scale, images can be taken at different but nearby locations based on a 3D virtual space of the area. Reference images can share objects of interest viewed from different perspectives based on local orientation. From this set of images, a 3D reconstruction can be created, and patterns can be observed about frequently visited locations producing high-quality photos at different times of the day or under different environmental conditions. The direction of the sunlight (which in turn affects exposure) can be incorporated into the 3D reconstruction. Using the 3D reconstruction information, the model can project the locations where photos were taken in the area based on given conditions, time, and date.
[0076] Camera settings update
[0077] Figure 3 The camera settings suggestion module 338 of the photo optimization model module 334 of the proposed solution is depicted. Since the EXIF data of the reference images can be extracted from the database, the camera settings used when these photos were taken are available. By referring to those photos taken under the same lighting and environmental conditions, at least one updated camera setting can be determined and suggested to the user or directly changed to the camera to obtain better exposure in the final photo taken. In addition, in the case that photos with the same lighting and environmental conditions do not exist in the cloud database, the model can be trained to estimate the updated camera settings under various conditions.
[0078] The proposed solution can provide users with immediate live photo feedback during a photo-taking session to obtain aesthetically pleasing photos.
[0079] When extracting information to decide which similar images to retrieve from the database, the system not only relies on machine learning models to interpret information from the images, but also uses data collected from hardware such as camera settings, GPS, and phone sensors to produce highly reliable information.
[0080] When deciding which images to retrieve, the proposed solution takes into account the user's preference on photo style based on the user's usage history.
[0081] The proposed solution provides feedback in terms of executable actions, rather than just providing similar reference photos. Thus, the solution enables users with less photography experience to take high-quality photos because specific guidance, navigation instructions and camera settings are directly updated to these users, or the user is explicitly informed which camera settings to update.
[0082] The proposed solution provides suggestions for global position changes based on a 3D spatial model of the user's nearby locations to guide the photographer to a better photo-taking position.
[0083] The system combines the position guidance with at least one updated camera setting to be modified to further improve the photo quality.
[0084] Those skilled in the art will appreciate that the various illustrative blocks, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative blocks, modules, elements, parts, methods and algorithms have been generally described above in terms of their functions. Whether this function is implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. Skilled technicians can implement the described functions in different ways for each specific application. Without departing from the scope of the subject technology, various components and blocks can be arranged differently (e.g., arranged in different orders, or divided in different ways).
[0085] It is understood that the specific order or hierarchy of steps in the disclosed process is an illustration of an exemplary method. Based on design preferences, it is understood that the specific order or hierarchy of steps in the process can be rearranged. Some steps can be performed simultaneously. The attached method claims present the elements of the various steps in a sample order and are not meant to be limited to the specific order or hierarchy presented.
[0086] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. The foregoing description provides various examples of the subject technology, and the subject technology is not limited to these examples. It is obvious to those skilled in the art that various modifications to these aspects are obvious, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language of the claims, wherein the reference to the singular element is not intended to mean "one and only one", unless specifically stated, but means "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. Pronouns in the male (e.g., his) include female and neutral (e.g., she and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the present invention. The predicates "configured to", "operable to" and "programmed to" do not imply any specific tangible or intangible modification to the subject, but are intended to be used interchangeably. For example, a processor configured to monitor and control an operation or component can also mean that the processor is programmed to monitor and control an operation, or that the processor is operable to monitor and control an operation. Likewise, a processor configured to execute code may be construed as a processor programmed to execute code or a processor operable to execute code.
[0087] Phrases such as "aspects" do not imply that such aspects are essential to the present technology, or that such aspects apply to all configurations of the subject technology. Disclosures relating to one aspect may apply to all configurations, or one or more configurations. An aspect may provide one or more examples. Phrases such as "aspects" may refer to one or more aspects, and vice versa. Phrases such as "embodiments" do not imply that such embodiments are essential to the subject technology, or that such embodiments apply to all configurations of the subject technology. Disclosures relating to one embodiment may apply to all embodiments, or one or more embodiments. An embodiment may provide one or more examples. Phrases such as "embodiments" may refer to one or more embodiments, and vice versa. Phrases such as "configurations" do not imply that such configurations are essential to the subject technology, or that such configurations apply to all configurations of the subject technology. Disclosures relating to one configuration may apply to all configurations, or one or more configurations. A configuration may provide one or more examples. Phrases such as "configurations" may refer to one or more configurations, and vice versa.
[0088] The word “exemplary” is used herein to mean “serving as an example or illustration.” Any aspect or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs.
[0089] All structural and functional equivalents to the elements of the various aspects described throughout the present invention that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, the disclosure herein is not intended to be dedicated to the public, regardless of whether such disclosure is explicitly recited in the claims. No element of any claim shall be interpreted under the provisions of 35 USC § 112, sixth paragraph, unless the element is expressly recited using the phrase "means for..." or, in the case of a method claim, the element is recited using the phrase "step for..." In addition, with respect to the use of "including," "having," or similar terms in the specification or claims, such terms are intended to be included in the manner of the term "comprising," similar to the interpretation of "comprising" when "including" is used as a conjunction in a claim.
[0090] References to "one embodiment," "an embodiment," "some embodiments," "various embodiments," or similar expressions indicate that a particular element or feature is included in at least one embodiment of the present invention. Although a phrase may appear in multiple places, the phrase does not necessarily refer to the same embodiment. In conjunction with the present invention, those skilled in the art will be able to design and incorporate any of a variety of mechanisms suitable for achieving the above-described functions.
[0091] It should be understood that the present invention teaches only one example of illustrative embodiments and that those skilled in the art may readily devise many variations of the invention after reading this disclosure and that the scope of the invention is determined by the following claims.
Claims
1. A photo feedback method, comprising: Input a query image and hardware sensor data of a camera, the hardware sensor data including camera settings, camera position, time stamp, and environmental conditions; detecting an object in the query image; Based on a 3D virtual space of the area of the object, a set of images are taken at different but adjacent positions, a 3D reconstruction is created from the set of images, and a position of the object where the photo was taken in the area is projected in the 3D virtual space based on given conditions, time and date by using information of the 3D reconstruction; classifying environmental conditions at the location of the object; extracting features from the query image and a cloud database based on a combination of the detection of the object, the projection of the location of the object, and the classification of the environmental condition; querying features of the query image of the object under environmental conditions at the projected position of the object; determining, from the cloud database, an image that is closest to the query image of the object under the environmental conditions of the projected position of the object and has a high rating value based on the features extracted from the cloud database, the features of the query image, the hardware sensor data of the camera, and the historical preferences of the user; optimizing the captured images based on the determined images that are closest to the query image of the object under the environmental conditions of the projected position of the object and have high rating values, the optimizing comprising changing the camera position and the camera settings; as well as An updated camera position for the updated image and at least one updated camera setting for the updated image are sent.
2. The photographing feedback method according to claim 1, wherein: The at least one updated camera setting includes at least one of ISO, aperture, and shutter speed.
3. The photographing feedback method according to claim 1, wherein: The detecting of the object in the query image is performed by means of the cloud database.
4. The photographing feedback method according to claim 1, wherein: The projecting of the position of the object is performed by means of the cloud database.
5. The photographing feedback method according to claim 1, wherein: Classifying the environmental conditions is performed by means of the cloud database.
6. The photographing feedback method according to claim 1, wherein: Querying the features of the query image is performed by means of the cloud database.
7. The photographing feedback method according to claim 1, wherein: The cloud database includes a collection of worldwide photos input under different environmental conditions.
8. The photographing feedback method according to claim 1, wherein: Determining the closest image is based on one of a user's usage history and a user preference.
9. The photographing feedback method according to claim 1, wherein: The updated position includes the pose of the camera.
10. The photographing feedback method according to claim 1, wherein: The updated position includes the orientation of the element in the updated image.
11. The photographing feedback method according to claim 1, wherein: The updated positions include relative proportions of elements in the updated image.
12. The photographing feedback method according to claim 1, wherein: The updated position includes scene content of the updated image.
13. The photographing feedback method according to claim 1, wherein: The updated position includes one of a position offset and a global position change.
14. The photographing feedback method according to claim 1, wherein: Transmitting the updated location and the at least one updated camera setting is performed in real time or near real time.
15. The photographing feedback method according to claim 1, wherein: The updated location includes navigation instructions.
16. The photographing feedback method according to claim 1, wherein: The at least one updated camera setting includes adjusting a camera setting via instructions sent from the cloud database.
17. The photographing feedback method according to claim 1, wherein: The closest image is selected based on geographic location.
18. The photographing feedback method according to claim 1, wherein: The closest image is selected based on features that map the query image to a cloud database image, and wherein a distance between the features indicates image similarity.
19. The photographing feedback method according to claim 1, wherein: The closest image is selected based on the focus point of the object in the query image, wherein the query image is similar to the photo in the cloud database in terms of scene content and focus point of the picture.
20. The photographing feedback method according to claim 1, wherein: The closest image is selected based on the geographic location and the environmental conditions.
