Computing device displaying image transformability information - Patents.com

The computing device predicts image convertibility by using metadata extraction and matching algorithms, addressing the inconvenience of preparing multiple images for stitching, thereby enhancing the efficiency of creating larger panoramic or spherical images.

JP2024544469APending Publication Date: 2024-12-03MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2024521856
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-29
Filing Date
2022-09-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing image stitching technologies require users to capture and prepare multiple images in advance to create larger panoramic or spherical images, which can be inconvenient and time-consuming.

Method used

A computing device that displays conversion possibility information, allowing users to predict whether their selected target image can be transformed into a larger image by stitching it with available images, using metadata extraction and image matching algorithms to determine compatibility and feasibility.

Benefits of technology

Enables users to efficiently determine if their images can be stitched into larger formats without the need for extensive pre-capture planning, saving time and resources by providing real-time conversion predictions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544469000001_ABST
    Figure 2024544469000001_ABST
Patent Text Reader

Abstract

A computing device is provided, the computing device including a processor coupled to a storage device configured to store image data and a memory storing a plurality of instructions that, when executed by the processor, cause the processor to select a target image from the image data. The processor is further configured to display convertibility information indicating that the target image can be converted into a larger image having a larger field of view by stitching at least a portion of the target image and an associated selector with another image. The processor is further configured to display the larger image upon receiving a user selection of the selector.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a computing device that displays image transformability information. [Background technology]

[0002] Various image stitching techniques and algorithms have been developed in recent years. Image stitching is the process of combining multiple photographic images with overlapping fields of view to generate a larger image, such as a spherical or panoramic image. Such image stitching techniques allow a larger image to be generated by stitching together multiple two-dimensional images of a scene taken substantially simultaneously from a two-dimensional camera. Since the images have slightly different views of a scene and contain visual features present in the scene that are shared among the images, a stitching algorithm can match the shared visual features and assemble a larger composite image that recreates a larger view of the scene than is shown in any of the individual images. Summary of the Invention

[0003] A computing device and method are provided. The computing device includes a storage device configured to store image data and a processor coupled to a memory storing instructions that, when executed by the processor, cause the processor to select a target image from the image data. The processor is further configured to display conversion possibility information, the conversion possibility information indicating that the target image can be converted into a larger image having a larger field of view by stitching another image together with at least a portion of the target image and an associated selector. The processor is further configured to convert the target image into the larger image upon receiving a user selection of the selector.

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description of the Invention. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Moreover, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure. [Brief description of the drawings]

[0005] [Figure 1] A schematic diagram of a computing system according to one exemplary implementation of the disclosure is shown, the computing system including a computing device configured to display convertibility information indicating that a target image can be converted into a larger image, and a server configured to generate the convertibility information and the larger image. [Figure 2A]FIG. 2 shows a schematic diagram of an exemplary graphical user interface (GUI) for displaying convertibility information adjacent to a target image and region of interest selected by a user using the computing system of FIG. [Figure 2B] 2 shows a schematic diagram of an exemplary graphical user interface (GUI) for displaying transformability information adjacent to a target image and a region of interest selected by a user using the computing system of FIG. 1; [Figure 2C] 2 shows a schematic diagram of an exemplary graphical user interface (GUI) for displaying transformability information adjacent to a target image and a region of interest selected by a user using the computing system of FIG. 1; [Figure 2D] 2 shows a schematic diagram of an exemplary graphical user interface (GUI) for displaying transformability information adjacent to a target image and a region of interest selected by a user using the computing system of FIG. 1; [Figure 2E] 2 shows a schematic diagram of an exemplary graphical user interface (GUI) for displaying transformability information adjacent to a target image and a region of interest selected by a user using the computing system of FIG. 1; [Figure 2F] 2 shows a schematic diagram of an exemplary graphical user interface (GUI) for displaying transformability information adjacent to a target image and a region of interest selected by a user using the computing system of FIG. 1; [Diagram 3] 2 is a schematic diagram of a map of a target image and other images displayed in one exemplary GUI displayed on the display of the computing system of FIG. 1. [Figure 4A]1 shows a schematic diagram of a number of exemplary GUIs of the computing system of FIG. 1 that are overlaid on the target image in the map of FIG. 3 to characterize other images that are within a threshold distance range of the target image to determine whether conversion of the target image to a larger image is possible. [Figure 4B] 1 shows a schematic diagram of a number of exemplary GUIs of the computing system of FIG. 1 that are overlaid on the target image in the map of FIG. 3 to characterize other images that are within a threshold distance range of the target image to determine whether conversion of the target image to a larger image is possible. [Figure 4C] 1 shows a schematic diagram of a number of exemplary GUIs of the computing system of FIG. 1 that are overlaid on the target image in the map of FIG. 3 to characterize other images that are within a threshold distance range of the target image to determine whether conversion of the target image to a larger image is possible. [Figure 4D] 1 shows a schematic diagram of a number of exemplary GUIs of the computing system of FIG. 1 that are overlaid on the target image in the map of FIG. 3 to characterize other images that are within a threshold distance range of the target image to determine whether conversion of the target image to a larger image is possible. [Figure 4E] 1 shows a schematic diagram of a number of exemplary GUIs of the computing system of FIG. 1 that are overlaid on the target image in the map of FIG. 3 to characterize other images that are within a threshold distance range of the target image to determine whether conversion of the target image to a larger image is possible. [Figure 5A] 2 shows a schematic diagram of one exemplary GUI displaying a larger image transformed from a target image using the computing system of FIG. 1; [Figure 5B]2 shows a schematic diagram of one exemplary GUI displaying a larger image transformed from a target image using the computing system of FIG. 1; [Figure 6] 1 shows a flowchart of a computerized method according to one exemplary implementation of the present disclosure. [Figure 7] 2 shows a block diagram of an example computing system that may be used to implement the computing system of FIG. 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As briefly described above, photo stitching tools and services may be used to create larger images, such as spherical or panoramic images, by stitching together multiple pictures taken at the same time in the same geographic location. One such photo stitching tool allows a user to upload various images captured at one time to a server, which stitches the various images together to form the larger image. However, a drawback with such an approach is that the user must be in possession of sufficient two-dimensional images to create the larger image, which is not always the case. If a user wishes to ensure the user's ability to use such tools, the user must plan ahead and ensure that the user has captured enough images to create the larger image at a later time. It may be inconvenient and time consuming for the user to prepare all of the images necessary to create the larger image in this manner.

[0007] To address this issue, a computing device is disclosed herein, which is configured to display conversion possibility information, which indicates whether it is possible to convert the target image into a larger image with a larger field of view by stitching together at least a portion of the target image with other images available to the platform. The conversion possibility information allows a user to foresee whether a target image that the user selects can be converted into a larger image before attempting to convert the target image into the larger image. This is beneficial because it allows a user to know whether enough images are available to stitch together the user's individual images into a larger image, since not all of the user's individual images can be converted into a larger image. For example, it may not be possible to create a larger image for a particular target image when not enough surrounding images are available in an associated image store, or when the location of the user's individual images is not identifiable.

[0008] FIG. 1 shows a schematic diagram of a computing system 1 according to one exemplary implementation of the disclosure, including a computing device 2 configured to display convertibility information 22 indicating that a target image 20 can be converted into a larger image 24, and a server 4 configured to generate the convertibility information 22 and the larger image 24. The computing device 2 may be any of various types of portable computing devices including integrated cameras, such as smartphones, tablet computing devices, digital cameras, and head-mounted display devices. Such configurable devices equipped with cameras may be carried by a user to various locations and used to capture images at the locations. Alternatively, the computing device 2 may not include an integrated camera and / or may not be portable. For example, the computing device 2 may be a laptop device, a desktop device, or a server computing device. Such devices may be used by a user to view stored images taken at other locations.

[0009] In the illustrated configuration, the computing device 2 includes a processor 10, a memory 6, and a storage device 14, which may be operatively coupled to one another by a communication bus. Each of the processor 10, the memory 6, and the storage device 14 may be configured as one or more physical components, such as one or more processor cores and / or one or more physical memory modules. The storage device 14 stores image data 16, such as digital photos and videos taken by a smartphone, tablet computing device, head mounted display device, or digital camera, downloaded from the web, or shared from other users. In this manner, it will be understood that the image data 16 not only includes the target image 20, but also other images 44.

[0010] Computing device 2 may further include an input device 8, which may be a keyboard, a mouse, a touch screen, a track pad, an accelerometer, a microphone, a camera, or any other suitable type of input device. In addition, computing device 2 may further include a display 12, such as a flat panel display or a virtual reality (VR) headset display configured to display images.

[0011] First, to obtain the convertibility information 22, the target image 20 is selected using the target image selector 18 of the processor 10 based on a user's selection. Alternatively, the processor 10 may be configured to programmatically or automatically select the target image 20 using the target image selector 18 that is suitable for generating a larger image without a user's selection. For example, the target image 20 may be programmatically or automatically selected using the target image selector 18 based on the location, time, subject, and user preferences of the target image 20. As some of several specific examples, one of such target images 20 may be a photo taken at a tourist spot or a historical landmark, and the user wants to convert the photo into a larger image 24, such as a spherical image or a panoramic image. In another example, a user may select a region of interest in an existing image to create a target image 20, such as a bounding box around an object, person, and scene person of interest. For example, a user may select a region of interest 23 to include a specific object, such as a building or tower, in the selected image and extract the region of interest to form a target image 20, as shown in FIG. 2F below. This allows a user to select only the portions of the image that are of interest and exclude from the target image 20 unnecessary or undesirable content, such as a building under construction, an unrelated person, undesirable parts of the scene, etc.

[0012] The selected target image 20 is provided to an image metadata extraction module 26 of the processor 10 to extract metadata information 34 for the target image 20. The metadata information 34 may include location (GPS) information and date and time information at which the target image 20 is captured. In addition to the GPS information, the metadata information may include compass information indicating the orientation of the camera relative to true north at the time the image is captured. To this end, it will be appreciated that the computing device may include an inertial measurement unit (IMU), which may include both a magnetic compass and an accelerometer configured to generate compass information, and which may be configured to output a six degree of freedom (6DOF) pose vector, which indicates the position and orientation of the computing device in world space. Alternatively, instead of an IMU, a magnetic compass may provide a compass bearing that is included in the metadata information 34. From this pose vector or compass bearing, the camera's field of view in world space may be estimated. The metadata information may further include detected GPS coordinates and associated place names associated with the location where the image was taken. Image filtering based on compass bearing is illustrated in Figure 3 and described below.

[0013] The metadata information 34 may further include object information of the target image 20. The object information may be obtained by running object recognition algorithms on the target image 20. For example, the object recognition algorithms may be configured to identify well-known architectures, monuments, fountains, statues, paintings, or other works of art, as some of several examples, and include their names, locations, or other identifying information as object information. For example, the algorithms may be trained to recognize the Taj Mahal, the White House, the Washington Monument, the Brandenburg Gate, the Mona Lisa painting, or various other well-known landmarks around the world. Figures 4A and 4B illustrate an exemplary object tag in the target image 20 indicating that the Taj Mahal has been recognized in the target image. In this example, the object information may include the name of the object and the location of the object in world space. The object information may also include image segmentation information, such as a bounding box surrounding a region of interest containing the object, as illustrated by the dashed lines in Figures 4A and 4B, which identifies the location of the recognized object in the target image.

[0014] The metadata information 34 may further include feature information of the target image 20, such as a binned representation of the image's tonal values, i.e., the RGB (e.g., red, green, and blue) values ​​of the image organized into a histogram, which can be useful for matching the target image 20 with other images 44 having similar color profiles.

[0015] The metadata information may further include extracted visual features such as so-called scale-invariant feature transform (SIFT) features extracted by the scale-invariant feature transform (SIFT) algorithm, binary robust independent elementary features (BRIEF) extracted by the binary robust independent elementary features (BRIEF) algorithm, gradients binned in histograms by the histogram of oriented gradient algorithm (HoG), etc. Examples of extracted visual features are illustrated in Figures 4A and 4B.

[0016] Referring to FIG. 1, if the server determines that the conversion is possible, at least a part of the extracted metadata information 34 of the target image 20 may be transmitted to the server 4 via a server communication module 32 of a larger image conversion possibility display module 28 in order to obtain conversion possibility information 22 indicating that the target image 20 can be converted into a larger image 24 having a larger field of view by stitching at least a part of the target image 20 and the other image 44. The system 1 may determine that the two images are stitchable if there is a sufficient correspondence between a number of extracted visual features in each image and the color histogram of each image. The system 1 determines whether one or more other images 44 can be stitched in such a way as to cover the entire field of view of the potential larger image 24, as shown in FIG. 4A. As shown in Fig. 4B, if there is any gap in the stitched image in the field of view of the larger image that is larger than a threshold size, the system 1 determines that conversion to the larger size candidate image is not possible. The threshold size may be set, for example, to a few (e.g., 1 to 10 pixels). The gaps may be filled with a color value obtained by averaging the color values ​​of the surrounding pixels, for example, as shown in Fig. 4C described below. Alternatively, no gaps may be allowed in the larger image.

[0017] 1, it should be understood that the particular size and format of the larger image 24 should not be limited. For example, the larger image 24 may be a spherical image, a 270 degree panoramic image, or a 180 degree panoramic image. Alternatively, other sizes or projection types of the larger image 24 may be generated. For example, the larger image 24 may also be a three-dimensional image or a 360 degree three-dimensional image.

[0018] In one example usage scenario, the other images 44 may include surrounding images associated with the target image 20 that were taken at different times and uploaded by different users to the server 4. In one particular example, the user may input a desired time range for matching the other images and may send the target image 20 and metadata information 34 including the desired time range for the other images 44 to the server 4 to obtain convertibility information 22 that the server calculates, as described below.

[0019] Although several of the examples described thus far refer to the other images 44 as being stored on the server 4, it should be understood that in other example configurations, the other images 44 may be stored on the computing device 2 and the convertibility information 22 may be calculated on the computing device 2 using components indicated by dashed lines in FIG. 1, as described further below.

[0020] In addition to or instead of transmitting the metadata information 34 from the computing device 2 to the server 4, the target image 20 may itself be transmitted from the computing device 2 to the server 4, particularly in situations where the additional bandwidth consumed by transmitting the target image 20 is undesirable. In this configuration, the target image 20 may be used by the convertibility information generator 36. For example, in such an alternative configuration, the image metadata extraction module 26 may alternatively be located at the server 4, and the convertibility information generator 36 may use metadata information 34 extracted from the target image 20 at the server 4.

[0021] Continuing with the illustrated embodiment of FIG. 1, when a request for convertibility information is sent from the computing device 2 to the server 4, the server 4 receives the request from the computing device 2 along with the accompanying metadata information 34 of the target image and / or along with the target image 20 in some of a number of configurations. Based on this information, a conversion possibility information generator 36 of the server 4 searches a collection of other images 44 stored in an image store 40. As used herein, an image store refers to a collection of images stored in one or more storage devices, such as a hard disk drive (HDD) or a solid state drive (SSD). The collection of images may be collected from various image sources, including public image aggregation sites on the Internet, other platform users uploading photos, and the like. In this manner, the image store 40 may store a large amount of image data, including photos and videos of various locations captured and uploaded by various users around the world. Typically, the image store 40 is accessible from and may be collocated with the server 4, although it will be appreciated that in alternative configurations described below, the image store may reside on a computing device, as described below.

[0022] The image selection module 52 of the convertibility information generator 36 selects images from the image store 40 to analyze the possibility of creating a larger image 24 for the target image 20. The image store 40 is searched to determine potential matches with the target image 20. The image selection module 52 utilizes various techniques for image selection. For example, the image selection module 52 may perform image selection by searching for matching images in a location matching phase and a content matching phase. During the location matching phase, GPS coordinates, camera pose information, and / or location place name information may be used to identify other images 44 captured within the vicinity of the target image 20. During the content matching phase, the content of the other images 44 is compared to the target image 20 by, for example, comparing object tags applied to each image by an object recognition algorithm and feature vectors including multiple visual features extracted from each image.

[0023] To illustrate these phases, FIG. 3 shows an example map of a target image 20 and other images 44 taken in the Agra region of India near the Taj Mahal. The alignment phase searches for other images 44 taken within a threshold distance TD of the target image 20. The threshold distance TD from the location of the target image 20 is represented as a dashed circle in FIG. 3. Images captured from locations within the threshold distance TD are first selected in the alignment phase. The threshold distance may be a specific value, such as 10 feet, or may be a relative value, such as 1 / 10 of the distance to a recognized object in the target image 20, such as the Taj Mahal in FIG. 3. In addition, if known, the alignment phase may also involve filtering images that are not taken from a similar orientation as the target image. In FIG. 3, the images taken within the threshold distance TD of the location of the target image 20 are further compared to determine whether they are within a target angle range TAR relative to the capture orientation of the target image 20. In one embodiment, the target angle range TAR is 60 degrees, i.e., 30 degrees that differ in a positive or negative direction from the camera pose (compass bearing orientation) whose vector is shown at T for the target image 20. Alternatively, a range of 90 degrees or 30 degrees may be used. In the illustrated example, two images, images P6 and P7, do not fall within the target angle range TAR and are consequently excluded from consideration for stitching with the target image, as indicated by the Xs by the pose vectors for images P6 and P7. The remaining images P1-P5 are selected for stitching with the target image 20, as shown in FIG. 4A and described below.

[0024] The content matching phase may involve searching for images that have one or more recognized objects that are the same as an object contained in the target image 20. For example, in Figures 4A and 4B, the recognized object TAJ MAHAL is surrounded by a bounding box (shown in dashed lines) and tagged with the semantic tag TAJ MAHAL. In this way, when searching for other images 44 during the content matching phase, other images 44 containing the same tag TAJ MAHAL that was applied by object recognition algorithms can be matched to the target image 20 containing the same tag.

[0025] Additionally or alternatively, the content matching phase may search for a set of visual features similar to the set of visual features contained in the target image. The sets of visual features of the target image 20 and the other images 44 may be identified using algorithms such as SIFT, BRIEF, or HoG described above for each image. A template matching approach may also be used, according to which a similarity measure of the two images is calculated, for example based on the sum square difference between the images or the cross correlation between the images. Figures 4A and 4B show a schematic illustration of a set of visual features extracted as plus signs sprinkled throughout the various constituent images. The image selection module 52 uses the positional correspondence between the sets of features to select other candidate images 44 for stitching with the target image 20.

[0026] Upon identifying a set of candidate matching images from the other images 44 by the techniques described above, the convertibility information generator 36 determines whether it is possible to create a larger image 24 for the target image 20 by determining whether it is possible to stitch the target image and the candidate matching images to achieve a field of view as large as the larger image 24. In some of the embodiments, multiple larger image sizes may be evaluated in parallel. Based on these calculations, the convertibility information generator 36 generates the convertibility information 22 and transmits the convertibility information 22 to the computing device 2. In the example of FIG. 4A, the convertibility information generator 36 determines that conversion to the 180-degree large image format is possible because the entire area within the boundary of the 180-degree large image format is covered by the stitched images. 4B, there are no images such as P2 in the image store that can fill the gap region G in the boundary of the 180 degree large image format, and therefore the convertibility information generator 36 determines that conversion to the 180 degree large image format is not possible. It will be noted that the gap region G is larger than a predetermined number of pixels, such as 1 to 10 pixels, that defines a permissible gap.

[0027] Referring to FIG. 1, a larger image conversion possibility display module 28 of the computing device 2 receives the conversion possibility information 22 from the server 4 and displays the conversion possibility information 22 and the associated selector 48 on the display 12. As described in detail below, the conversion possibility information 22 may be displayed as an icon 46 (such as a 360 degree icon) or text 90 (such as a 180 degree icon) on top of the target image 20 indicating one or more available options for image conversion. As shown, a respective selector 48 is provided for each available option. The user may select a particular conversion type from a plurality of available options for image conversion and then transmit a request for the larger image 24 together with the conversion type by the input device 8 using the associated selector 48. In a situation where there are insufficient images available to create the larger image 24 for the target image 20 and no conversion is possible, as is the case for the example of FIG. 4B described above, the icon of the conversion possibility information 22 may not be displayed or may be displayed with a visual indicator of its unavailability, such as an X marked by the icon.

[0028] When a user initiates a larger image request 30 for converting the target image 20 into a larger image 24 by selecting one of the available conversion options of the convertibility information 22 via a selector 48, the larger image request 30 for the larger image 24 is sent from the computing device 2 to the server 4. The larger image request 30 is received at the server 4 by a large image generator 38, which generates the larger image 24 using matching other images 44.

[0029] To generate the larger image 24, a stitching module 54 of the large image generator 38 is configured to overlay, orient, scale, and position the other images 44 on the target image 20 using the positional correspondence of the sets of extracted visual features of the matching other images 44 and the target image 20 described above, whereby the other images 44 and the target image 20 may be stitched together to form the larger image 24. Although the larger image 24 is described above as being generated by the stitching module 54 only after a larger image request 30 is received from the computing device 2, it will be appreciated that the stitching module 54 may be configured to stitch the larger image 24 earlier when evaluating the availability of a conversion to a larger image to avoid redundant processing, and to store the larger image 24 at the server 4 for future download to the computing device 2 upon receiving the larger image request 30.

[0030] In some embodiments, depth information may be encoded into the target image 20 and / or the other images 44. The depth information may be obtained from a depth camera at the time of image capture or may be ascertained using stereoscopic depth determination techniques by comparing images captured from adjacent locations in time and space. In such embodiments, the stitching module 54 of the large image generator 38 performs stitching in three-dimensional space, such that the resulting larger image 24 is a three-dimensional image. In one example, the large image generator 38 may further integrate such depth information into the large image 24 to generate a three-dimensional, 360-degree larger image viewable by a VR headset.

[0031] Once the larger image 24 is generated, the server 4 sends the generated larger image 24 back to the computing device 2. The computing device 2 receives the larger image 24 from the server 4 and displays the larger image 24 on the display 12 via a larger image transformability display module 28 of the processor 10. Alternatively or additionally, the larger image 24 may be transmitted from the computing device 2 to another computing device for display. For example, the computing device 2 may transmit the larger image to an external VR headset for 3D viewing.

[0032] In an alternative embodiment, the computing device 2 may perform the functions performed by the server 4 described above, i.e., generating the convertibility information 22 and the larger image 24. For this purpose, as indicated by the dashed lines in FIG. 1, the computing device 2 may implement a decision module 42 therein, as shown in FIG. 1, which decision module 42 includes a convertibility information generator 36 and a large image generator 38. This configuration eliminates the need to transmit the metadata information 34 and the target image 20 to the server 4. In this configuration, the decision module 42 executed by the processor 10 of the computing device 2 may generate the convertibility information 22 based on the target image 20 and / or the metadata information 34 of the target image, indicating whether the target image 20 can be converted into the larger image 24 by stitching at least a portion of the target image 20 and other images 44 stored in the storage device 14 of the computing device 2. In addition, the target image 20 may be converted into the larger image 24 by the large image generator 38 executed by the processor 10.

[0033] It will be understood that the computing system 1 may be configured with various functional modules to provide the user with a new image interaction experience, utilizing the above-described method for determining whether the possibility of a larger image transformation exists as described above. For example, the computing system 1 may be configured to allow the user to view a larger image 24, which is formed from selected other images 44 to achieve a particular look, feel, or effect, or to characterize a particular content. As three specific examples in this vein, the server 4 further includes an image filter 56 and an image filter criteria 58 that preselects the other images 44 according to image filter criteria 58 selected from the group consisting of date 60, weather 62, and historical events 64, allowing the user to personalize the larger image 24 according to a certain time period, weather type, or event. Since a large number of other images 44 are stored in the image store 40, multiple images may be stored for the same location and selected based on their characteristics as candidate images for the larger image 24. Alternatively, the candidate images may be preselected (i.e., filtered) by date 60, allowing the user to select the year in which the photo was taken. Candidate images may also be preselected (i.e., filtered) by weather 62, allowing the user to select the weather (e.g., sunny or rainy) in which the photo was taken.Weather information may be (1) tagged into the other images 44 based on weather data available for the location and date of the image, (2) classified from the image data of the other images 44 using a neural network trained to recognize weather conditions, and (3) matched between the target image 20 and the other images 44 by a neural network trained to match atmospheric, lighting, and terrain conditions between the images. Candidate images may be preselected (i.e., filtered) by historical events 64, allowing the user to select the historical event for which the photo was taken. The user may check and see if those options are available in the computing device 2 by the convertibility information 22, and select one of those options, as described below in connection with the example of FIG. 2E. One example scenario illustrating all of these filters is a user taking a photo of themselves at the Lincoln Memorial in Washington, DC, specifying that a larger image 24 be created from images of the famous March on Washington historical event that occurred at the Lincoln Memorial on an overcast day in 1963. If enough images are available in the image store 40, the large image generator 38 will generate a larger image 24 that will surround the user's target image 20 with historical images from other images 44 taken of the event in the specified weather conditions, as shown in FIG.

[0034] 4E shows a larger image formed from other images 44 taken at the Lincoln Memorial on August 28, 1963 within the vicinity of the user's target image 20 and oriented toward the front of the Lincoln Memorial. All of the other images 44 used to form the larger image 24 in this example depict scenes from Martin Luther King's famous "I Have a Dream" speech on August 28, 1963, while the target image 20 depicts members of the user's family in the same location today. This larger image 24 was generated in response to the user selecting the target image 20 of the user's family at the Lincoln Memorial, then selecting as an event criterion through a pop-up window 110C as shown in FIG. 2E that the other images 44 are filtered by the "I Have a Dream" event, viewing the transformability information 22 indicating that a larger image 24 can be created for the target image 20 based on the selected event, and selecting the transform selector of FIG. 2E.

[0035] 2A-2F show schematic diagrams of an exemplary GUI that displays convertibility information 22 over a target image 20 that a user selects using the computing device 2 of FIG. 1. The target image 20 is selected by the user or the processor 10 from the image data 16, and the target image 20 may be displayed as an image thumbnail on the display 12 of the computing device 2. The selected target image 20 may be displayed on the main screen 104 of the display 12. The convertibility information 22 displayed over the target image 20 may indicate available conversion options, such as a 360-degree image, a 270-degree image, a 180-degree image, and a 360-degree three-dimensional image. The convertibility information 22 may be displayed as text 90 and may include a general availability indicator 90A and multiple conversion options 90B as shown in FIG. 2A. The user may select a desired conversion option 90B from the multiple available conversion options 90B by a corresponding selector 48. The selector 48 is shown as multiple selection checkboxes, one or more of which may be selected, but may take any suitable alternative form. The user may further request conversion to the selected conversion option 90B by selecting a conversion selector 106 displayed adjacent to the target image 20. Once a larger image request 30 is made, the larger image request 30 is transmitted to the server 4 or, alternatively, to a decision module 42 of the computing device 2 for conversion to a larger image 24 by the large image generator 38.

[0036] 2B, the convertibility information 22 may also be displayed as a conversion indicator icon 46, with or without text, as shown in FIG 2B. In this example, the convertibility information 22 icon 46 indicates that conversion of 360 degree images, 360 degree three-dimensional images, and 180 degree images is available. The user may select the desired conversion option via a selector 48.

[0037] As shown in Figure 2C, the convertibility information 22 may include image conversion status information. An icon 46 in the convertibility information 22 shown in Figure 2C indicates that conversion of the target image 20 to a larger 360 degree image is possible and that a conversion request has not yet been made.

[0038] 2D, icon 46 of transformability information 22 may indicate that a transformation request has been made for an available transformation of target image 20 to a larger 360 degree image and that the transformation is in progress, as indicated in FIG 2D by both a cloud with an up arrow icon and a PROCESSING label located below the icon.

[0039] 2E, the convertibility information 22 indicating that it may be possible to convert the target image 20 into the larger image 24 using other images 44 from a certain date, weather, and / or historical event may be displayed by icons 46A, 46B, and 46C, from which the user may select using a selector 48 to request conversion. When the user selects one of the icons 46A, 46B, or 46C, a pop-up window 110A, 110B, or 110C may be displayed for further selecting criteria for deriving a more personalized larger image 24 for the target image 20. For example, the user may select the year in which the image was taken (e.g., 1900, 1950, or 2000, etc.), as shown in 110A. The user may also select the weather in which the image was taken (e.g., sunny, rainy, cloudy, etc.), as shown in 110B. Additionally, the user may select the historical event during which the image was taken, as shown at 110C. The event may be matched by matching the selected historical event with event tags labeled in other images 44 by human curators or trained artificial intelligence models. When the user selects one of the options using the selector 48 and clicks the transform button 106, a request may be transmitted to the server 4 for transformation of the larger image 24 using the selected image filter criteria 58 on the other images 44, and a personalized larger image 24 with the requested image effects or features may be generated.

[0040] 2F, the convertibility information 22 may indicate that a user-selected region of interest 23 in the target image 20 can be converted into a larger image 24. For example, the user may select the region of interest 23 to include a particular object, such as a building or a person, in the selected image and exclude a portion of the building under construction, and may extract the region of interest 23 to form the target image 20, i.e., based on the selection, the region of interest 23 becomes the target image 20 itself. Thus, upon receiving a user selection of the selector, the system is configured to convert the modified target image 20, cropped to the region of interest 23, into a larger image 24. The resulting larger image 24 includes the region of interest 23 and excludes the remainder of the original uncropped target image 20, meaning in this example that the resulting larger image 24 excludes a portion of the building under construction, as illustrated by the tractor in FIG. 4D.

[0041] Furthermore, multiple target images 20 and 21 taken at the same location may be selected from the image data 16 by the user or the processor 10, which may be displayed as image thumbnails 16 on the display 12 of the computing device 2. Once the multiple images 20 and 21 are selected, metadata information 34 of the selected multiple target images 20 and 21 may be extracted and transmitted to the server 4. Convertibility information 22 indicating the possibility of the multiple target images 20 and 21 being converted into a larger image 24 may consequently be obtained from the server by the computing device.

[0042] 5A-5B show schematic diagrams of an exemplary GUI displaying an example of a larger image 24 converted from a target image 20 using the computing device 2 of FIG. 1. Based on a user's request, as described above, the target image 20 is converted into a larger image 24 by a decision module 42 or a server 4 integrated in the computing device 2 and displayed on the display 12 of the computing device 2 as shown in FIG. 5A. An element of original image navigation information 94 indicating that the target image 20 has been converted into the larger image 24 may be displayed on the larger image 24. As part of the original image navigation information 94, the larger image 24 may be displayed using a visual differentiator 100 that visually distinguishes the target image 20 and other stitched images in the larger image 24 as shown in FIG. 5A. The visual discriminator 100 may be integrated into the original image navigation information 94, which includes an icon including the original image area and other image areas that are stitched with the original image to create the larger image 24. The icon may change orientation as the user pans the larger image with the image slider 96 to indicate whether the displayed portion of the larger image is the original image or another stitched image. Alternatively, the visual discriminator 100 may be a dashed line, a shaded area, or a highlighted area that is superimposed on the larger image 24. Referring to FIG. 5B, the icon of the original image navigation information 94 may indicate the user's current view of the larger image 24 with a "V" shaped marker 110. The marker 110 may change orientation as the user pans the larger image with the image slider 96.

[0043] As described above, the target image 20 intended to be converted to the larger image 24 may be selected by the user or automatically selected by the processor 10 of the computing system 2 and displayed in the GUI. Conversion possibility information 46 indicating whether there are enough other images 44 in the image store 40 to stitch the larger image 24 to the target image 20 is displayed on top of the target image 20 on the display 12 of the computing device 2. The conversion possibility information 46 indicates the availability of conversion options such as 180 degree image, 270 degree image, 360 degree image, and 360 degree three-dimensional image by text 90 or icons 46, allowing the user to visually predict which conversions are available and select a preferred option. The user may select a conversion option using the selector 48 and request the conversion. The icons 46 further allow the user to view the process of converting the target image 20 to the larger image 24 using the selected option. The icons 52 may further allow the user to apply filters to the other images 44 based on time, weather, and historical event criteria via pop-up windows 110A, 110B, and 110C to personalize the larger image 24. Once the target image 20 is converted to the larger image 24 based on the user's request, the larger image 24 may be displayed in the GUI along with source image navigation information 94, which includes a visual discriminator 100 that allows the user to visually discern the other stitched images and the target image in the larger image 24 using superimposed icons, dashed lines, shaded regions, or highlighted regions.In this manner, a user may predict available transformations of the target image 20 into the larger image 24, select a desired size and scale of the other image 44, and visually distinguish between the original target image 20 and the other images for a more personalized and navigable larger image 24.

[0044] FIG. 6 shows a flowchart of a computerized method 300 according to an exemplary implementation of the disclosure. In step 302, the method may include selecting a target image by a client computing device or a server of one or more computing devices. As described above, the computing device 2 may be any of various types of computing devices, such as a smartphone, a tablet computing device, a digital camera, and a head-mounted display device. In step 304, the method may further include executing a determination module by the client computing device or a server of one or more computing devices to generate convertibility information, the convertibility information indicating that the target image can be converted into a larger image having a larger field of view (e.g., a spherical image or a panoramic image) by stitching at least a portion of the target image and other images from the image store based on the target image and / or metadata information of the target image. As described above, the metadata information may include location information, time information, and / or object information associated with the target image. In step 306, the method may further include displaying the convertibility information and an associated selector by the client computing device. As described above, the conversion capability information may include icons indicating one or more available options for image conversion and icons indicating image conversion status information. At step 308, the method may further include converting the target image into a larger image upon receiving a user selection of the selector. At step 310, the method may further include outputting the larger image for storage or display on one or more of the multiple computing devices.As explained above, a larger image may be displayed, allowing the target image and other stitched images to be visually distinguished.

[0045] The systems and methods described above may be utilized to provide a user with advanced knowledge of whether a larger image can be generated from images in an image store for a particular target image selected by the user. By programmatically searching for matching images based on various location and content-based factors indicated in the metadata of the target image and returning user convertibility information, the system can avoid the need to upload several images captured at approximately the same time to a remote stitching service to determine whether they can be stitched into a larger image, thereby significantly saving processing time and bandwidth. Furthermore, the systems and methods described herein provide a technical solution for matching the target image with various other images in the image store based on various location-based and content-based criteria, thereby allowing the user's target image to be matched with stitchable images even when the user's own image library does not store such images. Finally, the systems and methods described herein do not require other images to be captured from the same location and time as the target image, increasing the potential number of images available for stitching.

[0046] In some of the embodiments, the methods and processes described herein may be coupled to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0047] 7 illustrates a schematic of a non-limiting embodiment of a computing system 400 capable of performing one or more of the methods and processes described above. The computing system 400 is illustrated in simplified form. The computing system 400 may embody the computing device 2 illustrated in FIG. 1 and described above. The computing system 400 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, and wearable computing devices, such as smart watches and head-mounted augmented reality devices.

[0048] Computing system 400 includes a logical processor 402, a volatile memory 404, and a non-volatile storage device 406. Computing system 400 may optionally include a display subsystem 408, an input subsystem 410, a communication subsystem 412, and / or other components not shown in FIG.

[0049] Logical processor 402 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical entities. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more components, achieve a technical effect, or otherwise arrive at a desired result.

[0050] A logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. A processor that is a logical processor 402 may be a single-core processor or a multi-core processor, and instructions executed by the processor may be configured for sequential, parallel, and / or distributed processing. Individual components of a logical processor may optionally be distributed across two or more separate devices that are remotely located and / or configured for cooperative processing. Aspects of a logical processor may be virtualized and executed by remotely accessible networked computing devices configured by a cloud computing configuration. In such cases, it will be understood that these virtualized aspects are executed by multiple different physical logical processors of various different machines.

[0051] The non-volatile storage device 406 includes one or more physical devices configured to hold instructions executable by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 406 may be transformed, for example, to hold different data.

[0052] The non-volatile storage device 406 may include physical devices that are removable and / or internal. The non-volatile storage device 406 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage device technologies. The non-volatile storage device 406 may include non-volatile devices, dynamic devices, static devices, read / write devices, read-only devices, sequential access devices, location addressable devices, file addressable devices, and / or content addressable devices. It will be appreciated that the non-volatile storage device 406 is configured to retain instructions even when power to the non-volatile storage device 406 is interrupted.

[0053] Volatile memory 404 may include physical devices including random access memory. Volatile memory 404 is typically utilized by logical processor 402 to temporarily store information during processing of software instructions. It will be understood that volatile memory 404 typically does not continue to store instructions when power to volatile memory 404 is interrupted.

[0054] Aspects of the logical processor 402, volatile memory 404, and non-volatile storage device 406 may be integrated together into one or more hardware logic components, which may include, for example, field programmable gate arrays (FPGAs), program specific and application specific integrated circuits (PASICs / ASICs), program specific and application specific standard products (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0055] The terms "module", "program", and "engine" may be used to describe aspects of the computing system 400 that are typically implemented in software by a processor to perform a particular function using portions of volatile memory, which function involves a transformation that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated by the logical processor 402 to execute instructions held by the non-volatile storage device 406 using portions of volatile memory 404. Multiple different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may include individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0056] When included, the display subsystem 408 may be used to present a visual representation of the data held by the non-volatile storage device 406. The visual representation may take the form of a graphical user interface (GUI). When the methods and processes described herein modify the data held by the non-volatile storage device and thus transform the state of the non-volatile storage device, the state of the display subsystem 408 may be similarly transformed to visually represent the change in the underlying data. The display subsystem 408 may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with the logical processor 402, the volatile memory 404, and / or the non-volatile storage device 406 in a shared enclosure, or such display devices may be peripheral display devices.

[0057] When included, the input subsystem 410 may include or interface with one or more user input devices, such as a keyboard, mouse, touch screen, or game controller. In some of the embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and conversion and / or processing of input actions may be handled on-board or off-board. Exemplary NUI components may include microphones for speech and / or language recognition, infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition, head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition, and electric field sensing components for assessing brain activity, and / or any other suitable sensors.

[0058] When included, the communications subsystem 412 may be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communications subsystem 412 may include wired and / or wireless communication devices compatible with one or more different communications protocols. As a non-limiting example, the communications subsystem may be configured for communication over a wired or wireless local or wide area network, such as a wireless telephone network or a Wi-Fi connection over an HDMI. In some of the embodiments, the communications subsystem may enable the computing system 400 to send and / or receive messages to and from other devices over a network, such as the Internet.

[0059] The following paragraphs describe some aspects of this disclosure. According to one aspect of this disclosure, a computing device is provided. The computing device may include a storage device configured to store image data. The computing device may further include a processor coupled to a memory storing a plurality of instructions that, when executed by the processor, cause the processor to select a target image from the image data. The processor may be further configured to display convertibility information, the convertibility information indicating that the target image can be converted into a larger image having a larger field of view by stitching at least a portion of the target image and an associated selector with another image. The processor may be further configured to display the larger image upon receiving a user selection of the selector.

[0060] According to this aspect, the convertibility information may be generated by a determination module of a processor of a computing device based on the target image and / or metadata information of the target image.

[0061] According to this aspect, the processor may be further configured to transmit the target image and / or metadata information of the target image to the server to cause the server to generate the convertibility information. The processor may be further configured to receive the convertibility information from the server. The processor may be further configured to receive a user selection of the selector. The processor may be further configured to transmit a larger image request to the server and receive the larger image from the server.

[0062] According to this aspect, the metadata information may include location information, time information, orientation information, and / or object information associated with the target image.

[0063] According to this aspect, the larger image may be a spherical image or a panoramic image.

[0064] According to this aspect, the other images may be pre-selected by an image filter according to filter criteria selected from the group consisting of date, weather, and historical events.

[0065] According to this aspect, the conversion possibilities information may include icons indicating one or more available options for converting the image.

[0066] According to this aspect, the conversion possibility information may include an icon indicating the image conversion status information.

[0067] According to this aspect, the computing device may be a device selected from the group consisting of a smartphone, a head mounted display device, a tablet computing device, and a digital camera.

[0068] According to this aspect, the larger image may be displayed using a visual discriminator that visually discriminates the other stitched images and the target image within the larger image.

[0069] According to another aspect of the disclosure, a computerized method is provided. The computerized method may include selecting, by one or more processors and / or servers of one or more computing devices, a target image. The computerized method may further include executing a determination module to generate convertibility information, the convertibility information indicating that the target image can be converted into a larger image having a larger field of view by stitching at least a portion of the target image with another image from an image store. The computerized method may further include displaying the convertibility information and an associated selector. The computerized method may further include converting the target image into the larger image upon receiving a user selection of the selector. The computerized method may further include outputting the larger image for storage or display on one or more of the computing devices.

[0070] According to this aspect, the target image may be selected by a client computing device of the one or more computing devices. The determination module may be executed by the client computing device or a server of the one or more computing devices. The convertibility information and associated selector may be displayed by the client computing device. The target image may be converted into a larger image by the client computing device or the server upon receiving a user selection of a selector.

[0071] According to this aspect, the convertibility information may be generated based on the target image and / or metadata information of the target image.

[0072] According to this aspect, the metadata information may include location information, time information, orientation information, and / or object information associated with the target image.

[0073] According to this aspect, the larger image may be a spherical image or a panoramic image.

[0074] According to this aspect, the conversion possibilities information may include icons indicating one or more available options for converting the image.

[0075] According to this aspect, the conversion possibility information may include an icon indicating the image conversion status information.

[0076] According to this aspect, the computing device may be a device selected from the group consisting of a smartphone, a head mounted display device, a tablet computing device, and a digital camera.

[0077] According to this aspect, the larger image may be displayed such that the target image and the other stitched images are visually distinguishable.

[0078] According to another aspect of the disclosure, the computing device may include a storage device configured to store image data. The computing device may further include a processor coupled to a memory storing a plurality of instructions that, when executed by the processor, cause the processor to detect a target image from the image data. The processor may be further configured to detect the target image from the image data. The processor may be further configured to transmit the target image or metadata information of the target image to a server to generate convertibility information, the convertibility information indicating that the target image can be converted into a larger image having a larger field of view by stitching at least a portion of the target image with another image. The processor may be further configured to receive the convertibility information from the server. The processor may be further configured to display the convertibility information and an associated selector. The processor may be further configured to programmatically select the target image. The processor may be further configured to send a request to the server to convert the target image into the larger image. The processor may be further configured to receive the larger image from the server and display the larger image.

[0079] It will be understood that the configurations and / or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, as numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various operations shown and / or described may be performed in the order shown and / or described, in other orders, in parallel, or may be omitted. Similarly, the order of the processes described above may be changed.

[0080] The subject matter of this disclosure includes all novel and non-obvious combinations and subcombinations of the various processes, systems, and configurations disclosed herein, as well as other features, functions, operations, and / or properties, as well as equivalents thereof.

Claims

1. 1. A computing device, comprising: a storage device configured to store image data; a processor coupled to a memory storing instructions that, when executed by the processor, cause the processor to: selecting a target image from the image data; displaying convertibility information, the convertibility information indicating that the target image can be converted into a larger image having a larger field of view by stitching at least a portion of the target image and an associated selector with another image; upon receiving a selection by a user of the selector, displaying the larger image. Computing device.

2. The computing device of claim 1 , wherein the convertibility information is generated by a decision module of the processor of the computing device based on the target image and / or metadata information of the target image.

3. The processor further comprises: transmitting the target image and / or metadata information of the target image to a server, causing the server to generate the convertibility information; receiving the convertibility information from the server; receiving a user selection of the selector; Transmitting a larger image request to the server; receiving the larger image from the server; The computing device of claim 1 configured to:

4. The computing device of claim 3 , wherein the metadata information comprises location information, time information, orientation information, and / or subject information associated with the target image.

5. The computing device of claim 1 , wherein the larger image is a spherical image or a panoramic image.

6. The computing device of claim 1 , wherein the other images are pre-selected by an image filter according to filter criteria selected from the group consisting of date, weather, and historical events.

7. The computing device of claim 1 , wherein the conversion capability information includes icons indicating one or more available options for converting an image.

8. The computing device of claim 1 , wherein the convertibility information includes an icon indicating image conversion status information.

9. The computing device of claim 1 , wherein the computing device is a device selected from the group consisting of a smartphone, a head mounted display device, a tablet computing device, and a digital camera.

10. 10. The computing device of claim 1, wherein the larger image is displayed using a visual discriminator that visually discriminates the target image and other stitched images within the larger image.

11. 1. A computerized method, the computerized method comprising: by one or more processors and / or servers of one or more computing devices; selecting a target image; executing a determination module to generate convertibility information, the convertibility information indicating that the target image can be converted into a larger image having a larger field of view by stitching at least a portion of the target image with other images from an image store; displaying the convertibility information and associated selectors; transforming the target image into the larger image upon receiving a selection by a user of the selector; outputting the larger image for storage or display on one or more of the computing devices. How it is computerized.

12. the target image is selected by a client computing device of the one or more computing devices; the determination module is executed by the client computing device or a server of the one or more computing devices; the convertibility information and associated selectors are displayed by the client computing device; the target image is transformed by the client computing device or a server into the larger image upon receiving a selection by the user of the selector; 12. The computerized method of claim 11.

13. 12. The computerized method of claim 11, wherein the transformability information is generated based on the target image and / or metadata information of the target image, the metadata information of the target image including location information, time information, orientation information, and / or object information associated with the target image.

14. The computerized method of claim 11 , wherein the conversion capability information includes icons indicating one or more available options for converting an image.

15. The computerized method of claim 11 , wherein the convertibility information includes an icon indicating image conversion status information.

Citation Information

Patent Citations

  • Panorama compositing of two or more image data

    JP2004135230A