Intelligent suggestion of image zoom region

Abstract

Described herein are techniques for providing intelligent suggestions for image zoom regions. One method includes receiving a search query; performing a search using the search query to identify search results including image search results, the image search results including a plurality of images responsive to the search query; for a given image of the plurality of images included in the image search results, determining at least one zoom region in the given image; and providing the search results including the image search results, including providing the given image and an indication of the at least one zoom region in the given image.

Description

Background Technology

[0001] Individuals (referred to herein as "users") typically conduct online research by submitting search queries, which may return search results that include images. Users can also search for specific images by submitting image search queries. When a user is browsing search results that include images or images returned in response to an image search query, the user may be interested in specific aspects of one or more images. For example, among other visual information, a user may be interested in a specific object of interest included in one or more images, or a specific piece of information included in one or more images. A specific object of interest could be, for example, a specific part in a drawing of vehicle parts or a specific toy.

[0002] Users can browse search results that include images, images returned in response to image search queries, and / or image collections on mobile devices or other computing devices. The screen fidelity of mobile devices or other computing devices is limited, making it impossible to display images at their native resolution. Therefore, users can zoom in on specific areas of an image to view finer details. By zooming in, users may be able to see more clearly a specific object of interest contained within one or more images, or a specific piece of information contained within one or more images. However, image zooming can be a time-consuming process, requiring full focus and flexibility from the user when fine-tuning the zoom position and zoom level, thus potentially leading to excessive use of the client device's power resources and / or other computing resources when manually performing image zooming. Summary of the Invention

[0003] The embodiments disclosed herein relate to providing intelligent suggestions for image zoom areas. The embodiments can automatically suggest one or more image zoom areas in one or more images, such as images returned in response to a search query. Each automatically suggested image zoom area can be indicated by a selectable target, and in response to a user's selection of a target (e.g., by tapping or clicking the target), the image can be automatically zoomed in on the display at a location in the image associated with the selectable target. Therefore, a specific portion of the image can be zoomed in, allowing the user to view finer details within that specific portion of the image. By zooming in on the image, the user is able to view more clearly a specific object of interest included within the image or a specific piece of information contained within the image.

[0004] In the implementation, the image zoom area can be: a region within an image relevant to a search query, the image being returned as a result of the search query; a region within an image typically zoomed in by other users or similar to a region within an image typically zoomed in by other users; and / or a region within an image frequently zoomed in by the user or similar to a region within an image frequently zoomed in by the user. For example, if a user has previously zoomed in to a region containing an image of a race car, the image zoom area can be automatically determined based on the region containing the image of a race car.

[0005] In one example, if a user searches for "toy X", the user is likely to be most interested in the portion of an image containing toy X included in the search results. In this implementation, zoomable areas of images containing toy X can be automatically suggested because these portions of the image respond most strongly to the "toy X" search query. In another example, if a user searches for "toy" based on prior indications of interest in toy X from a particular user (e.g., confirmed by the particular user repeatedly zooming in on portions of an image containing toy X), the implementation can automatically suggest zoomable areas of images containing toy X when displaying images to the particular user (e.g., as search results, or as part of a set of images the user is browsing). In yet another example, if a particular user searches for "toy" based on the frequency with which other users zoom in on areas of images containing toy X, the implementation can automatically suggest zoomable areas of images containing toy X.

[0006] By selecting a target associated with the image zoom area, users can quickly zoom in on specific areas of an image of interest, thereby improving user convenience, saving time, and reducing the use of power and / or other computing resources on the client device compared to manually zooming in on the image.

[0007] The implementation includes a method and associated interface for providing image zoom suggestions for images returned in response to a user query or included in a set of images browsed by the user. These image zoom suggestions may be based on zoom areas (e.g., determined based on the behavior of other users), may be implied by the search query, or may be based on preferences and interests learned from the user who issued the query. In the implementation, image zoom suggestions may be displayed as automatically suggested image zoom areas. Automatically suggested image zoom areas may be displayed as highlights, shadows, boxes, icons, or other graphic indicators overlaid on portions of one or more images in the image search results set or the set of images the user is browsing. In the implementation, tapping or clicking on an automatically suggested image zoom area may cause the zoom level of the relevant image to be automatically adjusted (e.g., adjusted to a specific zoom level associated with the automatically suggested image zoom area), and the image may be automatically centered around the relevant zoom area. In the implementation, the user can use zoom controls provided in the user interface to further adjust the zoom level of the image and / or navigate within the image to display different portions of the image on the user interface.

[0008] In some implementations, image zoom suggestions can be provided when a user is browsing their own photos (e.g., a photo library including photos taken on the user's phone). For example, automatically suggested image zoom areas can be determined based on regions within a given image that the user frequently zooms in on, or regions similar to those in other images frequently zoomed in on by the user (e.g., regions including the user's family members). The system can determine the suggested image zoom areas on a server-side device or locally (e.g., as part of a photo viewer application) based on the user's historical browsing behavior.

[0009] In some implementations, the system may wait until the user selects the image (e.g., by tapping or clicking the image) before determining and / or providing indications for one or more suggested image scaling areas (e.g., as highlights, glimpses, boxes, icons, or other graphic indications overlaid on the image).

[0010] In various embodiments, a method implemented by one or more processors may include: receiving a search query; performing a search using the search query to identify search results including image search results, the image search results including a plurality of images in response to the search query; for a given image among the plurality of images included in the image search results, determining at least one zoom area in the given image; and providing search results including image search results, which includes an indication of providing the given image and at least one zoom area in the given image.

[0011] In some embodiments, the search query is an image search query and the search results include only image search results. In some embodiments, the method further includes determining, for another image among a plurality of images included in the image search results, that no indication of a zoom area is provided. In some embodiments, providing an indication of at least one zoom area in a given image includes providing a graphical indication of the location of the zoom area within the given image for each of the at least one zoom area in the given image.

[0012] In some embodiments, determining at least one zoomable area includes, for each of the at least one zoomable area, determining: the location of the zoomable area, and the zoom level or bounding box of the zoomable area, and the method further includes receiving user interface input indicating a selected zoomable area in a given image; and, in response to receiving the user interface input, adjusting the zoom level of the given image to display a view of the given image magnified at the zoom level of the selected zoomable area or magnified based on the bounding box of the selected zoomable area, the magnified view being centered on the location of the selected zoomable area within the given image. In some embodiments, determining at least one zoomable area includes, for each of the at least one zoomable area, determining the location of the zoomable area by retrieving metadata associated with the given image, the metadata indicating the location of the zoomable area in the given image. In some embodiments, the metadata includes a mapping between a search query and areas in the given image, or a mapping between a semantic representation of a search query and areas in the given image.

[0013] In some implementations, at least one zoom area in a given image includes a zoom area associated with a search query. In some implementations, at least one zoom area in a given image is associated with a search query based on historical data reflecting previous zooming on the given image of past instances returned in response to a search query or related search query.

[0014] In some implementations, at least one zoomed region in a given image is further associated with a search query based on historical data of one or more other queries having at least a threshold similarity to the search query. In some implementations, determining at least one zoomed region in a given image includes determining the location of the zoomed region associated with the search query, but not the location of another zoomed region associated with another search query.

[0015] In some implementations, at least one zoom area in a given image includes a zoom area unrelated to the search query. In some implementations, at least one zoom area in a given image includes a zoom area associated with user preferences.

[0016] In some additional or alternative embodiments, the computer program product may include one or more computer-readable storage media having program instructions commonly stored on the one or more computer-readable storage media. The program instructions may be executable to: receive a search query; perform a search using the search query to identify search results including image search results, the image search results comprising a plurality of images in response to the search query; provide the search results including the image search results, the provision including each of the plurality of images in response to the search query; receive user interface input indicating a selected image among the plurality of images in response to the search query; determine the position of at least one zoom area in the selected image; and provide an indication of the position of each of the at least one zoom area in the selected image.

[0017] In some embodiments, providing an indication of the position of each of at least one zoom area in a selected image includes providing a graphical indication of the position of each zoom area within the selected image for each of the at least one zoom area in the selected image. In some embodiments, the program instructions are further executable to: receive user interface input indicating a selected zoom area in the selected image; and, in response to receiving the user interface input, adjust the zoom level of the selected image to display a magnified view of the selected image centered on the position of the selected zoom area within the selected image. In some embodiments, determining the position of at least one zoom area in the selected image includes retrieving metadata associated with the selected image that indicates the position of the at least one zoom area in the selected image.

[0018] In some additional or alternative embodiments, the system may include a processor, a computer-readable storage device, one or more computer-readable storage media, and program instructions commonly stored on one or more computer-readable storage media. The program instructions may be executable to: receive a search query; perform a search using the search query to identify search results including image search results, the image search results comprising a plurality of images in response to the search query; for a given image among the plurality of images included in the image search results, determine the location of a zoomed-in area in the given image; and provide the search results including the image search results, the provision including, for the given image, providing a zoom level adjusted to display a magnified view of the given image centered on the location of the zoomed-in area within the given image.

[0019] In some embodiments, the program instructions are further executable to: provide an instruction to zoom in on a given image; and provide user interface elements to adjust the zoom level of the given image. In some embodiments, the program instructions are further executable to: determine a confidence level associated with a zoom area; and adjust the zoom level to display a zoomed-in view of the given image in response to the confidence level associated with the zoom area meeting a confidence level threshold.

[0020] The above description serves as an overview of some embodiments of this disclosure. These embodiments and other embodiments are further described in more detail below.

[0021] Various embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), and / or tensor processing unit (TPU)) to perform methods such as one or more methods described herein. Other embodiments may include a client device including a processor operable to execute the stored instructions to perform methods such as one or more methods described herein. Other embodiments may include a system comprising one or more servers including one or more processors operable to execute the stored instructions to perform methods such as one or more methods described herein. Attached Figure Description

[0022] Figure 1 An example environment is schematically depicted in which selected aspects of this disclosure can be implemented according to various embodiments.

[0023] Figure 2 , Figure 3 and Figure 4 The flowchart illustrates example methods for automatically suggesting image scaling areas based on various implementation schemes.

[0024] Figure 5A and Figure 5B Example applications of the technologies described herein are depicted according to various implementation schemes.

[0025] Figure 6 The illustration shows an example architecture of a computing device. Detailed Implementation

[0026] Figure 1 An example environment 100 is schematically depicted, according to various embodiments, in which selected aspects of this disclosure may be implemented. Figure 1Any computing device depicted in or elsewhere in the figure may include logic such as one or more microprocessors (e.g., a central processing unit or “CPU”, a graphics processing unit or “GPU”) that execute computer-readable instructions stored in memory, or other types of logic such as application-specific integrated circuits (“ASICs”), field-programmable gate arrays (“FPGAs”), etc. Figure 1 Some of the systems depicted, such as the user intent prediction system 110, can be implemented using one or more server computing devices that form what is sometimes called “cloud infrastructure,” although this is not necessary.

[0027] In the implementation scheme, example environment 100 includes one or more user interface input devices 102, one or more user interface output devices 104, and a search system 112. Although the search system 112 is... Figure 1 The diagram illustrates the separation of the user interface input device 102 and user interface output device 104. However, in some embodiments, all or all aspects of the search system 112 may be implemented on a computing device that also includes the user interface input device 102 and / or the user interface output device 104. For example, all or all aspects of the rendering engine 132 of the search system 112 may be implemented on the computing device. In some embodiments, all or all aspects of the search system 112 may be implemented on a computing device that is separate from and remote from the computing device containing the user interface input device 102 and / or the user interface output device 104 (e.g., all or all aspects may be implemented "in the cloud"). In some of these embodiments, these aspects of the search system 112 may communicate with the computing device via one or more networks, such as a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet). In some embodiments, one or more aspects of the search system 112 and the user interface input / output devices 102, 104 may be assisted by a virtual assistant or components of a virtual assistant.

[0028] To name just a few, user interface input device 102 may include, for example, a physical keyboard, a touchscreen, a visual sensor such as a digital camera, an accelerometer (e.g., for capturing gestures), a fingerprint sensor, a radar sensor (e.g., for visually detecting gestures), and / or a microphone. User interface output device 104 may include, for example, a display screen, a haptic feedback device, and / or a speaker. User interface input and output devices 102 and 104 may be combined on one or more of a user's computing devices. For example, a user's mobile phone may include user interface input device 102 and user interface output device 104; or a standalone personal virtual assistant hardware device may include user interface input device 102 and user interface output device 104; or a first computing device may include user interface input device 102, and a separate computing device may include user interface output device 104, etc. A "standalone personal virtual assistant hardware device" can be a device primarily or specifically designed to allow a user to interact with a virtual assistant using free-form natural language input. These can take various forms, such as a standalone "smart" speaker, a smart display, etc.

[0029] like Figure 1 As illustrated, a user provides a search query to a search system 112 via a user interface input device 102. The search system 112 provides a response output to the user via a user interface output device 104. Each search query is a request for information. Search queries can take the form of text and / or other forms, such as, for example, audio and / or images. For example, in some embodiments, the user interface input device 102 may include a keyboard that generates text input in response to user interface input directed to the keyboard. Furthermore, for example, in some embodiments, the user interface input device 102 may include a microphone. In some such cases, a speech recognition module native to the search system 112 or in communication with the search system 112 can convert the audio speech input received at the microphone into text input. The text input can then be provided to the search system 112. For brevity, the input is displayed on... Figure 1 The input is illustrated as being provided directly to the search system 112 by the user interface input device 102, and the output is illustrated as being provided directly to the user interface output device 104 by the search system 112. However, it should be noted that in various embodiments, one or more intermediate software and / or hardware components may functionally lie between the search system 112 and the user decoding input device 102 and / or the user interface output device 104, and may optionally process the input and / or output. Therefore, in some embodiments, multiple client computing devices can communicate with all or all aspects of the search system 112, collectively forming a coordinated “ecosystem” of computing devices.

[0030] Search system 112 may additionally include search engine 142, ranking engine 152, and rendering engine 132. In some embodiments, one or more of engines 142, 152, and / or 132 may be omitted, combined, and / or implemented in components separate from search system 112. For example, one or more of engines 142, 152, and / or 132, or any operational portion thereof, may be implemented in a component executed by a client computing device, including user interface input device 102 and / or user interface output device 104, and separate from search system 112. Moreover, for example, search engine 142 and / or ranking engine 152 may be implemented wholly or partially by a system separate from search system 112 (e.g., a separate search system communicating with search system 112).

[0031] In some embodiments, a search query provided to the search system 112 via the user interface input device 102 is used by the search engine 142 during the execution of a search of one or more search databases 175. In some embodiments, once the search engine 142 generates search results for the search databases 175 based on the search query, the results are provided to a ranking engine 152. The ranking engine 152 can rank the results based on the relevance of the resources associated with the search results to the search parameters of the search query.

[0032] In some embodiments, the ranked search results are then provided to the rendering engine 132. The rendering engine 132 provides the ranked search results in response to the search query to one or more user interface output devices 104 for presentation to the user in various output formats. In some embodiments, the ranked search results may include one or more images. In some embodiments, the search query provided to the search system 112 via the user interface input device 102 may be an image search query, and the ranked search results may be multiple images returned in response to the image search query.

[0033] In some embodiments, rendering engine 132 will also utilize images provided to one or more user interface output devices 104 to provide indications of one or more automatically suggested image zoom areas in one or more images included in the ranked search results. In some embodiments, the indications may be visual indications, such as highlights, shadows, boxes, icons, or other graphic indications, which are overlaid on the images when they are displayed on one or more user interface output devices 104. The visual indications may identify selectable targets in one or more images. In response to a user's selection of a target using user interface input device 102 (e.g., by tapping or clicking the target), the image may be automatically zoomed in on a user interface output device 104 (e.g., a display) at the location in the image associated with the selectable target. In some embodiments, the user may also manually manipulate the zoom level and / or scroll position within the image by using user interface input device 102 to access controls in the user interface, providing touch input, etc., after selecting a target or without selecting a target. In some embodiments, information regarding the user's selection of one or more automatically suggested image zoom areas and / or information regarding the manually selected zoom level and / or scroll position within one or more images may be provided to search system 112.

[0034] In some implementations, the image zoom area may be a region within an image that is determined by the search engine 142 to be relevant to a search query and is returned as a result of the search query.

[0035] In other implementations, the image zoom area may be a region within an image that other users typically zoom in on, such as one identified by search engine 142 using historical data in search database 175. In response to receiving information from multiple users regarding user selections of one or more automatically suggested image zoom areas and / or information regarding manually selected zoom levels and / or scroll positions within one or more images, this information regarding regions within typically zoomed-in images stored in search database 175 may be updated. Search system 112 may anonymize and / or aggregate information regarding user selections of one or more automatically suggested image zoom areas and / or information regarding manually selected zoom levels and / or scroll positions within one or more images to protect user privacy.

[0036] In other implementations, the image zoom area may be a region within an image that is frequently zoomed in by the user, or a region within an image that is similar to a region within an image that is frequently zoomed in by the user, such as that identified by the search engine 142 using historical data in the user preference database 185. In response to receiving information about one or more automatically suggested image zoom areas and / or information about manually selected zoom levels and / or scroll positions within one or more images, this information about regions within images frequently zoomed in by the user, stored in the user preference database 185, may be updated.

[0037] In some implementations, when search system 112 performs an image search, it logs (records) when / how users zoomed in on the image results, and for which users the queries were performed, and uses this information to train and refine an offline model for automatically suggesting image zoom areas. Specifically, for a given image, search system 112 mines information about zooming behavior from logs generated from past searches of the same or similar queries to identify query-specific and query-independent common zoom areas. Each precise zoom location and / or level may vary slightly, so search system 112 may perform clustering and / or quantization to collapse a set of similar zooms from different users / sessions into one. Thus, search system 112 can determine common zoom areas for images in general (across different queries) and common zoom areas given a specific query (or cluster of queries). For example, any search that shows a specific image as a result will indicate query-independent zooming where users have similar zooming behavior.

[0038] In some implementations, the mining process described above is used to build a statistical model that enables the search system 112 to identify common user behaviors related to zooming based on log data, given a specific image and / or query. Alternatively or additionally, in some implementations, the mining process described above is used to train a machine learning model (e.g., a deep neural network) that can more broadly generalize common zooming behaviors across different images. The model can be a variant of a convolutional neural network. Specifically, the model can be an object localization model, such as a Single Shot Detector (SSD), a Recurrent Convolutional Neural Network (RCNN), etc. The input to the model can be an image, and the output can be a set of potential bounding boxes. In some implementations, the machine learning model can be multimodal and can take a side input to the query, which is used to bias the results toward content explicitly mentioned by the user as part of their text request. The query parameters for the model can be optional. If no query parameters are specified, the model may fall back to suggesting a general region of interest that is not query-specific.

[0039] In some implementations, a combination of statistical models and neural networks can be used to determine the suggested scaling region. In some implementations, the system applies previously observed behavior to a specific query while simultaneously generalizing more broadly across the image.

[0040] In some implementations, for each set of search results including images, zoom areas can be identified and displayed to the user. Search system 112 can identify query-independent zoom areas offline as a static process, serving as associated metadata for each image. Once the system has a query for each candidate image, search system 112 can find and / or calculate query-specific zoom areas online at query time.

[0041] In some implementations, search system 112 can use image search results with smaller sizes or cropped versions of larger images as learning sources. For example, if a user typically taps or clicks on a smaller image next to a larger one, the indication is based on the query and the user's preferences.

[0042] In some implementations, the image zoom area can also be personalized based on the user's interests. For example, search system 112 can determine that a user is interested in racing cars based on a user profile stored in user preference database 185. Search system 112 can also infer the user's interests based on the user's previous image zoom history and / or other signals such as web searches, video views, purchase history, travel history, etc. In this setting, search system 112 can learn a separate or extended model that allows search system 112 to adjust the convolutional neural network used to determine the suggested zoom area based on such preferences provided as input on the other side. In other implementations, search system 112 can use a general model that emits bounding boxes and related classes. Search system 112 can make the output of the general model intersect with the user's interests. For example, if one of the bounding boxes emitted by the general model is a bounding box of a racing car, search system 112 can select that bounding box of a racing car as a suggested image zoom area to provide to the user based on the user profile stored in user preference database 185, which identifies the user's interest in racing cars.

[0043] Figure 2 A flowchart illustrating an example method 200 that automatically suggests an image scaling area is provided. For convenience, the operation of method 200 is described with reference to a system performing the operation. This system of method 200 includes one or more processors and / or other components of various computer systems. Furthermore, although the operations of method 200 are shown in a specific order, this is not intended to be restrictive. One or more operations may be reordered, omitted, or added.

[0044] In box 210, the system receives a search query. In some embodiments, a search application or search webpage presented by a web browser may be displayed on one or more upper user interface output devices 104. The user may input a search query using one or more user interface input devices 102, which may be transmitted to the search system 112. The search query may be a text query, an image query, or any other type of search query.

[0045] In box 220, the system uses a search query to perform a search to identify search results that include image search results, which comprise multiple images in response to the search query. In some embodiments, the search engine 142 of search system 112 uses the search query received by search system 112 in box 210 to perform a search on one or more search databases 175. As a result of the search, search engine 142 identifies search results that include image search results, which comprise multiple images in response to the search query. In some embodiments, the search query received in box 210 is an image search query and the search results include only image search results. In other embodiments, the search results include images as well as other types of search results (e.g., web pages, text, etc.).

[0046] In box 230, for a given image among a plurality of images included in image search results, the system determines at least one zoomed area in the given image. In some embodiments, search engine 142 provides the search results identified in box 220 to ranking engine 152, which ranks the search results based on the relevance of the resources associated with the results to the search parameters of the search query. In some embodiments, ranking engine 152 then provides the ranked search results to rendering engine 132. In some embodiments, search engine 142, ranking engine 152, and / or rendering engine 132 determine at least one zoomed area in the given image.

[0047] Referring again to box 230, in some embodiments, determining at least one zoom area includes, for each of the at least one zoom area, determining the location of the zoom area and determining the zoom level of the zoom area or the bounding box of the zoom area.

[0048] Referring again to box 230, in some embodiments, determining at least one zoom area includes, for each of the at least one zoom area, determining the location of the zoom area by retrieving metadata associated with a given image, the metadata indicating the location of the zoom area within the given image. In some embodiments, the metadata includes a mapping between a search query and areas in the given image, or a mapping between a semantic representation of a search query and areas in the given image.

[0049] Referring again to box 230, in some embodiments, at least one zoomed area in a given image includes a zoomed area associated with a search query. In some embodiments, at least one zoomed area in a given image is associated with a search query based on historical data reflecting previous zooming on a given image of a past instance returned in response to a search query or a related search query. For example, a related search query may be a search query similar to or the same as the search query received at box 210. In other embodiments, at least one zoomed area in a given image is further associated with a search query based on historical data of one or more other queries having at least a threshold similarity to the search query. The historical data may be stored in search database 175 or any other location. In some embodiments, determining at least one zoomed area in a given image includes determining the location of a zoomed area associated with a search query, but not the location of another zoomed area associated with another search query.

[0050] Referring again to box 230, in some embodiments, at least one zoomable region in a given image includes a zoomable region unrelated to the search query. In some embodiments, at least one zoomable region in a given image includes a zoomable region associated with user preferences. For example, an image zoomable region may be a region within a given image that is frequently zoomed in by the user or is similar to a region within other images frequently zoomed in by the user (e.g., a region including a family member of the user), as identified by search engine 142 using historical data in user preference database 185.

[0051] Referring again to box 230, in some embodiments, the search system 112 uses statistical models, neural networks, or a combination of statistical models and neural networks to determine suggested image scaling regions. In some embodiments, the system uses previously observed behavior for a specific query while also generalizing more broadly across the image.

[0052] Referring again to box 230, in some implementations, for another image among multiple images included in an image search result, the system determines that no indication of a zoom area is provided.

[0053] At box 240, the system provides search results including image search results, including providing a given image and an indication of at least one zoom area within the given image. In some embodiments, rendering engine 132 provides search results to one or more user interface output devices 104 for presentation to a user in various output formats, the search results including image search results identified at box 220 and an indication of at least one zoom area within the given image defined at box 230. For example, search results may be displayed in a search application or a search webpage rendered by a web browser.

[0054] Referring again to frame 240, in some embodiments, providing an indication of at least one zoomable region in a given image includes providing a graphical indication of the location of the zoomable region within the given image for each of the at least one zoomable region in the given image. In some embodiments, the graphical indication may be a visual indication, such as a highlight, a shadow, a frame, an icon, etc., which are superimposed on the given image when the given image is displayed on one or more user interface output devices 104.

[0055] In box 250, the system receives user interface input indicating a selected zoom area in a given image. In some embodiments, the user interface input is provided via user interface input device 102. For example, a user can tap or click an indication of the location of the zoom area in a given image (e.g., highlight, shadow, box, icon, etc.) provided in box 240.

[0056] In box 260, in response to receiving user interface input in box 250, the system adjusts the zoom level of a given image to display a view of the given image magnified by the selected zoom level or based on the bounding box of the selected zoom area, the magnified view being centered on the position of the selected zoom area within the given image. In some embodiments, the zoom level and position of the given image displayed on one or more user interface output devices 104 are automatically adjusted based on the zoom level and position associated with the zoom area selected in box 250.

[0057] Referring again to box 260, in some implementations, after adjusting the scaling level and position (e.g., by zooming in on the bounding box), the system can emphasize specific objects within the bounding box and / or crop specific objects from the larger image. In some implementations, this is done by highlighting specific objects and / or by preserving specific objects but applying a certain level of transparency to the surrounding area in the image. For example, if the bounding box surrounds a bird appearing in an image depicting a landscape, the system can emphasize the heatmap corresponding to the bird (e.g., highlight the bird object against the background), or perform a “soft” crop of the bird from the surrounding image by applying transparency to the area around the bird.

[0058] In other implementations, the operation of method 200 can be performed with respect to video, which the system can treat as a collection of images. For example, the system can preprocess the video to identify and / or extract 10 keyframes, which can be indexed and processed according to the operation of method 200. In box 220, these keyframes can be included in the search results, and in box 230, the system can determine the zoom level within the keyframes. The video and / or keyframes from the video can be included in the image search results in box 240, can be selected in box 250, and the zoom level of the keyframe image or the entire video can be adjusted in box 260.

[0059] Figure 3 A flowchart illustrating an example method 300 that automatically suggests an image scaling area is provided. For convenience, the operation of method 300 is described with reference to a system performing the operation. This system of method 300 includes one or more processors and / or other components of various computer systems. Furthermore, although the operations of method 300 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0060] In box 310, the system receives a search query. In some embodiments, a search application or search webpage presented by a web browser may be displayed on one or more user interface output devices 104. The user may input a search query using one or more user interface input devices 102, which may be transmitted to the search system 112. The search query may be a text query, an image query, or any other type of search query.

[0061] In box 320, the system uses a search query to perform a search to identify search results that include image search results, which comprise multiple images in response to the search query. In some embodiments, the search engine 142 of search system 112 uses the search query received by search system 112 in box 310 to perform a search on one or more search databases 175. As a result of the search, search engine 142 identifies search results that include image search results, which comprise multiple images in response to the search query. In some embodiments, the search query received in box 310 is an image search query and the search results include only image search results. In other embodiments, the search results include images as well as other types of search results (e.g., web pages, text, etc.).

[0062] In box 330, the system provides search results including image search results, which include each of a plurality of images in response to a search query. In some embodiments, rendering engine 132 provides the search results including the image search results identified in box 320 to one or more user interface output devices 104 for presentation to a user in various output formats. For example, the search results may be displayed in a search application or a search webpage rendered by a web browser.

[0063] In box 340, the system receives user interface input indicating a selected image from a plurality of images in response to a search query. In some embodiments, the user interface input is provided to the search system 112 via user interface input device 102. For example, a user can scroll through a plurality of images provided in box 330 and then tap or click to select an image.

[0064] In box 350, the system determines the location of at least one zoomable area within the selected image. In some embodiments, in response to receiving user input in box 340, the search engine 142, ranking engine 152, and / or rendering engine 132 of the search system 112 determine the location of at least one zoomable area within the image selected in box 340. In some embodiments, determining the location of at least one zoomable area within the selected image includes retrieving metadata associated with the selected image that indicates the location of at least one zoomable area within the selected image. In some embodiments, the metadata includes a mapping between a search query and areas in the selected image, or a mapping between a semantic representation of a search query and areas in the selected image.

[0065] Referring again to box 350, in some embodiments, at least one zoom area in the selected image includes a zoom area associated with a search query. In some embodiments, at least one zoom area in the selected image is associated with the search query based on historical data reflecting previous zooming on the selected image of past instances of a given image returned in response to the search query. In other embodiments, at least one zoom area in the selected image is further associated with the search query based on historical data of one or more other queries having at least a threshold similarity to the search query. The historical data may be stored in search database 175 or any other location. In some embodiments, determining the location of at least one zoom area in the selected image includes determining the location of the zoom area associated with the search query, but not the location of another zoom area associated with another search query.

[0066] Referring again to box 350, in some embodiments, at least one zoomable region in the selected image includes a zoomable region unrelated to the search query. In some embodiments, at least one zoomable region in the selected image includes a zoomable region associated with user preferences. For example, the image zoomable region may be a region within the selected image that is frequently zoomed in by the user or is similar to a region within other images frequently zoomed in by the user (e.g., a region including a family member of the user), as identified by search engine 142 using historical data in user preference database 185.

[0067] Referring again to box 350, in some embodiments, the search system 112 uses statistical models, neural networks, or a combination of statistical models and neural networks to determine suggested image scaling regions. In some embodiments, the system uses previously observed behavior for a specific query while also generalizing more broadly across the image.

[0068] At frame 360, the system provides an indication of the position of each of at least one zoom area within the selected image. In some embodiments, rendering engine 132 provides one or more user interface output devices 104 with an indication of the position of at least one zoom area within the selected image determined at frame 350, for example, to be presented to a user within a search application or a search webpage rendered by a web browser.

[0069] Referring again to frame 360, in some embodiments, providing an indication of the position of at least one zoom area in the selected image includes providing a graphical indication of the position of the zoom area within the selected image for each of the at least one zoom area in the selected image. In some embodiments, the graphical indication may be a visual indication, such as a highlight, a shadow, a frame, an icon, etc., which are superimposed on the selected image when the given image is displayed on one or more user interface output devices 104.

[0070] In box 370, the system receives user interface input indicating a selected zoom area in the selected image. In some embodiments, the user interface input is provided via user interface input device 102. For example, a user can tap or click an indication of the location of the zoom area in the selected image provided in box 360 (e.g., highlight, dim light, box, icon, etc.).

[0071] In box 380, in response to receiving user interface input in box 370, the system adjusts the zoom level of the selected image to display a magnified view of the selected image centered on the position of the selected zoom area within the selected image. In some embodiments, the zoom level and position of the selected image displayed on one or more user interface output devices 104 are automatically adjusted based on the zoom level and zoom position associated with the zoom area selected in box 370.

[0072] Figure 4 A flowchart illustrating an example method 400 that automatically suggests an image scaling area is provided. For convenience, the operation of method 400 is described with reference to a system performing the operation. This system of method 400 includes one or more processors and / or other components of various computer systems. Furthermore, although the operations of method 400 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, or added.

[0073] In box 410, the system receives a search query. In some embodiments, a search application or search webpage presented by a web browser may be displayed on one or more user interface output devices 104. The user may input a search query using one or more user interface input devices 102, which may be transmitted to the search system 112. The search query may be a text query, an image query, or any other type of search query.

[0074] In box 420, the system performs a search using a search query to identify search results that include image search results, which comprise multiple images in response to the search query. In some embodiments, search engine 142 of search system 112 performs a search on one or more search databases 175 using the search query received by search system 112 in box 410. As a result of the search, search engine 142 identifies search results that include image search results, which comprise multiple images in response to the search query. In some embodiments, the search query received in box 410 is an image search query and the search results comprise only image search results. In other embodiments, the search results include images as well as other types of search results (e.g., web pages, text, etc.).

[0075] In box 430, for a given image among multiple images included in image search results, the system determines the location of the zoom area within the given image. In some embodiments, search engine 142 provides the search results identified in box 420 to ranking engine 152, which ranks the search results based on the relevance of the resources associated with the results to the search parameters of the search query. In some embodiments, ranking engine 152 then provides the ranked search results to rendering engine 132. In some embodiments, search engine 142, ranking engine 152, and / or rendering engine 132 determine the location of the zoom area within the given image.

[0076] Referring again to box 430, in some embodiments, determining the location of a zoom area within a given image includes retrieving metadata associated with the given image that indicates the location of the zoom area within the given image. In some embodiments, the metadata includes a mapping between a search query and areas in the given image, or a mapping between a semantic representation of a search query and areas in the given image.

[0077] Referring again to box 430, in some embodiments, the zoom area in a given image includes a zoom area associated with a search query. In some embodiments, the zoom area in a given image is associated with the search query based on historical data reflecting previous zooming on a given image of a past instance returned in response to a search query or a related search query. For example, a related search query could be a search query similar to or the same as the search query received in box 410. In other embodiments, the zoom area in a given image is further associated with the search query based on historical data of one or more other queries having at least a threshold similarity to the search query. The historical data may be stored in search database 175 or any other location. In some embodiments, determining the location of a zoom area in a given image includes determining the location of a zoom area associated with a search query, but not the location of another zoom area associated with another search query.

[0078] Referring again to box 430, in some embodiments, the zoom area in a given image includes zoom areas unrelated to the search query. In some embodiments, the zoom area in a given image includes zoom areas associated with user preferences. For example, the zoom area may be a region in a given image that is frequently zoomed in by the user or is similar to a region in other images frequently zoomed in by the user (e.g., a region including a family member of the user), as identified by search engine 142 using historical data in user preference database 185.

[0079] Referring again to box 430, in some embodiments, the search system 112 uses statistical models, neural networks, or a combination of statistical models and neural networks to determine the scaling region. In some embodiments, the system uses previously observed behavior for a specific query while also generalizing more broadly across the image.

[0080] In box 440, the system provides search results including image search results, including, for a given image, a zoom level adjusted to display a magnified view of the given image centered on the position of the zoom area within the given image defined in box 430. In some embodiments, rendering engine 132 provides search results to one or more user interface output devices 104 for presentation to a user in various output formats, the search results including image search results identified in box 420 and a given image with a zoom level automatically adjusted to display a magnified view of the given image. For example, search results may be displayed in a search application or a search webpage rendered by a web browser. In some embodiments, the zoom level of the given image is automatically adjusted without requiring the user to first select a zoom area in the given image.

[0081] Referring again to box 440, in some embodiments, the system provides an indication to zoom in on a given image and provides user interface elements to adjust the zoom level of the given image. The indication may be a graphical element (e.g., an icon) displayed in the user interface or any other visual indication.

[0082] Referring again to box 440, in some embodiments, the system determines a confidence level associated with the zoom area. The confidence level may be based, for example, historical data reflecting previous zooming of a given image within the zoom area related to the search query and / or similar search query received in box 410. In some embodiments, the confidence level may be determined based on the output of a statistical model and / or a neural network. In some embodiments, in response to a confidence level threshold being met within the zoom area, the system adjusts the zoom level to display a magnified view of the given image.

[0083] Figure 5A and 5B Examples of how an image zoom area and a magnified image can be displayed on a user interface output device 104 (e.g., via a search application or a web browser). Figure 5A and 5B The scenario described is for illustrative purposes only. Figure 5A and 5B In the middle, the user interface output device 104 displays webpage 500. For example... Figure 5A As illustrated, webpage 500 includes a search box 510 where a user provides a search query for "bird"; and search result images 520-1 and 520-2, which are provided by search system 112 in response to the search query. Search result image 520-1 includes an indication 530 of the position of an image zoom area 540. In response to receiving user interface input (e.g., a user tapping or clicking the image zoom area 540) via user interface input device 102, the zoom level of search result image 520-1 is adjusted to display a magnified view of search result image 520-1 as a portion of the zoom area 540, such as... Figure 5B As shown in the illustration. Users can further adjust the zoom level using the zoom control 550.

[0084] Figure 6This is a block diagram of an example computing device 610 that can be optionally used to perform one or more aspects of the techniques described herein. The computing device 610 typically includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices may include: a storage subsystem 624, which includes, for example, a memory subsystem 625 and a file storage subsystem 626; a user interface output device 620; a user interface input device 622; and a network interface subsystem 616. The input and output devices allow user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and couples to corresponding interface devices in other computing devices.

[0085] User interface input device 622 may include a keyboard, pointing devices such as a mouse, trackball, touchpad or graphics tablet, scanner, touchscreen integrated into a display, audio input devices such as a voice recognition system, microphone, and / or other types of input devices. Generally, the term "input device" is used to include all possible types of means and methods for inputting information into computing device 610 or a communication network.

[0086] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for producing a visible image. The display subsystem may also provide a non-visual display, for example, via an audio output device. Generally, the term "output device" is used to encompass various possible types of means and methods for outputting information from computing device 610 to a user or another machine or computing device.

[0087] Storage subsystem 624 stores the programming and data structures that provide some or all of the functionality of the modules described herein. For example, storage subsystem 624 may include programming and data structures for performing... Figure 2 , 3 The selection of methods and implementation in 4 Figure 1 The logic of each component described in the text.

[0088] These software modules are typically executed by processor 614 alone or in combination with other processors. Memory subsystem 625, included in storage subsystem 624, can include multiple memories, including main random access memory (RAM) 630 for storing instructions and data during program execution and read-only memory (ROM) 632 for storing fixed instructions. File storage subsystem 626 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives, and associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing the functionality of certain embodiments may be stored by file storage subsystem 626 in storage subsystem 624 or in other machines accessible by processor 614.

[0089] Bus subsystem 612 provides mechanisms for enabling the various components and subsystems of computing device 610 to communicate with each other as intended. Although bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0090] The computing device 610 can be of various types, including workstations, servers, computing clusters, blade servers, server groups, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, therefore... Figure 6 The description of the computing device 610 depicted herein is intended only as a specific example for illustrating some embodiments. Many other configurations of the computing device 610 may have more... Figure 6 The computing device depicted in the text has more or fewer components.

[0091] While several embodiments have been described and illustrated herein, various other ways and / or structures may be used to perform functions and / or obtain results and / or one or more advantages described herein, and each of these variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications to which the teachings are applied. Those skilled in the art will recognize or be able to identify many equivalents of the specific embodiments described herein using only conventional experimentation. Therefore, it should be understood that the foregoing embodiments are presented by way of example only, and that embodiments may be practiced in ways other than those specifically described and claimed within the scope of the appended claims and their equivalents. Embodiments of this disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of this disclosure if such features, systems, articles, materials, kits, and / or methods are not contradictory.

Claims

1. A method implemented by one or more processors, the method comprising: Receive search queries; The search query is used to perform a search to identify search results that include image search results, the image search results comprising a plurality of images in response to the search query; For a given image among the plurality of images included in the image search results, at least one zoom area in the given image is determined, wherein determining at least one zoom area in the given image includes automatically determining the position of at least one zoom area in the given image without user input; and The search results include the image search results, including providing the given image and an indication of at least one zoom area in the given image, wherein the at least one zoom area is selectable to automatically zoom in on the view of the given image, such that when a zoom area in at least one zoom area in the given image is selected, the selected zoom area is centered and enlarged in the view.

2. The method according to claim 1, wherein, The search query is an image search query and the search results include only the image search results.

3. The method of claim 1, further comprising, for another image among the plurality of images included in the image search results, determining that no indication of a zoom area is provided.

4. The method according to claim 1, wherein, Providing the indication of the at least one zoomed area in the given image includes, for each of the at least one zoomed area in the given image, providing a graphical indication of the location of the zoomed area within the given image.

5. The method according to claim 1, wherein, Determining the at least one scaling region includes, for each of the at least one scaling region, determining: The location of the scaling area, and The scaling level of the scaling area or the bounding box of the scaling area, and Further includes: Receive user interface input, the user interface input indicating a selected zoom area in the given image, and In response to receiving the user interface input, the zoom level of the given image is adjusted to display a magnified view of the given image, magnified at the zoom level of the selected zoom area or based on the bounding box of the selected zoom area, the magnified view being centered on the position of the selected zoom area within the given image.

6. The method according to claim 1, wherein, Determining the at least one zoom area includes, for each of the at least one zoom area, determining the location of the zoom area by retrieving metadata associated with the given image, the metadata indicating the location of the zoom area in the given image.

7. The method according to claim 6, wherein, The metadata includes a mapping between a search query and regions in the given image, or a mapping between a semantic representation of a search query and regions in the given image.

8. The method according to claim 1, wherein, The at least one zoom area in the given image includes the zoom area associated with the search query.

9. The method according to claim 8, wherein, The at least one zoomed area in the given image is associated with the search query based on historical data, which reflects previous zooming on the given image of past instances returned in response to the search query or related search queries.

10. The method according to claim 9, wherein, The at least one zoomed area in the given image is further associated with the search query based on historical data of one or more other queries that have at least a threshold similarity to the search query.

11. The method according to claim 1, wherein, Determining the at least one zoomed area in the given image includes determining the location of the zoomed area associated with the search query, but not the location of another zoomed area associated with another search query.

12. The method according to claim 1, wherein, The at least one zoomed area in the given image includes a zoomed area that is independent of the search query.

13. The method according to claim 1, wherein, The at least one zoom area in the given image includes a zoom area associated with user preferences.

14. A method implemented by one or more processors, the method comprising: Receive search queries; The search query is used to perform a search to identify search results that include image search results, the image search results comprising a plurality of images in response to the search query; Provide the search results that include the image search results, including each of the plurality of images in response to the search query; Receive user interface input, the user interface input indicating a selected image from the plurality of images in response to the search query; Determining the position of at least one zoom area in the selected image, wherein determining the position of at least one zoom area in the selected image includes automatically determining the position of at least one zoom area in the selected image without user input; as well as Provides an indication of the position of each of the at least one zoom area in the selected image. The provision of an indication of the position of each of at least one zoom area in the selected image includes, for each of the at least one zoom area in the selected image, providing a graphical indication of the position of the zoom area within the selected image.

15. The method of claim 14, further comprising: Receive user interface input, which indicates a selected zoom area in the selected image; as well as In response to receiving the user interface input, the zoom level of the selected image is adjusted to display a magnified view of the selected image, the magnified view being centered on the selected zoom area at the position within the selected image.

16. The method according to any one of claims 14 to 15, wherein, Determining the location of the at least one zoom area in the selected image includes retrieving metadata associated with the selected image, the metadata indicating the location of the at least one zoom area in the selected image.

17. A method implemented by one or more processors, the method comprising: Receive search queries; The search query is used to perform a search to identify search results that include image search results, the image search results comprising a plurality of images in response to the search query; For a given image among the plurality of images included in the image search results, the position of the zoom area within the given image is automatically determined without user input; The search results include the image search results, including providing the given image with a zoom level adjusted to display a magnified view of the given image, the magnified view being centered at the position of the zoom area within the given image; as well as Determine the confidence level associated with the scaling region. In response to the confidence level associated within the zoom area satisfying a confidence level threshold, the zoom level is adjusted to display the magnified view of the given image.

18. The method of claim 17, further comprising: Provides an indication that the given image has been magnified; as well as Provide user interface elements to adjust the zoom level of the given image.

19. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 18.

20. A computer-readable storage medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 18.

21. A system comprising a processor, a computer-readable storage medium, one or more computer-readable storage media, and program instructions commonly stored on the one or more computer-readable storage media, the program instructions being executable to perform the method according to any one of claims 1 to 18.

Images