Method, system, and program for performing location-based search

Abstract

A technique for performing a search based on location information, wherein a search request providing location information regarding a region of a screen image is received, a selection of a type indicator is received, where the type indicator indicates one of a text element and an image, one or more of the text element and a date and time are received in response to the type indicator indicating the text element, a search is performed using the location information and one or more of the text element and the date and time to identify one or more screen image identifiers of one or more corresponding screen images of a plurality of screen images, where the one or more screen image identifiers are used to retrieve one or more corresponding screen images, and the one or more corresponding screen images are displayed as search results.

Description

[Technical Field]

[0001] SUMMARY OF THE INVENTION Embodiments of the present invention relate to performing location-based searches. [Background technology]

[0002] A user may remember an object (e.g., a subject, a photograph, etc.) by associating it with a location. For example, when reading a document on a computer screen (e.g., a personal computer or smartphone), a user may recall that certain information about the subject was located in the upper right corner of the screen.

[0003] Users may also remember objects relative to their surroundings (e.g., relative to an image) rather than using absolute screen locations (e.g., coordinates). For example, a user may recall seeing an article about a subject on the left side of a newspaper page that had a photo in the bottom right corner of the page.

[0004] In such cases, in conventional search systems, a user executes a search by specifying a particular word or meta information. Summary of the Invention

[0005] According to certain embodiments, a computer-implemented method for performing a location-based search is provided. The computer-implemented method includes the operations: above Area of A search request providing location information is received. A selection of a type indicator is received, the type indicator indicating one of a text element and an image. In response to the type indicator indicating a text element, one or more of a text element and a date and time are received. A search is performed using the location information and one or more of the text element and the date and time. hand, identifying one or more screen image identifiers for one or more corresponding screen images of the plurality of screen images; RThe one or more screen image identifiers are used to retrieve one or more corresponding screen images, which are displayed as search results.

[0006] According to another embodiment, a computer program product for performing location-based searching is provided. The computer program product comprises a computer-readable storage medium having program code embodied therein, the program code being executable by at least one processor to perform operations. above Area of A search request providing location information is received. A selection of a type indicator is received, the type indicator indicating one of a text element and an image. In response to the type indicator indicating a text element, one or more of a text element and a date and time are received. A search is performed using the location information and one or more of the text element and the date and time. hand identifying one or more screen image identifiers for one or more corresponding screen images of the plurality of screen images; R The one or more screen image identifiers are used to retrieve one or more corresponding screen images, which are displayed as search results.

[0007] According to yet another embodiment, a computer system for performing location-based searching is provided, the computer system comprising one or more processors, one or more computer-readable memories, and one or more tangible computer-readable storage devices, and program instructions stored in at least one of the one or more tangible computer-readable storage devices for execution by at least one of the one or more processors via at least one of the one or more memories to perform operations. above Area ofA search request providing location information is received. A selection of a type indicator is received, the type indicator indicating one of a text element and an image. In response to the type indicator indicating a text element, one or more of a text element and a date and time are received. A search is performed using the location information and one or more of the text element and the date and time. hand identifying one or more screen image identifiers for one or more corresponding screen images of the plurality of screen images; R The one or more screen image identifiers are used to retrieve one or more corresponding screen images, which are displayed as search results.

[0008] Reference is now made to the drawings wherein like reference numbers represent corresponding parts throughout. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates, in block diagram form, a computing environment described in certain embodiments. [Figure 2] FIG. 2 illustrates example columns of a table in a database of positioned text elements, as described in certain embodiments. [Figure 3] FIG. 2 illustrates exemplary columns of a table in a geo-referenced image database, as described in certain embodiments. [Figure 4] FIG. 2 illustrates exemplary columns of a table in a location information database, as described in certain embodiments. [Figure 5] FIG. 4 is a flowchart illustrating operations for processing a screen image, as described in certain embodiments. [Figure 6] FIG. 1 is a flowchart illustrating operations for extracting and processing one or more text elements in a screen image, as described in certain embodiments. [Figure 7] FIG. 10 is a flowchart illustrating operations for extracting and processing one or more images within a screen image, as described in certain embodiments. [Figure 8A] FIG. 10 illustrates an example of storing location information for text elements and images, as described in certain embodiments. [Figure 8B] FIG. 10 illustrates an example of storing location information for text elements and images, as described in certain embodiments. [Figure 9A] FIG. 10 is a flowchart illustrating operations for performing a search using location information, as described in certain embodiments. [Figure 9B] FIG. 10 is a flowchart illustrating operations for performing a search using location information, as described in certain embodiments. [Figure 10] FIG. 2 illustrates an exemplary text element search, as described in certain embodiments. [Figure 11] FIG. 2 illustrates an exemplary image search, as described in certain embodiments. [Figure 12] FIG. 1 illustrates a computing node according to certain embodiments. [Figure 13] FIG. 1 illustrates a cloud computing environment according to certain embodiments. [Figure 14] FIG. 1 illustrates abstract model layers according to certain embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] The disclosure of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications that do not depart from the scope and spirit of the disclosed embodiments will be apparent to those skilled in the art. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond those found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0011] Embodiments store text elements (e.g., strings of characters representing words and / or phrases) and images displayed on the screen of a computer (e.g., electronic terminal, personal computer, smartphone, etc.) along with location information for the on-screen text and images, so that the location information can be used for information retrieval. In this manner, embodiments enable a user to request a search using on-screen location information stored by the user. In certain embodiments, the location information specifies coordinates. In certain embodiments, the coordinates relate to an upper-left position (X1, Y1) and a lower-right position (X2, Y2).

[0012] 1 illustrates a block diagram of a computing environment in accordance with certain embodiments. Computing device 100 includes a location engine 110, a screen 120 for displaying text elements and images, and a graphical user interface (GUI) for receiving location-based search input and displaying search results. Computing device 100 is connected to a data store 150.

[0013] The data store 150 includes screen images 160, extracted text elements 162 extracted from each of the screen images, extracted images 164 extracted from each of the screen images, a location-annotated text element database 170, a location-annotated image database 180, and a location information database 190.

[0014] In certain embodiments, computing device 100 is an electronic terminal, personal computer, smartphone, etc., and includes a screen on which text and images are displayed. In certain embodiments, location information database 190 includes location information for text elements and images. In certain embodiments, location-annotated text element database 170 and location-annotated image database 180 are optional.

[0015] The location engine 110 acquires a screen image of the screen 120. For example, the screen image may be a screenshot of a story displayed on a web page. The location engine 110 identifies each text element and the coordinates on the screen image where the text element is located, which reflect the coordinates on the screen 120 where the text element is located. The location engine 110 may use optical character recognition, web page crawling, or other techniques to identify the text elements. The location engine 110 stores each text element, along with its coordinates as metadata, in the data store 150. The location engine 110 identifies images (e.g., photographs, drawings, etc.) and the coordinates on the screen image where each image is located, which reflect the coordinates on the screen 120 where the image is located. The location engine 110 classifies the images (e.g., using a trained model for classification). The location engine 110 stores each image, along with its image classification and its coordinates as metadata, in the data store 150.

[0016] 2 illustrates exemplary columns of a table 200 in the location-based text element database 170, as described in certain embodiments. In FIG. 2, table 200 includes columns for a text element 210 (e.g., a word or phrase), a top-left location coordinate 220, a bottom-right location coordinate 230, a date and time 240, and a screen image identifier 250 (which identifies the screen image of the text element). The date and time may be represented as metadata for the stored image and may be used in search requests.

[0017] 3 illustrates exemplary columns of a table 300 in geo-referenced image database 180, according to certain embodiments. In FIG. 3, table 300 includes columns for the image's image identifier 310, top-left location coordinate 320, bottom-right location coordinate 330, image classification 340, date and time 350, and screen image identifier 360. In certain embodiments, image identifier 310 is used to retrieve the image from extracted images 164.

[0018] 4 illustrates exemplary columns of a table 400 of location information database 190, as described in certain embodiments. In FIG. 4, table 400 includes columns for record identifier 410, type indicator 420 (indicating whether the record is for a text element or an image, and in the case of an image, providing an image classification), value 430 (providing the text element if it is a text element type, or an image identifier if it is an image type), top left position coordinate 440, bottom right position coordinate 450, date and time 460, and screen image identifier 470.

[0019] In certain embodiments, tables 200, 300 in each database 170, 180 are merged to generate table 400 in database 190. In other embodiments, data is stored in table 400 in database 190 without creating each database 170, 180.

[0020] FIG. 5 illustrates, in a flowchart, operations for processing a screen image, according to certain embodiments. Control begins at block 500, where location engine 110 acquires a screen image from the screen of a computing device. At block 502, location engine 110 generates a screen image identifier for the screen image. In certain embodiments, the screen image identifier is unique. In other embodiments, a group of related screen images may have the same screen image identifier, but the screen image identifier, along with the date and time, uniquely identifies each screen image in the group. At block 504, location engine 110 acquires the date and time of the screen image. The date and time indicate when the screen image was acquired. At block 506, location engine 110 stores the screen image with the screen image identifier and the date and time. At block 508, location engine 110 processes one or more text elements of the screen image. At block 510, location engine 110 processes one or more images of the screen image. At block 512, location engine 110 determines whether a predetermined amount of time has passed or whether there has been a change on the screen. If so, processing continues at block 500; if not, processing continues at block 512 to check again. In particular embodiments, instead of returning to block 512 to check again, location engine 110 waits for a period of time before proceeding to block 512 to check again. The predetermined amount of time reflects the amount of time between screen images being captured.

[0021] FIG. 6 illustrates, in a flowchart, operations for extracting and processing one or more text elements in a screen image, according to certain embodiments. The operations of FIG. 6 expand on the processing of block 508. Control begins at block 600, where location engine 110 extracts one or more text elements from a screen image, the screen image having a screen image identifier and an associated date and time. In certain embodiments, the one or more text elements are a series of words. At block 602, location engine 110 selects the next text element of the one or more text elements, starting with the first text element. At block 604, location engine 110 identifies coordinates of the text element on the screen image. At block 606, location engine 110 stores the text element, the coordinates, the date and time of the screen image, and the screen image identifier of the screen image in database 170 and / or database 190. At block 608, location engine 110 determines whether another text element exists to process. If so, processing proceeds to block 602 to select another text element; if not, processing is complete.

[0022] FIG. 7 illustrates, in a flowchart, operations for extracting and processing one or more images within a screen image, according to certain embodiments. The operations of FIG. 7 detail the processing of block 510. Control begins at block 700, where the location engine 110 extracts one or more images from the screen image, the screen image having a screen image identifier and an associated date and time. At block 702, the location engine 110 selects the next image of the one or more images, starting with the first image. At block 704, the location engine 110 identifies the coordinates of the image on the screen image. At block 706, the location engine 110 classifies the image to generate an image classification. In certain embodiments, the location engine 110 classifies the image using an image classifier (e.g., a machine learning model). Examples of image classifications are person, animal, building, road, river, etc.

[0023] At block 708, location engine 110 stores the image, coordinates, image classification, date and time of the screen image, and screen image identifier of the screen image in database 180 and / or database 190. At block 710, location engine 110 determines whether there is another image to process. If there is, processing continues to block 702 to select another image; if not, processing is complete.

[0024] 8A and 8B illustrate examples of storing location information for text elements and images, as described in certain embodiments. In FIG. 8A, screen image 800 includes text elements "Topic_ABC" and "Memory," and includes an image of a photograph of a person. Screen image 800 also has a date and time and a screen image identifier (SS-0001). Location engine 110 extracts each of the text elements, determines the coordinates of those text elements, and stores this information in database table 810. Location engine 110 extracts the image, determines the image classification of "person," determines the coordinates, and stores this information in database table 820. Location engine 110 then links information about the text elements and images with the date and time and screen image identifier of the screen image and stores this information in database table 830, which is an example of a table in location database 190.

[0025] After the location information of the text elements and images is stored, the location engine 110 provides a GUI to allow a user to search the screen by specifying a range of location information as a search parameter (e.g., by using a mouse, finger, etc.). The location engine 110 receives a type attribute via the GUI, where type is another search parameter that specifies text elements or images within the specified range. The location engine 110 may also receive input of a text element or image classification as an additional search parameter. The location engine 110 may also receive a date and time as yet another search parameter. The location engine 110 then attempts to identify one or more screen images using the input from the user. The location engine 110 displays thumbnails of the identified screen images as search results. The user may then select a thumbnail to view the screen image.

[0026] 9A and 9B illustrate, in a flow chart, operations for performing a search using location information, as described in certain embodiments. Control begins at block 900, where the location engine 110 searches for a screen image. above Area of A search request providing location information is received. In certain embodiments, the search request is provided by a user selecting an area via a GUI with a mouse, finger, etc. In other embodiments, the user may select an area by providing a description of the area (e.g., the top left quarter). The area may be represented as an area or portion of a screen image.

[0027] In block 902, the location engine 110 may receive a type indicator of a search request from a user, the type indicator indicating one of a text element and an image. The user uses the type indicator to indicate whether the user is searching for a text element or an image. In particular embodiments, the location engine 110 provides a drop-down box in a GUI to allow the user to select a type indicator.

[0028] At block 904, the location engine 110 determines whether the type indicator indicates a text element. If so, processing continues at block 906; if not, processing continues at block 914 (FIG. 9B). At block 906, the location engine 110 receives one or more of a text element and a date and time. That is, a user may provide input for a text element (e.g., "memory"), a date and time, or both. In particular embodiments, the location engine 110 provides a text box and a calendar-time box in a GUI to allow a user to provide the text element, a date and time, or both.

[0029] At block 908, the location engine 110 performs a search using the location information and one or more of the text elements and date and time to identify one or more screen image identifiers for one or more corresponding screen images. At block 910, the location engine 110 retrieves one or more corresponding screen images using the one or more screen image identifiers. At block 912, in response to the search request, the location engine 110 displays the one or more corresponding screen images as search results. In particular embodiments, the location engine 110 provides the search results as thumbnails of the screen images, and when a thumbnail is selected, the screen image of the thumbnail is displayed in full in the GUI. In other embodiments, the location engine 110 provides the search results as a list of screen image identifiers for selection.

[0030] At block 914, the location engine 110 receives one or more of the image classification and the date and time. That is, the user may provide input for the image classification, the date and time, or both. In particular embodiments, the location engine 110 provides an image classification drop-down box and a calendar time box in the GUI to allow the user to provide the image classification, the date and time, or both.

[0031] At block 916, the location engine 110 performs a search using the location information and one or more of the image classification and date and time to identify one or more screen image identifiers for one or more corresponding screen images. R At block 918, the location engine 110 uses the one or more screen image identifiers to retrieve one or more corresponding screen images. At block 920, in response to the search request, the location engine 110 displays the one or more corresponding screen images as search results. In particular embodiments, the location engine 110 provides the search results as thumbnails of the screen images, and when a thumbnail is selected, the screen image of the thumbnail is displayed in full in the GUI. In other embodiments, the location engine 110 provides the search results as a list of screen image identifiers for selection.

[0032] FIG. 10 illustrates an exemplary text element search, according to certain embodiments. In certain embodiments, the GUI displays an input (e.g., on the left) to allow a user to provide input for a search and an output (e.g., on the right) of search results found for that input. In GUI 1010, the location engine 110 receives a user selection of a region. In this example, the user selected a rectangular region using a mouse or finger. However, in other embodiments, a circular or other shaped region may be selected. In GUI 1020, the location engine 110 indicates that no search results were found based on region alone. In GUI 1030, the location engine 110 receives a user selection of a text element type indicator. In this example, the user selected a text element for the type indicator using a dropdown. In GUI 1040, the location engine 110 indicates that no search results were found based on region and text element type indicator. In GUI 1050, the location engine 110 receives a user selection of the text element "Memory." In this example, the user entered the text element "Memory" into the text element input field. After performing a search using the region and "Memory", the location engine 110 displays in the GUI 1060 thumbnails of screen images that contain the text element "Memory" in the region.

[0033] 11 illustrates an exemplary image search, as described in certain embodiments. In GUI 1110, location engine 110 receives a user selection of a new region and image type indicator. After performing a search using the region and image type indicator, location engine 110 displays thumbnails of two screen images containing images within the region in GUI 1120. Each of these two screen images contains the image within the region.

[0034] In GUI 1130, location engine 110 receives a user selection of an image classification of "people." In this example, the user selected the image classification of "people" using a dropdown. In GUI 1140, location engine 110 displays the screen image of the two screen images (from GUI 1120) that contains the image of a person.

[0035] In alternative embodiments, the user may optionally provide a hand-drawn image of a person, a photograph, etc. In GUI 1150, location engine 110 received user input of a more specific image. In this example, the user drew the image freehand. In particular embodiments, the user may specify metadata for the drawn image, including color, number of entities (e.g., people, animals, buildings, etc.).

[0036] In certain embodiments, the location engine 110 acquires a screen image of the electronic device and stores the screen image along with a screen image identifier generated for the screen image. The location engine 110 collects strings within the screen image and stores the strings along with coordinate information and a screen image identifier. The location engine 110 collects images within the screen image, classifies the images according to type using a trained model, and stores the images by adding coordinate information and a screen image identifier. The location engine 110 searches for screen images based on a user's designation of an area or range for the user to search on the screen. The location engine 110 receives input of a type indicator of the string or image within the area along with either the string or image classification. The location engine 110 then searches for screen image identifiers in a database that contain the desired string or image classification within the area (i.e., based on the coordinates of the string or image classification near the specified area). The location engine 110 displays screen images with screen capture identifiers.

[0037] The location engine 110 uses location information to enable searches when a user remembers the location of the desired information. For example, a user may use embodiments to perform a search based on memory, such as: "I think I saw information about the treatment in the documents that were displayed in the upper right corner of the screen." "I saw an article about computers on the left side of the newspaper page on my screen." "I think I saw a picture in the bottom right corner of the web page on my screen."

[0038] The location engine 110 may be provided as part of an application's functionality or may be built into the operating system. Screen images may be captured and stored at fixed time intervals or based on changes to the screen. In particular embodiments, the entire screen may be captured, or a portion of the screen (e.g., where a change is detected) may be captured.

[0039] In particular embodiments, the location engine 110 stores a subset of text elements, thereby avoiding storing all text elements and conserving storage and other computational resources. For example, the location engine 110 may determine the term frequency-inverse document frequency of each of the text elements and store the text elements that are more frequently used resources. As another example, the location engine 110 may determine that text elements included in a dictionary or other source can be stored. In yet another example, the location engine 110 may store text elements not found in a dictionary or other source.

[0040] In certain embodiments, text elements on a web page are detected by crawling the website rather than using optical character recognition.

[0041] In particular embodiments, the location engine 110 may selectively exclude certain elements in the screen image, such as commonly displayed toolbars or particular icons, or prioritize such elements to determine whether to include the elements in the screen image, in order to conserve storage.

[0042] Embodiments may be used to capture any screen. For example, embodiments are applicable to electronic medical records (e.g., where information is entered electronically or where paper medical records are scanned and stored as images). In such an example, a doctor can search the electronic medical record based on his or her memory (e.g., of the paper medical record).

[0043] Unlike conventional techniques that search for targets from documents stored in data sources such as databases, the present embodiment searches for targets from documents that the user has viewed and that are displayed in a GUI by allowing the user to specify on-screen location information as a search criterion.

[0044] Unlike prior art techniques that search based on text, metadata, or text linked to an image, where a user must know about related words or images (input) to perform a search, embodiments provide searching based on input of on-screen location information. Embodiments may be used in conjunction with prior art techniques to enhance search accuracy by further using metadata or text as additional input.

[0045] 12 illustrates a computing environment 1210 according to certain embodiments. In certain embodiments, the computing environment is a cloud computing environment. With reference to FIG. 12, computer node 1212 is merely one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments of the invention described herein. In any event, computer node 1212 may implement and / or perform any of the functions described above.

[0046] Computer node 1212 may be a computer system capable of operating in numerous other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, suitable for use with computer node 1212 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, microcomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of these systems or devices.

[0047] Computer node 1212 may be described in the general context of computer system executable instructions, such as program modules being executed by the computer system. Typically, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer node 1212 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0048] 12, computer node 1212 is shown in the form of a general-purpose computing device. Components of computer node 1212 may include, but are not limited to, one or more processors or processing units 1216, a system memory 1228, and a bus 1218 that couples various system components including the system memory 1228 to the one or more processors or processing units 1216.

[0049] Bus 1218 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example only, an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnects (PCI) bus.

[0050] Computer node 1212 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer node 1212 and includes both volatile and nonvolatile media, removable and non-removable media.

[0051] System memory 1228 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 1230 and / or cache memory 1232. Computer node 1212 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 1234 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to removable, non-volatile magnetic disks (e.g., "floppy disks"), and an optical disk drive may be provided for reading from and writing to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media. In such examples, each may be connected to bus 1218 by one or more data media interfaces. As shown and described in detail below, the system memory 1228 may include at least one program product comprising a series of (e.g., at least one) program modules configured to perform the functions of embodiments of the present invention.

[0052] For example, programs / utilities 1240 including a set of (at least one) program modules 1242 may be stored in system memory 1228, including, but not limited to, an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or a combination thereof, may include an implementation of a network environment. The program modules 1242 typically perform the functions and / or methods of embodiments of the present invention described herein.

[0053] Computer node 1212 may also communicate with one or more external devices 1214, such as a keyboard, pointing device, display 1224, one or more devices that allow a user to interact with computer node 1212, or any device (e.g., network card, modem, etc.) that allows computer node 1212 to communicate with one or more other computing devices, or combinations thereof. Such communication may occur via input / output (I / O) interface 1222. Additionally, computer node 1212 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or combinations thereof, via network adapter 1220. As shown, network adapter 1220 communicates with other components of computer node 1212 via bus 1218. It should be understood that other hardware and / or software components, not shown, may be used with computer node 1212. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0054] In certain embodiments, computing device 100 has a computer node 1212 architecture. In certain embodiments, computing device 100 is part of a cloud infrastructure. In certain alternative embodiments, computing device 100 is not part of a cloud infrastructure.

[0055] Cloud Implementation Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings presented herein is not limited to cloud computing environments. Rather, embodiments of the present invention may be implemented in conjunction with any other type of computing environment now known or later developed.

[0056] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computational resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) and for rapidly provisioning and releasing these resources with minimal administrative effort or interaction with a service provider. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0057] The features are as follows: On-demand self-service: Cloud customers can automatically provision server time, network storage, and other computing power as needed, without requiring unilateral, human interaction with the service provider. Wide network access: Cloud capabilities are available over the network and can be accessed using standard mechanisms, facilitating usage by heterogeneous thin- or thick-client platforms (e.g., mobile phones, laptops, and PDAs). Resource Pool: The provider's computing resources are pooled and offered to multiple consumers using a multi-tenant model. Various physical and virtual resources are dynamically allocated and reallocated according to demand. There is a location-independent sense; consumers typically have no control or knowledge regarding the exact location of the resources offered, although at a higher level of abstraction, they may be able to specify a location (e.g., country, state, or data center). Rapid Elasticity: Cloud capacity can be quickly and elastically provisioned, in some cases automatically, to scale out quickly, and quickly released to scale in quickly. Capacity available for provisioning often appears to consumers as unlimited, available for purchase in any quantity at any time. Metered Services: Cloud systems leverage metering capabilities to automatically control and optimize resource usage at an abstraction level appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0058] The service model is as follows: SaaS (Software as a Service): The consumer is provided with the ability to use the provider's applications running on a cloud infrastructure. Those applications can be accessed from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application features, except for the possibility of setting limited user-specific application configuration settings. PaaS (Platform as a Service): The ability offered to a consumer is to deploy applications they create or acquire, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The capability provided to a customer is the provisioning of processing, storage, network, and other basic computing resources, over which the customer can deploy and run any software, which may include operating systems and applications. The customer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, and deployed applications, and in some cases, limited control over selected network components (e.g., host firewalls).

[0059] The deployment model is as follows: Private Cloud: This cloud infrastructure is operated solely for one organization and can be managed by that organization or a third party, and can reside on-premise or off-premise. Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with shared interests (e.g., mission, security requirements, policy, and compliance considerations). It can be managed by these organizations or a third party and can reside on-premises or off-premises. Public cloud: This cloud infrastructure is available for use by the general public or large industry organizations and is owned by an organization that sells cloud services. Hybrid cloud: This cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain distinct but are joined together by standardized or proprietary technologies that allow for data and application portability (e.g., cloud bursting to balance load between clouds).

[0060] A cloud computing environment is a service-oriented environment that emphasizes statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0061] Referring now to FIG. 13 , an exemplary cloud computing environment 1320 is illustrated. As illustrated, the cloud computing environment 1320 includes one or more cloud computing nodes 1310 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1354A, a desktop computer 1354B, a laptop computer 1354C, or an automobile computer system 1354N, or any combination thereof, may communicate. The nodes 1310 may communicate with each other. The nodes 1310 may be physically or virtually grouped in one or more networks (not shown), such as a private cloud, community cloud, public cloud, or hybrid cloud, or any combination thereof, as previously described herein. This allows the cloud computing environment 1320 to provide an infrastructure, platform, and / or SaaS that does not require cloud consumers to maintain resources on their local computing devices. The types of computing devices 1354A-N shown in FIG. 13 are intended to be illustrative only, and it is understood that the computing node 1310 and cloud computing environment 1320 can communicate with any type of computer-controlled device via any type of network and / or network-addressable connection (e.g., a connection using a web browser).

[0062] Referring now to Figure 14, a set of functional abstraction layers provided by cloud computing environment 1320 (Figure 13) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 14 are intended to be illustrative only, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0063] Hardware and software layer 1460 includes hardware and software components. Examples of hardware components include mainframe 1461, reduced instruction set computer (RISC) architecture-based server 1462, server 1463, blade server 1464, storage device 1465, and network and network components 1466. In some embodiments, software components include network application server software 1467 and database software 1468.

[0064] The virtualization layer 1470 comprises an abstraction layer that can provide virtual entities such as virtual servers 1471 , virtual storage 1472 , virtual networks including virtual private networks 1473 , virtual applications and operating systems 1474 , and virtual clients 1475 .

[0065] By way of example, management layer 1480 may provide the following functions: Resource provisioning 1481 dynamically procures computing and other resources used to execute tasks within the cloud computing environment. Metering and pricing 1482 tracks the costs of resources used within the cloud computing environment and generates and sends bills for the use of those resources. By way of example, those resources may include application software licenses. Security verifies the identity of cloud users and tasks and protects data and other resources. User portal 1483 provides users and system administrators with access to the cloud computing environment. Service level management 1484 allocates and manages cloud computing resources to meet required service levels. Service level agreement (SLA) planning and execution 1485 proactively prepares and procures cloud computing resources in accordance with SLAs in anticipation of upcoming demand.

[0066] The workload layer 1490 shows examples of functionality available in a cloud computing environment. Examples of workloads and functionality provided by this layer include mapping and navigation 1491, software development and lifecycle management 1492, virtual classroom instruction delivery 1493, data analytics processing 1494, transaction processing 1495, and performing location-based search 1496.

[0067] Thus, in certain embodiments, software or programs implementing location-based search execution, according to embodiments described herein, are provided as a service in a cloud environment.

[0068] Additional Embodiment Details The present invention may be a system, a method, or a computer program product, or any combination thereof, at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium containing computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0069] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device, such as, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves in which instructions are recorded, and any suitable combination thereof. As used herein, a computer-readable storage medium should not itself be construed as a transitory signal such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0070] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or any combination thereof. This network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or any combination thereof. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within each computing / processing device.

[0071] Computer-readable program instructions for carrying out the operations of the present invention may be source or object code written in any combination of one or more programming languages, including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present invention, electronic circuitry including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to customize the electronic circuitry by utilizing state information of the computer-readable program instructions.

[0072] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0073] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to create a machine, where the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium and capable of directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer-readable storage medium on which the instructions are stored comprises an article of manufacture containing instructions for performing aspects of the functions / acts specified in the blocks of the flowcharts and / or block diagrams.

[0074] Computer-readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / acts specified in the flowchart and / or block diagram blocks, thereby causing a series of operable steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process.

[0075] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be executed concurrently as a single step, or may be executed substantially concurrently in a partially or fully overlapping manner in time, or may even be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks included in the block diagrams and / or flowchart diagrams, may be implemented by a dedicated hardware-based system that performs the specified function or operation or executes a combination of dedicated hardware and computer instructions.

[0076] The terms "an embodiment," "embodiment," "embodiments," "the embodiment," "the embodiment," "the embodiment," "one or more embodiments," "some embodiments," and "one embodiment" mean "one or more (but not all) embodiments of the present invention," unless expressly specified otherwise.

[0077] The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

[0078] An enumerated list of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise.

[0079] The terms "a," "an," and "the" mean "one or more," unless expressly specified otherwise.

[0080] Devices that are in communication with each other need not be in continuous communication with each other unless explicitly specified otherwise. In addition, devices that are in communication with each other may communicate directly or indirectly through one or more intermediaries.

[0081] A description of an embodiment including multiple components in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0082] Where a single device or item is described herein, it will be readily apparent that two or more devices / items (whether cooperating or not) may be used in place of the single device / item. Similarly, where two or more devices or items (whether cooperating or not) are described herein, it will be readily apparent that a single device / item may be used in place of two or more devices or items, or that a different number of devices / items may be used in place of the number of devices or programs shown. The functionality and / or features of a device may alternatively be embodied by one or more other devices not explicitly described as having such functionality / features. Thus, other embodiments of the present invention need not include the device itself.

[0083] The foregoing description of various embodiments of the invention has been provided for purposes of illustration and description. It is not intended that these descriptions be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above. It is intended that the scope of the invention be limited not by this Detailed Description, but by the claims appended hereto. The above specification, examples, and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, embodiments of the invention reside in the claims appended hereto. The foregoing description provides examples of embodiments of the invention, and modifications and substitutions may be made in other embodiments.

Claims

1. 1. A computer-implemented method, the computer storing text elements and images displayed on a screen of the computer along with on-screen text position information and image position information; The method comprises: receiving a search request providing location information of an area on a screen image, the search request being a search request using on-screen location information stored by a user when the user viewed the screen image; receiving a type indicator of the search request, the type indicator indicating one of a text element and an image; In response to the type indicator indicating the text element: receiving one or more of the text element and the date and time; searching a database using the location information and the one or more of the text elements and the date and time to identify one or more screen image identifiers for one or more corresponding screen images of a plurality of screen images, wherein the database has location information, the text elements, the date and time at which the screen image was captured, and the screen image identifiers; retrieving the one or more corresponding screen images from an image database using the identified one or more screen image identifiers; and displaying the one or more corresponding screen images as search results; The method comprising:

2. The method of claim 1 , wherein the search is performed by matching coordinates of the region with coordinates in a database associated with the text element.

3. In response to the type indicator indicating the image, receiving one or more of an image classification and another date and time; performing a new search using the location information and the one or more of the image classification and the different date and time to identify one or more additional screen image identifiers for one or more corresponding screen images; The method of claim 1 further comprising:

4. The method of claim 3 , wherein the new search is performed by matching coordinates of the region with coordinates in a database associated with the image classification.

5. The method comprises: obtaining the plurality of screen images by periodically capturing each of the screen images; further comprising each of said screen images having an associated date and time and a screen image identifier; The method of claim 1.

6. identifying text elements in screen images of the plurality of screen images; storing said text elements together with coordinates of said text elements on said screen image, a date and time of said screen image, and a screen image identifier of said screen image; The method of claim 5 further comprising:

7. Identifying an image of the plurality of screen images; classifying the image to determine an image classification; storing said image together with the coordinates of said image on said screen image, said image classification, the date and time of said screen image, and a screen image identifier of said screen image; The method of claim 5 further comprising:

8. The method of claim 1 , wherein the method is performed in Software as a Service (SaaS).

9. A computer program causing a computer to execute the method according to any one of claims 1 to 8.

10. 10. A computer readable storage medium having the computer program according to claim 9 stored thereon.

11. A computer system, comprising a processor and a memory, that executes the method according to any one of claims 1 to 8.

Images