Map search recommendation system based on geographic semantic index driven by image content analysis
Generate geosemantic indexes through image content analysis, which solves the inconsistency caused by unmapped and volatile facility attributes in map search, and realizes accurate recommendations for specific activity locations, improving the comprehensiveness and accuracy of search results.
Patent Information
- Application Number
- CN202411172518.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-20
- Filing Date
- 2019-01-07
- Publication Date
- 2024-12-06
AI Technical Summary
Existing map search techniques are difficult to accurately identify locations associated with human activities, especially inconsistent or incorrect search results due to unmapped facilities and attributes, excessive or volatile.
Image content analysis technology is used to generate geosemiconductor indexes, and semantic elements are identified by analyzing and numerically identifiable elements to construct a geosemiconductor index containing rich semantic context information, which is used to provide location recommendations for specific activities.
It improves the accuracy and comprehensiveness of map searches, can identify and recommend locations suitable for specific activities, and solves the problem of inconsistent query results caused by unmapped and variable facility attributes in the prior art.
Smart Images

Figure CN119088995A_ABST
Abstract
Description
[0001] This application is a divisional application of an invention patent application with an application date of January 7, 2019, application number 201980051889.7, and invention name “Map search recommendation system based on geographic semantic indexing driven by image content analysis”. Technical Field
[0002] The present disclosure generally relates to search recommendation. More specifically, the present disclosure relates to implementing search recommendation using image content analysis and geo-semantic indexing. Background Art
[0003] Users often wish to find locations that can accommodate certain human activity needs. However, when searching for nearby locations with certain facilities and / or physical attributes, typical search queries may not produce results that are valuable to the user. For example, a user may utilize a mapping application to view a map of an area and attempt to visually determine locations for a particular activity or search for manually marked elements that may indicate the possibility of a sporting activity. Such search techniques may be cumbersome and may produce inconsistent or incorrect results. For example, facilities and / or attributes may be unmapped and / or too numerous or very susceptible to change. Therefore, there is a need for an improved system for map searching to identify locations associated with an activity. Summary of the invention
[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.
[0005] An example aspect of the present disclosure is directed to a computer-implemented method, the method comprising obtaining, by one or more computing devices, a vocabulary of image feature types associated with user activities. The method comprises obtaining, by the one or more computing devices, a collection of pictures. The method comprises performing, by the one or more computing devices, an image content analysis on the collection of pictures based on the vocabulary of image feature types. The method comprises generating, by the one or more computing devices, at least one activity score for each of a plurality of location units in a geo-semantic index based at least in part on the vocabulary of image feature types. The method comprises populating, by the one or more computing devices, a geo-semantic index of location units, the geo-semantic index of location units comprising data indicating the at least one activity score for each location unit. The method comprises providing, by the one or more computing devices, a geo-semantic index of location units for generating location recommendations in response to a query.
[0006] Another example aspect of the present disclosure is directed to a computer-implemented method, the method comprising: obtaining, by one or more computing devices, a vocabulary of image feature types associated with user activities; obtaining, by one or more computing devices, a collection of pictures; performing, by one or more computing devices, image content analysis on the collection of pictures based on the vocabulary of image feature types; generating, by one or more computing devices, at least one activity score for each of a plurality of locations based at least in part on the vocabulary of image feature types; and providing, by one or more computing devices, a location recommendation based on the activity scores of the plurality of locations in response to a query associated with a particular activity.
[0007] Another example aspect of the present disclosure is directed to a computing system, comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions, which, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising: obtaining a vocabulary of image feature types associated with user activities; obtaining a collection of pictures; performing image content analysis on the collection of pictures based on the vocabulary of image feature types; generating at least one activity score for each of a plurality of locations based at least in part on the vocabulary of image feature types; and providing location recommendations based on the activity scores of the plurality of locations in response to a query associated with a particular activity.
[0008] Another example aspect of the present disclosure is directed to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing system, cause the computing system to: obtain a vocabulary of image feature types associated with user activities; obtain a collection of pictures; perform image content analysis on the collection of pictures based on the vocabulary of image feature types; generate at least one activity score for each of a plurality of locations based at least in part on the vocabulary of image feature types; and provide location recommendations based on the activity scores for the plurality of locations in response to a query associated with a particular activity.
[0009] Other aspects of the present disclosure are directed to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices. It will be understood that the aspects described above in the context of one form can be implemented in any other convenient form. Features described in the context of one aspect can be implemented in the context of another aspect.
[0010] These and other features, aspects and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] A detailed discussion of embodiments for those of ordinary skill in the art is set forth in this specification with reference to the accompanying drawings, in which:
[0012] Figure 1 depicts a block diagram of a computing system according to an example embodiment of the present disclosure;
[0013] Figure 2 depicts an example of a street-level image with identified features according to an example embodiment of the present disclosure;
[0014] Figure 3A-3B depicts an example of a street-level image according to an example embodiment of the present disclosure;
[0015] Figure 4 A flowchart of an example method for providing search recommendations based on a geographic semantic index according to an example embodiment of the present disclosure is depicted.
[0016] Figure 5 A flowchart depicting an example method for providing search recommendations based on a geographic semantic index according to an example embodiment of the present disclosure; and
[0017] Figure 6A-6B Another example of a street-level image is depicted according to an example embodiment of the present disclosure.
[0018] Reference numerals repeated in various figures are intended to identify like features in the various embodiments. DETAILED DESCRIPTION
[0019] Overview
[0020] In general, the present disclosure is directed to a system and method for map search recommendations based on a geographic semantic index developed using image content analysis. Specifically, the system and method of the present disclosure can facilitate the generation of a geographic semantic index of features and / or attribute types associated with user activities for the identification of locations and the use of such geographic semantic indexes to provide map search recommendations including one or more geographic locations that can be adapted to specific user activities. For example, a user may want to search for nearby locations that can be adapted to specific user activities (e.g., nearby child-friendly places, playgrounds with shaded areas, places where people can sit and read, etc.). A geographic semantic index of a location can be constructed based on image content analysis (e.g., analyzing a periodic collection of street-level pictures, etc. to identify features / attributes associated with an activity, etc.), and the geographic semantic index includes rich semantic context information for each location. Then, the geographic semantic index can be used to identify location units within a certain geographic area (e.g., a fixed radius of a user's location, etc.), which can adapt to desired activities (e.g., activity scores based on location units, etc.). The identified location can be provided to the user as part of a query result recommendation (e.g., a mapped location, a location list, etc.). In some cases, location recommendations may include descriptions of locations based on features / attributes and activities associated with the locations. Thus, the systems and methods of the present disclosure may support answering user queries for locations with requested features and / or attributes that may be suitable for specific activities in a more comprehensive manner and directing users to such locations.
[0021] More specifically, according to one aspect of the present disclosure, a computing system can facilitate the construction and periodic updating of a geo-semantic index of locations (e.g., on a grid of location cells) having rich semantic context information for each location generated using an image content analysis engine. The image content analysis system can identify a wide range of semantic elements (e.g., on the order of tens of thousands of elements), many of which reference features and / or attributes of physical and urban landscapes. Such an image content analysis system can be tasked with periodically processing very large image corpora. The geo-semantic index can use actual knowledge about physical locations derived from collected images to provide a semantic context map of the world and facilitate search recommendations for places that can support desired activities and / or recommendations for activities within a particular area.
[0022] According to one aspect of the present disclosure, a computing system may support cataloguing of vocabularies of known feature and / or attribute types (e.g., from image content analysis dictionaries, etc.) and user activities to which they may be adapted (e.g., to search for users and applications that may be of interest). For example, such a feature / attribute vocabulary may include "bench" for reading; "alley" for walking; "loading area" or "parking meter" for passengers to pick up or park; "statue", "fountain", or "boutique" for gatherings; etc. The feature / attribute vocabulary may have a many-to-many relationship between activities and vocabulary terms. In some embodiments, the feature / attribute vocabulary may include verb attribute terms in addition to physical features / attributes, for example, identifying that people are reading in a picture, etc.
[0023] The computing system may periodically obtain a collection of pictures (e.g., street-level pictures, aerial pictures, etc.), for example, from a continuously updated picture corpus, publicly available pictures, user-generated pictures, etc. The computing system may perform image content analysis on the obtained pictures to identify feature / attribute type terms from the generated vocabulary. Based on the identified features / attributes, the computing system may populate and / or update a geo-semantic index of location cells associated with the pictures (e.g., on a grid of location cells mapped to a geographic area). The geo-semantic index may also include data related to the collection time of pictures with the identified features / attributes. In some embodiments, the index may be continuously updated based on user feedback, for example, via manual updates from a human operator to match features / attributes to activities. In some embodiments, the image content analysis may include filtering out certain image data, such as features / attributes identified in private and / or restricted areas, etc.
[0024] For each activity supported by the vocabulary, the computing system may generate a weighted activity score for each location unit based at least in part on the number, diversity and / or recency of semantic terms corresponding to the activity associated with the location unit. In some embodiments, the activity score may be generated based on one or more metrics defined by a client application associated with the activity. For example, many user applications focus on specific human activities, which may be supported by search recommendations using a geo-semantic index as described herein. In some embodiments, some activities in the geo-semantic index and / or vocabulary may be defined based only on the needs of specific user applications that define activities and metrics for activity scores. In some embodiments, activity scores may be different based on the type of activity, the type of application, etc., for example, based on elements that may be important or necessary for the activity. The generation of such activity scores and the repeated refreshing of information provided by image content analysis from a large picture corpus (e.g., a street-level picture corpus, etc.) can enable the function of geo-semantic indexing to provide users with more comprehensive suggestions based on desired activities.
[0025] According to another aspect of the present disclosure, a computing system may obtain a query from a user about nearby locations that may be adapted to a particular user activity. The computing system may analyze a geo-semantic index to identify one or more high-scoring location cells nearby (e.g., within a fixed radius of a search area indicated by the user, around the user's current location, etc.), and provide one or more of those locations that match the desired activity as search recommendations. In some embodiments, for each location, the computing system may generate a text description based on features / attributes and / or activities associated with the location. For example, in some embodiments, such a description may be generated based on the application of natural language processing, may be curated by a human operator, and so on. These location descriptions may be provided to the user as part of a location recommendation to help the user select and / or find a desired location.
[0026] For example, a user query may be submitted to the computing system, such as "find a place where I can go for a walk and enjoy some shade." The computing system may filter the query submission for any known activity words associated with the vocabulary, for example, the system may identify "walk" and "enjoy the shade" as activity terms in the vocabulary. The computing system may analyze the index to identify one or more location cells that have high matches for certain vocabulary terms based on the presence of these activities in the cell. Thus, the identified cell locations may have among their descriptive feature terms terms such as "alley," "tree," "sidewalk," "gravel path," and / or other similar terms. The computing system may return one or more of the identified locations as recommendations to the user, for example based in part on the activity scores associated with the identified location cells.
[0027] In some implementations, the geo-semantic index may be used to provide recommendations for one or more activities within a certain geographic area, as opposed to recommendations for locations that accommodate a particular activity. For example, instead of answering the query "where can I do <specific activity>", the computing system may use the geo-semantic index to answer a request for "please recommend activities that I can do within X meters of my location". In this case, the same geo-semantic index may use the user's current location and the semantic elements stored in the index (e.g., developed based on image content analysis) to provide recommendations for certain activities within the specified geographic area supported by those elements.
[0028] In some cases, such activity queries may not be issued directly by users through desktop search or voice assistants. Instead, potential users can also interact with the recommendation system through their use of various client applications (e.g., via RPC calls). Many applications focus on specific human activities that can be supported by geographic semantic indexing. Some good examples can include applications for gatherings, calling cars, sports and outdoor sports, virtual reality games, dog walking, bird watching, etc. Each type of application may need to define its activity terms and specific weights and measurement functions with any other algorithm filtering criteria (e.g., avoiding roads or residential buildings with a specific distance margin, etc.). According to some aspects, the system and method of the present disclosure can provide an interface for any application that requires knowledge about public spaces to support its activity-based features set for users. As an example, in the case of a passenger ride, the computing system can return a location with a "loading zone" sign, a parking meter, a highly visible landmark, and other items that make it easy for drivers and passengers to find a place and easily and legally park and pick up people at the place. In the party example, the activity score of a location unit may take into account items that make the place highly "walkable," for example, having elements such as "walking paths" and / or various pedestrian amenities, "fountains," "sculptures," etc.
[0029] According to some aspects, a computing system may obtain a vocabulary of image feature types associated with user activities. The vocabulary may include data identifying image feature types. Multiple activity types may each be associated with a corresponding subset of image feature types, the corresponding subset being from the vocabulary of image feature types. The computing system may obtain a collection of pictures, such as a collection of images. The computing system may perform image content analysis on the collection of pictures based on the vocabulary of image feature types. For example, the computing system may determine whether a corresponding picture of each of a plurality of location units of a geo-semantic index includes a feature type from the vocabulary of image feature types. The computing system may generate at least one activity score for each location unit in the geo-semantic index based on the vocabulary of image feature types. For each location unit, the computing system may determine an activity score for each activity type based at least in part on whether the location unit includes some corresponding subsets of image feature types for such activity types. The computing system may populate the geo-semantic index of the location unit with data indicating the at least one activity score for each location unit. The computing system may provide the geo-semantic index of the location unit for generating location recommendations in response to a query.
[0030] The systems and methods described herein provide many technical effects and benefits. For example, the systems and methods of the present disclosure can provide improvements to search recommendation systems. The systems and methods can, for example, enable queries to nearby locations based on the presence of certain facilities and / or attributes that may not yet be mapped and / or are too many or very easy to change. The latter may especially make it infeasible to rely solely on fixed permanent records of such entities (e.g., user reports) without being able to automatically refresh them. Therefore, the systems and methods of the present disclosure can support answering user queries for locations with requested features and / or attributes that can adapt to specific activities in a more comprehensive manner, and guide users to such locations.
[0031] According to some aspects, an improved search recommendation system and / or related user interface can be provided by associating locations with activities based on image content analysis. Image content analysis of pictures associated with locations can be used to identify the presence of features of a vocabulary corresponding to image features. Image features in the vocabulary can be associated with activities or activity types. Thus, the geo-semantic index described herein can include information associated with activities at specific locations, where these activities are determined based on image content analysis. Such techniques can reflect improvements over manual or other processes.
[0032] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.
[0033] Example devices and systems
[0034] Figure 1A block diagram of an example computing system 100 is depicted that can facilitate search recommendations based on a geo-semantic index developed using image content analysis, in accordance with an example embodiment of the present disclosure. System 100 is provided as an example only. Other computing systems including different components may be used in addition to or in place of system 100. System 100 may include one or more user computing devices (such as user computing device 102) and one or more remote computing devices (such as server computing system 120) communicatively coupled via one or more networks (such as communication network 140). Although in Figure 1 Only one user computing device is shown in FIG. 1 , but any number of user computing devices may be connected to server computing system 120 via network 140 .
[0035] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.
[0036] The user computing device 102 includes one or more processors 104 and one or more memories 106. The one or more processors 104 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and can be a processor or multiple processors connected in operation. The one or more memories 106 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The one or more memories 106 can store data 108 and instructions 110, which are executed by the processor 104 to enable the user computing device 102 to perform operations such as described herein. More specifically, as described herein, the user computing device 102 can generate query requests through the communication network 140 and receive response data from the server computing system 120.
[0037] The user computing device 102 may also include one or more input / output interfaces 112. The one or more input / output interfaces 112 may include, for example, devices for receiving information from a user or providing information to a user, such as through a display device, a touch screen, a touch pad, a mouse, data entry keys, an audio output device (such as one or more speakers), a microphone, a camera, a tactile feedback device, etc. The user computing device 102 may also include one or more communication / network interfaces 114 for communicating with one or more systems or devices, including systems or devices located remotely from the user computing device 102.
[0038] The server computing system 120 may include one or more processors 122 and one or more memories 124. The one or more processors 122 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be a processor or multiple processors operatively connected. The one or more memories 124 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The one or more memories 124 may store data 126 and instructions 128, which are executed by the processor 122 to enable the server computing system 120 to perform operations such as those described herein, including supporting image content analysis, generating and / or maintaining geographic semantic indexes, facilitating search recommendations, and the like.
[0039] In some implementations, the server computing system 120 may include or otherwise be implemented by one or more server computing devices. Where the server computing system 120 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.
[0040] In some embodiments, the server computing system 120 can host a geographic information system 130, such as a geographic information system associated with a mapping service. The geographic information system (GIS) 130 can implement a mapping application, a virtual earth application, or any other suitable GIS. GIS130 can support the archiving, retrieval, and manipulation of geospatial data that has been indexed and stored according to geographic coordinates (such as latitude coordinates, longitude coordinates, and altitude coordinates) associated with the geospatial data. GIS130 can combine satellite images, photos, maps, models, and other geographic data with Internet search functions to enable users to view pictures of the planet (e.g., map data associated with multiple geographic regions) and related geographic information (e.g., places such as islands and cities, points of interest such as local restaurants, hospitals, parks, hotels, and schools). GIS130 can further allow users to conduct local searches to obtain directions to a location or directions between two locations, or otherwise retrieve map data for a selected area. The results can be displayed in a two-dimensional or three-dimensional presentation of the area of interest. In some embodiments, the user can pan, tilt, and rotate the view to view three-dimensional terrain and buildings.
[0041] GIS 130 may provide search results, recommendations, and the like (e.g., in response to a user query, etc.) based at least in part on a geo-semantic index as described herein. For example, GIS 130 may include or otherwise be provided access to a geo-semantic index (such as geo-semantic index 134), which may include semantic context associated with a plurality of location units (e.g., features, attributes, activities, etc. associated with each location unit), wherein the geo-semantic index may be populated and / or updated based on image content analysis, manual input, and the like.
[0042] According to one aspect of the present disclosure, the server computing system 120 may include an image content analysis engine 132, which may facilitate implementation of embodiments of the present disclosure. For example, the server computing system 120 may obtain data including a collection of pictures, such as, for example, street-level pictures collected periodically. The image content analysis engine 132 may be assigned a task of periodically processing the obtained picture data to identify semantic elements that reference features and / or attributes of a physical landscape, which features and / or attributes are, for example, associated with one or more human activities and catalogued in a vocabulary of feature / attribute types as described herein. For example, a feature / attribute type may be associated with or indicate a physical object and / or physical landscape such as a structure, infrastructure, building, object, etc. within a picture. For example, features / attributes may include "playground", "park", "alley", "path", "boutique", "loading area", "parking meter", "bench", "fountain", "regulations", "artwork", "tree", "bush", etc. In some embodiments, the data indicating a feature within a picture may include one or more feature vectors indicating one or more aspects of the identified feature.
[0043] According to one aspect of the present disclosure, the server computing system 120 may include a geo-semantic index 134. The geo-semantic index 134 may include a geographic index of locations, the geographic index including rich semantic context information for each location built and maintained based on image content analysis, such as provided by the image content analysis engine 132. As an example, the geo-semantic index may include, for each location unit, data indicating one or more features / attributes (e.g., benches, paths, fountains, murals, parking meters, playgrounds, etc.) identified in a picture associated with the location unit, data related to the collection time (s) of the analyzed image having the identified features / attributes (e.g., timestamps, etc.), one or more activity scores for user activities associated with and / or adapted by the identified features / attributes in the location unit, one or more text descriptions based on the features / attributes and / or activities associated with the location unit, etc.
[0044] For example, an obtained picture associated with a location unit in a geo-semantic index may be processed by an image content analysis system (e.g., image content analysis engine 132) to identify one or more features in the image that are included in a feature vocabulary associated with the geo-semantic index (e.g., a playground, a bench, a fountain, a parking meter, a tree, a path, etc.). One or more features associated with the location unit identified in the picture may be matched with one or more user activities associated with and / or adapted by the feature (e.g., using a feature vocabulary associated with the geo-semantic index). For example, one or more activity scores may be generated for each location unit based on the type of identified features corresponding to the activity associated with the location unit, the number of identified features, the diversity of identified features, the recency of the identified features (e.g., the time since a picture including the feature was collected), etc.
[0045] The geo-semantic index 134 may be used to identify locations within a certain geographic region that may accommodate expected user activity (e.g., based on data obtained from the user computing device 102 associated with the search query, etc.). The server computing device 120 may provide the identified location data to the user computing device 102, for example, as part of a query result recommendation (e.g., a mapped location, a list of locations, etc.).
[0046] Server computing system 120 may also include one or more communication / network interfaces for communicating with one or more systems or devices including systems or devices located remotely from server computing system 120 (e.g., such as user computing device 102). Server computing system 120 may also include one or more input / output interfaces.
[0047] The network 140 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications through the network 140 may be carried via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).
[0048] Example images
[0049] Figure 2 An example street-level image 200 is depicted with features identified using an image content analysis system according to an example embodiment of the present disclosure. Figure 2 As shown, street-level images 200 may be obtained as part of a picture corpus and analyzed by an image content analysis system (e.g., Figure 1 The image content analysis engine 132 of the image processing system 130 processes the street-level image 200 to identify one or more features and / or attributes within the image 200. Specifically, the image content analysis system can identify various features within the image 200 to facilitate the creation and / or updating of a geo-semantic index as described herein. The image 200 can be associated with a location cell within the geo-semantic index, and the identified features / attributes can be used to periodically update data associated with the location cell within the geo-semantic index. In some embodiments, the image content analysis system can identify features within the image 200 and indicate a confidence level for the presence of the identified features, such as Figure 2 The identified features / attributes may be compared to a vocabulary of features / attributes associated with the geo-semantic index (e.g., feature / attribute types associated with user activity types maintained within the geo-semantic index), and the identified features / attributes may be used to facilitate updating of the geo-semantic index (e.g., generating / updating data associated with the location unit, generating / updating activity scores for the location unit associated with the image 200, etc.).
[0050] For example, the obtained picture (e.g., image 200) can be processed by the image content analysis system to identify one or more features / attributes located within the picture that are associated with a vocabulary of a geo-semantic index (e.g., a vocabulary that associates features / attributes with activities to which the features / attributes apply). The one or more features / attributes identified within the picture and one or more user activities associated with and / or to which the features apply can be matched, for example, using a vocabulary associated with the geo-semantic index. One or more activity scores can be generated for each location unit in the geo-semantic index, for example, based on the type of identified features corresponding to activities associated with the location unit, the number of identified features, the diversity of identified features, the recency of identified features (e.g., the time since a picture including the feature was collected), etc.
[0051] Figure 3A and Figure 3B Depicts an example street-level picture according to an example embodiment of the present disclosure. Figure 3A As shown, street-level image 300A may be obtained as part of a picture corpus and may be analyzed by an image content analysis system (e.g., Figure 1 The image content analysis engine 132 processes the street-level image 300A to identify features and / or attributes, such as Figure 3BAs shown in image 300B. Specifically, the image content analysis system can identify various features / attributes within image 300A / 300B, such as identifying that the image is associated with attributes of a public space such as a park, identifying that the image contains features such as water features, fountains, plants, etc. The identified features / attributes can be associated with a vocabulary of the geo-semantic index (e.g., a vocabulary that associates features / attributes with activities to which the features / attributes adapt). Image 300A / 300B can be associated with a location unit within the geo-semantic index, and the identified features / attributes can be used to generate and / or update data associated with the location unit within the geo-semantic index (e.g., generate / update feature data associated with the location unit, generate / update activity scores for the location unit associated with the image, etc.).
[0052] Fig. 6A and Figure 6B Depicts an example street-level picture according to an example embodiment of the present disclosure. Fig. 6A As shown, street-level image 600A may be obtained as part of an image corpus and may be analyzed by an image content analysis system (e.g., Figure 1 The image content analysis engine 132 processes the street-level image 600A to identify features and / or attributes, such as Figure 6B Specifically, the image content analysis system may identify various features within image 600A / 600B, such as identifying that the image is associated with attributes of a public space and / or a playground, identifying that the image contains features identified as playground equipment, fences, etc. Image 600A / 600B may be associated with a location unit within the geo-semantic index, and the identified features / attributes may be used to generate and / or update data associated with the location unit within the geo-semantic index (e.g., generate / update activity scores for location units associated with the image, etc.).
[0053] Example Method
[0054] Figure 4 A flowchart of an example method 400 for providing search recommendations based on a geographic semantic index according to an example embodiment of the present disclosure is depicted. Figure 4 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the methods of the present disclosure are not limited to the order or arrangement specifically shown. The various steps of method 400 may be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure. Method 400 may be performed by one or more computing devices (such as Figure 1 The method may be implemented using one or more of the computing devices depicted in FIG.
[0055] At 402, method 400 may include, for example, Figure 1A computing device of the server computing system 120 obtains a vocabulary of feature types (e.g., a vocabulary that associates features / attributes identified in a picture with activity types to which the features / attributes can adapt). Specifically, the computing device can facilitate, for example, a catalog classification of a vocabulary of known feature and / or attribute types from an image content analysis dictionary, etc., and one or more user activity types to which the feature and / or attribute types can adapt. For example, the vocabulary of feature / attribute types can include "bench" for reading; "alley" for walking; "loading area" or "parking meter" for passengers to pick up or park; "statue", "fountain" or "boutique" for gatherings; etc. The feature / attribute vocabulary can have a many-to-many relationship between activities and feature / attribute terms.
[0056] At 404, method 400 may include obtaining, by a computing device, a corpus of periodically collected images, such as street-level images, aerial images, etc. For example, the computing device may periodically obtain a collection of images, such as from a continuously updated corpus of images, publicly available images, user-generated images, etc.
[0057] At 406, method 400 may include performing, by the computing device, image content analysis on the picture collection based at least in part on the vocabulary of feature types. For example, the computing device may perform image content analysis on the obtained pictures to identify feature / attribute type terms from the generated vocabulary (e.g., identifying semantic elements that reference features and / or attributes of a physical landscape associated with a user activity). In some embodiments, for example, the image content analysis may include using one or more machine learning models to identify one or more features / attributes within images included in the picture collection.
[0058] As an example, the obtained picture can be processed by the image content analysis system to identify one or more features / attributes associated with a vocabulary of a geo-semantic index (e.g., a vocabulary that associates features / attributes with activities to which the features / attributes apply) located within the picture. The one or more features / attributes identified within the picture and one or more user activities associated with and / or to which the features apply can be matched, for example, using the vocabulary associated with the geo-semantic index.
[0059] At 408, method 400 may include populating and / or updating a geo-semantic index of a location unit based on image content analysis of a collection of pictures by a computing device. Specifically, images within a collection of pictures may be identified using specific location units mapped to geographic areas. Based on features / attributes identified via image content analysis of pictures associated with the location units, the computing device may populate and / or update the geo-semantic index of the location unit based in part on a vocabulary of feature types. For example, the geo-semantic index may be populated based on data associated with: a feature / attribute type associated with each location unit, an activity type associated with the feature / attribute in each location unit, and the like. In some embodiments, the geo-semantic index may also include data related to the collection time of pictures having the identified features / attributes. In some embodiments, the image content analysis and / or population of the geo-semantic index may include filtering out certain image data, such as features / attributes identified in private and / or restricted areas, and the like.
[0060] At 410, method 400 may include generating, by the computing device, one or more activity scores for each place cell (e.g., associated with each activity to which the features / attributes identified in the place cell may apply). Specifically, the computing device may generate one or more weighted activity scores for each place cell (e.g., for each activity supported by the vocabulary) based at least in part on the number, diversity, and / or recency of the feature / attribute types corresponding to the activities associated with the place cell.
[0061] In some embodiments, an activity score may be generated based on one or more metric definitions provided by a client application associated with the activity. For example, in some embodiments, some activities in a geo-semantic index and / or vocabulary may be defined based solely on the needs of a specific user application that defines the activity and the metric for the activity score. In some embodiments, the activity score may differ based on the activity type, application type, etc., for example, based on elements that may be important or necessary for the activity.
[0062] As an example, in some embodiments, the geo-semantic index may include, for each location unit, data indicating one or more features / attributes identified in a picture associated with the location unit (e.g., a bench, a path, a fountain, a mural, a parking meter, a playground, etc.), data related to the collection time (s) of the analyzed image having the identified features / attributes (e.g., timestamps, etc.), one or more activity scores for user activities associated with and / or adapted by the identified features / attributes in the location unit, one or more text descriptions based on the features / attributes and / or activities associated with the location unit, and the like.
[0063] At 412 , method 400 may include providing, by a computing device, for example, in response to a user query received from a user computing device regarding a desired activity or the like, a geo-semantic index for generating map recommendations.
[0064] For example, a computing device may obtain a query from a user regarding nearby locations that may be tailored to a particular user activity. The computing device may analyze the geo-semantic index to identify one or more high-scoring location cells nearby (e.g., within a fixed radius of a search area indicated by the user, around the user's current location, etc.), and provide data associated with one or more of those locations that match the desired activity as a search suggestion.
[0065] According to aspects of the present disclosure, image content analysis and updating of geo-semantic indexes may be iteratively performed based on a periodic set of new pictures.
[0066] Figure 5 A flowchart of an example method 500 for providing search recommendations based on a geographic semantic index according to an example embodiment of the present disclosure is depicted. Figure 5 The steps are depicted as being performed in a particular order for purposes of illustration and discussion, but the methods of the present disclosure are not limited to the order or arrangement specifically shown. The various steps of method 500 may be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of the present disclosure. Method 500 may be performed by one or more computing devices (such as Figure 1 The method may be implemented using one or more of the computing devices depicted in FIG.
[0067] At 502, method 500 may include generating, by a server computing device, a vocabulary of feature types. For example, the server computing device may facilitate building a vocabulary of known feature and / or attribute types, such as from an image content analysis dictionary, and one or more user activity types to which the feature and / or attribute types may be adapted. For example, the vocabulary of feature types may include "bench" for reading; "alley" for walking, "loading zone" or "parking meter" for passengers to pick up or park; "statue", "fountain", or "boutique" for gatherings; etc. The feature / attribute vocabulary may have a many-to-many relationship between activities and feature / attribute terms.
[0068] At 504, method 500 may include obtaining, by the server computing device, a corpus of images, such as street-level images, aerial images, etc. For example, the computing device may periodically obtain a collection of images, such as from a continuously updated corpus of images, publicly available images, user-generated images, etc.
[0069] At 506, method 500 may include performing, by the server computing device, image content analysis on the corpus of pictures based at least in part on the vocabulary of feature types. For example, the server computing device may perform image content analysis on the obtained pictures to identify feature / attribute type terms from the generated vocabulary (e.g., identifying semantic elements that reference features and / or attributes of a physical landscape associated with a user activity). For example, the server computing device may determine whether a corresponding picture for each of a plurality of location cells of the geo-semantic index includes a feature / attribute type from the vocabulary.
[0070] At 508, method 500 may include generating and / or updating, by the server computing device, a geo-semantic index for location cells based on image content analysis of images within the picture collection. Specifically, images within the picture collection may be identified using specific location cells mapped to geographic areas. Based on features / attributes identified via image content analysis of each image associated with the location cell, the server computing device may populate and / or update the geo-semantic index for the location cell based in part on a vocabulary of feature types. The server computing device may also generate / update one or more activity scores for each location cell of the geo-semantic index (e.g., associated with each activity to which the features / attributes identified in the location cell may apply).
[0071] At 510, method 500 may include a user computing device obtaining data associated with a query (e.g., from a user, an application, etc.) regarding nearby locations that may be tailored to a particular user activity. For example, a user may submit a search query on a user computing device, such as "find a place where I can take a walk and enjoy some shade." At 512, method 500 may include the user computing device providing data associated with the query to a server computing device.
[0072] At 514, method 500 may include the server computing device analyzing the geo-semantic index to determine one or more locations that may match the desired user activity. For example, the server computing device may analyze the geo-semantic index to identify one or more location cells that are nearby (e.g., within a fixed radius of a search area indicated by the user, around the user's current location, etc.). At 516, method 500 may include the server computing device selecting one or more locations from the identified location cells. For example, the server computing device may select one or more location cells from the identified location cells that have an activity score associated with the desired activity that is above a certain threshold score (e.g., a certain number of location cells from the identified cells that have the highest activity score for the desired activity).
[0073] For example, the server computing device may filter query submissions for any known activity words associated with the vocabulary. For example, the server computing device may identify "taking a walk" and "enjoying the shade" as activity terms in the vocabulary. The server computing device may analyze the index to identify one or more location cells based on image content analysis of the picture that have a high match for these activities based on the presence of certain vocabulary terms associated with these cells. Thus, the identified cell locations may have among their descriptive feature terms terms such as "alley," "tree," "sidewalk," "gravel path," and / or other similar terms. The computing system may select one or more of the identified locations based, for example, in part on the activity scores associated with the identified location cells.
[0074] At 518, method 500 may include the server computing device providing data associated with one or more recommended locations to the user computing device. For example, the server computing device may provide data indicating a map of selected locations, a list of selected locations, etc. In some embodiments, for each location, the server computing system may generate a text description based on features / attributes and / or activities associated with the location. For example, in some embodiments, such descriptions may be generated based on the application of natural language processing, may be curated by a human operator, etc. These location descriptions may be provided to the user as part of the location recommendations to help the user select and / or find a desired location.
[0075] At 520 , method 500 may include the user computing device receiving the recommendation data from the server computing device and providing the recommendation data to the user.
[0076] Additional disclosure
[0077] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems and the actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.
[0078] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided as an illustration rather than a limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, can easily make changes, variations and equivalents to these embodiments. Therefore, it will be readily apparent to those of ordinary skill in the art that the present disclosure does not exclude the inclusion of such modifications, variations and / or additions to the subject matter. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, it is intended that the present disclosure covers such changes, variations and equivalents.
[0079] An example embodiment of the present disclosure is directed to a computer-implemented method, which includes obtaining, by one or more computing devices, a vocabulary of image feature types associated with user activities. The method includes obtaining, by the one or more computing devices, a collection of pictures. The method includes performing, by the one or more computing devices, an image content analysis on the collection of pictures based on the vocabulary of image feature types. The method includes generating, by the one or more computing devices, at least one activity score for each of a plurality of location units in a geo-semantic index based at least in part on the vocabulary of image feature types. The method includes populating, by the one or more computing devices, a geo-semantic index of location units, the geo-semantic index of location units including data indicating at least one activity score for each location unit. The method includes providing, by the one or more computing devices, a geo-semantic index of location units for generating location recommendations in response to a query.
[0080] The method may further include, wherein generating, by the one or more computing devices, the at least one activity score for each location unit includes generating, for each location unit, the at least one activity score based at least in part on a corresponding image feature type identified by the image content analysis. The method may further include associating, by the one or more computing devices, each of a plurality of activity types with a corresponding subset of image feature types, the corresponding subset from a vocabulary of image feature types; wherein performing, by the one or more computing devices, the image content analysis on the set of pictures includes determining whether a corresponding picture of each location unit includes a feature type from the vocabulary of image feature types; wherein generating, by the one or more computing devices, the at least one activity score for each location unit includes determining, for each activity type, an activity score based at least in part on whether such location unit includes the corresponding subset of image feature types for such activity type. The method may further include, wherein a first location unit may include a first activity score associated with a first activity type and a second activity score for a second activity type, and wherein the first activity score is based in part on identifying a first subset of feature types associated with the first location unit, and the second activity score is based in part on identifying a second subset of feature types associated with the first location unit. The method may further include: obtaining, by the one or more computing devices, a query for location recommendations for a desired activity; obtaining, by the one or more computing devices, a geo-semantic index of location units; determining, by the one or more computing devices, one or more location units associated with the query within a geographic area based on activity scores associated with the desired activity from the geo-semantic index of the location units; and providing, by the one or more computing devices, a recommendation including one or more selected locations based on the determined location units. The method may further include: obtaining, by the one or more computing devices, a query for activity recommendations within a defined geographic area; obtaining, by the one or more computing devices, a geo-semantic index of location units; determining, by the one or more computing devices, one or more user activities associated with location units within a defined geographic area based in part on activity scores associated with one or more user activities in the geo-semantic index of the location units; and providing, by the one or more computing devices, a recommendation including the one or more user activities determined in part based on the activity scores and a location within the defined geographic area for each of the one or more user activities. The method may further include generating, by the one or more computing devices, a description for the place cell based in part on the feature types and activities associated with the place cell identified through the image content analysis.The method may further include filtering out, by the one or more computing devices, feature types identified in a restricted geographic location identified by image content analysis so that they are not populated in the geo-semantic index. The method may further include, wherein generating, by the one or more computing devices, an activity score for each location unit in the geo-semantic index based on supported user activities includes generating a weighted activity score for each location unit based at least in part on the number, diversity, and recency of identified image feature types corresponding to activities associated with the location unit. The method may further include, wherein obtaining, by the one or more computing devices, a vocabulary of image feature types associated with user activities includes cataloging a vocabulary of known feature types from an image content analysis dictionary and user activities to which the feature types can adapt. The method may further include, wherein the collection of images includes a periodic collection of street-level images.
[0081] Another example embodiment of the present disclosure is directed to a computing device. The computing device includes one or more processors. The computing device includes one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to perform operations. The operations include obtaining a vocabulary of image feature types associated with user activities. The operations include obtaining a collection of pictures. The operations include performing image content analysis on the collection of pictures based on the vocabulary of image feature types. The operations include generating at least one activity score for each of a plurality of location units in a geo-semantic index based at least in part on the vocabulary of image feature types. The operations include populating a geo-semantic index of location units, the geo-semantic index of location units including data indicating the at least one activity score for each location unit. The operations include providing a geo-semantic index of location units for generating location recommendations in response to a query.
[0082] Another example embodiment of the present disclosure is directed to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations. The operations include obtaining a vocabulary of image feature types associated with user activities. The operations include obtaining a collection of pictures. The operations include performing image content analysis on the collection of pictures based on the vocabulary of image feature types. The operations include generating at least one activity score for each of a plurality of location cells in a geo-semantic index based at least in part on the vocabulary of image feature types. The operations include populating a geo-semantic index of location cells, the geo-semantic index of location cells including data indicating the at least one activity score for each location cell. The operations include providing a geo-semantic index of location cells for generating location recommendations in response to a query.
Claims
1. A computer-implemented method, the method comprising: obtaining, by one or more computing devices, a vocabulary of image feature types associated with user activities; Obtaining, by the one or more computing devices, a collection of pictures; performing, by the one or more computing devices, an image content analysis on the collection of pictures based on a vocabulary of image feature types; generating, by the one or more computing devices, at least one activity score for each of a plurality of locations based at least in part on a vocabulary of image feature types; as well as In response to a query associated with a particular activity, location recommendations are provided by the one or more computing devices based on the activity scores for the plurality of locations.
2. The computer-implemented method of claim 1 , wherein: The plurality of locations are represented by a plurality of place cells in a geo-semantic index of place cells, and the method further comprises: populating, by the one or more computing devices, a geo-semantic index of location cells, the geo-semantic index of location cells comprising data indicative of the at least one activity score for each location cell; and A geo-semantic index of location units is provided by the one or more computing devices for use in generating location recommendations in response to a query.
3. The computer-implemented method of claim 2, wherein: Generating, by the one or more computing devices, the at least one activity score for each location unit comprises: The at least one activity score is generated for each location cell based at least in part on a corresponding image feature type identified through the image content analysis.
4. The computer-implemented method of claim 2, wherein: The method further comprises: associating, by the one or more computing devices, each activity type of the plurality of activity types with a corresponding subset of image feature types from a vocabulary of image feature types; wherein performing, by the one or more computing devices, image content analysis on the picture set comprises determining whether a corresponding picture of each position unit includes a feature type from a vocabulary of image feature types; and Wherein generating, by the one or more computing devices, the at least one activity score for each location cell comprises determining an activity score for each activity type based at least in part on whether such location cell includes a respective subset of image feature types for such activity type.
5. The computer-implemented method of claim 2, wherein: The first location unit includes a first activity score associated with a first activity type and a second activity score for a second activity type, and wherein the first activity score is based at least in part on a first subset of identification feature types associated with the first location unit and the second activity score is based at least in part on a second subset of identification feature types associated with the first location unit.
6. The computer-implemented method of claim 2, wherein: The method further comprises: obtaining, by the one or more computing devices, a query for location recommendations for a desired activity; Obtaining, by the one or more computing devices, a geo-semantic index of the location unit; determining, by the one or more computing devices, one or more location cells within a geographic area associated with the query based on activity scores associated with desired activities from a geo-semantic index of the location cells; and Recommendations are provided, by the one or more computing devices, based on location cells within the geographic area associated with the query, including one or more selected locations.
7. The computer-implemented method of claim 2, wherein: The method further comprises: obtaining, by the one or more computing devices, a query for activity recommendations within a defined geographic area; Obtaining, by the one or more computing devices, a geo-semantic index of the location unit; determining, by the one or more computing devices, one or more user activities associated with a location unit within a defined geographic area based in part on activity scores associated with the one or more user activities in a geo-semantic index of the location unit; and A recommendation is provided, by the one or more computing devices, the recommendation including the one or more user activities determined based in part on the activity score and a location within a defined geographic area for each of the one or more user activities.
8. The computer-implemented method of claim 2, wherein: The method further comprises: A description for the place cell is generated by the one or more computing devices based in part on the feature types and activities associated with the place cell identified through the content image analysis.
9. The computer-implemented method of claim 2, wherein: The method further comprises: The one or more computing devices filter out feature types identified in a restricted geographic location through image content analysis, so that the feature types are not populated in the geographic semantic index.
10. The computer-implemented method of claim 1, wherein: Generating, by the one or more computing devices, an activity score for each location based on the supported user activity includes generating a weighted activity score for each location based at least in part on a quantity, diversity, and recency of identified image feature types corresponding to activity associated with the location.
11. The computer-implemented method of claim 1 , wherein: Obtaining, by the one or more computing devices, a vocabulary of image feature types associated with user activities includes cataloguing a vocabulary of known feature types from an image content analysis dictionary and user activities to which the known feature types are adaptable.
12. The computer-implemented method of claim 1, wherein: The picture collection includes a periodic collection of street-level pictures.
13. A computing system comprising: one or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause a computing system to perform operations comprising: Obtaining a vocabulary of image feature types associated with user activities; Get a collection of images; performing image content analysis on a collection of images based on a vocabulary of image feature types; generating at least one activity score for each of the plurality of locations based at least in part on a vocabulary of image feature types; and In response to a query associated with a particular activity, location recommendations are provided based on the activity scores for the plurality of locations.
14. The computing system of claim 13, wherein: The plurality of locations are represented by a plurality of place cells in a geo-semantic index of place cells, and the operations further include: populating a geo-semantic index of location cells, the geo-semantic index of location cells comprising data indicative of the at least one activity score for each location cell; and A geo-semantic index of location units is provided for use in generating location recommendations in response to a query.
15. The computing system of claim 14, wherein: Generating the at least one activity score for each location cell includes generating the at least one activity score for each location cell based at least in part on a corresponding image feature type identified through image content analysis.
16. The computing system of claim 14, the operations further comprising: A query to obtain location recommendations for a desired activity; Obtaining the geographic semantic index of the location unit; determining one or more location cells within the geographic area associated with the query based on activity scores associated with the desired activity in the geo-semantic index of the location cells; as well as Recommendations are provided that include one or more selected locations based on location cells within a geographic area associated with the query.
17. The computing system of claim 14, the operations further comprising: A description for the place cell is generated based in part on the feature types and activities associated with the place cell identified through the content image analysis.
18. The computing system of claim 14, wherein: Generating an activity score for each place cell in the geo-semantic index based on the supported user activities includes generating a weighted activity score for each place cell based at least in part on the number, diversity, and recency of semantic terms corresponding to activities associated with the place cell.
19. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing system, cause the computing system to: Obtaining a vocabulary of image feature types associated with user activities; Get the image collection; performing image content analysis on a collection of images based on a vocabulary of image feature types; generating at least one activity score for each of the plurality of locations based at least in part on a vocabulary of image feature types; as well as In response to a query associated with a particular activity, location recommendations are provided based on the activity scores for the plurality of locations.
20. The one or more non-transitory computer-readable media of claim 19, wherein: The picture collection includes a periodic collection of street-level pictures.
