Shape-based graph search

Abstract

A method for shape-based graphical search is described. Each graphical object in a set of graphical objects is analyzed. The analysis includes determining an outline of the graphical object from graphical data forming the graphical object. The outline of the graphical object is sampled to obtain sampling points that capture the outline of the graphical object. A shape descriptor of the graphical object is determined that captures local and global geometric characteristics of the sampling points. Search results for a search query are determined based on a comparison between a shape descriptor of a graphical object identified by a user and a shape descriptor of at least one graphical object in the set of graphical objects. At least one of the search results may be presented on a user device associated with the search query.

Description

Background Art

[0001] Example-based image search techniques allow users to provide an example image and receive search results that include visually similar images. In some traditional approaches, a large number of raster images are labeled using neural networks based on whether a particular image depicts a particular type of object or displays a particular feature. Search results can be determined based on how similar the image label is to the example image label. This approach can be suitable for finding images that contain similar themes. However, a user may be interested in search results that contain shapes similar to the example image.

[0002] Traditional shape-based image search is optimized for photos and other detailed graphics by utilizing shading and pixel color. However, many vector graphics and some raster graphics (e.g., sketches) lack detail, which can lead to inaccurate search results. For example, these graphics may lack internal details and be strongly characterized by overall footprints that are not well captured using traditional methods. Histogram of oriented gradients is an example of a traditional method for capturing shapes in an image, which is more suitable for detailed images. All pixels of an image are divided into a cell grid, and a histogram of gradient directions is created for each cell from its pixels. These gradient directions may not be sufficient to produce accurate shape-based search results for less detailed graphics. Summary of the invention

[0003] Aspects of the present disclosure provide improvements to a computer's ability to identify and compare shapes of graphical objects to produce more accurate graphical search results. Specifically, the present disclosure provides shape descriptors of graphical objects that form a detailed representation of the overall footprint of the object, thereby allowing more accurate shape-based graphical searches.

[0004] In various implementations, shape descriptors of graphical objects are determined and stored in a data repository. A user may provide an example graphical object for a search query (e.g., by drawing a vector outline), and the shape descriptor of the example object may be compared with the stored shape descriptors to determine graphical objects of similar shapes for search results. The shape descriptor of a graphical object may be determined by determining an outline of the graphical object (e.g., a set of connected Bezier curves). The outline is sampled to generate sampling points that capture the outline. For each sampling point, local and global geometric properties of the sampling point may be determined. As used herein, local and global geometric properties of a sampling point of a graphical object refer to one or more values ​​representing a geometric relationship between the sampling point relative to other sampling points of the graphical object (e.g., by referencing the positional geometry of other sampling points to the sampling point). The local and global geometric properties of each sampling point may be stored in a corresponding feature vector. The feature vectors of the sampling points may be combined into one or more aggregate feature vectors that form a shape descriptor.

[0005] In some cases, any suitable method may be used to label the graphical objects, depending on whether the graphical objects depict a particular type of object or show particular features (e.g., using machine learning and / or manual labeling). When determining search results, graphical objects having shapes similar to the example objects may be determined by analyzing their shape descriptors. Other graphical objects may be included in the search results based on sharing a label that is similar or identical to the determined graphical object. Using this approach, shape descriptors do not need to be determined and / or analyzed for at least some of these other graphical objects, thereby reducing processing requirements for generating search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present invention is described in detail below with reference to the accompanying drawings, in which:

[0007] Figure 1 is a block diagram illustrating an example of an operating environment according to an embodiment of the present disclosure;

[0008] Figure 2 A block diagram of a graphical search system according to an embodiment of the present disclosure is shown;

[0009] FIG3 shows a block diagram of a graphical user interface according to an embodiment of the present disclosure;

[0010] Figure 4A shows a representation of an outline of a graphical object according to an embodiment of the present disclosure;

[0011] Figure 4B shows point sampling of the outline of a graphic object according to an embodiment of the present disclosure;

[0012] Figure 5 is a flowchart illustrating a method for determining a shape descriptor of a graphic object according to an embodiment of the present disclosure;

[0013] Figure 6 is a flowchart illustrating another method for searching an image based on a shape descriptor according to an embodiment of the present disclosure; and

[0014] Figure 7 is a block diagram of an exemplary computing environment suitable for implementing embodiments of the present disclosure. DETAILED DESCRIPTION

[0015] The subject matter of the present invention is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this patent. Rather, the inventors have contemplated that the claimed subject matter may also be embodied in other ways in conjunction with other current or future technologies to include different steps or combinations of steps similar to the steps described in this document. In addition, although the terms "step" and / or "box" may be used herein to imply different elements of the method employed, unless and except when the order of the individual steps is explicitly described, these terms should not be interpreted as implying any particular order among or between the various steps disclosed herein.

[0016] Traditional shape-based image search is optimized for photos and other detailed graphics by utilizing the shading and color of pixels. For example, photos can capture important details of all visible areas of real-world objects, which can be used for shape recognition. Many vector graphics and some raster graphics lack similar information, which may lead to inaccurate search results. For example, when a user sketches (e.g., hand-drawing) an object, it may be strongly characterized by an overall footprint that cannot be well captured using traditional methods, resulting in lower quality search results. To illustrate the above, the sketched object may not include too many colors, shading and / or other details, so that traditional methods may not be enough to capture the object for search. However, it may be desirable to allow users to quickly sketch example objects and retrieve similar but more detailed search results.

[0017] The Histogram of Oriented Gradients is an example of a traditional method for capturing shapes in an image. All pixels of the image are divided into a grid of cells, and a histogram of gradient directions is created for each cell from its pixels. When the graphics lack sufficient detail, these gradient directions may not capture the overall shape of many vector graphics and some raster graphics well enough to produce accurate search results.

[0018] Aspects of the present disclosure provide improvements to the ability of a computer to identify and compare the shapes of graphical objects to produce more accurate graphical search results. Specifically, the present disclosure provides shape descriptors of graphical objects that form a detailed representation of the overall footprint of the object, allowing for more accurate shape-based graphical searches. The methods described herein can utilize vector art representations of the outlines of graphical objects (e.g., connected Bezier curves and / or other vector primitives) to calculate shape descriptors of graphical objects. Shape descriptors can be used to search for similar vector graphics that will be stored in vector files or other data objects.

[0019] In various implementations, shape descriptors of graphical objects are determined and stored in a database (or other suitable data repository). A user may provide an example graphical object for a search query, such as by drawing a vector sketch on a user device. The user may initiate a search for similar graphical objects on the user device, where a shape descriptor of the example object may optionally be generated or otherwise determined. The shape descriptor of the example object may be compared to stored shape descriptors to determine a graphical object of similar shape to the search result.

[0020] A shape descriptor of a graphical object may be determined by determining an outline of the graphical object. An outline may be a vector object formed by a collection of vector primitives such as connected Bezier curves. The outline may be sampled to generate sampling points, which capture the outline using a sampling technique such as adaptive point sampling, where a number of sampling points for a segment of the outline is based on the length of the segment. Local and global geometric properties of the sampling points may be determined and may be stored in corresponding feature vectors. The shape descriptor includes feature vectors of the sampling points, which may be combined into one or more aggregate feature vectors.

[0021] The feature vector of the sampling point can be determined using a variety of possible methods. These include capturing local and global geometric characteristics of the sampling point in the spatial and / or spectral domain. In an exemplary spatial method, a spatial histogram is created for each sampling point that captures the relative position of other sampling points. For example, a geometric object (e.g., a circle) can be centered at the sampling point. The geometric object can be divided into equal sectors, which can be further divided into multiple partitions (bins). The feature vector of the sampling point can include a value for each partition, which is a count of the number of other sampling points that fall within the partition. Other suitable methods for capturing local and global geometric characteristics include determining the thermal kernel signature of the sampling point, skeletonization, etc.

[0022] Then, the shape descriptor of the graphical object (e.g., the example object) can be compared with similarly determined shape descriptors of other graphical objects to generate search results. As described in further detail later, in some cases, other graphical objects are marked to facilitate the determination of search results. Depending on whether the graphical object depicts a particular type of object or shows a particular feature (e.g., using machine learning and / or manual marking), any suitable method can be used to mark the graphical object. When determining the search results, a graphical object having a similar shape to the example object can be determined by comparing its shape descriptor with the shape descriptor of the example object. Based on sharing a tag that is similar or identical to the determined graphical object, the additional graphical object can be included in the search results.

[0023] In some cases, shape descriptors do not need to be determined and / or analyzed for at least some of these other graphical objects, thereby reducing processing requirements for generating search results. For example, the tags can be determined using a conventional method of assigning confidence scores to each tag and object or image pair. A subset of items (e.g., one or more graphical objects or images) can be selected for each tag based on the confidence scores of the tags (e.g., the top ten scored items for each tag). Shape descriptors can then be determined for this subset of items, and other items can be included in the search results with items from the subset based on having the same or similar tags as the item.

[0024] Now go to Figure 1 , a block diagram showing an example of an operating environment in which some implementations of the present disclosure may be adopted is provided. It should be understood that this and other arrangements described herein are set forth only as examples. In addition to or in place of those arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, sequences, and functional groupings, etc.) may be used, and for clarity, some elements may be omitted completely. In addition, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or combined with other components, and implemented in any suitable combination and position. The various functions described herein as being performed by one or more entities may be performed by hardware, firmware, and / or software. For example, some functions may be performed by a processor executing instructions stored in a memory.

[0025] Among other components not shown, operating environment 100 includes a plurality of user devices, such as user devices 102A and 102B to 102N, network 104, and server 108. It should be understood that Figure 1 The operating environment 100 shown in FIG. 1 is an example of a suitable operating environment. For example, Figure 1 Each component shown in the figure can be implemented via any type of computing device, such as a combination of Figure 7One or more of the computing devices 700 described. These components can communicate with each other via a network 104, which can be wired, wireless, or both wired and wireless. The network 104 may include multiple networks or one network in a network, but is shown in a simple form to avoid confusing the various aspects of the present disclosure. As an example, the network 104 may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet, and / or one or more private networks. In the case where the network 104 includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) may provide wireless connections. Network environments are common in offices, enterprise-wide computer networks, intranets, and the Internet. Therefore, the network 104 is not described in detail.

[0026] It should be understood that within the scope of the present disclosure, any number of user devices, servers, and other disclosed components may be employed within operating environment 100. Each may include a single device or multiple devices cooperating in a distributed environment.

[0027] User devices 102A to 102N include any type of computing device capable of being operated by a user. For example, in some implementations, user devices 102A to 102N are Figure 7 Type of computing device described. By way of example and not limitation, a user device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a global positioning system (GPS) or device, a video player, a handheld communication device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, any combination of these depicted devices, or any other suitable device.

[0028] The user equipment may include one or more processors and one or more computer-readable media. The computer-readable media may include computer-readable instructions that may be executed by the one or more processors. These instructions may be provided by, for example, Figure 1 For simplicity, application 110 is referred to as a single application, but in practice its functionality may be implemented by one or more applications. As described above, other user devices may include one or more applications similar to application 110.

[0029] The application may generally be any application capable of facilitating the exchange of information between a user device and the server 108 when performing aspects of the present disclosure. In some implementations, the application comprises a web application, which may be run in a web browser and may be hosted at least in part on the server side of the operating environment 100. Additionally or alternatively, the application may comprise a dedicated application, such as media editing software, examples of which include Etc. In some cases, one or more applications represented by applications 110 are integrated into an operating system (eg, as a service). Thus, it is contemplated herein that "application" is to be interpreted broadly.

[0030] The server 108 also includes one or more processors, and one or more computer-readable media. The computer-readable media includes computer-readable instructions that can be executed by one or more processors. These instructions can optionally implement one or more components of the graphic search system 106, which will be further described in detail below. However, the components can generally be implemented on any combination of the server 108, the user device 102A and / or the application 110. In addition, other components not shown in the figure can also be included in the distributed environment.

[0031] Therefore, it should be understood that the graph search system 106 can be provided via multiple devices arranged in a distributed environment, which together provide the functionality described herein, or can be implemented on a single device. In some cases, the graph search system 106 can be at least partially integrated into a user device such as user device 102A. In addition, the graph search system 106 can be implemented at least in part as a cloud computing service.

[0032] For cloud-based implementations, instructions on server 108 may implement one or more components of graph search system 106, and application 110 may be used by a user to interface with the functionality implemented on server 108. As described above, in some cases, application 110 includes a web browser. In other cases, server 108 may not be necessary. For example, components of graph search system 106 may be implemented entirely on a user device such as user device 102A. Thus, graph search system 106 may be implemented at least in part by instructions corresponding to application 110.

[0033] The graphic search system 106 may determine a ranking score (e.g., ranking score 244) for one or more graphic items including a graphic object (such as an image including a graphic object and / or a graphic data object of an individual graphic object). The graphic search system 106 may determine these ranking scores based on determining how similar the shape descriptors of the graphic objects are to the shape descriptors of the example graphic objects corresponding to the search query. Based on the ranking scores, the graphic search system 106 may determine which graphic objects to include in the search results for the search query. In some embodiments, from a user device (e.g., a user device 102A using an application 110), a user may initiate a search, such as by outlining or otherwise identifying an example graphic object on the user device and optionally selecting a search button. The search results are determined by the server 108 and returned to the user device for display. The user may then optionally select one or more search results to import the corresponding graphic objects into the application 110 (e.g., for editing or other graphic composition).

[0034] Figure 2 A block diagram of a graphical search system according to an embodiment of the present disclosure is shown. Figure 1 The graphic search system 206 of the graphic search system 106 includes an interface manager 212 , a contour determiner 216 , a contour sampler 218 , a shape descriptor determiner 220 , a rendering manager 222 , a search engine 224 , a data object processor 228 , and a storage device 230 .

[0035] The storage device 230 may include a computer-readable medium and is configured to store computer instructions (e.g., software program instructions, routines, or services), data, and / or models used in the embodiments described herein. In some implementations, the storage device 230 stores information or data received via various components of the graph search system 206 and / or generated by various components of the graph search system 206, and provides access to the information or data to various components as needed. In an implementation, the storage device 230 includes a data repository (or a computer data storage device). Although depicted as a single component, the storage device 230 may be implemented as one or more data repositories (e.g., a database) and may be at least partially in the cloud. Some information such as data objects may be stored on a user device (e.g., user device 102A), and other information may be stored externally (e.g., on server 108). Therefore, it should be understood that the information in the storage device 230 may be distributed across one or more data repositories for storage (which may be externally hosted) in any suitable manner.

[0036] Storage device 230 includes, for example, search results 232, data objects 234, graphic objects 238, shape descriptors 240, tags 242, and ranking scores 244, among other potential information. In this example, each data object 234 includes a digital image, but any number of images may be included. As an example, each data object 234 may be in an image file format (e.g., a bitmap and / or vector file format) or a video file format. For example, each data object 234 may be in a vector image format, such as scalable vector graphics (SVG), Adobe Illustrator (AI), Encapsulated PostScript (EPS), etc.

[0037] In some embodiments, the graphic search system 206 includes a data object processor 228 to extract a graphic object 238 from the graphic data of the data object 234. As used herein, a data object refers to a storage structure for one or more graphic objects. A graphic object refers to a predetermined set of graphic data (e.g., vector graphic data and / or raster graphic data) that collectively represent an image (e.g., geometry, shading, etc.).

[0038] As an overview, the interface manager 212 is configured to provide an interface between the graphical search system 206 and other entities such as user devices, users, and / or other systems. The interface manager 212 can receive data indicating an example graphical object for use by a search engine 224 in a search query to search for graphical objects similar to the example object. The interface manager 212 can receive this data from one or more user devices, such as in response to a user selection of an example image for an image-based search. In response to the search query, the search engine 224 can search for graphical objects 238 having shape descriptors similar to the example graphical objects to include in the search results 232. The presentation manager 222 is configured to cause presentation of information on one or more user devices based on the search results 232.

[0039] The shape descriptor determiner 220 is configured to determine shape descriptors 240 of the graphical object 238 and the example graphical object at any suitable time. The shape descriptor determiner 220 may determine the outline of each of the graphical object 238 and the example graphical object using the outline determiner 216. The shape descriptor determiner 220 may further use the outline sampler 218 to determine the point sampling that defines the outline. Based on the point sampling of each graphical object, the shape descriptor determiner 220 may determine the shape descriptor of the graphical object.

[0040] As described above, the interface manager 212 is configured to provide an interface between the graphic search system 206 and other entities such as user devices, users and / or other systems. The interface manager 212 can receive data indicating an example graphic object for use by the search engine 224 in a search query to search for a graphic object similar to the example object. For example, data can be generated based on user input to a user device such as the user device 102A (e.g., via the application 110). For example, the user input can correspond to a user selection of an option for searching for a graphic object similar to the example object (e.g., a search button). In some embodiments, the user identifies the example graphic object for search to the interface manager 212. This can include providing a user input that results in a sketch or drawing of the example graphic object. For example, the user can construct the graphic object by free hand drawing and / or using object primitives. As another example, the user can select a data object containing the graphic object to identify the graphic object. For example, the user can select the data object from a file browser or other interface that provides multiple options for the example object.

[0041] According to the implementation of the present disclosure, Figure 3A and Figure 3B One such example described above illustrates an example of a graphical interface. The interface manager 212 may process information about Figure 3A and 3B Any combination of the various interfaces described between users, user devices, and / or servers. Figure 3A A screen 300A showing a graphical interface, and Figure 3B Screen 300B of a graphical interface is shown. As shown, the graphical interface includes a drawing board 350, which is configured to display graphical objects. It also includes multiple tools 352 in a toolbar that a user can use to compose graphical objects. For example, screen 300A represents an intermediate version 354A of a graphical object 354 of screen 300B defined by user input. For example, using a freehand vector drawing tool 352A, a user has provided user input that defines an intermediate version 354A of a graphical object 354. This can be accomplished by manipulating a cursor 356 to define a shape through user input. As an example, user input can come from a finger, a mouse or a stylus or other input object and / or device. Graphical object 354 can include vector data and can be formed by multiple Bezier curves. For example, Bezier curves and / or other vector primitives can be interconnected to define a graphical object.

[0042] After completing the graphical object 354, the user may provide a user selection of option 358, which instructs the graphical search system 206 to search for similar graphical objects using the identified graphical object 354. In this example, option 358 is a search button, but other graphical interface elements (e.g., menus) may be employed. In some cases, the search may be automatically initiated when the graphical object 354 is completed (e.g., by the interface manager 212 automatically detecting the completion of the graphical object 354).

[0043] Selection of option 358 causes the transmission of a corresponding search query from the user device to a search engine 224, which may be located on a server (e.g., server 108). The search query may identify the graphical object 354 as an example object. For example, the search query may include graphical data defining the graphical object 354. Additionally or alternatively, the search query may include a shape descriptor of the graphical object 354. For example, the shape descriptor determiner 220, the contour determiner 216, and / or the contour sampler 218 may be implemented at least in part on the user device to determine a shape descriptor of an example graphical object on the client side of the operating environment 100. As another example, the shape descriptor may be determined server-side. In other embodiments, the search query may include an identifier of the graphical object, such as in the case where the graphical object is a predefined object selected by the user for search (e.g., one of the graphical objects 238).

[0044] As described above, in response to the search query, the search engine 224 may search for a graphic object 238 having a shape descriptor similar to the example graphic object to include in the search results 232. The presentation manager 222 is configured to cause presentation of information on one or more user devices based on the search results 232. This may be accomplished by the presentation manager 222 sending data that causes the presentation. For example, where the presentation manager 222 is located on a user device such as the user device 102A (e.g., as part of the application 110), the presentation manager 222 may communicate the data to a service on the device to cause the presentation. Where the presentation manager 222 is at least partially located on a server such as the server 108, the data may be transmitted over the network 104. Examples of the data may include, for example, any combination of at least one of the following: a graphic object, a data object containing a graphic object, a ranking score, a preview image, an order in which the graphic object is presented, an API call to present the object, and an operating system call to present the preview image.

[0045] As an example, the transmission of data may result in Figure 3B. In addition, rather than displaying multiple images for a particular image effect, in some cases, the highest ranking graphic objects and / or data objects may be indicated by presentation. The sorting order of the graphic objects of the example graphic objects may be indicated by presentation (e.g., the most similar to the image may be sorted first, and the least suitable may be sorted last, as indicated by the sorting score).

[0046] exist Figure 3B In the embodiment of the present invention, for example, presentation manager 222 has caused presentation of search results 360 in search results area 362 of the graphical user interface. As an example, search results 360 include graphical objects presented as preview images 360A, 360B, and 360C of their corresponding data objects. For example, preview image 360B corresponds to multiple graphical objects in graphical objects 238, but may be included in search results 360 based on the similarity of graphical object 364 to graphical object 354. In other implementations, the preview image may represent only graphical object 364, rather than each graphical object in the data object.

[0047] In various embodiments, one or more graphical objects represented in the search results may be user-selectable to import the corresponding graphical objects into the application 110 (e.g., for editing or other graphical composition). For example, the interface manager 212 receives a user input corresponding to a user selection of one or more of the graphical objects represented in the search results area 362. The user input may be provided to the preview image 360A, for example, to select the corresponding graphical object, for example, for import into the artboard 350 or a new artboard. In some implementations, the user may drag and drop the preview image from the search results area 362 into the artboard 350 via the user input to cause the graphical object to be imported. Optionally, this may automatically cause the replacement of the graphical object 354 with the imported graphical object. The user may use the tool 352 in some embodiments to edit the imported graphical object.

[0048] The data object processor 228 may be used to identify graphical objects within the data objects so that search descriptors may be identified for those graphical objects and those graphical objects may be searched for. For example, the data object processor 228 may identify multiple discrete graphical objects (e.g., a large, tree, and a rabbit) within a single vector art file and process those graphical objects to make them individually searchable.

[0049] As described above, in some embodiments, the graphical search system 206 includes a data object processor 228 that extracts graphical objects 238 from the graphical data of the data objects 234. For example, each of the data objects 234 includes graphical data, such as graphical data 236. In some embodiments, the data object processor 228 can identify multiple graphical objects from a single data object (e.g., a file) and extract and store these graphical objects separately as graphical objects 238. Each of the graphical objects 238 can include a pointer or other identifier of the corresponding graphical data extracted from the data object and stored as a new data object and / or a corresponding portion of the graphical data in the data object. Using this approach, as opposed to treating the data objects as single graphical objects, the graphical search system 206 can search individually for graphical objects similar to the example objects represented within the same data object. For example, assume that the graphical data 236 includes a natural scene with a bird, a fox, a sun, and a fish. Using the methods described herein, the fish can be individually matched to the graphical object 354 and included in the search results 360 as an individual graphical object and / or an original data object.

[0050] The data object processor 228 can use various methods to identify the graphical objects in the data object for extraction. In some implementations, each graphical object can be identified in the data object data based on the description of the graphical object in the data object. For example, a vector file such as an SVG file allows the definition of groups of graphical elements (e.g., vector elements). The data object processor 228 can identify each group of graphical elements as a corresponding graphical object. It should be noted that a group can have multiple sub-groups that also describe lower-level objects that create a complete graphical object. For example, a group can be part of a tree with nodes that include groups and / or vector elements. The data object processor 228 can analyze the groups in the data object to determine which groups correspond to the graphical objects to be extracted and / or can select groups based on their hierarchy as graphical objects (e.g., selecting each top-level group).

[0051] It should be noted that in some implementations, such as where data object 234 corresponds to a bitmap image (e.g., a Joint Photographic Experts Group (JPEG) file), at least some of graphics data 236 may include raster data. In these cases, data object processor 228 may use any of a variety of object recognition algorithms to identify sub-portions of the bitmap image as graphics objects (e.g., using a computer vision algorithm for object recognition). Therefore, it should be understood that in some embodiments, one or more of graphics objects 238 may be formed of or include raster graphics data in addition to or in lieu of vector graphics data. However, in Figure 3A and 3B In the example, each graphics object is represented using vector graphics data.

[0052] As described above, shape descriptor determiner 220 is configured to determine shape descriptors 240 of graphical object 238 and example graphical objects. For example, this may occur when graphical object 238 is extracted from data object 234 (e.g., simultaneously) and / or after graphical object 238 is extracted from data object 234.

[0053] The shape descriptor determiner 220 may use the contour determiner 216 to determine the contour of each of the graphic objects 238. Various methods may be used to identify the contour. One method uses Canny edge detection, which calculates the magnitude and angle of the gradient vector of the graphic object. The contour may include a continuous representation of the footprint of the graphic object without internal details. As an example, the contour may be formed by a plurality of Bezier curves that may be interconnected. In various implementations, the contour is a vector contour represented using vector elements. For example, in the case where the graphic object is a raster object, the contour generated by the contour determiner 216 may be a vector contour. Similarly, in the case where the graphic object is a vector object, the contour generated by the contour determiner 216 may be a vector contour.

[0054] Figure 4A 4. A representation of the outline of a graphic object according to an embodiment of the present disclosure is shown. Specifically, an outline 406 of a graphic object 404 can be generated by an outline determiner 216. The graphic object 404 is a vector object formed by vector elements. The outline determiner 216 can generate the outline 406 by analyzing the vector elements and / or other graphic data defining the graphic object 404. The outline 406 is formed by a plurality of interconnected curves.

[0055] The shape descriptor determiner 220 may further use the contour sampler 218 to determine the point samples that define the contour. Figure 4B Point sampling of the outline of a graphical object according to an embodiment of the present disclosure is shown. For example, given a contour determined using the contour determiner 216, the shape descriptor determiner 220 can analyze the contour to determine point samples 408A, 408B, and 408N. As another example, the contour sampler 218 can determine the point sampling when the contour is generated. The contour sampler 218 can determine the point sampling using a point sampling algorithm that captures the contour of the graphical object as multiple points. In various implementations, the contour sampler 218 determines the point sampling based on the curvature of the contour. For example, when sampling the contour, the point density of the sampling points can be based on the curvature of the contour. As shown in FIG. Figure 4BAs shown, the contour sampler 218 can provide a lower point density for a segment of a contour having a higher curvature than for a segment of a contour having a lower curvature. Additionally or alternatively, the contour sampler 218 can determine the number of points for a particular segment based on the length of the segment. For example, the contour sampler 218 can sample more points for a segment of a longer contour than for a segment of a shorter contour. As shown, segment 410A includes eleven point samples, while segment 410B is shorter and therefore includes fewer point samples (e.g., four point samples). A suitable algorithm for determining the sampling points is adaptive point sampling.

[0056] Based on the point sampling of each graphics object, the shape descriptor determiner 220 can determine a shape descriptor of the graphics object. The shape descriptor determiner 220 can determine local and global geometric characteristics of each sampling point, which can be stored as a corresponding feature vector for that particular sampling point. The shape descriptor is based on the feature vectors of the sampling points of the graphics object, which can be determined by combining the feature vectors into one or more aggregate feature vectors that constitute the shape descriptor.

[0057] The shape descriptor determiner 220 may determine the feature vector of the sampling point using a variety of possible methods. These include capturing local and global geometric properties of the sampling point in the spatial and / or spectral domains. As described above, the local and global geometric properties of a sampling point of a graphical object refer to one or more values ​​that represent the geometric relationship of the sampling point relative to other sampling points of the graphical object (e.g., by referencing the positional geometry of the other sampling points to the sampling point). For example, the one or more values ​​of the sampling point may be based on the position of the sampling point relative to other sampling points (e.g., each other sampling point) in the graphical object. The one or more values ​​may each be based on an angle between the sampling point and another sampling point, the proximity of the sampling points, and / or the distance between the sampling points (e.g., directly or along the contour of the graphical object).

[0058] In an example spatial method, a spatial histogram is created for each sampling point that captures the relative positions of other sampling points. For example, a geometric object (e.g., a circle) can be centered at a sampling point. The geometric object can be divided into equal sectors, which can be further divided into multiple partitions. The feature vector of the sampling point can include a value for each partition, which is a count of the number of other sampling points that fall within the partition.

[0059] Figure 4BAn example of this method is illustrated. As shown, the geometry object 412 is centered on the sampling point 408A. The sampling points of the geometry object 412 and / or the contour 406 can be scaled or resized so that no matter which sampling point the geometry object 412 is centered on, all sampling points will fall into the geometry object 412. The geometry object 412 includes equal sectors, such as sector 414, and the sector is formed by multiple partitions. For example, sector 414 is formed by partitions 414A, 414B, 414C and 414D. In some embodiments, the geometry object 412 includes twelve sectors, each containing five partitions, for a total of sixty partitions. In the example shown, there are eight sectors, each containing four partitions, to more clearly describe the geometry object. The geometry object 412 is centered on the sampling point 408A, and the feature vector of the sampling point 408A can include the value of each partition, in this example, the value is a count of the number of other sampling points falling into the partition. For example, sample point 408D is located in partition 422 along with three other sample points for a total count of three. Therefore, partition 422 may have a value of four in the feature vector for sample point 408A. As shown, twenty-six partitions do not contain sample points, so that the feature vector may include twenty magnitude zeros, one magnitude zero for each partition that does not contain a sample point. The feature vector may be normalized so that each value varies from 0 to 1.

[0060] The shape descriptor determiner 220 can similarly determine the feature vector of each other sampling point. The shape descriptor determiner 220 can combine those feature vectors to form a shape descriptor of the graphical object. In some embodiments, the shape descriptor includes at least two aggregate feature vectors. The shape descriptor determiner 220 can calculate an aggregate feature vector that contains the standard deviation of each value in the sampling point feature vector. The shape descriptor determiner 220 can calculate another aggregate feature vector that contains the mean of each value in the sampling point feature vector. This is just an example, and any number of feature vectors can be used as a shape descriptor (or another form of aggregation metric). By combining the feature vectors of the sampling points, the shape descriptor 240 can consume lower storage space and can reduce the processing power required by the search engine 224 for comparing shape descriptors.

[0061] Thus, for each graphical object, the shape descriptor determiner 220 may determine, in some embodiments, two feature vectors that form a shape descriptor of the graphical object. The search engine 224 may search for graphical objects having shape descriptors similar to the example graphical object for inclusion in the search results. Specifically, the shape descriptor may be used by the search engine 224 as at least one factor for evaluating the similarity between the graphical objects. The search engine 224 may determine the similarity between a given graphical object and the example object based on a comparison between the shape descriptors of the graphical objects. For example, the more similar the shape descriptors are, the more similar the overall shapes of the graphical objects are, thereby increasing the likelihood of including the graphical objects as search results. In various embodiments, the search engine 224 determines the similarity between the shape descriptors by determining the cosine similarity between the feature vectors, which may result in a similarity score for each graphical object. The similarity score may be used to rank the graphical objects, and the graphical objects (and / or their corresponding data objects) may be selected based on the similarity score to be included in the search results.

[0062] In some implementations, the search engine 224 may include at least some of the graphical objects in the search results using tags assigned to those graphical objects. For example, at least some of the graphical objects 238 may be tagged using any suitable method, depending on whether the graphical object depicts a particular type of object or shows a particular feature (e.g., using machine learning and / or manual tagging). As described above, when determining the search results, the search engine 224 may determine that the graphical object has a shape similar to the example object by analyzing its shape descriptor. Other graphical objects 238 in the graphical objects 238 may be included in the search results based on sharing tags that are similar or identical to the determined graphical object. For example, if the graphical object is tagged with a bird, one or more other graphical objects tagged with a bird may be included in the search results.

[0063] Using this approach, in some cases, the shape descriptor determiner 220 does not need to determine the shape descriptors of at least some of the graphical objects 238 and / or the search engine 224 does not need to analyze the shape descriptors of at least some of the graphical objects 238. This can reduce processing and storage requirements for generating search results. For example, tags can be determined using a conventional method that assigns confidence scores to each tag and object or image pair for each graphical object. A subset of items (e.g., one or more graphical objects or images) can be selected for each tag based on the confidence scores of the tags (e.g., the top ten scoring items for each tag). Then, shape descriptors can be determined only for the subset of items, and other items can be included in the search results, and the search results have items from the subset based on having the same or similar tags as the items. Thus, for example, shape descriptors of one or more graphical objects that are highly associated with a particular tag can be determined without having to determine a shape descriptor for each graphical object tagged with the tag. However, when those graphical objects that do not have shape descriptors have tags similar to tags of graphical objects included in the search results based on shape descriptors, those graphical objects that do not have shape descriptors can still be included in the search results.

[0064] Reference now Figure 5 , a flow chart illustrating an embodiment of a method 500 for determining a shape descriptor of a graphical object is provided. Each block of the method 500 and other methods described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method can also be implemented as computer-usable instructions stored on a computer storage medium. These methods can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with other hosted services), or a plug-in to another product, etc.

[0065] At block 510, method 500 includes determining an outline of a graphical object. For example, outline determiner 216 may determine an outline of a graphical object of graphical object 238 from graphical data 236 forming the graphical object.

[0066] At block 520, the method 500 includes sampling the contour of the graphic object to obtain sampling points. For example, the contour sampler 218 may sample the contour of the graphic object to obtain sampling points that capture the contour of the graphic object.

[0067] At block 530, method 500 includes determining a shape descriptor for the graphics object, the shape descriptor capturing local and global geometric characteristics of the sampling points. For example, shape descriptor determiner 220 may determine a shape descriptor for the graphics object, wherein the shape descriptor captures local and global geometric characteristics of each sampling point.

[0068] Reference Figure 6 , a flow chart illustrating an embodiment of a method 600 for searching an image based on shape descriptors is provided.

[0069] At block 610, method 600 includes determining a shape descriptor for a graphics object. For example, shape descriptor determiner 220 may determine shape descriptor 240 for graphics object 238. Each shape descriptor may be represented using, for example, Figure 5 The method 500 is determined.

[0070] At block 620, method 600 includes receiving a search query corresponding to a user-identified graphical object. For example, interface manager 212 may receive a search query corresponding to a user-identified graphical object. The graphical object may be graphical object 354 identified by a user of user device 102A using application 110.

[0071] At block 630, the method 600 includes generating search results based on comparing the shape descriptor to the shape descriptor of the identified object. For example, the search engine 224 may generate search results 232 for the search query based on a comparison between the shape descriptor of the graphical object identified by the user and the shape descriptor of at least one of the graphical objects 238. In some embodiments, the shape descriptor determiner 220 is at least partially on the user device 102A (e.g., part of the application 110), which calculates the shape descriptor of the graphical object 354 and sends the shape descriptor to the server 108 including the search engine 224 in the search query.

[0072] At block 640, method 600 includes causing presentation of at least one graphical object based on the search results. For example, presentation manager 222 may cause presentation of one of the search results on user device 102A. This may include using interface manager 212 to send at least some of the search results to user device 102A for display on the device or otherwise sending data causing display. For example, search result 360 may be displayed on user device 102A.

[0073] refer to Figure 7 , computing device 700 includes bus 710 that directly or indirectly couples the following devices: memory 712, one or more processors 714, one or more presentation components 716, input / output (I / O) ports 718, input / output components 720, and a schematic power supply 722. Bus 710 represents something that can be one or more buses (such as an address bus, a data bus, or a combination thereof). Although shown with lines for clarity, Figure 7, but in reality, it is not so clear to depict the various components, and metaphorically, the lines would more accurately be gray and fuzzy. For example, a presentation component such as a display device can be considered an I / O component. In addition, a processor has memory. The inventors recognize that this is the nature of the art, and reiterate that Figure 7 The figures are merely illustrations of exemplary computing devices that may be used in conjunction with one or more embodiments of the present invention. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., as all of these are referred to in the same manner as the present invention. Figure 7 The scope of the present invention is contemplated and referenced to a "computing device".

[0074] The computing device 700 typically includes various computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 700, and includes volatile and non-volatile media, removable and non-removable media. As an example and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media include both volatile and non-volatile, removable and non-removable media implemented in any method or technology, for storage devices such as computer-readable instructions, data structures, program modules or other data. Computer storage media include but are not limited to RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, magnetic tapes, disk storage or other magnetic storage devices or any other media that can be used to store the required information and can be accessed by the computing device 700. Computer storage media do not contain the signal itself. Communication media usually implement computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information delivery medium. The term "modulated data signal" refers to a signal having one or more characteristics that set or change its characteristics in a way that encodes the information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection and wireless media such as acoustic, RF, infrared and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0075] Memory 712 includes computer storage media in the form of volatile and / or non-volatile memory. Memory may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. Computing device 700 includes one or more processors that read data from various entities such as memory 712 or I / O components 720. Presentation component 716 presents data indications to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, etc.

[0076] I / O port 718 allows computing device 700 to be logically coupled to other devices including I / O components 720, some of which may be built-in therein. Schematic components include microphones, joysticks, game controllers, satellite antennas, scanners, printers, wireless devices, etc. I / O components 720 may provide a natural user interface (NUI) that processes air gestures, voices, or other physiological inputs generated by a user. In some cases, the input may be sent to an appropriate network element for further processing. NUI may implement any combination of the following: voice recognition, touch and stylus recognition, facial recognition, biometrics, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition associated with a display on computing device 700. Computing device 700 may be equipped with a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, and a combination thereof, for gesture detection and recognition. In addition, computing device 700 may be equipped with an accelerometer or gyroscope capable of detecting motion. The output of the accelerometer or gyroscope may be provided to a display of computing device 700 to draw an immersive augmented reality or virtual reality.

[0077] The present invention has been described with respect to particular embodiments, which are intended in all respects to be illustrative rather than restrictive. Alternative embodiments will become apparent to those skilled in the art to which the present invention pertains without departing from the scope of the present invention.

[0078] Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the scope of the appended claims. Embodiments of the invention have been described for purposes of illustration and not limitation. Alternative embodiments become apparent to the reader of the disclosure after and because of reading the disclosure. Alternative ways of implementing the above may be accomplished without departing from the scope of the appended claims. Certain features and subcombinations have utility and may be employed without reference to other features and subcombinations and are contemplated to be within the scope of the claims.

Claims

1. A computer-implemented method for searching a graph, the method comprising: Analyzing each graphic object in the graphic object set, the analyzing comprising: Sampling the contour of the graphic object to obtain sampling points that capture the contour of the graphic object; and Determining a shape descriptor of the graphics object, the shape descriptor capturing local and global geometric characteristics of each of the sampling points, wherein determining the shape descriptor of the graphics object comprises: For each of said sampling points, determining a feature vector representing said local and global geometric characteristics of said sampling point; and combining corresponding values ​​of the feature vectors from each sampling point into a combined feature vector, wherein the shape descriptor comprises the combined feature vector; receiving a search query corresponding to a graphical object identified by a user; and Search results for the search query including at least one graphical object in the set of graphical objects selected are generated based on a comparison between the shape descriptor of the user-identified graphical object and the shape descriptor of each graphical object in the set of graphical objects. 2 . The computer-implemented method of claim 1 , wherein the outline of the graphical object comprises a vector graphic.

3. The computer-implemented method of claim 1 , further comprising: loading a data object including vector graphics data, the vector graphics data including graphics data for each graphics object in the set of graphics objects; identifying the set of graphics objects in the vector graphics data of the loaded data object, each graphics object corresponding to a respective vector graphic in the vector graphics data; as well as Based on identifying the set of graphical objects, the analyzing of each graphical object in the set of graphical objects is performed.

4. The computer-implemented method of claim 1 , further comprising: Extracting the set of graphic objects from a vector graphic file, wherein each graphic object is a vector graphic; as well as Each extracted graphical object and the shape descriptor of the graphical object are stored in a database, wherein generating the search results comprises processing the search query against the database.

5. The computer-implemented method of claim 1 , wherein: The generating of the search results includes: identifying a tag assigned to a given graphical object, wherein the comparing is between the shape descriptor of the graphical object identified by the user and the shape descriptor of the given graphical object; Based on the comparing, identifying a plurality of graphical objects assigned to the tag; and The identified plurality of graphical objects are added to a set of search results for the search query. The computer-implemented method of claim 1 , wherein the sampling of the contour comprises a number of sampling points for a segment of the contour based on a length of the segment.

7. The computer-implemented method of claim 1, wherein the outline of a given graphical object is formed by a plurality of Bezier curves.

8. The computer-implemented method of claim 1 , comprising: determining the local and global geometric characteristics of the sampling point of the sampling point using a spatial histogram, wherein partitions of the spatial histogram represent corresponding spatial regions and values ​​of the partitions are based on the number of neighboring sampling points of the sampling point within the corresponding spatial region; as well as A spatial descriptor is determined from the spatial histogram.

9. The computer-implemented method of claim 1, further comprising causing presentation of the search results associated with the search query on a user device, the presentation comprising sending an indicator of the search results to the user device.

10. At least one non-transitory computer-readable medium having executable instructions embodied thereon, the executable instructions, when executed by at least one processor, causing the at least one processor to perform a method for searching a graph, the method comprising: Sampling the contour of the graphic object identified by the user to obtain the sampling points capturing the contour of the graphic object identified by the user; Determining a shape descriptor of the graphical object identified by the user, the shape descriptor capturing local and global geometric characteristics of each of the sampling points, wherein determining the shape descriptor of the graphical object comprises: For each of said sampling points, determining a feature vector representing said local and global geometric characteristics of said sampling point; and combining corresponding values ​​of the feature vectors from each sampling point into a combined feature vector, wherein the shape descriptor comprises the combined feature vector; sending a search query including the shape descriptor to a search engine; receiving search results responsive to the search query from the search engine, the search results identifying at least one graphical object based on a similarity between the shape descriptor of the graphical object identified based on the user and at least one other shape descriptor of the at least one graphical object determined by sampling at least one other contour of the at least one graphical object; and The at least one graphical object is presented on a user device associated with the search query.

11. The computer-readable medium of claim 10, wherein said presenting said at least one graphical object comprises: A plurality of preview images of graphical objects including the search results are presented.

12. The computer-readable medium of claim 10, wherein said presenting said at least one graphical object comprises: The at least one graphic object is imported into a drawing board of a graphic editing application.

13. The computer-readable medium of claim 10, further comprising: converting a user input stream into a digital rendering of a graphical object including the user identification, the digital rendering being displayed in a graphical editing application; receiving a user selection of a search button displayed with the digital outline; as well as Based on the receiving of the user selection, the rendering of the at least one graphical object is performed in the graphics editing application.

14. The computer-readable medium of claim 10, wherein the sampling of the contour comprises a number of sampling points of a segment of the contour based on a length of the segment.

15. A computer-implemented system for searching a graph, the system comprising: at least one processor; as well as at least one data storage device communicatively coupled to the at least one processor, wherein the data storage device comprises a non-transitory computer storage medium storing computer usable instructions executable by the at least one processor, wherein the computer usable instructions comprise: A contour sampler, for sampling the contour of the graphic object to obtain sampling points capturing the contour of the graphic object; a shape descriptor determiner, configured to determine shape descriptors of the graphic objects, each shape descriptor capturing local and global geometric characteristics of the sampling points of the corresponding graphic object, wherein determining the shape descriptors comprises: For each of said sampling points, determining a feature vector representing said local and global geometric characteristics of said sampling point; and combining corresponding values ​​of the feature vectors from each sampling point into a combined feature vector, wherein the shape descriptor comprises the combined feature vector; and A search engine for generating search results for a search query based on a comparison between the shape descriptors of the graphic objects corresponding to the search query and the shape descriptors of at least one of the graphic objects, wherein the search results include the at least one of the graphic objects.

16. The computer implemented system of claim 15, the computer usable instructions further comprising an interface manager for: receiving the search query from a user device; and The search results are provided to the user device for presentation of at least some of the search results.

17. The computer implemented system of claim 15, the computer usable instructions further comprising an interface manager of the server for: receiving, at the server, the search query in at least a first one or more network communications from a user device; and In at least a second one or more network communications, the search results are provided to the user device for presentation of at least some of the search results.

Images