Method, computer program and apparatus for visual search

Abstract

A technique for performing visual search includes updating a probability distribution based on successive frames containing an object image until a specified condition is met, and generating search results for the object only after the specified condition is met. When a user uses a device to capture an image of a scene, a front-end visual search application running on the device acquires successive image frames and sends the first image frame to a back-end computer configured to perform classification on the image frames. The back-end computer acquires a prior probability distribution and generates a likelihood function indicating whether the image frame contains the object. The back-end computer then updates the prior probability distribution by adding respective values ​​of parameters associated with the prior probability distribution and the likelihood function.

Description

[Technical field]

[0001] Technical Field The present description relates to performing a visual search for objects in an image. [Background technology]

[0002] background Computer vision is a technique used to classify images of objects in a scene. For example, some search engines are configured to generate search results based on input images of objects. In such search engines, a machine learning engine, such as a convolutional neural network, is trained as a classifier to classify images as belonging to one of several classes. For example, an image of a four-legged animal may be classified as a dog, a cat, a horse, a sheep, or a cow. Summary of the Invention

[0003] overview An implementation provides a back-end visual search function configured to present robust search results based on objects in image data sent from a device performing a visual search application or operation. For example, a mobile device (e.g., a smartphone) may use a sensing device (e.g., a camera) to capture an image of a scene including an object (e.g., a menu in a restaurant). A visual search application causes the device to compress the image and send the image to a client computer (e.g., a digital capture server). The client computer retrieves a prior (i.e., a Bayesian prior) probability distribution (e.g., a beta distribution) of objects belonging to a certain object class (e.g., a class of menus). The client computer then initiates a coarse classification of the image data using the image data at a back-end server, which generates a current distribution of the probability that the image contains an object belonging to the object class. In response, the client computer updates the prior distribution based on the previous prior distribution and the current distribution. If the prior distribution is a beta distribution, the current distribution may be considered as a conjugate prior distribution, i.e., a binomial distribution whose parameters indicate whether an object belongs to the object class as a result of the classification. In this case, updating the prior probability distribution includes adding the value of the parameter of the current probability distribution to the corresponding value of the parameter of the prior probability distribution. Once updated, the client computer compares the measure of the prior probability distribution (e.g., the mean of the distribution) to a threshold value. The client computer returns search results for the object only if the updated measure of the prior probability distribution is greater than the threshold value. In this way, the visual search function is more stable and efficient, improving the user experience. [Means for solving the problem]

[0004] In one general aspect, a method may include receiving first image data and second image data from a device during a visual search operation for an object in a scene, the first image data representing a first image of the scene at a first time, and the second image data representing a second image of the scene at a second time. The method may also include generating a first visual match probability (i.e., a probability of a match between the object and an object in a coarse object class) based on the first image data, the first visual match probability indicating a likelihood that the object included in the first image of the scene belongs to an object class. The method may further include updating the first visual match probability based on the second image data to generate a second visual match probability in response to determining that the first visual match probability does not satisfy a criterion. The method may further include transmitting a digital capture associated with the object to the device as part of the visual search operation after determining that the second visual match probability satisfies the criterion.

[0005] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method. The method can include receiving first image data and second image data from a device during a visual search operation for an object in a scene, the first image data representing a first image of the scene at a first time, and the second image data representing a second image of the scene at a second time. The method can also include generating a first visual match probability based on the first image data, the first visual match probability indicating a likelihood that the object included in the first image of the scene belongs to a coarse object class in the first image of the scene. The method can further include updating the first visual match probability based on the second image data to generate a second visual match probability in response to determining that the first visual match probability does not satisfy a first criterion. The method may further include determining a likelihood that the object belongs to a fine object class after determining that the second visual match probability satisfies the first criterion. The method may further include, in response to determining that the likelihood that the object belongs to the fine object class satisfies a second criterion, transmitting a digital capture associated with the object to the device as part of the visual search operation.

[0006] In another general aspect, an electronic device configured to generate a recrawling strategy includes a memory and a control circuit coupled to the memory. The control circuit can be configured to receive first image data and second image data from a device during a visual search operation for an object in a scene, the first image data representing a first image of the scene at a first time, and the second image data representing a second image of the scene at a second time. The control circuit can be further configured to generate a first visual match probability based on the first image data, the first visual match probability indicating a likelihood that the object included in the first image of the scene belongs to an object class in the first image of the scene. The control circuit can be further configured to update the first visual match probability based on the second image data to generate a second visual match probability in response to determining that the first visual match probability does not satisfy a criterion. The control circuit can be further configured to transmit a digital capture associated with the object to the device as part of the visual search operation after determining that the second visual match probability satisfies the criterion.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0008] [Figure 1A] FIG. 1 illustrates an exemplary electronic environment in which the improved techniques described herein may be implemented. [Figure 1B] FIG. 1 illustrates an example scene including an object and a device configured to capture an image of the object. [Figure 1C] A diagram showing an example digital capture retrieved by a device associated with an object in a scene, according to a disclosed implementation. [Diagram 2]1 is a flowchart illustrating an example method for performing a visual search, according to a disclosed implementation. [Diagram 3] FIG. 2 is a sequence diagram of an exemplary visual search according to a disclosed implementation. [Figure 4] 1 is a flowchart illustrating an example visual search decision process. [Diagram 5] FIG. 1 shows a plot of a continuous prior probability distribution. [Figure 6] FIG. 1 illustrates an example evaluation process for a recrawl strategy. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description Some classifiers used in visual search applications or operations output a probability associated with each such classification, based on an image input, indicating the likelihood that the image actually contains an object belonging to a particular object class. However, some classifiers have more fine-grained classification outputs. For example, assuming a classifier associates an image with a dog, the classifier can then determine whether the dog is a hound, and if so, whether it is a beagle, a basset, or a large hound, such as for wolf hunting. The classifier may further assign probabilities to each of these subclass branches by the classifier, and these probabilities may generally vary with the branches. In this case, the classifier may assign a distribution of probabilities to each class, rather than a single probability value.

[0010] Conventional classifiers used in visual search applications return a classification corresponding to the most probable class above a detection threshold, and after assigning an image to a class, update their probability distribution for each of the classes. In this way, if the initial classification of an image is indicated by the user to be incorrect, the classifier can use new data provided by that indication.

[0011] A technical problem in performing visual search is that the above-mentioned conventional classifiers generate classification results for each input image. Each classification operation performed by the classifier uses significant computational resources, and the search engine served by the classifier may appear sluggish to the user. Furthermore, in many cases, coarse object detection (i.e., classification based on the distribution in the first branch of the search tree) may fluctuate when the probability distribution average is close to a threshold. Such fluctuations may also cause obvious instability in the search engine behavior, which may further worsen the user experience. Thus, coarse object classes include less specific object classes.

[0012] According to implementations described herein, the technical solution to the above technical problem includes updating a probability distribution based on successive frames including an object image until a specified condition is met, and generating search results for the object only after the specified condition is met. For example, when a user points a device, such as a smartphone camera, at a poster including a map of a country, a front-end visual search application running on the smartphone obtains successive image frames including an image of the map, compresses each frame, and transmits a first compressed frame to a back-end computer configured to perform classification on the frames. Based on previous training results, the back-end computer obtains an initial prior probability distribution. Based on the received image frame, the back-end computer generates a current probability distribution indicating whether the image frame includes an object, i.e., a map of the country. The back-end computer then updates the prior probability distribution by calculating a posterior probability distribution. In some implementations, if the prior distribution is a beta distribution and the current distribution is the conjugate of the prior distribution, i.e., a binomial distribution, the updating involves adding the respective values ​​of the parameters associated with the distribution. In some implementations, the updating includes incrementing the value of one of the parameters of the beta distribution according to whether the frame is classified as including an object at a coarse level. The back-end computer then evaluates the probability measure of the updated prior distribution (e.g., the mean of the distribution) and compares the probability measure to a threshold, i.e., the minimum value of the probability measure for which the image frame is likely to contain an object in that object class. If the threshold is exceeded, the back-end computer fetches the digital complement from the search server and delivers the search results to the smartphone.

[0013] In some implementations, the object class is a coarse object class, and further refinement of the coarse object class is performed prior to sending the digital complement to the device. For example, if the object is a map of Martha's Vineyard, the coarse object class may be "map". More finely refined classes may include "map of the country is the United States", "map of the United States includes map of Massachusetts", and "map of Massachusetts includes Martha's Vineyard". The back-end computer iterates the derivation of visual search probability distributions; these distributions are expected to become narrower as the object class becomes more finely refined. In some implementations, the back-end computer may determine a level of refinement of the object class, at which the prior probability distribution satisfies the refined criteria, and the back-end computer may obtain and send the digital complement to the device. In some implementations, the determination of satisfaction of the coarse and finely refined criteria is performed in parallel. In some implementations, there are further criteria for determining the final level of refinement.

[0014] A technical advantage of the disclosed implementations is that by capturing successive frames and updating the probability distributions in real time with each frame, much of the back-end communication between the front-end application running on the device and the back-end computer performing the classification on the images is eliminated, thus reducing the time and resources required to return correct search results. Furthermore, the search process may be more stable because the update process on the server is more likely to suppress information from the user that may change rapidly.

[0015] 1A illustrates an exemplary electronic environment 100 in which the above technical solutions may be implemented. A computer 120 is configured to perform a visual search on images provided by a display device 170.

[0016] 1B illustrates an example scene 10 including an object 20 and a user 100 having a device 170 configured to capture an image of the object, where the user 100 aims (e.g., points) a camera on the device 170 at the object 20 to obtain a digital capture, e.g., information about the object 20 from a network such as the World Wide Web. In some implementations, the user 100 aims the camera on the device 170 at the object 20 until such a digital capture is received at the device 170.

[0017] In some implementations, device 170 compresses each image, for example, using an encoder for a lossy compression scheme, before transmitting each image over the network to a computer that obtains digital capture of object 20. In such implementations, the images may differ significantly from each other because some data contained in the images may be lost in a lossy compression scheme. The data lost in one image may differ from the data lost in another image because such compression may vary widely based on very small differences between images. Thus, a first image captured by device 170 at a first time will differ from a second image captured by device 170 at a second time because the lossy compression of the first image differs from the lossy compression of the second image. For example, user 100 is unlikely to be completely still, and small movements may result in differences in camera position and orientation, and therefore small differences in the images.

[0018] 1C illustrates an example digital supplement 110 retrieved (e.g., received) by device 170 associated with object 20 in scene 10. In some implementations, digital supplement 110 takes the form of a web page that provides information about object 20. In some implementations, digital supplement 110 takes the form of static text, audio, video, interactive content, etc.

[0019] Returning to FIG. 1A, computer 120 includes a network interface 122, one or more processing units 124, and memory 126. Network interface 122 includes, for example, an Ethernet adapter, a token ring adapter, etc., for converting electronic and / or optical signals received from network 150 into an electronic format for use by computer 120. Set of processing units 124 includes one or more processing chips and / or assemblies. Memory 126 includes both volatile memory (e.g., RAM) and non-volatile memory, such as one or more ROMs, disk drives, solid state drives, etc. Together, set of processing units 124 and memory 126 form control circuitry configured and arranged to perform various methods and functions as described herein.

[0020] In some implementations, one or more of the components of the computer 120 may be or include a processor (e.g., processing unit 124) configured to process instructions stored in the memory 126. Examples of such instructions as shown in FIG. 1 include an entity manager 130, a prediction manager 140, a recrawl manager 150, and a recrawl strategy manager 160. Further, as shown in FIG. 1A, the memory 126 is configured to store various data, which are described with respect to each manager that uses such data. Note that in some implementations, the entity page corresponds to an offer page that includes an offer for selling a product.

[0021] Image manager 130 is configured to receive image data 132. In some implementations, image manager 130 receives image data 132 from display device 170 via network interface 122, i.e., via a network (such as network 190). In some implementations, image manager 130 receives image data 132 from local storage (e.g., a disk drive, flash drive, SSD, etc.).

[0022] Image data 132 represents an image of a scene as viewed through display device 170, for example, via an image acquisition manager 172 configured to acquire images through a camera and transmit the image data 132 to computer 120. In some implementations, the image data takes the form of a series of frames of compressed image data 134(1), 134(2), ..., 134(N).

[0023] Compressed images 134(1), ..., 134(N) represent the encoded form of image data generated by image acquisition manager 172. Display device 170 is configured to perform an encoding operation on the generated image data before transmitting the generated image data to computer 120. In some implementations, the type of encoding (e.g., JPEG) used for compression is important only to the effect that any classifier configured to determine whether an image contains an object belonging to an object class is trained using the similarly compressed images. In this way, no decoding operation is required to be performed for classification. Furthermore, a successive image frame, e.g., compressed image 134(2), is unlikely to be the same as the previous frame, e.g., compressed image 134(1), because small changes in the camera orientation or position can cause large changes in the compressed image. However, in some implementations, there may be a decoding step before any classification is performed.

[0024] In some implementations, image acquisition manager 172 is also configured to identify textual descriptions of objects in the scene. For example, image acquisition manager 172 may associate certain image data with textual descriptors, such as names of people, places, products, artwork, etc. Image acquisition manager 172 may then include such textual descriptions with image data 132 such that identification of possible objects for classification is simplified.

[0025] The prior distribution manager 140 is configured to obtain prior distribution data 142 representing a prior probability distribution. In some implementations, the prior distribution data 142 is generated by the prior distribution manager 142 based on training data from an object classifier. In some implementations, the object classifier includes a convolutional neural network (CNN) and the training data includes previous visual search results.

[0026] The prior distribution data 142 represents a prior probability distribution indicating the likelihood that an object included in an image frame, e.g., 134(1), belongs to a certain object class. In some implementations, the prior distribution data 142 includes a distribution identifier (e.g., “beta”, “gamma”, “binomial”, etc.) and values ​​of a parameter associated with the distribution identifier. In some implementations, the prior distribution data includes probability density values ​​that together form a probability distribution. In some implementations, the prior probability distribution is distributed over possible probability values ​​between 0 and 1. That is, the coarse classification indicates whether an object belongs to a coarsely defined object class, with further layers of object classes forming a tree structure. Various paths along the tree structure provide a distribution of probabilities of the coarsely defined object classes.

[0027] In some implementations, determining whether an image containing an object belonging to a coarse class belongs to a finer class is performed using a set of heuristics derived from empirical statistical data. For example, an object determined to be a dog may be a poodle or a hound. Determining whether a dog is a poodle or a hound includes determining the higher likelihood that any dog ​​is a poodle or a hound. In some implementations, there is a whitelist or blacklist that indicates whether such finer classes should be considered in the visual search.

[0028] Current distribution manager 144 is configured to indicate whether image data 132, e.g., compressed image 132(1), includes objects that belong to an object class. Current distribution manager 144 is configured to generate current distribution data 146 based on whether an object belongs to an object class.

[0029] The current distribution data 146 represents the likelihood that the image data 132, specifically the compressed image 134(1), contains an object belonging to a certain object class. In some implementations, the current distribution data 146 is a binomial distribution based on a parameter indicating whether an object in an image belongs to the object class. That is, the parameter of the binomial distribution, in some configurations, is the number of times that successive image frames have an object belonging to a certain object class, and the number of times that those successive image frames do not have an object belonging to the object class.

[0030] The distribution update manager 150 is configured to update the prior probability distribution based on the previous prior probability distribution 142 and the current probability distribution 146 to generate updated distribution data 152. In some implementations, the distribution update manager 150 is configured to multiply the previous prior distribution 142 by the current probability distribution 146. In such implementations, the distribution update manager 150 is configured to normalize the product by dividing it by the sum of the products over all probabilities. In some implementations, if the previous prior distribution 142 is a beta distribution and the current distribution 146 is a binomial distribution (and its conjugate to the prior probability distribution), the distribution update manager 150 is configured to add the respective values ​​of the parameters associated with the current distribution 146 to the parameters associated with the prior distribution 142. In this case, the updated prior distribution 152 is a beta distribution with updated parameter values.

[0031] The updated distribution data 152 indicates a new prior distribution resulting from the update by the distribution update manager 150. If the previous prior distribution 142 is a beta distribution and the current distribution 146 is a binomial distribution, then the updated prior distribution 152 is a beta distribution. Further, in this case, if the image frame contains an object belonging to a certain object class, then the first parameter of the beta distribution is incremented and the second parameter remains unchanged. In contrast, if the image frame does not contain an object belonging to that object class, then the first parameter of the beta distribution remains unchanged and the second parameter is incremented.

[0032] The information acquisition manager 160 is configured to determine whether the updated prior probability distribution 152 satisfies the criteria represented by the information criteria data 162. In some implementations, the information acquisition manager 160 is configured to derive a probability measure from the updated prior probability distribution 152. For example, if the criteria states that the probability measure is greater than a threshold, the information acquisition manager 160 is configured to compare the derived probability measure to the threshold. In some implementations, the probability measure is the mean of the probability distribution. In some implementations, the probability measure is the mean of the probability distribution.

[0033] Information acquisition manager 160 is also configured to obtain information from search server 180 about the object based on information criteria data 162. For example, if the object is determined to be a menu from a particular restaurant, the information may take the form of reviews of that restaurant. The reviews may be taken from indexed search results generated by the search server. Information acquisition manager 160 further transmits this information to display device 170 in response to the criteria defined in information criteria data 162 being met.

[0034] The information criteria data 162 represents one or more criteria used to determine whether to transmit the retrieved information to the display device 170. In some implementations, the criterion is that the mean of the prior probability distribution is greater than a threshold. In this case, the information criteria data 162 may take the form of a threshold.

[0035] The components of the user device 120 (e.g., modules, processing units 124) may be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which may include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of the computer 120 may be configured to operate within a cluster of devices (e.g., a server farm). In such implementations, the functionality and processing of the components of the computer 120 may be distributed across several devices of the cluster of devices.

[0036] The components of computer 120 may be or include any type of hardware and / or software configured to process attributes. In some implementations, one or more portions of the components shown in the components of computer 120 of FIG. 1 may be or include hardware-based modules (e.g., digital signal processor (DSP), field programmable gate array (FPGA), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-readable instructions that can be executed on a computer). For example, in some implementations, one or more portions of the components of computer 120 may be or include software modules configured for execution by at least one processor (not shown). In some implementations, the functionality of the components may be included in different modules and / or different components than those shown in FIG. 1, including combining functionality shown as two components into a single component.

[0037] Although not shown, in some implementations, components of computer 120 (or portions thereof) may be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some implementations, components of computer 120 (or portions thereof) may be configured to operate within a network. Thus, components of computer 120 (or portions thereof) may be configured to function within various types of network environments, which may include one or more devices and / or one or more server devices. For example, the network may be or include a local area network (LAN), a wide area network (WAN), etc. The network may be or include, for example, a wireless network and / or a wireless network implemented using gateway devices, bridges, switches, etc. The network may include one or more segments and / or have portions based on various protocols, such as Internet Protocol (IP) and / or proprietary protocols. The network may include at least a portion of the Internet.

[0038] In some implementations, one or more of the components of computer 120 may be or include a processor configured to process instructions stored in a memory. For example, image manager 130 (and / or portions thereof), prior distribution manager 140 (and / or portions thereof), current distribution manager 144 (and / or portions thereof), distribution update manager 150 (and / or portions thereof), and information acquisition manager 160 (and / or portions thereof) may be a combination of a processor and memory configured to execute instructions associated with a process to achieve one or more functions.

[0039] In some implementations, the memory 126 may be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, the memory 126 may be implemented as multiple memory components (e.g., multiple RAM components or disk drive memory) associated with components of the VR server computer 120. In some implementations, the memory 126 may be a database memory. In some implementations, the memory 126 may be or include a non-local memory. For example, the memory 126 may be or include a memory shared by multiple devices (not shown). In some implementations, the memory 126 may be associated with a server device (not shown) in a network and configured to serve components of the computer 120. As shown in FIG. 1, the memory 126 is configured to store various data, including image data 132, prior distribution data 142, current distribution data 146, updated distribution data 152, and information criteria data 162.

[0040] The beta distribution is defined as:

[0041]

number

[0042] where α and β are hyperparameters of the beta distribution and q is a probability; that is, B(α,β) is the beta function and Γ(α) is the gamma function. In some implementations, the prior probability distribution 142 and the posterior probability distribution 152 take this mathematical form. Thus, the prior distribution 142 is the conjugate prior probability distribution of the likelihood function, which in some implementations takes the form of a binomial distribution. One such binomial distribution takes the form:

[0043]

number

[0044] The posterior probability is given by Bayes' theorem:

[0045]

number

[0046] If the prior is a beta distribution, then computing the posterior probability reduces to a Bayesian update of the prior as follows:

[0047]

number

[0048] That is, updating the prior probability distributions involves adding the respective values ​​of the parameters of each distribution. Examples of these distributions are discussed in further detail with respect to FIG.

[0049] 2 is a flow chart illustrating an example method 200 for performing a visual search according to the improved techniques described above. The method 200 may be performed by a software configuration described in connection with FIG. 1 , resident in the memory 126 of the computer 120 and executed by the set of processing units 124.

[0050] At 202, image manager 130 receives first and second image data (e.g., compressed image 134(1,2)) from a device (e.g., display device 170), where the first image data represents a first image of a scene, where the first image of the scene includes an object. For example, the scene may include a building that displays a menu, and the object may be the menu.

[0051] At 204, information acquisition manager 160 generates a first probability measure based on the first image data, the first probability measure indicating the likelihood that an object included in the first image of the scene belongs to an object class, the first probability measure not meeting a specified criterion (in information criteria data 162). In some implementations, the first probability measure is the mean of the prior probability distribution obtained by prior distribution manager 140 and represented by prior distribution data 142.

[0052] At 206, in response to the first probability measure not meeting the specified criteria, the distribution update manager 150 updates the first probability measure based on the second image data (e.g., 134(2)) to generate a second probability measure, where the second probability measure meets the specified criteria. Again, if the first probability measure is based on a beta distribution, the current distribution manager 144 generates a binomial distribution based on whether the classifier determines that an object included in the second image of the scene belongs to that object class. In some implementations, the second probability measure is the mean of the updated prior probability distribution, i.e., the updated prior probability.

[0053] At 208, in response to the second probability measure meeting the specified criteria, the information acquisition manager 160 transmits information associated with the object, such as a digital supplement, to the device. In some implementations, the information takes the form of web content about the object. For example, if the object is a menu from a restaurant, the information may take the form of a review of the restaurant taken from a restaurant review website.

[0054] 3 is a sequence diagram of an exemplary visual search 300 including a display device 170, a computer 120, and a search server 180. The visual search 300 may be performed by a software configuration described in connection with FIG. 1 , which resides in memory 126 of the computer 120 and is executed by a set of processing units 124.

[0055] At 302, the display device 170 transmits image data to the computer 120, as described above with respect to FIGS.

[0056] At 304, the computer 120 retrieves a prior probability distribution p(q) that indicates that objects in the image data belong to a certain object class.

[0057] At 306, computer 120 receives an initial coarse classification result in the form of a likelihood function (or current distribution) p(s,f|q). In some implementations, data representing the likelihood function is stored locally on computer 120. At 306, computer 120 receives an initial coarse classification result in the form of a likelihood function (or current distribution) p(s,f|q). In some implementations, data representing the likelihood function is stored locally on computer 120.

[0058] At 308, the computer 120 generates the posterior probability distribution p(q|s,f) and evaluates its mean against a specified threshold, where the mean is less than the threshold.

[0059] At 312, computer 120 replaces the previous prior distribution with the posterior distribution, i.e., p(q)←p(q|s,f).

[0060] At 314, the computer 120 receives new coarse classification results based on the new image data in the form of a likelihood function (or current distribution) p(s,f|q).

[0061] At 316, the computer 120 generates the posterior probability distribution p(q|s,f) and evaluates its mean against a specified threshold, where the mean is greater than the threshold.

[0062] At 318 , computer 120 retrieves search results for the object, eg, a digital supplement, from search server 180 .

[0063] At 320 , the computer 120 transmits the search results to the display device 170 .

[0064] 4 is a flow chart illustrating an example visual search decision process 400. The visual search decision process 400 may be performed by a software configuration described in connection with FIG. 1 , which resides in the memory 126 of the computer 120 and is executed by the set of processing units 124.

[0065] At 402 , the computer 120 receives compressed image data from the display device 170 .

[0066] At 404, the computer 120 obtains a prior distribution, which may take the form of a beta distribution as defined above. The prior distribution indicates the likelihood of the object class.

[0067] At 406, computer 120 updates the beta distribution according to whether the classifier determines that an object belonging to an object class is represented in the image data, e.g., if the image data does not contain such an object, computer 120 increments the value of β, otherwise computer 120 increments the value of α.

[0068] At 408, the computer 120 determines whether the mean of the updated prior distribution is greater than a specified threshold. If the mean is greater than the threshold, the process 400 proceeds to 410. If not, the process 400 returns to 404.

[0069] At 410, the computer 120 obtains and transmits search results associated with the object.

[0070] 5 is a plot of a continuous prior probability distribution 500. For example, curve 510 represents a beta distribution with α=8 and β=20, which gives the mean:

[0071]

number

[0072] If the specified threshold is 0.33, the beta distribution will need to be updated. Considering that computer 120 sends new image data, computer 120 will update the distribution according to whether an object belonging to a certain object class is determined to be in the new image data. In this case, computer 120 increments α. The new curve 520 has a mean of:

[0073]

number

[0074] In this case, the average is still below the threshold, so process 400 (FIG. 4) repeats itself and computer 120 transmits another new image data. In response, computer 120 determines whether an object belonging to that object class is determined to be within this other new image data. If so, computer 120 again increments α, and the average of curve 530 becomes:

[0075]

number

[0076] , which exceeds the threshold, the computer 120 can transmit the search results to the display device 170.

[0077] 6 shows an example of a general purpose computing device 600 and a general purpose mobile computing device 650 that may be used with the techniques described herein. The computing device 600 is an example of a configuration of the computer 120 of FIGS.

[0078] As shown in Figure 6, computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed herein.

[0079] Computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 that connects to memory 604 and a high-speed expansion port 610, and a low-speed interface 612 that connects to a low-speed bus 614 and storage device 606. Each of components 602, 604, 606, 608, 610, and 612 are interconnected using various buses and may be implemented on a common motherboard or in other manners as desired. Processor 602 can process instructions for execution within computing device 600, including instructions stored in memory 604 or storage device 606, to display graphical information for a GUI on an external input / output device, such as a display 616 coupled to high-speed interface 608. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as desired. Multiple computing devices 600 may also be connected, with each device providing a portion of the required operations (e.g., as a server bank, as a group of blade servers, or as a multiprocessor system).

[0080] The memory 604 stores information within the computing device 600. In one implementation, the memory 604 is one or more volatile memory units. In another implementation, the memory 604 is one or more non-volatile memory units. The memory 604 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0081] The storage device 606 can provide mass storage for the computing device 600. In one implementation, the storage device 606 may be or include a computer-readable medium, such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. The computer program product can be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 604, the storage device 606, or a memory on the processor 602.

[0082] The high-speed controller 608 manages bandwidth-intensive operations for the computing device 500, and the low-speed controller 612 manages less bandwidth-intensive operations. Such an allocation of functions is merely exemplary. In one implementation, the high-speed controller 608 is coupled to the memory 604 (e.g., via a graphics processor or accelerator), to a display 616, and to a high-speed expansion port 610 that may accept various expansion cards (not shown). In this implementation, the low-speed controller 612 is coupled to the storage device 506 and to a low-speed expansion port 614. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), and may be coupled, e.g., via a network adapter, to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router.

[0083] The computing device 600 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 620, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 624. In addition, it may be implemented in a personal computer, such as a laptop computer 622. Alternatively, components from the computing device 600 may be combined with other components in a mobile device (not shown), such as device 650. Each such device may include one or more of the computing devices 600, 650, and the entire system may be composed of multiple computing devices 600, 650 in communication with each other.

[0084] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0085] These computer programs (also known as programs, software, software applications or codes) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or in an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device used to provide machine instructions and / or data to a programmable processor (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)), including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0086] To provide for user interaction, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may be used to provide for user interaction, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input.

[0087] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), a middleware component (e.g., an application server), a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0088] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0089] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the specification.

[0090] It is also understood that when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to other elements, it may be directly on, directly connected to, or directly coupled to other elements, or there may be one or more intervening elements. In contrast, when an element is referred to as being directly on, directly connected to, or directly coupled to other elements, there are no intervening elements. Although the terms directly on, directly connected, or directly coupled may not be used throughout the detailed description, elements that are shown as being directly on, directly connected, or directly coupled may be so referred to. The claims of this application may be amended to describe the exemplary relationships described in the specification or shown in the drawings.

[0091] While certain features of the described implementations are shown as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It should therefore be understood that the claims are intended to cover all such modifications and changes that are within the scope of the implementations. It should be understood that these are presented only as examples and not as limitations, and that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination except mutually exclusive combinations. The implementations described herein may include various combinations and / or subcombinations of the functions, components and / or features of the different implementations described.

[0092] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Additionally, other steps may be provided or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the claims.

[0093] Some examples are described below. Example 1: A method comprising: receiving first image data and second image data from a device during a visual search operation for an object in a scene, the first image data representing a first image of the scene at a first time and the second image data representing a second image of the scene at a second time, the method further comprising: generating a first visual match probability based on the first image data, the first visual match probability indicative of a likelihood that the object included in the first image of the scene belongs to a coarse object class in the first image of the scene, the method further comprising: in response to determining that the first visual match probability does not satisfy a criterion, updating the first visual match probability based on the second image data to generate a second visual match probability; and after determining that the second visual match probability meets the criterion, transmitting a digital capture associated with the object to the device as part of the visual search operation.

[0094] Example 2: The method of example 1, wherein the criteria include a visual search probability greater than or equal to a threshold. Example 3: The first visual search probability is a mean of a first probability distribution over the probability that the object in the first image belongs to the object class, the first probability distribution having the mean as the first probability measure including a first set of parameter values; 3. The method of example 2, wherein the second visual search probability is a mean of a second probability distribution, the second probability distribution having the mean as the second visual search probability including a second set of parameter values.

[0095] Example 4: After receiving the second image data, the first probability distribution is a prior distribution; Updating the first probability measure comprises: 4. The method of example 3, comprising multiplying the prior distribution by a current probability distribution, the current probability distribution representing a distribution of probability that a parameter of the current probability distribution has a particular value given a probability that the object included in the second image of the scene represented by the second image data belongs to the object class.

[0096] Example 5: The method of example 4, wherein the current probability distribution is a binomial distribution. Example 6: After receiving the second image data, the first probability distribution is a prior distribution, and the prior distribution is based on a value of a first parameter and a value of a second parameter; The method further comprises: generating a current probability distribution, the current probability distribution representing a distribution of probabilities that a parameter of the current probability distribution has a particular value given a probability that the object included in the second image of the scene represented by the second image data belongs to the object class, the current probability distribution being based on a value of a third parameter and a value of a fourth parameter; Updating the first visual match probability includes: 4. The method of example 3, comprising adding the value of the first parameter to the value of the third parameter, and adding the value of the second parameter to the value of the fourth parameter.

[0097] Example 7: After receiving the second image data, the first probability distribution is a prior distribution, and the prior distribution is based on a value of a first parameter and a value of a second parameter; Updating the first visual search probability in response to determining that the object is included in the object class, incrementing the value of the first parameter and not incrementing the value of the second parameter; 4. The method of example 3, comprising incrementing the value of the second parameter and not incrementing the value of the first parameter in response to determining that the object is included in the object class.

[0098] Example 8: The method of claim 3, wherein the first probability distribution and the second probability distribution are beta distributions.

[0099] Example 9: A method according to at least one of the preceding examples, wherein the digital capture includes data about the object not included in the image data, and the digital capture includes data from the World Wide Web and / or a database.

[0100] Example 10: A computer program product comprising a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit of a computer, causes the processing circuit to perform a method, the method comprising: receiving first image data and second image data from a device during a visual search operation for an object in a scene, the first image data representing a first image of the scene at a first time and the second image data representing a second image of the scene at a second time, the method further comprising: generating a first visual match probability based on the first image data, the first visual match probability indicative of a likelihood that the object included in the first image of the scene belongs to a coarse object class in the first image of the scene, the method further comprising: in response to determining that the first visual match probability does not satisfy a first criterion, updating the first visual match probability based on the second image data to generate a second visual match probability; determining a likelihood that the object belongs to a fine object class after determining that the second visual match probability satisfies the first criterion; and and in response to determining that the likelihood that the object belongs to the fine object class satisfies a second criterion, sending a digital capture associated with the object to the device as part of the visual search operation.

[0101] Example 11: The computer program product of Example 10, wherein the first criterion includes the probability measure being greater than or equal to a threshold.

[0102] Example 12: The first probability measure is a mean of a first probability distribution over the probability that the object belongs to the coarse object class, the first probability distribution having the mean as the first probability measure with a first set of parameter values; 12. The computer program product of Example 11, wherein the second probability measure is a mean of a second probability distribution, the second probability distribution having the mean as the second probability measure comprising a second set of parameter values.

[0103] Example 13: After receiving the second image data, the first probability distribution is a prior distribution; Updating the first probability measure comprises: 13. The computer program product of Example 12, further comprising: multiplying the prior distribution by a current probability distribution, the current probability distribution representing a distribution of probability that a parameter of the current probability distribution has a particular value given a probability that the object included in the second image of the scene represented by the second image data belongs to the coarse object class.

[0104] Example 14: The computer program product of Example 13, wherein the current probability distribution is a binomial distribution.

[0105] Example 15: After receiving the second image data, the first probability distribution is a prior distribution, and the prior distribution is based on a value of a first parameter and a value of a second parameter; The method further comprises: generating a current probability distribution, the current probability distribution representing a distribution of probabilities that a parameter of the current probability distribution has a particular value given a probability that the object included in the second image of the scene represented by the second image data belongs to the coarse object class, the current probability distribution being based on a value of a third parameter and a value of a fourth parameter; Updating the first probability measure comprises: adding the value of the first parameter to the value of the third parameter, and adding the value of the second parameter to the value of the fourth parameter.

[0106] Example 16: After receiving the second image data, the first probability distribution is a prior distribution, and the prior distribution is based on a value of a first parameter and a value of a second parameter; Updating the first probability measure comprises: incrementing the value of the first parameter and not incrementing the value of the second parameter in response to the object in the second scene being classified as belonging to the coarse object class; and and in response to the object included in the second scene being classified as not belonging to the coarse object class, incrementing the value of the second parameter and not incrementing the value of the first parameter.

[0107] Example 17: The computer program product of Example 12, wherein the first probability distribution and the second probability distribution are beta distributions.

[0108] Example 18: A computer program product described in any one of Examples 10 to 17, wherein the digital supplement includes data about the object not included in the image data, and the digital supplement includes data from the World Wide Web and / or a database.

[0109] Example 19: An electronic device, Memory, a processing circuit coupled to the memory, the processing circuit comprising: and configured to receive first image data and second image data from a device during a visual search operation for an object in a scene, the first image data representing a first image of the scene at a first time and the second image data representing a second image of the scene at a second time, the processing circuitry further comprising: and configured to generate a first visual match probability based on the first image data, the first visual match probability indicating a likelihood that the object included in the first image of the scene belongs to a coarse object class in the first image of the scene, the processing circuitry further comprising: in response to determining that the first visual match probability does not satisfy a criterion, updating the first visual match probability based on the second image data to generate a second visual match probability; The electronic device is configured to, after determining that the second visual match probability satisfies the criterion, transmit a digital capture associated with the object to the device as part of the visual search operation.

[0110] Example 20: The first visual search probability is a mean of a first probability distribution over the probability that the object in the first image belongs to the object class, the first probability distribution having the mean as the first probability measure including a first set of parameter values; 20. The electronic device of example 19, wherein the second visual search probability is a mean of a second probability distribution, the second probability distribution having the mean as the second visual search probability including a second set of parameter values.

[0111] Example 21: After receiving the second image data, the first probability distribution is a prior distribution, and the prior distribution is based on a value of a first parameter and a value of a second parameter; The processing circuit further comprises: generating a current probability distribution, the current probability distribution representing a distribution of probabilities that a parameter of the current probability distribution has a particular value given a probability that the object included in the second image of the scene represented by the second image data belongs to the object class, the current probability distribution being based on a value of a third parameter and a value of a fourth parameter; The processing circuit configured to update the first visual match probability further comprises: 21. The electronic device of example 20, configured to add the value of the first parameter to the value of the third parameter, and add the value of the second parameter to the value of the fourth parameter.

[0112] Example 22: After receiving the second image data, the first probability distribution is a prior distribution, and the prior distribution is based on a value of a first parameter and a value of a second parameter; The processing circuit configured to update the first visual match probability further comprises: incrementing the value of the first parameter and not incrementing the value of the second parameter in response to the object in the second scene being classified as belonging to the object class; The electronic device of Example 20, configured to increment the value of the second parameter and not increment the value of the first parameter in response to the object included in the second scene being classified as not belonging to the object class.

Claims

1. 1. A method comprising: receiving, from a device during a visual search operation for an object in a scene, a first image of the scene at a first time and a second image of the scene at a second time; generating a first probability based on the first image of the scene, the first probability indicating a likelihood that the object included in the first image of the scene belongs to an object class, the method further comprising: in response to determining that the first probability does not meet a criterion, updating the first probability based on the second image of the scene to generate a second probability; and after determining that the second probability satisfies the criterion, transmitting a digital capture associated with the object to the device as part of the visual search operation; the first probability is a function of a first probability distribution over the probability that the object in the first image belongs to the object class; The method, wherein the second probability is a function of a second probability distribution.

2. The method of claim 1 , wherein the criteria include the first probability being greater than or equal to a threshold and the second probability being greater than or equal to the threshold.

3. after receiving the second image, the first probability distribution is a prior distribution; Updating the first probability comprises:

2. The method of claim 1 , comprising multiplying the prior distribution by a current probability distribution, the current probability distribution representing a distribution of probability that a parameter of the current probability distribution has a particular value given a probability that the object included in the second image of the scene belongs to the object class.

4. The method of claim 3 , wherein the current probability distribution is a binomial distribution.

5. after receiving the second image, the first probability distribution is a prior distribution, the prior distribution being based on a value of a first parameter and a value of a second parameter; The method further comprises: generating a current probability distribution, the current probability distribution representing a distribution of probabilities that a parameter of the current probability distribution has a particular value given a probability that the object in the second image of the scene belongs to the object class, the current probability distribution being based on a value of a third parameter and a value of a fourth parameter; Updating the first probability comprises:

2. The method of claim 1, comprising adding the value of the first parameter to the value of the third parameter, and adding the value of the second parameter to the value of the fourth parameter.

6. after receiving the second image, the first probability distribution is a prior distribution, the prior distribution being based on a value of a first parameter and a value of a second parameter; Updating the first probability comprises: in response to determining that the object is included in the object class, incrementing the value of the first parameter and not incrementing the value of the second parameter; 2. The method of claim 1, further comprising: in response to determining that the object is included in the object class, incrementing the value of the second parameter and not incrementing the value of the first parameter.

7. The method of claim 1 , wherein the first probability distribution and the second probability distribution are beta distributions.

8. 8. The method of claim 1, wherein the digital capture includes data about the object not included in the first image of the scene and the second image of the scene, and the digital capture includes data from the World Wide Web and / or a database.

9. and code that, when executed by a processing circuit of a computer, causes the processing circuit to perform a method, the method comprising: receiving, from a device during a visual search operation for an object in a scene, a first image of the scene at a first time and a second image of the scene at a second time; generating a first probability based on the first image of the scene, the first probability indicating a likelihood that the object included in the first image of the scene belongs to a coarse object class, the method further comprising: responsive to determining that the first probability does not satisfy a first criterion, updating the first probability based on the second image of the scene to generate a second probability; determining a likelihood that the object belongs to a fine object class after determining that the second probability satisfies the first criterion; and in response to determining that the likelihood that the object belongs to the fine object class satisfies a second criterion, transmitting a digital capture associated with the object to the device as part of the visual search operation; the first probability is a function of a first probability distribution over the probability that the object belongs to the coarse object class; The second probability is a function of a second probability distribution.

10. The computer program product of claim 9 , wherein the first criterion comprises the first probability being greater than or equal to a threshold value.

11. after receiving the second image, the first probability distribution is a prior distribution; Updating the first probability comprises:

10. The computer program product of claim 9, further comprising multiplying the prior distribution by a current probability distribution, the current probability distribution representing a distribution of probability that a parameter of the current probability distribution has a particular value given a probability that the object included in the second image of the scene belongs to the coarse object class.

12. The computer program product of claim 11 , wherein the current probability distribution is a binomial distribution.

13. after receiving the second image, the first probability distribution is a prior distribution, the prior distribution being based on a value of a first parameter and a value of a second parameter; The method further comprises: generating a current probability distribution, the current probability distribution representing a distribution of probabilities that a parameter of the current probability distribution has a particular value given a probability that the object in the second image of the scene belongs to the coarse object class, the current probability distribution being based on a value of a third parameter and a value of a fourth parameter; Updating the first probability comprises:

13. The computer program product of claim 12, further comprising adding the value of the first parameter to the value of the third parameter, and adding the value of the second parameter to the value of the fourth parameter.

14. after receiving the second image, the first probability distribution is a prior distribution, the prior distribution being based on a value of a first parameter and a value of a second parameter; Updating the first probability comprises: incrementing the value of the first parameter and not incrementing the value of the second parameter in response to the object in the second image being classified as belonging to the coarse object class; 10. The computer program product of claim 9, further comprising: in response to the object in the second image being classified as not belonging to the coarse object class, incrementing the value of the second parameter and not incrementing the value of the first parameter.

15. 10. The computer program product of claim 9, wherein the first probability distribution and the second probability distribution are beta distributions.

16. 16. A computer program product according to any one of claims 9 to 15, wherein the digital capture includes data about the object not included in the first image of the scene and the second image of the scene, and the digital capture includes data from the World Wide Web and / or a database.

17. 1. An electronic device comprising: Memory, and a processing circuit coupled to said memory, said processing circuit configured to perform the method of any one of claims 1 to 8.

Images