Multi-angle object recognition

Abstract

Methods, systems, and apparatus for controlling smart devices are described. In one aspect, a method includes: capturing multiple consecutive images via a camera on a user device for display in an application environment of an application executed on the user device, performing an object recognition process on the images, the object recognition process including determining that multiple images, each depicting a specific object, are required to perform object recognition on the specific object; and in response to the determination, generating a user interface element indicating a camera operation to be performed, the camera operation capturing two or more images; determining that a user has caused the indicated camera operation to be performed to capture the two or more images in response to the user interface element, and in response, determining whether the specific object is positively recognized from the multiple images.

Description

Technical Field

[0001] This specification describes techniques related to using user interface elements to present an object recognition process in real time to a user viewing an image on a mobile device. Background Art

[0002] Many mobile devices, such as smartphones, are equipped with cameras for capturing pictures and digital images. Mobile devices provide access to a wide variety of information. This information can be viewed in a web browser or in native applications running on the mobile device. Users can also use images to obtain additional information about objects or locations. For example, digital image files, video and / or audio files, and resources on specific topics can be accessed through the Internet.

[0003] Object recognition software is used to detect and identify objects in images or video sequences. The process for object recognition in images or video sequences typically uses a variety of different steps to analyze the image to positively identify the object within the image.

[0004] Once the user selects an image in which to search for an object of interest, object recognition software running in the background and / or server of the mobile device will operate without user feedback during the process of identifying the object within the image. Typically, the lack of user feedback during the object recognition software results in imperfect object recognition, including incorrect recognition, no positive recognition, or recognition of unexpected objects within the image or video sequence. However, without a direct visual indicator of the process of object recognition displayed within the candidate image on the mobile device, the user may have little understanding of the object recognition process and cannot give important feedback to assist the object recognition process. Summary of the invention

[0005] In general, an innovative aspect of the subject matter described in this specification can be embodied in a computer-implemented method, the computer-implemented method comprising capturing multiple consecutive images through a camera on a user device for display in an application environment of an application executed on the user device; performing an object recognition process on the images, the object recognition process comprising: determining that multiple images each depicting a specific object are required to perform object recognition on the specific object; and in response to the determination, generating a user interface element within the application environment indicating a camera operation to be performed, the camera operation capturing two or more images; determining that a user has caused the indicated camera operation to be performed to capture two or more images in response to the user interface element, and in response, determining whether the specific object is positively identified from the multiple images; and in response to determining that the specific object is positively identified, eliminating the user interface element indicating the camera operation to be performed and generating a user interface element indicating positive identification of the specific object; and in response to determining that the specific object is not positively identified, maintaining the user interface element indicating the camera operation to be performed. Other embodiments of this aspect include corresponding apparatus, methods, systems, and computer programs encoded on a computer storage device configured to perform the actions of the method.

[0006] These and other aspects may each optionally include one or more of the following features. In some aspects, the user interface element includes a first interface element that indicates that images of the specific object are captured from multiple angular positions relative to the camera's current angular position to the specific object. In some aspects, the user interface element includes a second user interface element that is different from the first user interface element and indicates that images of the specific object are captured from specific angular positions relative to the camera's current angular position to the specific object.

[0007] In some aspects, the user interface element indicates that an image of a particular object is captured at a zoom level that is different than a current zoom level of the camera.

[0008] In some aspects, determining whether a particular object is positively identified from the plurality of images comprises: sending the collected plurality of images to an object recognition server via an application executed on a user device; and receiving an indication from the object recognition server at the user device whether the particular object is positively identified. In some aspects, sending the collected plurality of images to the object recognition server comprises: determining, based on bandwidth criteria, a number of images from the plurality of images depicting the particular object from a particular angle; and sending the number of images to the object recognition server.

[0009] In general, another innovative aspect of the subject matter described in this specification can be embodied in a computer-implemented method, the computer-implemented method comprising receiving, by an image processing component, an image depicting an object obtained from a camera device; determining that a specific object is not positively identified from the image, and in response: providing data causing an application environment of an application executed on a user device in which successive images captured by a camera on the user device are displayed to display a user interface element indicating a camera operation to be performed; in response to the camera operation being performed, receiving multiple images captured by the performed camera operation; determining whether the specific object is positively identified from the multiple images; and in response to determining that the specific object is positively identified, providing data causing the user interface element indicating the camera operation to be performed to be eliminated and causing the user interface element indicating the positive identification of the specific object to be displayed in the application environment; and in response to determining that the specific object is not positively identified, providing data causing the user interface element indicating the camera operation to be performed to be maintained within the application environment. Other aspects of this aspect include corresponding apparatus, methods, systems, and computer programs encoded on a computer storage device configured to perform the actions of the method.

[0010] These aspects and other aspects may each optionally include one or more of the following features. In some aspects, determining whether a particular object is positively identified from multiple images includes: iteratively updating a weighted average until a weighted average threshold is exceeded. In some aspects, each iteration includes: performing object recognition on each image to determine whether a particular object is identified as one or more identified objects, wherein: for each particular object identified as an identified object of a particular image, a weight is assigned to the one identified object of the particular image; for each particular object identified as two or more identified objects of the particular image, an equal weight is assigned to each of the two or more identified objects of the particular image; determining whether the weighted average exceeds the weighted average threshold.

[0011] In some aspects, determining whether a particular object is positively identified from the plurality of images includes: creating a composite image from the plurality of images; and performing object recognition on the composite image.

[0012] In some aspects, determining whether a particular object is positively identified from the plurality of images includes: creating a panoramic image from the plurality of images; and performing object recognition on the panoramic image.

[0013] In some aspects, providing data for causing an application environment of an application executed on a user device in which continuous images captured by a camera on the user device are displayed to display a user interface element indicating a camera operation to be performed includes: determining one of a plurality of camera operations to be performed, wherein each camera operation has a corresponding user interface element that indicates the camera operation and is different from each user interface element used for each other camera operation; and providing data to the user device for causing the user interface element for the determined one of the plurality of camera operations to be displayed.

[0014] In general, another innovative aspect of the subject matter described in this specification can be embodied in a computer-implemented method, the computer-implemented method comprising providing an image captured by a camera on a user device; sending the image to an object recognition server at a first frequency and from the user device; receiving an indication at the user device and from the object recognition server that a specific object within the image was not positively identified from the image sent to the server; in response to receiving the indication that the specific object was not positively identified, sending the image to the object recognition server at a second frequency higher than the first frequency; receiving an indication at the user device from the object recognition server that the specific object was positively identified; and in response to receiving the indication that the specific object was positively identified, ceasing to send the image from the user device to the object recognition server at the second frequency. Other aspects of this aspect include corresponding apparatus, methods, systems, and computer programs encoded on a computer storage device configured to perform the actions of the method.

[0015] These and other aspects may each optionally include one or more of the following features. In some aspects, the method may further include: in response to receiving an indication that a specific object was not positively identified, generating a user interface element indicating a camera operation to be caused by a user during the object recognition process, wherein the camera operation involves capturing two or more images, and wherein the step of sending the images to the object recognition server at a second frequency is conditional on the performance of the camera operation. In some aspects, the method may further include: in response to receiving an indication that a specific object was positively identified, eliminating the user interface element indicating the camera operation to be performed, and generating a user interface element indicating positive identification of the specific object.

[0016] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages. Providing a real-time user interface element or visual indicator to indicate to a user on a mobile device displaying an image of interest that an object recognition process is in progress allows the user to provide feedback and assist the object recognition process. Presenting user interface elements for intermediate steps in the object recognition process to the user can help identify potential problems in the object recognition process during one or more of the steps. In addition, if the object recognition process fails to identify an object, using user interface elements can help the user understand at which point the object recognition process failed. This reduces or eliminates the need to instrument the object recognition process to provide, for example, a data record describing pass events and fail events during the process. Therefore, the technical field of object recognition processing and process troubleshooting is improved.

[0017] In addition, if it is determined that the first image is not suitable for the object recognition process, providing visual indicators about the multiple angles from which the object was acquired can assist the user in selecting a second image. Visual indicators selected to represent the steps of the object recognition process can be built around a human-computer interaction model to provide an improved intuitive user experience. In addition, introducing latency into the object recognition process can allow the user to more easily visually understand a process that otherwise may occur too quickly for the user to understand.

[0018] According to some aspects described herein, bandwidth usage can be controlled by guiding a user through a user interface element in a user interface to capture key image frames (e.g., different angles), and only sending selected key frames to a server for analysis for object recognition. This can result in more efficient use of bandwidth and computing resources.

[0019] Utilizing conventional user interfaces, the user may not know to move the camera of the user device to different angles of the candidate object in order to perform object recognition. An effective user interface will ideally enable the user to easily and quickly access the correct data / functions. The object recognition application of the present system can present visual instructions to the user to obtain images of the object from different angles. For example, a user can use an object recognition application to view an object of interest on a mobile device, but the current angle or view of the object cannot use the current image data to identify the object. The object recognition application presents user interface elements (e.g., virtual indicators, arrows, etc.) to the user through a user interface element generator to obtain an image of the object from another angle. Therefore, the present system enables users to easily and quickly access the angles of the object with easy and fast navigation, particularly on small screen devices.

[0020] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a block diagram of an example environment in which an object recognition process for multiple images displayed on a mobile device can occur on the mobile device.

[0022] Figure 2 is a flow chart of an example process for performing object recognition on an image displayed on a mobile device.

[0023] Figure 3 is a block diagram of an example environment in which an object recognition process for multiple images displayed on a mobile device can occur between a server and the mobile device.

[0024] Figure 4 is a flow diagram of an example process 400 for performing object recognition on multiple images from a mobile device.

[0025] Figure 5 is a flow chart of an example process for transmitting image data between a mobile device and a server for object recognition.

[0026] Figure 6 is a block diagram of an example computer system that may be used to implement the methods, systems, and processes described in this disclosure.

[0027] Like reference numerals and designations in the various drawings refer to like elements. DETAILED DESCRIPTION

[0028] Described below are systems and methods for generating a user interface element within an application environment where multiple images are displayed continuously on a user device, the user interface element indicating a camera operation to be invoked by a user during an object recognition process performed on the image. An image may be a single image captured by a camera operation, or a constituent image of a video captured by a video process. Thus, when used in this specification, the term "image frame" may be a single image, such as a still image of a photograph, or may be an "image frame," such as one of multiple frames from a video. For simplicity, the term "image frame" is used for both image types.

[0029] In some aspects, the technology relates to systems and methods for receiving image frames to determine whether a particular object is positively identified from a plurality of image frames and determining whether a user interface element guiding a user to capture additional image frames of the particular object should be maintained. Alternatively, the system can determine whether a user interface element should be shown to guide the user to obtain a different image of the object.

[0030] In particular, the present technology focuses on object recognition applications that use user interface elements to guide users to take multiple images (e.g., from multiple angles or multiple zoom levels), as well as the underlying technology that fuses the results from these individual frames. In some aspects, the object recognition application on the mobile device can perform the object recognition process and select and generate user interface elements to be displayed on the mobile device. In other aspects, the server can receive several images (received within a short time frame) from the user device and then perform object recognition to determine whether a specific object is positively identified from multiple image frames. In some aspects, the server can also determine which user interface elements should be displayed to the user and send this information to the object recognition application on the user device.

[0031] Performing object recognition may include a weighted average routine, a composite image routine, a panoramic image routine, or a combination thereof. The weighted average routine may be used to calculate a percentage of likelihood for each individual frame and calculate whether the weighted average is above a certain threshold. The composite image routine may create a composite image from the image frames using known techniques to create a higher definition image (e.g., reduce glare, remove artifacts), and then perform object recognition with a higher likelihood of positively identifying a specific object. The panoramic image routine may be used to create a wide angle view around a specific object, and then perform object recognition with a higher likelihood of positively identifying a specific object. For example, the panoramic image routine may identify a certain object from a specific angle, such as a unique pocket representing a specific brand on the other side of a wallet.

[0032] In some aspects, image frames from a camera of a mobile device may be sent to an object recognition server at a first frequency (e.g., every tenth image frame) until such time that the server determines that a particular object cannot be positively identified from one or more image frames sent to the server. In response, images may be sent from the mobile device to the server at a second frequency (e.g., the second frequency may be every second image frame) that is higher than the first frequency until such time that a positive identification can be made. This increase in frequency may be dependent on the performance of a camera operation that may be indicated by a user interface element.

[0033] To perform the above operations, the system and method utilize an object recognition application to display user interface elements on a mobile device to recognize objects from different angles. In particular, the system includes an object recognizer component and a user interface element generator component. Each component can reside on a mobile device, on a server, or a combination thereof. In operation, the object recognizer component processes an image or multiple images to recognize an object of interest. The user interface element generator component creates a user interface element displayed on a user interface of the mobile device.

[0034] In some aspects, the method described herein is performed entirely on a mobile device. In other aspects, the system may include an object recognition system connected to a mobile device via a network. In particular, the object recognition system may include a back-end server having an object recognition engine and a user interface element selector engine. The object recognition engine may process one or more images sent from a mobile device and identify an object of interest. The user interface element selector engine may send data to a user interface element generator within an object recognition application on a mobile device to generate a user interface element to instruct a user to obtain additional images of an object of interest from different angles.

[0035] These and additional features are described in more detail below.

[0036] Figure 1 is a block diagram of an example environment 100 in which an object recognition application 120 can perform an object recognition process on a plurality of image frames displayed in an application environment of a mobile device. Figure 1 Also depicted is an example screenshot 130 of the mobile device 110 showing an object 140 within an application environment of the object recognition application 120 .

[0037] The object recognition application 120 may be installed on and / or executed by the mobile device 110. The mobile device 110 is an electronic device capable of requesting and receiving resources over a data communication network. Example mobile devices 110 include smart phones, tablet computing devices, wearable computing devices (e.g., smart watches), and other devices capable of sending and receiving data over a data communication network.

[0038] The mobile device 110 may also run various applications. The applications on the mobile device 110 may include an application environment, such as a graphical user interface (GUI) 115, in which images may be shown. Examples of such applications are camera-enabled applications that may capture images using a camera on the device, such as camera 112, applications that may receive images and display them, and applications that may access and display images stored on the mobile device 110. For example, an application may access a repository of image data 114 stored on the mobile device 110, wherein the application environment may load an image from the image data 114.

[0039] The object recognition application 120 may be implemented as a native application developed for a specific platform or a specific device, a web browser providing a web interface, or another appropriate type of application. The object recognition application 120 performs object recognition on an image or multiple image frames within an application environment on the mobile device 110. For example, the object recognition application 120 may include an object recognizer 122 that attempts to detect and recognize (e.g., identify) objects in pixel data (or images). The object recognizer 122 may use edge detection and / or other object recognition techniques to detect various objects, such as people, barcodes, landmarks, paintings, vehicles, and / or other types of objects. The object recognizer 122 may access the displayed image data by means of an application programming interface (API).

[0040] The object recognition application 120 can generate a user interface element within an application environment in which multiple images are continuously displayed, and the user interface element indicates the camera operation to be caused by the user during the object recognition process. For example, the object recognition application 120 may include a user interface element generator 124, which generates, updates and presents user interface elements 126A-126C (hereinafter generally referred to as user interface elements 126). The interface element generator 124 can display the user interface element 126 through the user interface 115 on the mobile device 110. For illustrative purposes, specific user interface elements 126A, 126B and 126C are shown at the same time. For example, user interface element 126A can be generated separately to cause the user to obtain an image from one side of the object pointed to by the arrow. User interface element 126B can be generated separately to cause the user to obtain an image from one side of the object pointed to by the arrow, which side is the opposite side to which the user interface element 126A points. User interface element 126C can be generated separately to indicate the positive recognition of object 140 to the user.

[0041] In some aspects, a combination of user interface elements 126 may be shown simultaneously to cause a particular camera operation. For example, multiple indicators may be shown to make it clear to the user that multiple different angular positions may be required to produce a panoramic image, for example. According to some aspects, the user interface element may be an animation, a video, an audio instruction, an instruction to the user in a text box, any other type of media content, or any combination thereof.

[0042] The user interface element generator 124 can select and configure the user interface element 126 based on the object recognition process. For example, the screenshot 130 of the mobile device 110 depicts the object 140 within the application environment of the object recognition application 120. After the object 140 is shown using the object recognition application 120, and if the object is not positively recognized, the user interface element generator 124 can generate and display the user interface element 126A or 126B, for example, to teach the user to capture images from angular positions of different angles relative to the current angular position of the camera to a particular object. When the object is positively recognized, the user interface elements such as 126A or 126B that teach the camera operation to be performed are eliminated, and the user interface element 126C can be generated and displayed. The user interface element 126C indicates that there is a positive recognition of the object 140 during the object recognition process.

[0043] Figure 2 is a flow chart of an example process 200 for performing object recognition on multiple images displayed on a mobile device.

[0044] Process 200 provides a plurality of images for display in an application environment of an application executed on a user device, each of the images being one of a set of consecutive image frames captured by a camera on the user device (202). Figure 1 As shown, an object recognition application 120 is being executed and images from image frames captured by a camera 112 of a mobile device 110 are displayed on a user interface 115. The images so collected may be stored locally on the mobile device 110 in a repository of image data 114, such as in a photo application on the mobile device 110. The images may also be stored on a user's cloud service or cloud-based image application. The images may also be "live" images, where each image is composed of pixel data received by the camera in the active viewport of the camera application environment on the mobile device 110, without capturing the image and saving it locally or on cloud-based storage.

[0045] During the object recognition process, process 200 determines that multiple image frames each depicting a particular object are needed to perform object recognition on the particular object (204). For example, object recognizer 122 of object recognition application 120 on mobile device 110 can access and execute multiple different object recognition models to perform object recognition on object 140.

[0046] In response to determining that multiple image frames are required to perform object recognition on a particular object, process 200 generates a user interface element within an application environment in which multiple images are displayed continuously, the user interface element indicating camera operations to be performed, wherein each camera operation involves capturing two or more image frames (206). For example, Figure 1 As shown in screenshot 130 of , user interface element generator 124 creates user interface element 126A to indicate to the user that the image frame is captured from a different angular perspective.

[0047] Process 200 determines that the user has caused the indicated camera operation to be performed in response to the user interface element (208). After determining that the user has caused the indicated camera operation to be performed in response to the user interface element, process 200 collects a plurality of image frames captured by the performed camera operation (210). For example, after displaying user interface element 126A to the user, the system can detect whether the user has acquired a new image using camera 112 of mobile device 110.

[0048] Process 200 determines whether a particular object is positively identified from a plurality of image frames captured after the user causes the indicated camera operation to be performed (212). If the determination on the decision branch at step 212 is "yes", process 200 proceeds to eliminate the user interface element indicating the camera operation to be performed (214). Process 200 then generates a user interface element indicating positive identification of the particular object (216). For example, as shown in screenshot 130, if user interface element 126A was initially shown to the user to capture additional data from the indicated angular position, and object 140 was positively identified, object recognition application 120 would remove user interface element 126A and display user interface element 126C, such as "Positive ID".

[0049] If the determination on the decision branch at step 212 is "no", then process 200 proceeds to maintain the user interface element indicating the camera operation to be performed (218). Process 200 then returns to step 208 to determine that the user has caused the indicated camera operation to be performed in response to the user interface element. For example, if user interface element 126A was initially shown to the user to capture additional data from the indicated angular position, and object 140 was not positively identified, object recognition application 120 will maintain user interface element 126A. In some aspects, object recognition application 120 may display additional user interface elements 126 to make it clearer to the user that additional image frames may be required for positive identification.

[0050] In accordance with some aspects, the user interface element may include a first interface element that indicates that image frames of a particular object are captured from multiple angular positions relative to the current angular position of the camera to the particular object. In accordance with some aspects, the user interface element includes a second user interface element that is different from the first user interface element and indicates that image frames of the particular object are captured from a specific angular position relative to the current angular position of the camera to the particular object. For example, the user interface element may be an animated image that indicates that multiple pictures are taken from several different angles, rather than just one other angle as shown for user interface element 126A or 126B in screenshot 130. The user interface element as an animated image may depict a person holding a mobile device or other camera device over a real-time view of the application environment, such as, for example, an augmented reality (AR) sticker. For example, an animated character may be shown walking around a particular object and collecting image frames as they walk around. Other indicators may also be used for user interface element 126 depending on the camera operation that the system determines is needed to identify the object. In some embodiments, the interface element is shown adjacent to the detected object, such as Figure 1 Element 126A in. In other embodiments, the interface element may be descriptive text shown at a fixed location, for example, at the bottom of the camera window and labeled "Take images of the side and back of the object you are trying to detect."

[0051] According to some aspects, user interface element 126 may include capturing an image frame of a particular object at a zoom level different from the current zoom level of the camera. For example, user interface element 126 may be an icon, animation, etc., which shows that the user wants to zoom in or out.

[0052] According to some aspects, the processes described herein are all performed on the mobile device 110. However, according to some aspects, determining whether a particular object is positively identified from a plurality of image frames includes sending the collected plurality of image frames to an object recognition server through an application executed on a user device, and receiving an indication from the object recognition server at the user device whether the particular object is positively identified. According to some aspects, sending the collected plurality of image frames to the object recognition server includes: determining, based on bandwidth criteria, a plurality of image frames from a plurality of image frames depicting a particular object from a particular angle, and sending the plurality of image frames to the object recognition server. These aspects of using an object recognition server, also referred to herein as an object recognition engine, which is a backend server within an object recognition system, are further described below.

[0053] Figure 3 3 is a block diagram of an example environment 300 in which an object recognition process for multiple image frames displayed on a mobile device may occur. A computer network 302, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, connects mobile devices 110 and an object recognition system 310. The online environment may include thousands of mobile devices 110.

[0054] The object recognition system 310 can perform object recognition for the mobile device 110 based on image data received from the mobile device 110. In some aspects, the object recognition system 310 can select which user interface elements to be displayed by the mobile device 110. The object recognition system 310 includes one or more front-end servers 312 and one or more back-end servers 314. The front-end server 312 can receive image data from the mobile device 110 and provide the data to the back-end server 314. The front-end server 312 can also send data to the mobile device 110 in response to receiving the image data.

[0055] The backend server 314 includes an object recognition engine 315 that can perform an object recognition process in response to image data received from the mobile device 110. As used herein, the term engine refers to a data processing device that performs a set of tasks. The object recognition engine 315 can use object data from an object data storage unit 316 (e.g., one or more hard drives, flash memory, etc.), which stores data that the object recognition engine 315 can use to perform object recognition.

[0056] In some aspects, the backend server 314 includes a user interface element selection engine 317 that can select one or more user interface elements that are most relevant to the image data received from the mobile device 110 from the data stored in the user interface element data storage unit 318. The user interface element selection engine 317 can then provide the data to the user interface element generator 124 of the object recognition application 120 of the mobile device 110 to generate the selected user interface element to be displayed on the user interface 115.

[0057] Figure 4 is a flow chart of an example process 400 for object recognition of multiple images from a mobile device. Figure 3 Process 400 is implemented by object recognition system 310 in communication with mobile device 110.

[0058] Process 400 receives an image frame obtained from a camera device depicting an object (402). For example, object recognition system 310 may receive an image captured by camera 112 from mobile device 110 via network 302.

[0059] Process 400 determines that a particular object is not positively identified from the image frames (404), and provides data that causes an application environment of an application executed on a user device in which successive image frames captured by a camera on the user device are displayed to display a user interface element indicating a camera operation to be performed (406). For example, object recognition engine 315 can analyze images received from mobile device 110 and perform object recognition. If object recognition engine 315 does not positively identify an object, such as object 140, user interface element selection engine 317 can provide data to object recognition application 120 to generate a user interface element such as user interface element 126A (e.g., Figure 1 ) to indicate the camera operation that the user wants to perform.

[0060] In response to the camera operation being performed, the process 400 receives a plurality of image frames captured by the camera operation performed (408). For example, in presenting the user Figure 1 After selecting the user interface element 126A, the user then uses the camera 112 of the mobile device 110 to capture additional images. The object recognition application 120 will then trigger the mobile device to send the newly acquired images to the object recognition system 310.

[0061] Process 400 determines whether a specific object is positively identified from a plurality of image frames captured after the user causes the indicated camera operation to be performed (410). If the determination on the decision branch at step 410 is "yes", process 400 proceeds to provide data causing the user interface element indicating the camera operation to be performed to be eliminated (412). Then, process 400 provides data causing the user interface element indicating positive identification of the specific object to be displayed in the application environment (414). For example, Figure 1 As shown in screenshot 130 in , if user interface element 126A is shown to the user to capture additional data from the indicated angular position, and object recognition engine 315 positively identifies object 140, user interface element selection engine 317 will send data to user interface element generator 124 of object recognition application 120 to remove user interface element 126A and display user interface element 126C, “Positive ID”. This will indicate to the user that object 140 is positively identified.

[0062] If the determination of the decision branch at step 410 is "no", the process 400 proceeds to provide data (416) that causes a user interface element indicating the camera operation to be performed to be maintained within the application environment. The process 400 then returns to step 408 to receive a plurality of image frames captured by the camera operation performed in response to the camera operation being performed. For example, if the user interface element 126A is initially shown to the user to capture additional data from the indicated angular position, and the object recognition engine 315 does not positively identify the object 140, the user interface element selection engine 317 sends data to the user interface element generator 124 of the object recognition application 120 to maintain the user interface element 126A. In some aspects, the user interface element selection engine 317 may send data to the user interface element generator 124 to generate additional user interface elements 126 to make it more obvious to the user that additional image frames are needed for positive recognition. In some aspects, the object recognition application 120 may determine whether to display the additional user interface element 126.

[0063] According to some aspects, performing object recognition and determining whether a particular object is positively recognized from multiple image frames can include a weighted average routine, a composite image routine, a panoramic image routine, or a combination thereof. According to some aspects, these routines can be performed by Figure 1 According to some aspects, these routines can be performed by the object recognition application 120 on the mobile device 110. Figure 3 The object recognition engine 315 executes.

[0064] According to some aspects, determining whether a particular object is positively identified from a plurality of image frames may include iteratively updating a weighted average until a weighted average threshold is exceeded. Each iteration may include performing object recognition on each image frame to determine whether the particular object is identified as one or more identified objects. For each particular object identified as one identified object of a particular frame, a weight may be assigned to the one identified object of the particular frame. For each particular object identified as two or more identified objects of a particular frame, a distribution weight may be assigned to each of the two or more identified objects of the particular frame. Finally, if the weighted average exceeds the weighted average threshold, it may be determined whether the particular object is positively identified from the plurality of image frames. For example, for four frames of image data, the object recognition analysis may determine that for Figure 1 , frame 1 is 85% vendor-1 package, frame 2 is 50% vendor-1 package and 50% vendor-2 package, frame 3 is 90% vendor-1 package, and frame 4 is 50% vendor-1 package and 50% vendor-3 package. For this example, the weighted average threshold is 60%. Therefore, for this example of four image frames, the weighted average will be calculated as 68.75% for the vendor-1 package. Based on the weighted average threshold, the weighted average routine will determine that object 140 is a vendor-1 package. In some aspects, a user interface element can be displayed that will provide the name of the vendor-1 package as well as additional information about the product, such as hyperlinks to multiple web resources related to the product or native applications installed on the mobile device 110.

[0065] According to some aspects, determining whether a particular object is positively identified from the plurality of image frames may include creating a composite image from the plurality of image frames and performing object recognition on the composite image. For example, known techniques for creating high-quality composite images may be used to combine such Figure 1 The composite image may be combined with multiple frames of a particular object of the object 140 to reduce glare, remove artifacts, etc. This higher quality composite image may then provide a higher likelihood of positive identification in the object recognition analysis.

[0066] According to some aspects, determining whether a particular object is positively identified from the plurality of image frames can include creating a panoramic image from the plurality of image frames and performing object recognition on the composite image. For example, the system can generate a user interface element to indicate to the user that they need to walk around the object and obtain images from all around the object to create the panoramic image for analysis.

[0067] According to some aspects, providing an application environment of an application executed on a user device in which successive image frames captured by a camera on the user device are displayed to display data indicating a user interface element of a camera operation to be performed includes determining one of a plurality of camera operations to be performed. Each camera operation has a corresponding user interface element that indicates the camera operation and is different from each user interface element for each other camera operation. Then, data can be provided to the user device, which causes a user interface element for the determined one of the plurality of camera operations to be displayed on the user device. For example, the user interface element selection engine 317 can determine which user interface element should be selected from the user interface element data storage unit 318 based on an object recognition analysis of the image data from the object recognition engine 315. Then, the user interface element selection engine 317 can send data to the user interface element generator 124 to generate and display the selected user interface element on the user interface 115 of the mobile device 110.

[0068] Figure 5 is a flow chart of an example process 500 for object recognition of multiple images from a mobile device. Figure 3 The object recognition system 310 communicates with the mobile device 110 to implement the process 500 .

[0069] Process 500 provides image frames captured by a camera on a user device (502). For example, when object recognition application 120 is executed on mobile device 110, images from image frames captured by camera 112 of mobile device 110 can be displayed on user interface 115.

[0070] Process 500 sends image frames from a user device to an object recognition server at a first frequency (504). For example, mobile device 110 communicates with object recognition system 310 via network 302, and object recognition application 120 can send images to object recognition engine 315 at a first frequency. For example, images can be sent at one frame per second (FPS).

[0071] Process 500 receives at the user device an indication from the object recognition server that a particular object within the image frame was not positively identified from one or more image frames sent to the server (506). For example, object recognition engine 315 may send a notification to object recognition application 120 that an object in the image frame was not identified.

[0072] In response to receiving an indication that a particular object was not positively identified, process 500 sends image frames to an object recognition server at a second frequency that is higher than the first frequency (508). For example, after receiving a notification from object recognition system 310 that object recognition engine 315 was unable to identify an object, object recognition application 120 on mobile device 110 can adjust the rate at which images are sent at the second frequency. For example, the next set of images can be increased and sent at three frames per second (FPS). According to some aspects, object recognition system 310 can request a specific frequency for the images to be sent. According to some aspects, object recognition application 120 can determine what frequency the second frequency should be adjusted to.

[0073] Process 500 receives an indication from an object recognition server at a user device that a particular object is positively recognized (510). For example, once object recognition engine 315 positively recognizes an object in an image frame, object recognition system 310 sends a notification of the positive recognition to object recognition application 120.

[0074] In response to receiving an indication that a particular object is positively identified, process 500 stops sending image frames from the user device to the object recognition server at a second frequency (512). For example, after object recognition application 120 receives notification of positive identification from object recognition system 310, object recognition application 120 stops sending new images.

[0075] According to some aspects, in response to receiving an indication that a particular object was not positively identified, a user interface element can be generated indicating a camera operation to be caused by the user during the object recognition process, wherein the camera operation involves capturing two or more image frames. According to some aspects, the step of sending the image frames to the object recognition server at a second frequency is conditioned on the performance of the camera operation. For example, the object recognition application 120 will first determine whether the user has performed the indicated camera operation before sending the image to the object recognition system 310. This action will reduce bandwidth because only newly acquired images that the system determines to be obtained by using the user interface element that teaches the user which images to capture will then be sent from the mobile device 110 to the object recognition system 310.

[0076] According to some aspects, in response to receiving an indication that a particular object is positively identified, a user interface element indicating a camera operation to be performed is eliminated and a user interface element indicating positive identification of the particular object is generated. Figure 1 As shown and discussed herein, user interface element 126A may be initially shown to the user to obtain more images, then after the object recognition engine recognizes the object, user interface element 126A may be removed, and then user interface element 126C may be added to indicate to the user that the object was positively recognized.

[0077] Additionally, in some embodiments, the user interface and visual indicators may be customized depending on the type of object the user is trying to identify. In particular, for a first object vertical, a first user interface and a first visual indicator may be used, and for a second object vertical that is different from the first object vertical, a second user interface and a second user indicator that are different from the first user interface and the first visual indicator may be used. For example, if the user is trying to identify a plant, the user interface may initially identify that the object is a plant, but it is not possible to determine which plant it is without acquiring an image of a specific perspective or specific features of the plant. For example, the user interface may instruct the user to first acquire an image of the flower portion of the plant, then acquire an image of the leaves, and then finally acquire an image of the roots. For certain objects, the visual indicators may be customized to provide specific angles or areas of the object. For example, for a shoe, the visual indicator may instruct the user to acquire an image from the side, a top view, and then a close-up of the logo. All of this information may then be used by the system described herein to improve the object recognition process.

[0078] Figure 6 6 is a block diagram of an example computer system 600 that can be used to perform the above operations. System 600 includes a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of the processor 610, the memory 620, the storage device 630, and the input / output device 640 can be interconnected, for example, using a system bus 650. The processor 610 is capable of processing instructions for execution within the system 600. In one embodiment, the processor 610 is a single-threaded processor. In another embodiment, the processor 610 is a multi-threaded processor. The processor 610 is capable of processing instructions stored in the memory 620 or on the storage device 630.

[0079] The memory 620 stores information within the system 600. In one embodiment, the memory 620 is a computer readable medium. In one embodiment, the memory 620 is a volatile memory unit. In another embodiment, the memory 620 is a non-volatile memory unit.

[0080] The storage device 630 can provide mass storage for the system 600. In one embodiment, the storage device 630 is a computer-readable medium. In various aspects, the storage device 630 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices over a network (e.g., a cloud storage device), or some other mass storage device.

[0081] The input / output device 640 provides input / output operations for the system 600. In one embodiment, the input / output device 640 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another embodiment, the input / output device may include a driver device configured to receive input data and send output data to other input / output devices, such as a keyboard, a printer, and a display device 660. However, other aspects may also be used, such as a mobile computing device, a mobile communication device, a set-top television client device, etc.

[0082] Although already Figure 6 An example processing system is described in the specification, but aspects of the subject matter and functional operations described in this specification may be implemented in other types of digital electronic circuitry or computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them.

[0083] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transient program carrier for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), which is generated as encoded information to be transmitted to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or sequential access memory, or a combination of one or more of them.

[0084] The term "data processing apparatus" covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. The apparatus may include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0085] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program being targeted, or in multiple collaborative files (e.g., files storing some portions of one or more modules, subroutines or codes). A computer program may be deployed to execute on one or more computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0086] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit (e.g., FPGA (field programmable gate array), ASIC (application-specific integrated circuit) or GPGPU (general purpose graphics processing unit)), and the device can be implemented as the dedicated logic circuit.

[0087] The computer suitable for executing a computer program includes, for example, a central processing unit that can be based on a general or special microprocessor or both, or any other type. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for completing or executing instructions and one or more memory devices for storing instructions and data. In general, a computer will also include one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data or will be operatively connected to one or more large-capacity storage devices to receive data from it or transfer data to it or receive and transmit both. However, a computer does not have to have such a device. In addition, a computer can be embedded in another device (e.g., to name a few examples, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive)).

[0088] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0089] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device through which the user can provide input to the computer, such as a mouse or trackball. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0090] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface and a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0091] A computing system may include clients and servers. Generally speaking, clients and servers are remote from each other and typically interact through a communication network. The relationship of client and server is formed by computer programs running on respective computers and having a client-server relationship to each other.

[0092] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on any invention or the scope of possible protection, but should be interpreted as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in the context of a separate embodiment in this specification may also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. In addition, although features may be described as working in certain combinations as described above and initially requested for protection, in some cases, one or more features from the combination requested for protection may be excluded from the combination, and the combination requested for protection may point to a variant of a sub-combination or a sub-combination.

[0093] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order or requiring all of the operations shown to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, but it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0094] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some aspects, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method comprising: capturing, by a camera on a user device, a plurality of sequential images for display in an application environment of an application executing on the user device; An object recognition process is performed on the image, the object recognition process comprising: determining that a plurality of images, each depicting a particular object, are required to perform object recognition on the particular object; and responsive to the determining, generating within the application environment a user interface element indicating a camera operation to be performed, the camera operation capturing two or more images; determining that the user has caused an indicated camera operation to be performed to capture two or more images in response to the user interface element, and in response, determining whether a particular object is positively identified from the plurality of images; and In response to determining that the specific object is positively identified, eliminating the user interface element indicating a camera operation to be performed and generating a user interface element indicating positive identification of the specific object; and In response to determining that the particular object is not positively identified, maintaining the user interface element indicating a camera operation to be performed.

2. The computer-implemented method of claim 1 , wherein: The user interface elements include a first user interface element that indicates that images of the particular object are captured from a plurality of angular positions relative to a current angular position of the camera to the particular object.

3. The computer-implemented method of claim 2, wherein: The user interface elements include a second user interface element that is different from the first user interface element and indicates that an image of the specific object is captured from a specific angular position relative to a current angular position of the camera to the specific object.

4. The computer-implemented method of claim 1 , wherein: The user interface element indicates that an image of the particular object is captured at a zoom level different from a current zoom level of the camera.

5. The computer-implemented method of claim 1 , wherein: Determining whether a particular object is positively identified from the plurality of images includes: sending the collected plurality of images to an object recognition server by the application executed on the user device; and An indication is received at the user equipment from an object identification server whether the particular object is positively identified.

6. The computer-implemented method of claim 5, wherein: Sending the collected plurality of images to the object recognition server comprises: determining, based on a bandwidth criterion, a number of the plurality of images that depict the particular object from a particular angle; and The plurality of images are sent to the object recognition server.

7. A computer-implemented method comprising: receiving, by an image processing component, an image depicting an object obtained from a camera device; as well as A determination is made that a particular object is not positively identified from the image, and in response: providing data causing an application environment of an application executed on a user device in which successive images captured by the camera on the user device are displayed to display a user interface element indicating a camera operation to be performed; In response to the camera operation being performed, receiving a plurality of images captured by the performed camera operation; determining whether a particular object is positively identified from the plurality of images; as well as In response to determining that the specific object is positively identified, providing data causing a user interface element indicating a camera operation to be performed to be eliminated and causing a user interface element indicating positive identification of the specific object to be displayed in the application environment; as well as In response to determining that the particular object is not positively identified, data is provided that causes a user interface element indicative of a camera operation to be performed to be maintained within the application environment.

8. The computer-implemented method of claim 7, wherein: Determining whether the specific object is positively identified from the plurality of images includes: Iteratively update the weighted average until it exceeds the weighted average threshold. Each iteration includes: Object recognition is performed on each image to determine whether the particular object is recognized as one or more recognized objects, wherein: For each specific object identified as an identified object of a specific image, assigning a weight to the identified object of the specific image; for each specific object identified as two or more identified objects of a specific image, assigning a distributed equal weight to each of the two or more identified objects of the specific image; and Determine whether the weighted average exceeds the weighted average threshold.

9. The computer-implemented method of claim 7, wherein: Determining whether the specific object is positively identified from the plurality of images includes: creating a composite image from the plurality of images; and Object recognition is performed on the composite image.

10. The computer-implemented method of claim 7, wherein: Determining whether the specific object is positively identified from the plurality of images includes: creating a panoramic image from the plurality of images; and Object recognition is performed on the panoramic image.

11. The computer-implemented method of claim 7, wherein: Providing data that causes an application environment of an application executed on the user device in which successive images captured by a camera on the user device are displayed to display the user interface element indicating a camera operation to be performed, comprising: determining one of a plurality of camera operations to perform, wherein each camera operation has a corresponding user interface element that indicates the camera operation and is different from each user interface element for each other camera operation; and Data is provided to the user device causing a user interface element for the determined one of the plurality of camera operations to be displayed.

12. A computer-implemented method comprising: Provide images captured by a camera on the user's device; transmitting the image at a first frequency and from the user device to an object recognition server; receiving, at the user device and from the object recognition server, an indication that a particular object within an image was not positively identified from an image sent to the server; in response to receiving an indication that the particular object was not positively identified, sending images to the object recognition server at a second frequency higher than the first frequency; receiving, at the user device, from the object identification server, an indication that the particular object is positively identified; as well as In response to receiving an indication that the particular object is positively identified, sending images from the user device to the object identification server at the second frequency is stopped.

13. The computer-implemented method of claim 12, further comprising: In response to receiving an indication that the particular object was not positively identified, generating a user interface element indicating a camera operation to be induced by the user during an object identification process, wherein the camera operation involves capturing two or more images, and The step of sending the image to the object recognition server at the second frequency is conditional on the execution of the camera operation.

14. The computer-implemented method of claim 13, further comprising: In response to receiving an indication that the specific object is positively identified, a user interface element indicating a camera operation to be performed is eliminated, and a user interface element indicating positive identification of the specific object is generated.

15. A system for object recognition, comprising: Data processing equipment; as well as a non-transitory computer-readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and which, when so executed, causes the data processing apparatus to perform operations comprising: capturing, by a camera on a user device, a plurality of sequential images for display in an application environment of an application executing on the user device; An object recognition process is performed on the image, the object recognition process comprising: determining that a plurality of images, each depicting a particular object, are required to perform object recognition on the particular object; and responsive to the determining, generating within the application environment a user interface element indicating a camera operation to be performed, the camera operation capturing two or more images; determining that the user has caused an indicated camera operation to be performed to capture two or more images in response to the user interface element, and in response, determining whether a particular object is positively identified from the plurality of images; and In response to determining that the specific object is positively identified, eliminating the user interface element indicating a camera operation to be performed and generating a user interface element indicating positive identification of the specific object; and In response to determining that the particular object is not positively identified, maintaining the user interface element indicating a camera operation to be performed.

16. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising: capturing, by a camera on a user device, a plurality of sequential images for display in an application environment of an application executing on the user device; An object recognition process is performed on the image, the object recognition process comprising: determining that a plurality of images, each depicting a particular object, are required to perform object recognition on the particular object; and responsive to the determining, generating within the application environment a user interface element indicating a camera operation to be performed, the camera operation capturing two or more images; determining that the user has caused an indicated camera operation to be performed to capture two or more images in response to the user interface element, and in response, determining whether a particular object is positively identified from the plurality of images; and In response to determining that the specific object is positively identified, eliminating the user interface element indicating a camera operation to be performed and generating a user interface element indicating positive identification of the specific object; and In response to determining that the particular object is not positively identified, maintaining the user interface element indicating a camera operation to be performed.

17. A system for object recognition, comprising: Data processing equipment; as well as a non-transitory computer-readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and which, when so executed, causes the data processing apparatus to perform operations comprising: receiving, by the image processing component, an image depicting an object obtained from a camera device; and A determination is made that a particular object is not positively identified from the image, and in response: providing data causing an application environment of an application executed on a user device in which successive images captured by the camera on the user device are displayed to display a user interface element indicating a camera operation to be performed; In response to the camera operation being performed, receiving a plurality of images captured by the performed camera operation; determining whether a particular object is positively identified from the plurality of images; and In response to determining that the specific object is positively identified, providing data causing a user interface element indicating a camera operation to be performed to be eliminated and causing a user interface element indicating positive identification of the specific object to be displayed in the application environment; and In response to determining that the particular object is not positively identified, data is provided that causes a user interface element indicative of a camera operation to be performed to be maintained within the application environment.

18. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising: receiving, by an image processing component, an image depicting an object obtained from a camera device; as well as A determination is made that a particular object is not positively identified from the image, and in response: providing data causing an application environment of an application executed on a user device in which successive images captured by the camera on the user device are displayed to display a user interface element indicating a camera operation to be performed; In response to the camera operation being performed, receiving a plurality of images captured by the performed camera operation; determining whether a particular object is positively identified from the plurality of images; as well as In response to determining that the specific object is positively identified, providing data causing a user interface element indicating a camera operation to be performed to be eliminated and causing a user interface element indicating positive identification of the specific object to be displayed in the application environment; as well as In response to determining that the particular object is not positively identified, data is provided that causes a user interface element indicative of a camera operation to be performed to be maintained within the application environment.

19. A system for object recognition, comprising: Data processing equipment; as well as a non-transitory computer-readable storage medium in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and which, when so executed, causes the data processing apparatus to perform operations comprising: Provide images captured by a camera on the user's device; transmitting the image at a first frequency and from the user device to an object recognition server; receiving, at the user device and from the object recognition server, an indication that a particular object within an image was not positively identified from an image sent to the server; in response to receiving an indication that the particular object was not positively identified, sending images to the object recognition server at a second frequency higher than the first frequency; receiving, at the user device, from the object identification server, an indication that the particular object is positively identified; and In response to receiving an indication that the particular object is positively identified, sending images from the user device to the object identification server at the second frequency is stopped.

20. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising: Provide images captured by a camera on the user's device; transmitting the image at a first frequency and from the user device to an object recognition server; receiving, at the user device and from the object recognition server, an indication that a particular object within an image was not positively identified from an image sent to the server; in response to receiving an indication that the particular object was not positively identified, sending images to the object recognition server at a second frequency higher than the first frequency; receiving, at the user device, from the object identification server, an indication that the particular object is positively identified; as well as In response to receiving an indication that the particular object is positively identified, sending images from the user device to the object identification server at the second frequency is stopped.

Images