Object classification for image recognition processing

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for receiving user input of an object of interest classification of image data (e.g., a single frame image, a continuous video, etc.) from a user device, and for each object specified by the identified object data belonging to the object of interest classification, displaying data for each object of interest and / or text block on the user device.

Description

Technical Field

[0001] The present application relates to image recognition processing. Background Art

[0002] Many user devices, such as smart phones, are equipped with cameras for capturing pictures and digital images. User devices provide access to a variety of information. This information can be viewed in a web browser or native application running on the user device. Users can also use images to obtain additional information related to objects or locations. For example, digital image files, video and / or audio files, and resources for specific topics are accessible through the Internet.

[0003] A user may desire additional information related to content that the user is viewing on a user device. Such content may be text or images. A user may desire to have an object of interest and / or a block of text highlighted on the user device or to indicate some other feature that the information may be useful to the user. For example, a user may encounter a food menu at a restaurant and desire to know additional information about a particular item of interest, or to highlight a particular subset of items based on a classification. Summary of the invention

[0004] This specification describes technology involving an object classifier that can receive image data and user input on a user device requesting to find items of interest in the image data, determine whether objects and / or text blocks of interest are present within the image data, and display those objects and / or text blocks of interest on the user device.

[0005] In general, one innovative aspect of the subject matter described in this specification can be implemented in a system, the system comprising: a specification executor component, which: receives as input object mapping data from an image recognition processing component, wherein the object mapping data, for each object identified in an image and according to a defined data structure, specifies the area of ​​the image occupied by the object and data identifying the object; receives as input a specification file, the specification file including data specifying a class of objects of interest; determines whether the object mapping data generated by the image recognition processing component specifies an object belonging to the class of objects of interest; and generates as output identified object data, the identified object data specifying each object determined to belong to the class of objects of interest; and a presentation component, which: receives as input the identified object data, and, for each object determined to belong to the class of objects of interest specified by the identified object data, the object mapping data specifying the area of ​​the image occupied by the object; and, for each object specified by the identified object data belonging to the class of objects of interest, generates presentation data for presenting the object on a user device. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs encoded on computer storage devices, configured to perform the actions of the method.

[0006] These and other embodiments can each optionally include one or more of the following features. In some aspects, the system can include an image recognition processing component. The image recognition processing component can receive image data as input, and process the image data depicting an image of multiple objects, and generate object mapping data as output, the object mapping data, according to a defined data structure and for each object identified in the image, specifying the area of ​​the image occupied by the object and data identifying the object.

[0007] In some aspects, the system may include a library generator. The library generator may receive user input specifying one or more object of interest categories; receive user input specifying a rendering mode for rendering each object depicted in the image that is determined to belong to the object of interest category; and generate a specification file based on the user input specifying the one or more object of interest categories and the user input specifying the rendering mode. In some aspects, the data generated by the rendering component to render the object of interest is data generated according to the rendering mode specified in the specification file.

[0008] In some aspects, the presentation component causes the user device to highlight within the image an object depicted in the image that is determined to belong to a class of objects of interest. In some aspects, the presentation component causes the user device to audibly describe an object depicted in the image that is determined to belong to a class of objects of interest.

[0009] In some aspects, the object of interest category defines an interesting text category, and wherein the presentation component causes the user device to highlight within the image an object depicted in the image that is determined to belong to the interesting text category. In some aspects, the object is one or more of text or a real-world physical object.

[0010] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages. By facilitating a platform through which developers can deploy object classifiers and specification executors, the system allows for rapid development and deployment of specialized object recognition and rendering capabilities for end users. This unconventional development and execution architecture brings about technical advances in the field of image processing.

[0011] In some embodiments, the object classifier can receive image data (single frame image, continuous video, etc.) and user input requesting to find items of interest in the image data on the user device, determine whether there are objects and / or text blocks of interest in the image data, and display those objects and / or text blocks of interest on the user device. In some embodiments, displaying objects and / or text blocks of interest can be based on user input defining a rendering mode. The rendering mode can include any type of visual or auditory feedback to signal to the user that the objects and / or text blocks of interest are in the image data. In addition, the object classifier can provide the user with additional information related to the objects and / or text blocks of interest.

[0012] In some embodiments, the more users that use the object classifier system, the more valuable the object classifier system will be by leveraging network effects. The object classifier system described herein is more like a mobile application platform that allows programmers to build various plug-in modules that users can add for access by the object classifier application on the user's device. In some embodiments, a user can make an identification request for an item of interest that may not have been created for a plug-in module, which allows programmers and / or developers to create a plug-in module to meet the requested need.

[0013] Furthermore, by providing the user with an indication of the user's object of interest, which the user might not otherwise notice in an image or in the phone's viewfinder, the object classification system provides the user with information that can assist the user in a number of different ways. For example, by providing auditory feedback as opposed to providing only visual feedback, some people who might have visual impairments (e.g., farsightedness and temporarily without corrective lenses) might be able to distinguish an object of interest or a block of text that they might not otherwise be able to distinguish.

[0014] Furthermore, by using previously used classifier searches or already created modules, the system reduces the number of classifier processing cycles because users are better informed of the information presented to them. This reduces the processing resources required and reduces overall system bandwidth requirements (or, alternatively, enables a greater number of users to be served without a proportional increase in processing resources). Thus, this improvement in the art of search processing technology is another significant advantage achieved by the systems and methods described below.

[0015] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A is a block diagram of an example environment in which an object classifier may be implemented.

[0017] Figure 1B is a system flow diagram of an example user device that may implement an object classifier.

[0018] Figure 2 is a flow chart of an example process for using an object classifier.

[0019] Figure 3 is a block diagram of an example computer system that may be used to implement the methods, systems, and processes described in this disclosure.

[0020] Like reference numbers and numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] Systems, methods, and computer program products are described for receiving user input from a user device for a classification of objects of interest for image data (e.g., a single frame image, a continuous video, etc.), and for each object belonging to the classification of objects of interest specified by the identified object data, displaying data for each object of interest and / or text block on the user device.

[0022] The system can determine how to present objects and / or blocks of text of interest to a user based on the arrangement and / or other visual characteristics of the text or objects within the image. For example, based on user input specifying one or more objects or text of interest and user input specifying a presentation mode (e.g., highlight, illuminate, circle, etc.), the system determines whether the image data contains an object or text of interest to the user. These and other features described herein may provide insight into the context of an image and its text / objects contained therein.

[0023] To perform the above operations, the system and method utilize configurable object classification for image recognition processing. In particular, the system includes a specification executor component and a rendering component. The specification executor component processes a specification file. The specification file includes data specifying an object classification of interest and what rendering will be generated when an object belonging to the classification is identified.

[0024] In operation, the image recognition processing component processes the image and generates object mapping data. The object mapping data specifies the area of ​​the image occupied by the object. The specification executor receives the object mapping data from the image recognition processing component and processes the specification file. If the specification file specifies that the detected object is an object belonging to the object of interest classification, then the specification executor component generates the identified object data specifying each object determined to belong to the object of interest classification. The presentation component receives the identified object data and the data specifying each object determined to belong to the object of interest classification.

[0025] The identified object data is provided to a presentation component that receives the identified object data and the object mapping data. For each object designated by the identified object data as belonging to the object classification of interest, the presentation component generates presentation data for presenting the object on a user device.

[0026] The system may also include a library generator that receives object of interest classifications and / or rendering modes from a developer to generate a specification file and sends the specification file to a specification executor component. When an object depicted in an image is determined to belong to an object of interest classification, the system may identify a specific portion of interest in the image by surrounding the object with a circle, highlighting the object, or the system may audibly describe the object. The received image data may be a single frame photograph or a recorded video. In other embodiments, the received image data will be from a live continuous video requiring the system to perform on-the-fly processing, which will continue to identify (multiple) objects of interest even if the screen moves, as long as the object remains in the viewfinder of the user device.

[0027] These and additional features are described in more detail below.

[0028] Figure 1A is a block diagram of an example environment 100A in which the object classifier 115 may be implemented. Figure 1B is a system flow diagram of an example environment 100B in which the object classifier 115 may be implemented using the user device 110 .

[0029] A computer network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, connects a classification server 104 and user devices 110. The classification server 104 may represent a combination of an application server, a database server, a communication server, a network server, etc., which comprises a mobile platform provider's system for collecting classification data from the various user devices 110 described herein, controlling and managing the classification applications and classification modules used on the various user devices 110 described herein.

[0030] The user device 110 is an electronic device capable of requesting and receiving resources over the network 102. Example user devices 110 include personal computers, mobile communication devices, and other devices capable of sending and receiving data over the network 102. The user device 110 typically includes a user application, such as a web browser or some other application, to facilitate sending and receiving data over the network 102. The user device 110 can present media using the application. Media is some combination of images, video, audio, text, or the like that a user consumes using an application running on the user device 110. A web browser can enable a user to display and interact with text, images, video, music, and other information on a web page, typically located at a website on the World Wide Web or a local area network.

[0031] The library server 104 includes a library generator 150. Although the library generator 150 is shown as being implemented in the library server 104, the library generator 150 may be implemented in some other computer system and the resulting specification file 152 generated by it may be provided to the library server 104. For example, if the library server 104 is part of an application store, the specification file 152 may be developed by utilizing the library generator 150 hosted by a developer server.

[0032] The library generator 150 receives the object of interest classification from the developer to generate the specification file 152. The specification file 152 is later provided to the user device that executes the specification executor 130. The specific object of interest classification will depend on the intent of the specification file. For example, if the specification file 152 is to be used to identify gluten-free menu items, the object of interest classification can be a text item or symbol indicating a gluten-free item. Similarly, if the specification file 152 is to be used to identify a dog in an image, the object of interest classification can be image data that is used by the image processing system so that the image processing system processes the image using a model trained to identify dogs.

[0033] The data indicating the classification of the object of interest stored in the specification file 152 may depend on the image processing system used. For example, if a text processing model receives such data as input, the text classification may include the text to be recognized and the classification of the text. Similarly, the image processing system may access a plurality of different object recognition models, each identified by a unique identifier. If the specification file indicates that a dog is to be identified, then the specification file may store an identifier of a model trained to recognize dogs. Of course, multiple model identifiers may be stored in the specification file 152 for the recognition of multiple different object types.

[0034] The specification files 152 may also specify presentation data. For example, when an object depicted in an image is determined to belong to a class of objects of interest, the system may identify the particular portion of interest in the image by drawing a circle around the object, highlighting the object, or the system may audibly describe the object. Which form of presentation is used is specified by the presentation data. Thus, each specification file 152 may be used to identify a particular object in an image and present a positive identification result in a particular manner, such as specified by its respective developer.

[0035] User device 110 using object classifier system 115 includes image recognition processor 120, specification executor 130, rendering engine 140, and specification file 150. Image recognition processor 120 may be implemented on user device 110, or alternatively, in a separate computer system, such as classification server 105.

[0036] Figure 2 is a flow chart of an example process 200 for using an object classifier.

[0037] Process 200 receives image data depicting an image of a plurality of objects (202). According to some embodiments, the plurality of objects may be one or more of text or real-world physical objects. Figure 1B As shown in , the image recognition processing component 120 of the object classifier 115 can receive image data 116 from the user device 110. The image data 116 can include objects 113A and 113B and text blocks 114A and 114B as viewed within the user interface 112A. For example, the image data 116 can be an image of a menu that can include pictures of menu items, text blocks, or a combination thereof.

[0038] Process 200 processes image data and generates object mapping data that, for each object identified in the image, specifies the area of ​​the image that the object occupies and data identifying the object according to a defined data structure (e.g., X and Y components) or some other data structure that can be used to map to a location in the image (204). Figure 1BAs shown in , the image recognition processing component 120 receives image data 116 from the user device 110 and generates and sends object mapping data 122 to the specification executor 130. For example, the image recognition processor 120 will process the image data 116 of the user interface 112A and identify objects and text blocks in the image, such as objects 113A and 113B and text blocks 114A and 114B, based on the defined data structure.

[0039] Process 200 accesses a specification file that includes data specifying a classification of an object of interest (206). Figure 1B As shown in , specification file 152 is accessed by specification executor 130. Specification file 152 may have been downloaded from library server 104 by a user at an earlier time.

[0040] After receiving the object mapping data (204) and the specification file (206), process 200 determines whether the object mapping data specifies an object that belongs to a class of objects of interest (208). Figure 1B As shown in , the specification executor 130 receives the object mapping data 122 and the specification file 152 from the image recognition processor 120, and then the image recognition processor 120 determines whether any object or text block marked in the object mapping data 122 belongs to the content specified as being of interest by the user in the user input 111. For example, referring to Figure 1B 112A, specification executor 130 may determine whether object 113A, object 113B, text block 114A, and / or text block 114B are identified as content selected by the user as being of interest. Further, for example, returning to the “gluten-free” example, specification executor 130 may determine whether any mapped text object from menu image data 116 is identified as “gluten-free.” For example, a menu item may include the word “gluten-free” or an image symbol indicating that the menu item is gluten-free.

[0041] Process 200 generates identified object data that specifies each object determined to belong to a class of objects of interest (210). As used in this document, identified object data is data indicating that an identified object belongs to a class of objects of interest. Figure 1B, the specification executor 130 generates and sends the identified object data 132 to the rendering component 140. The rendering engine 140 receives the identified object data 132 and, for each object specified by the identified object data 132 that is determined to belong to the class of objects of interest, also receives the object mapping data 122 that specifies the area of ​​the image occupied by the object. For example, continuing with the example below, where the text block 114B is the only object in the object mapping data 122 from the image recognition processor 120 that will be determined by the specification executor 130 as an object of interest based on the specification file 152, then the specification executor 130 will send the identified object data 132 that specifies only a portion of the object mapping data 122, the portion of the object mapping data 122 specifying the area of ​​the image of the user interface 112A occupied by the text block 114B.

[0042] Process 200 generates presentation data that presents the object on the user device for each object that the identified object data specifies as belonging to the object classification of interest (212). Figure 1B , the presentation engine 140 receives the identified object data 132 and the specific object mapping data 122, and generates and sends presentation data 142 to the user device 110 for display on the user device 110 via the user interface 112B. For example, continuing that the text block 114B is an example of the only object determined to be of interest to the user, the presentation data 142 now shows the text block 114B as illuminated or highlighted on the user interface 112B, compared to the same appearance of the user interface 112A. This example assumes that the user has selected to highlight the object of interest as part of the presentation mode included in the user input 111 as discussed herein.

[0043] According to some implementations, the object of interest category defines a text of interest category, and the presentation engine 140 may cause the user device 110 to highlight within the image objects depicted in the image that are determined to belong to the text of interest category.

[0044] Figure 3300 is a block diagram of an example computing device 300, 350 that can be used to implement the systems and methods described in this document, which can act as a client or as a server or multiple servers. The computing device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The computing device 300 is also intended to represent any other typically non-mobile device, such as a television or other electronic device with one or more processors embedded therein or attached thereto. The computing device 350 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other computing devices. The components shown here, their connections and relationships, and their functions are examples only, and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0045] The computing device 300 includes a processor 302, a memory 304, a storage device 306, a high-speed controller 308 connected to the memory 304 and a high-speed expansion port 310, and a low-speed controller 312 connected to a low-speed bus 314 and the storage device 306. Each of the components 302, 304, 306, 308, 310, and 312 are interconnected using various buses and can be mounted on a common motherboard or otherwise installed as appropriate. The processor 302 is capable of processing instructions for execution within the computing device 300, including instructions stored in the memory 304 or on the storage device 306 to display graphical information for a GUI on an external input / output device (such as a display 316 coupled to the high-speed controller 308). In other embodiments, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memory. Moreover, multiple computing devices 300 may be connected, with each device providing a portion of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0046] The memory 304 stores information within the computing device 300. In one embodiment, the memory 304 is a computer-readable medium. In one embodiment, the memory 304 is a volatile memory unit or multiple volatile memory units. In another embodiment, the memory 304 is a non-volatile memory unit or multiple non-volatile memory units.

[0047] Storage device 306 can provide mass storage for computing device 300. In one embodiment, storage device 306 is a computer readable medium. In various embodiments, storage device 306 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device or device array, including a device in a storage area network or other configuration. In one embodiment, a computer program product is tangibly implemented in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as a memory 304, a storage device 306, or a memory on a processor 302.

[0048] The high-speed controller 308 manages bandwidth-intensive operations for the computing device 300, while the low-speed controller 312 manages less bandwidth-intensive operations. This division of tasks is merely an example. In one embodiment, the high-speed controller 308 is coupled to the memory 304, the display 316 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 310, which can accept various expansion cards (not shown). In this embodiment, the low-speed controller 312 is coupled to the storage device 306 and the low-speed bus 314. Various communication ports (e.g., USB, A low-speed bus 314 (e.g., a low-speed expansion port) of Ethernet, wireless Ethernet) can be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or network device, such as a switch or router, for example, through a network adapter.

[0049] The computing device 300 can be implemented in many different forms, as shown in the drawings. For example, it can be implemented as a standard server 320, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 324. In addition, it can be implemented in a personal computer, such as a laptop computer 322. Alternatively, components from the computing device 300 can be combined with other components in a mobile device (not shown) such as a computing device 350. Each of such devices can contain one or more of the computing devices 300, 350, and the entire system can be composed of multiple computing devices 300, 350 that communicate with each other.

[0050] Computing device 350 includes, among other components, a processor 352, a memory 364, an input / output device (such as a display 354), a communication interface 366, and a transceiver 368. Computing device 350 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 350, 352, 364, 354, 366, and 368 are interconnected using various buses, and several of the components may be mounted on a common motherboard or otherwise as appropriate.

[0051] Processor 352 can process instructions for execution within computing device 350, including instructions stored in memory 364. Processors can also include separate analog and digital processors. The processor can provide, for example, coordination of other components of computing device 350, such as control of a user interface, applications run by computing device 350, and wireless communications performed by computing device 350.

[0052] The processor 352 can communicate with the user through the control interface 358 and the display interface 356 coupled to the display 354. The display 354 can be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. The display interface 356 may include appropriate circuits for driving the display 354 to present graphics and other information to the user. The control interface 358 can receive commands from the user and convert them for submission to the processor 352. In addition, an external interface 362 can be provided in communication with the processor 352 to enable the computing device 350 to communicate with other devices in a near area. The external interface 362 can be provided, for example, for wired communication (e.g., via a docking process) or wireless communication (e.g., via or other such techniques).

[0053] The memory 364 stores information within the computing device 350. In one embodiment, the memory 364 is a computer-readable medium. In one embodiment, the memory 364 is a volatile memory unit or multiple volatile memory units. In another embodiment, the memory 364 is a non-volatile memory unit or multiple non-volatile memory units. An expansion memory 374 may also be provided, and the expansion memory 374 may be connected to the computing device 350 through an expansion interface 372, and the expansion interface 372 may include, for example, a user identification module (SIM) card interface. Such an expansion memory 374 may provide additional storage space for the computing device 350, or may also store applications or other information of the computing device 350. In particular, the expansion memory 374 may include instructions for executing or supplementing the processes described above, and may also include security information. Therefore, for example, the expansion memory 374 may be provided as a security module of the computing device 350, and may be programmed with instructions that allow the safe use of the computing device 350. In addition, a security application may be provided via a SIM card together with additional information, such as placing identification information on the SIM card in an unhackable manner.

[0054] The memory may include, for example, flash memory and / or MRAM memory, as discussed below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 364, expansion memory 374, or memory on processor 352.

[0055] The computing device 350 may communicate wirelessly via the communication interface 366, which may include digital signal processing circuitry, where necessary. The communication interface 366 may provide for communication in a variety of modes or protocols, such as GSM voice calls, SMS, EMS or MMS messages, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communications may occur, for example, via a transceiver 368 (e.g., a radio frequency transceiver). Additionally, short-range communications may occur, such as using WiFi or other such transceiver (not shown). In addition, GPS receiver module 370 may provide additional wireless data to computing device 350, which may be used as appropriate by applications running on computing device 350.

[0056] Computing device 350 may also communicate audibly using audio codec 360, which may receive voice information from a user and convert it into usable digital information. Audio codec 360 may likewise generate audible sounds for the user, such as through a speaker, e.g., in a telephone handset of computing device 350. Such sounds may include sounds from voice telephone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on computing device 350.

[0057] The computing device 350 may be implemented in many different forms, as shown in the drawings. For example, it may be implemented as a cellular phone 380. It may also be implemented as part of a smart phone 382, ​​a personal digital assistant, or other mobile device.

[0058] In situations where the systems discussed herein collect personal information about a user or can utilize personal information, the user may be provided with an opportunity to control whether an application or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current location), or to control whether and / or how to receive content that may be more relevant to the user. In addition, certain data may be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (e.g., to a city, zip code, or state level) in the case of obtaining location information so that the user's specific location cannot be determined. Thus, the user may control how information about the user is collected and how it is used by the content server.

[0059] Embodiments of the subject matter and operations described in this specification can be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on computer storage media for execution by data processing apparatus or to control the operation of data processing apparatus.

[0060] A computer storage medium may be, or may be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, although a computer storage medium is not a propagated signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or may be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0061] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0062] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality of the foregoing or a combination of the foregoing. The apparatus may include dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment enable a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0063] A computer program (also referred to as a program, software, software application, script or code) can be written in any form of programming language, including compiled or interpreted languages, illustrative or procedural languages, and can be deployed in any form, including as a separate program or as a module, component, subroutine, object or other unit suitable for use in a computing environment. A computer program can but need not correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines or parts of code). A computer program can be deployed to execute on one computer, or executed on multiple computers located at one location, or distributed across multiple locations and interconnected by a communication network.

[0064] The processes and logic flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits), and the apparatus can also be implemented as the special purpose logic circuits.

[0065] For example, processors suitable for the execution of computer programs include both general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors in any type of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, or be operatively coupled to one or more mass storage devices to receive data from it or to transmit data to it or both, such as a magnetic disk, a magneto-optical disk, or an optical disk. However, a computer need not have such a device. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive, etc.). Devices suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable magnetic disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0066] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user device of the user in response to a request received from the web browser.

[0067] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, for example, as a data server, or includes a middleware component, for example, an application server, or includes a front-end component, for example, a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an interconnected network (e.g., the Internet), and a peer-to-peer network (e.g., a self-organizing peer-to-peer network).

[0068] The computing system may include a client and a server. The client and the server are usually far away from each other and typically interact through a communication network. The relationship between the client and the server is generated by means of a computer program that runs on a corresponding computer and has a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input therefrom). The data generated at the client device (e.g., the result of user interaction) can be received from the client device at the server.

[0069] Although this specification includes many specific implementation details, these details should not be interpreted as limitations on any feature or limitations on what may be claimed, but rather as descriptions of features specific to a particular embodiment. In this specification, certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations, even initially claimed as such, in some cases, one or more features in the combination may be removed from the claimed combination, and the claimed combination may involve a sub-combination or a variation of a sub-combination.

[0070] Similarly, although operation is described in the accompanying drawings in a specific order, this should not be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or performing all the operations shown, to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Moreover, the separation of the various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and system can be integrated together in a single software product or encapsulated in a plurality of software products usually.

[0071] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

Claims

1. A system implemented on a user device, comprising: An image recognition processing component, which: receiving as input image data depicting an image of a plurality of objects and a user input requesting that a classification of an object of interest be found in the image data, the image data being captured by the user device; and processing the image data and generating as an output object map data according to a defined data structure and specifying, for each object identified in the image, a region of the image occupied by the object and data identifying the object; Specifications for actuator components which: receiving as input the object mapping data from the image recognition processing component; receiving as input a specification file, the specification file comprising data specifying a model, the model being configured to recognize a class of objects of interest specified in the user input; determining, based on the specification file, whether the object mapping data generated by the image recognition processing component specifies an object belonging to a class of objects of interest; and generating as output identified object data specifying each object determined to belong to the class of objects of interest; and Renders a widget that: receiving as input the identified object data and, for each object determined to belong to a class of objects of interest specified by the identified object data, object mapping data specifying a region of an image occupied by the object; and For each object specified by the identified object data as belonging to the class of objects of interest, presentation data is generated for presenting the object on the user device.

2. The system according to claim 1, wherein: The specification file is generated by a library generator, which: receiving input from a developer specifying one or more object classes of interest; receiving input from a developer specifying a rendering mode for rendering each object depicted in the image that is determined to belong to a classification of objects of interest; as well as The specification file is generated based on developer input specifying one or more object of interest classes and developer input specifying a rendering mode.

3. The system according to claim 2, wherein: The data generated by the presentation component for presenting the object of interest is data generated according to the presentation mode specified in the specification file.

4. The system according to claim 1, wherein: The presentation component causes the user device to highlight within the image objects depicted in the image that are determined to belong to a class of objects of interest.

5. The system according to claim 1, wherein: The presentation component causes the user device to audibly describe objects depicted in the image that are determined to belong to the class of objects of interest.

6. The system according to claim 1, wherein: The objects are one or more of text or real-world physical objects.

7. The system according to claim 1, wherein: The object of interest category defines a text of interest category, and wherein the presentation component causes the user device to highlight within the image an object depicted in the image that is determined to belong to the text of interest category.

8. A computer-implemented method implemented on a user device, comprising: receiving image data depicting an image of a plurality of objects and a user input requesting to find a classification of an object of interest in the image data, the image data being captured by the user device; processing the image data and generating object map data, the object map data being in accordance with a defined data structure and specifying, for each object identified in the image, a region of the image occupied by the object and data identifying the object; receiving as input a specification file, the specification file comprising data specifying a model, the model being configured to recognize a class of objects of interest specified in the user input; determining, based on the specification file, whether the object mapping data specifies an object belonging to a classification of objects of interest; generating identified object data specifying each object determined to belong to the class of objects of interest; as well as For each object specified by the identified object data that belongs to the class of objects of interest, presentation data is generated for presenting the object on the user device based on the identified object data and object mapping data specifying an area of ​​the image occupied by the object.

9. The computer-implemented method of claim 8, further comprising: receiving input from a developer specifying one or more object classes of interest; receiving input from a developer specifying a rendering mode for rendering each object depicted in the image that is determined to belong to a classification of objects of interest; as well as A specification file is generated based on developer input specifying one or more object classes of interest and developer input specifying a rendering mode.

10. The computer-implemented method of claim 9, wherein: The data generated to present the object of interest is data generated according to the presentation mode specified in the specification file.

11. The computer-implemented method of claim 8, further comprising: The user device is caused to highlight within the image an object depicted in the image that is determined to belong to the class of objects of interest.

12. The computer-implemented method of claim 8, further comprising: The user device is caused to audibly describe objects depicted in the image that are determined to belong to the class of objects of interest.

13. The computer-implemented method of claim 8, wherein: The objects are one or more of text or real-world physical objects.

14. The computer-implemented method of claim 8, wherein: The object of interest category defines a text of interest category, and the method further includes causing a user device to highlight within an image an object depicted in the image that is determined to belong to the text of interest category.

15. A non-transitory computer storage medium encoded with a computer program, the computer program comprising instructions which, when executed by a data processing apparatus in a user device, cause the data processing apparatus to perform operations comprising: receiving as input image data depicting an image of a plurality of objects and user input requesting that a classification of an object of interest be found in the image data, the image data being captured by the user device; processing the image data and generating object map data, the object map data being in accordance with a defined data structure and specifying, for each object identified in the image, a region of the image occupied by the object and data identifying the object; receiving as input a specification file, the specification file comprising data specifying a model, the model being configured to recognize a class of objects of interest specified in the user input; determining, based on the specification file, whether the object mapping data specifies an object belonging to a classification of objects of interest; generating identified object data specifying each object determined to belong to the class of objects of interest; as well as For each object specified by the identified object data as belonging to the class of objects of interest, presentation data is generated for presenting the object on a user device based on the identified object data and object mapping data specifying an area of ​​the image occupied by the object.

16. The non-transitory computer storage medium of claim 15, further comprising: receiving input from a developer specifying one or more object classes of interest; receiving input from a developer specifying a rendering mode for rendering each object depicted in the image that is determined to belong to a classification of objects of interest; as well as A specification file is generated based on developer input specifying one or more object classes of interest and developer input specifying a rendering mode.

17. The non-transitory computer storage medium of claim 16, wherein: The data generated to present the object of interest is data generated according to the presentation mode specified in the specification file.

Images