Processing electronic documents bearing images using a multi-modal fusion framework

Abstract

A computer-implemented technique uses a multi-modal fusion approach using one or more neural networks to identify at least one item name associated with an input image. The technique is referred to as multi-modal because it collects and processes different kinds of evidence about each detected item name. The technique is referred to as employing a fusion approach because it fuses the multi-modal evidence into an output conclusion that identifies at least one item name associated with the input image. In one example, a first modality collects evidence by identifying and analyzing regions of the input image that can include item name relevant information. A second modality collects and analyzes any text that appears as part of the input image itself. A third modality collects and analyzes text that is not included in the input image itself but is still associated with the input image.

Description

Background Technology

[0001] Some computer-implemented tools attempt to automatically detect specific kinds of objects depicted in or otherwise associated with an input image. For example, there are computer-implemented tools for detecting entities that manufacture or otherwise supply products depicted in images. Traditionally, developers could accomplish this task by creating handcrafted sets of objects to detect features. However, this solution is labor-intensive. Furthermore, developers may struggle to generate feature sets that interpret the countless ways object-related information might appear in an image. Factors controlling the presence of object-related information can include: the size of the object-related information, its orientation, its resolution, its brightness level, the presence of one or more objects in the image that obscure it, etc. These challenges can lead to unsatisfactory detection results. Other tools use machine learning models to detect the presence of object-related information in images. These tools may perform better than image analysis using handcrafted features. However, due to the factors mentioned above, these tools may also produce inaccurate results. Summary of the Invention

[0002] This paper describes a computer-implemented technique that uses a multimodal fusion approach to identify at least one item name associated with an input image. An item name refers to the name of an item depicted by or otherwise associated with the input image. The technique is called multimodal because it collects and processes different kinds of evidence regarding item names. The technique is called an fusion approach because it fuses multimodal evidence into an output conclusion that identifies at least one item name associated with the input image.

[0003] According to one illustrative aspect, the first mode collects evidence by identifying and analyzing target regions in the input image that may contain information related to item names. The second mode collects and analyzes any text that appears as part of the input image itself. The third mode collects and analyzes text that is not included in the input image itself but is still associated with it. This text is referred to herein as external text. For example, external text may correspond to a caption or heading in an electronic document in which the input image appears or is otherwise associated with it.

[0004] According to another illustrative aspect, the technology can perform the aforementioned functions using one or more neural networks combined with optical character recognition (OCR) components. For example, the technology can use one or more neural networks to identify and analyze regions in an input image. The technology can use one or more other neural networks to encode external text. Furthermore, the technology can use one or more neural networks to fuse evidence collected from their multiple patterns.

[0005] According to another illustrative aspect, when a user accesses an electronic document or a decision is made to send an electronic document to the user, the technology can invoke its analysis of the electronic document. After identifying the item names associated with the input image, the technology can determine the supplementary content items associated with the item names. The technology then sends the supplementary content items to the user's computing device operated by the user.

[0006] The aforementioned technologies can be embodied in various types of systems, devices, components, methods, computer-readable storage media, data structures, graphical user interface presentations, artifacts, etc.

[0007] This overview is provided to introduce the selection of concepts in a simplified form; these concepts are further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0008] Figure 1 An illustrative item name identifier system is shown for classifying at least one item name associated with an input image.

[0009] Figure 2 It shows the relationship with Figure 1 The framework shown is compared to alternative frameworks for synthesizing feature information.

[0010] Figure 3 This shows what can be used to implement Figure 1 The item name identifier system of computing devices.

[0011] Figure 4 It shows that it can be used Figure 1 Different applications of the item name identifier system.

[0012] Figure 5 It shows Figure 1 This is an illustrative application of the item name identifier system.

[0013] Figure 6 It shows Figure 5 How to represent supplementary content items within an electronic document.

[0014] Figure 7This shows what can be used to implement Figure 1 A convolutional neural network (CNN) with one or more elements in a system of item name identifiers.

[0015] Figure 8 and Figure 9 It shows the use of in Figure 1 The item name identifier system uses two different region proposal components.

[0016] Figure 10 An implementation of a text encoder neural network is shown. The text encoder neural network transforms external text into encoded context information.

[0017] Figure 11 The training by is shown Figure 1 The item name identifier system is the training framework of one or more models used by the system.

[0018] Figure 12 and Figure 13 Together they show the description Figure 1 The process of descriptive operation of an item name identifier system.

[0019] Figure 14 The description in Figure 1 The process of integrating multimodal evidence in an item name identifier system.

[0020] Figure 15 The description in Figure 1 Another process of integrating multimodal evidence in an item name identifier system.

[0021] Figure 16 An illustrative type of computing device is shown that can be used to implement any aspect of the features shown in the foregoing figures.

[0022] The same numbers are used throughout the published content and accompanying drawings to refer to similar components and features. Series 100 numbers refer to those originally published in [the original text]. Figure 1 The features found, series 200 numbers refer to those initially found in Figure 2 The features found, series 300 numbers refer to those originally found in Figure 3 The features found in [the context] can be used to further illustrate this. Detailed Implementation

[0023] This disclosure is organized as follows. Section A describes a computer-implemented item name identifier system. Section B explains an illustrative method for illustrating the operation of the item name identifier system described in Section A. Section C describes illustrative computational functions that can be used to implement any aspect of the features described in Sections A and B.

[0024] As a preliminary consideration, the term "hardware logic circuit system" corresponds to a technology that includes one or more hardware processors (e.g., CPU, GPU, etc.) that execute machine-readable instructions stored in memory, and / or one or more other hardware logic units (e.g., FPGA) that perform operations using a task-specific set of fixed and / or programmable logic gates. Section C provides additional information about one implementation of a hardware logic circuit system. In certain contexts, each of the terms "component," "engine," "system," and "tool" refers to a portion of the hardware logic circuit system that performs a particular function.

[0025] In one case, the illustration's separation of the various parts into different units can reflect the use of the corresponding different physical and tangible parts in actual implementation. Alternatively or additionally, any single component shown in the illustration can be implemented by multiple actual physical components. Alternatively or additionally, the depiction of any two or more individual components in the illustration can reflect different functions performed by a single actual physical component.

[0026] Other accompanying figures illustrate these concepts in the form of flowcharts. In this form, certain operations are described as constituting different blocks that are executed in a specific order. Such implementations are illustrative and not restrictive. Some blocks described herein may be combined and executed in a single operation, some blocks may be decomposed into multiple component blocks, and some blocks may be executed in an order different from that described herein (including in the form of parallel execution of blocks). In one implementation, the blocks shown in the flowcharts that are related to processing-related functions may be implemented by the hardware logic circuitry system described in Section C, which in turn may be implemented by one or more hardware processors and / or other logic units comprising a task-specific set of logic gates.

[0027] Regarding the terminology, the phrase "configured to" encompasses various physical and tangible mechanisms for performing the identified operations. These mechanisms can be configured to perform the operations using the hardware logic circuitry system of Section C. The term "logic" similarly includes various physical and tangible mechanisms for performing tasks. For example, each processing-related operation shown in the flowchart corresponds to a logical component for performing that operation. Logic components can perform their operations using the hardware logic circuitry system of Section C. When implemented by a computing device, a logic component represents an electrical element that is a physical part of the computing system, regardless of how it is implemented.

[0028] Any storage resource or any combination of storage resources described herein can be considered a computer-readable medium. In many cases, a computer-readable medium means some form of physical and tangible entity. The term computer-readable medium also includes propagated signals, for example, signals transmitted or received through physical channels and / or air or other wireless media. However, the specific term "computer-readable storage medium" explicitly excludes propagated signals themselves while including all other forms of computer-readable medium.

[0029] The following interpretations may identify one or more features as “optional.” This type of statement should not be interpreted as an exhaustive description of features that can be considered optional; that is, other features may be considered optional, even if not explicitly stated in the text. Furthermore, any description of a single entity is not intended to exclude the use of multiple such entities; similarly, a description of multiple entities is not intended to exclude the use of a single entity. Moreover, while the description may interpret certain features as alternative ways of performing the identified function or implementing the identified mechanism, these features may also be combined in any combination. Furthermore, unless explicitly stated otherwise, the term “multiple” refers to two or more items and does not necessarily imply “all” items of a particular kind. Furthermore, unless otherwise stated, descriptive terms such as “first,” “second,” “third,” etc., are used to distinguish different items and do not indicate an order among the items. Finally, the terms “exemplary” or “illustrative” refer to one of many potential implementations.

[0030] A. Descriptive computing system

[0031] A.1. Overview

[0032] Figure 1 An implementation of an item name identifier system 102 is shown. The item name identifier system 102 is configured to identify one or more item names associated with the input image 104. For example, Figure 1 The input image 104 shown depicts a person holding a can of food 106 produced by a company called "Sam's Soda". The item name identifier system 102 therefore identifies the item name associated with the input image as "Sam's Soda".

[0033] The term "item name" generally refers to the name assigned to a product, service, or other type of item, or a name associated with an attribute of that product, service, or other type of item. In some cases, an item name also directly and / or indirectly conveys at least the entity associated with the item. For example, in some cases, an item name directly and / or indirectly identifies a specific entity that manufactures or otherwise provides the item. In some cases, an item name is synonymous with an item's brand. Here, people use an item name to identify an item as belonging to a specific provider and to distinguish it from similar items produced by other providers. In many cases, some legal bodies formally recognize item names associated with an item. In other cases, an item name does not receive legal protection. For example, an artisan may create a fashion accessory with distinctive visual characteristics that allow consumers to identify the accessory as coming from that particular artisan and not someone else; however, the artisan may not have registered the distinctive visual characteristics of their accessory with any government agency. The item name in this example could identify the artisan or their company.

[0034] In some cases, the item name is also associated with one or more flags, any of which may also appear in the input image 104. In some of these cases, the flag may include text that directly conveys the item name. In other cases, the flag may not include text, or it may include text that does not directly identify the item name. Figure 1 In the example, input image 104 includes a marker 108 associated with the item name appearing on jar 106. More generally, input images may convey information associated with an item name, with or without the marker associated with that item name. From another perspective, the subject matter constituting the item name information is controlled by the types of objects and image features identified as related to the item name in the training image set; the training system uses these training images to train various models used by the item name identifier system 102.

[0035] exist Figure 1 In the example, input image 104 appears as part of electronic document 110. Electronic document 110 may correspond to a webpage hosted by the manufacturer of Sam's Soda beverages. Alternatively, electronic document 110 may correspond to a digital advertisement to be presented to a user. Alternatively, electronic document 110 may correspond to a document submitted by a user to a search engine or other processing system. Section A.2 (hereinafter) provides additional information regarding the application of the item name identifier system 102, which may be combined or otherwise used. This section clarifies the context in which the item name identifier system 102 operates on electronic document 110. In yet another context, an electronic document may include a set of information items that are not necessarily displayed simultaneously.

[0036] Figure 1The specific electronic document 110 shown includes in-image text and external text 112. In-image text includes alphanumeric information that is part of the image itself. For example, in-image text includes the name "Sam's Soda," "USDA," "organic," ingredient and nutritional information, etc. External text 112 provides information about the product shown in the input image 104. External text 112 may typically appear as a title or title bar associated with the input image 104, descriptive text adjacent to the input image 104, etc., or any combination thereof. Although not shown, external text 112 may also include metadata associated with the input image 104. Metadata does not necessarily need to be visually presented on the electronic document 110 itself, if available. For example, metadata associated with the input image 104 may provide one or more key terms associated with the product depicted in the input image 104.

[0037] Figure 1 The components of the item name identifier system 102 shown will be described below in a generally top-down manner. First, the document parsing component 114 provides logic for extracting information from the electronic document 110. The document parsing component 114 operates by extracting the input image 104 from the electronic document 110 and routing the input image 104 to the image processing logic of the item name identifier system 102, as described below. The document processing component 114 also extracts external text 112 based on one or more predetermined rules and forwards the external text 112 to the context processing logic of the item name identifier system 102, as described below.

[0038] According to one rule, document parsing component 114 extracts all text appearing in input image 104 from electronic document 110. According to another rule, document parsing component 114 extracts the title and / or title bar of input image 104. According to another rule, document parsing component 114 extracts all text on electronic document 110 within a predetermined distance from input image 104. According to another rule, document parsing component 114 may select only text portions that have an identifiable semantic relationship with the topic of the input image. Document parsing component 114 may perform this task in different ways. For example, document parsing component 114 may extract text portions in electronic document 110 that include one or more key terms that also appear in the title of input image 104. Alternatively or additionally, document parsing component 114 may use a deep neural network (DNN) to map input image 104 to a first vector in a low-dimensional semantic space, map portions of text to a second vector in the semantic space, and then select the text if the two vectors are within a predetermined distance in the semantic space, for example, measured by cosine similarity or some other distance metric. According to another rule, document parsing component 114 may limit the external text 112 it extracts from electronic document 110 to a predetermined number of words, such as 512 words in a purely illustrative case. According to another rule, document parsing component 114 may select multiple external texts associated with multiple corresponding categories (e.g., information extracted from the title of input image 104 and information extracted from the text body appearing in electronic document 110), and apply different weights to these corresponding texts. The above selection rules are proposed for illustrative rather than restrictive purposes; other implementations may adopt other rules.

[0039] The item name identifier system 102 includes three processing function branches associated with three corresponding modes. Therefore, the item name identifier system 102 may also be referred to as a multimodal fusion framework. The region analysis branch 116 identifies and analyzes regions in the input image 104 that may contain logo information or other image information related to the object of interest. The in-image text analysis branch 118 extracts and analyzes any text appearing in the input image 104; as previously mentioned, this text is referred to herein as in-image text. The external text analysis branch 120 analyzes external text 112. These three branches (116, 118, 120) are described below in the context of detecting brand names; however, as will be clarified below, the item name identifier system 102 can be applied to determine other types of item names.

[0040] Starting with the region analysis branch 116, the region proposal component 122 identifies zero or more candidate regions, which can include objects of any type, including but not limited to those containing flag information. Figure 1In the example, the region proposal component 122 generates an annotated image 124 that identifies regions 126 associated with a hand, 128 associated with a can, 130 associated with a sign, 132 associated with a nutrition-related label, and so on. This is a simplified set of candidate regions presented for illustrative purposes; in practice, the region proposal component 122 can generate a large number of candidate regions.

[0041] A convolutional neural network (CNN) maps an annotated image 124 to outputs that identify one or more target regions. Each target region contains an object of interest and is associated with a product name. In some cases, the item name also conveys the entity to which the target region belongs. For example, each target region contains identification information (the object of interest in this example) that identifies the entity that manufactures and / or supplies the item. Target regions can display identification information independently or as part of an attachment to the product. Figure 1 In a specific example, CNN 134 indicates that candidate region 130 corresponds to target region. The output of CNN 134 specifies information for each target region. CNN 134 also provides feature information associated with each target region, referred to herein as region feature information. The region feature information associated with the target region may correspond to feature values ​​generated by the last layer of CNN 134. Additional information regarding the region analysis branch 116 is provided in Section A.3 below.

[0042] In an alternative scenario, region analysis branch 116 may determine that input image 104 does not include a target region associated with an object of interest (in this example, the marker information). In this case, region analysis branch 116 may generate default information that conveys the fact that input image 104 does not contain the relevant target region.

[0043] For the in-image text analysis branch 118, the Optical Character Recognition (OCR) component 136 performs optical character recognition on the input image 104 to produce an OCR output. The OCR output identifies all the text contained in the input image 104 (here, "Sam's Soda", "USDA", "organic", etc.). The OCR component 136 can use any technique to perform this task, such as DNN, Hidden Markov Model (HMM), etc. Then, the word encoder 138 maps the OCR output to in-image text information. The word encoder 138 can be implemented in different ways, such as by the encoder calculating the term frequency inverse document frequency (tf-idf) score for each word in the OCR output and then formulating an output vector that conveys these scores. The tf-idf score identifies the number of times (tf) a term appears in a document. It also determines the number d of documents in a corpus containing N documents that include the term. The tf-idf score is equal to tf·log(N / d). In another scenario, the word encoder 138 uses any type of neural network to map the OCR output to an output vector, such as the well-known Word2vec model. Other implementations are also possible. Note that the intra-image text analysis branch 118 operates on the input image 104 as a whole and similarly produces intra-image text information associated with the input image 104 as a whole. Conversely, the region analysis branch 116 identifies individual regions in the input image 104 and generates region feature information associated with each individual target region.

[0044] In an alternative scenario, OCR component 136 may determine that input image 104 does not contain internal text. In this case, word encoder 138 generates default in-image text information that conveys the fact that input image 104 does not contain text.

[0045] The external text analysis branch 120 includes a text encoder neural network (referred to as "text encoder" for brevity) 140 that maps the external text 112 to encoded context information. Different implementations may be used to implement the text encoder. Non-limitingly, the text encoder 140 may correspond to a transformer neural network. Additional information regarding this non-limiting implementation of the text encoder 140 will be described in subsection A.3 below. In other implementations, the text encoder 140 may be implemented as any other type of neural network, such as a CNN, a recurrent neural network (RNN), or any combination thereof. If the electronic document 110 does not contain external text 112, the external text analysis branch 120 can provide default context information that conveys this fact.

[0046] Fusion logic 142 combines instances of region feature information, intra-image information, and encoding context information to produce combined fused information. Fusion logic 142 can be implemented in different ways. In one approach, a first concatenation component 144 concatenates a vector providing intra-image text information with a vector providing encoding context information to produce a first concatenation vector 146. Fusion logic 142 then maps the first concatenation vector 146 to text fused information using a first fusion neural network 148. A second concatenation component 150 concatenates a vector providing text fused information with a vector providing region feature information associated with the target region 130 to produce a second concatenation vector 152. A second fusion neural network 154 maps the second concatenation vector 152 to combined fused information. In one implementation, the first fusion neural network 148 and the second fusion neural network 154 can correspond to respective fully connected (FC) neural networks, each having two or more layers, and each network can use any activation function (e.g., ReLU). In this fusion operation, any one or more of the region feature information, intra-image text information, and encoding context information can correspond to default information. For example, the encoding context information corresponds to the default information in the case that the external document 110 does not have any external text.

[0047] Classifier 156 classifies specific item names associated with target region 130 based on combined and fused information. Classifier 156 can be implemented in different ways, such as using the softmax function, support vector machine (SVM), logistic regression model, etc. Figure 1 In the example, the output of classifier 156 indicates that target region 130 is associated with the specific item name "Sam's Soda". In one implementation, classifier 156 can determine the item name by determining the probability of each of a plurality of predetermined item names and by selecting the item name with the highest probability, provided that the probability of that name is above a predetermined threshold.

[0048] The item name identifier system 102 can repeat the above-described fusion and classification operations for each target region identified by the branch analysis branch 116. For example, suppose the input image includes objects associated with two or more item names, which in turn are associated with two or more target regions. The item name identifier system 102 can determine all item names by processing each target region sequentially. In another implementation, the item name identifier system 102 can process multiple target regions in parallel.

[0049] Compared to the methods described above, other implementations can combine the three modes in different ways. For example, another implementation can fuse region feature information with text fusion information, and then fuse the result of this combination with encoding context information. Another implementation can use a single pipeline to operate on the label information, in-image text, and external text 112.

[0050] In one implementation, the item name identifier system 102 can selectively weight the information it generates using its various patterns. For example, the item name identifier system 102 can apply weights to the information generated by each branch based on the confidence level associated with that information. The weights applied to a piece of information establish its relevance to other information in subsequent fusion and classification operations. That is, a piece of information with a high confidence level will be considered more relevant than a piece of information with a lower confidence level.

[0051] Consider an example where input image 102 contains a relatively small sign associated with a product, or a sign that is otherwise difficult to discern (e.g., because of its orientation toward the viewer, and / or because it is partially occluded by another object, and / or because it has low resolution, etc.). CNN 134 can generate a confidence metric that reflects the level of confidence it has in detecting the sign. In this case, the confidence metric will be relatively low. As a result, item name identifier system 102 can reduce the relevance of the information generated by region analysis branch 116 to the weights imposed by the other two branches (118, 120). This also means that, in this example, compared to another case where the sign is clearly discernible in input image 104, item name identifier system 102 will rely more heavily on in-image text (if any) and / or external text 112 (if any).

[0052] Figure 2 The method for fusion is shown. Figure 1Another architecture 202 for the type of information collected. Region branch classifier 204 generates a first evaluation of the item names depicted in the target region based on region feature information provided by region analysis branch 116. OCR branch classifier 206 generates a second evaluation of the item names based on in-image text information provided by in-image text analysis branch 118. External text classifier 208 generates a third evaluation of the item names conveyed by encoded context information provided by external text analysis branch 120. Multimodal classifier 210 generates a final determination of the item names associated with the target region based on the three evaluations provided by the above three classifiers (204, 206, 208). Each of the classifiers identified above can be implemented in any way, such as a softmax function, support vector machine (SVM), deep neural network (DNN), logistic regression classifier, etc. In this example, multimodal classifier 210 can be described as embodying both fusion logic and classification logic.

[0053] Figure 3 This shows what can be used to implement Figure 1 and Figure 2 The device is a computing device that provides the functionality of a computer network. This device includes multiple user computing devices 302 that interact with one or more servers 304 via a computer network 306. User computing devices 302 may include any of the following: desktop computing devices, laptop computing devices, any type of handheld computing device (smartphone, tablet computing device), game consoles, cable TV boxes, mixed reality devices, wearable devices, etc. The computer network 306 may correspond to a wide area network (e.g., the Internet), a local area network (LAN), etc.

[0054] The elements of the item name identifier system 102 can be distributed between the user computing device 302 and the server 304 in any manner. For example, in one implementation, each user computing device implements a local instantiation of the item name identifier system 102. In another implementation, one or more servers implement the entire item name identifier system 102. In yet another implementation, the functional features of the item name identifier system 102 are distributed between the local computing device 302 and the server 304.

[0055] In the example above, the item name identifier system 102 identifies the brand name associated with an object within the input image 104. However, the item name identifier system 102 can use the aforementioned trimodal method to detect the names of other types of objects of interest in the image. For example, consider a case where the input image shows two types of dogs; there may be other objects. Furthermore, assume that the item name identifier system 102 is specifically configured to identify the names of dogs in an electronic document. In this case, the region analysis branch 114 can detect two target regions, each containing one of the two dogs, which are objects of interest. The in-image text analysis branch 118 can detect any text within the input image 108, some of which may be related to dogs. For example, the input image may show the name of a pet store in the background, or be part of the title bar of the image itself. The external text analysis branch 120 can extract external text from the electronic document, some of which may be related to the breed of dog shown in the image. The item name identifier system 102 can synthesize all this information in the manner described above to generate an output that provides the names of the two dogs, such as “Shiatzu” and “Labrador Retriever”.

[0056] In other examples, an item name can identify the name of a feature of a product, rather than representing the product as a whole. For example, item name identifier system 102 can be used to identify a fashion style present in an image. For instance, item name identifier system 102 can apply the methods described above to output the item name "houndstooth" when a garment appears in an input image displaying the pattern. Similarly, item name identifier system 102 can select evidence from the three analytical channels described above to reach this conclusion. This example more generally shows that the term "item name" as used herein can be considered synonymous with "attribute name." A brand name is simply an attribute of the input image.

[0057] Furthermore, the example above describes the use of three analysis branches. However, other implementations can typically include multiple analysis branches, for example, by using two or more analysis branches instead of being limited to three.

[0058] In summary, the item name identifier system 102 produces better classification results compared to classification systems that rely on a single analytical pattern. For example, consider an instance where the classification system itself uses image-based analysis to detect objects of interest in an input image. This system may produce unsatisfactory results when the objects of interest are difficult to discern in the input image. In contrast, the item name identifier system 102 uses multiple patterns to gather information about the objects of interest and employs fusion and classification logic where these different pieces of information support each other. This allows the item name identifier system 102 to produce satisfactory classification results, even when the objects of interest are difficult to discern in the input image.

[0059] A2. Explanatory Application

[0060] Figure 4 A non-exhaustive set of applications 402 that can utilize the item name identifier system 102 is shown. Applications 402 include an item service engine 404, a trend analysis engine 406, an index update engine 408, a consistency check engine 410, etc. Like the item name identifier system 102 itself, applications 402 can be distributed in any way between the user computing device 302 and the server 304.

[0061] Figure 5 It shows about Figure 4 Further details of the item service engine 404 are provided. The user interacts with the item service engine 404 using the user computing device 502. The user computing device 502 further includes a document viewer 504 that enables the user to interact with electronic documents. For example, the document viewer 504 may correspond to a browser implemented natively by the user computing device 502. The browser enables the user to view web pages hosted by one or more websites.

[0062] Figure 5 The operation of the application shown will be combined below. Figure 6 The description is illustrated by the electronic document 602. In one implementation, a user inputs an instruction to load the electronic document 602. For example, the user may click a link associated with the electronic document 602 in a document viewer 504. The document viewer 504 responds by displaying the electronic document 602 on a display device provided by the user's computing device 502. In operation 5.1, based on the aforementioned triggering event, the item service engine 404 receives an instruction to operate on the electronic document 602. For example, in response to the triggering event, the item service engine 404 may receive a Uniform Resource Locator (URL) associated with the electronic document 602 to be processed, and an instruction to operate on the electronic document 602.

[0063] exist Figure 6In illustrative terms only, electronic document 602 is a webpage associated with beverages. Input image 604 depicts a beverage. As previously mentioned, input image 604 includes text that is part of input image 604 itself. Electronic document 602 also includes external text 606 that is outside of input image 604 and is not part of input image 604.

[0064] In another scenario, a triggering event occurs when a user performs an action that indirectly implies the electronic document 602. For example, a user might enter a search query into a search engine. The search engine can determine that the top-ranked document 602 matches the user's search query and that the top-ranked document 602 includes the input image 604. In this example, the triggering event could correspond to the submission of the search query, which occurs before the electronic document 602 is rendered on the display device. Other triggering events may invoke the services of the item service engine 404; the examples above are presented for illustrative purposes and not for limitation.

[0065] In operation 5.2, item service engine 404 uses item name identifier system 102 to parse electronic document 602. In operation 5.3, item service engine 404 uses item name identifier system 102 to determine at least one item name associated with at least one target area. In operation 5.4, item service engine 404 uses specific item names to identify one or more matching supplementary content items. Item service engine 404 can perform this retrieval option based solely on item name information or in combination with one or more other features extracted by item service engine 404 from electronic document 602. For example, item service engine 404 can use one or more neural networks to perform additional analysis on input image 604 to determine the type of object depicted therein, the color of the object, the pattern displayed by the object, etc. The neural networks (multiple) can represent each of these additional attributes as a keyword or key phrase. Item service engine 404 can use specific item names to combine additional key terms to identify one or more matching supplementary content items.

[0066] More specifically, in a non-limiting case, data storage 506 stores a set of candidate supplementary content items, for example, corresponding to digital advertisements, etc. It is assumed that each supplementary content item is associated with one or more keywords. Item service engine 404 can perform retrieval options by identifying one or more supplementary content items that have a keyword closest to the keyword extracted from electronic document 602. In operation 5.6, item service engine 404 sends the supplementary content items(s) or links to these items(s) to user computing device 502. This causes document viewer 504 to display the supplementary content items(s) to the user. Figure 6In the example, document viewer 504 displays supplementary content item 608 in the same electronic document 602 where input image 604 appears, although this is not the case in all implementations.

[0067] return Figure 4 The trend analysis engine 406 can use the item name identifier 102 to identify the item name associated with each document in the document set. After identifying the item names associated with the document set, the trend analysis engine 406 can generate any statistical conclusions about the documents. For example, the trend analysis engine 406 can determine the item names that appear most frequently in the document set. Or, the trend analysis engine 406 can determine the item names that most often appear together in the document set, etc.

[0068] The trend analysis engine 406 can use any number of factors to define the members of a document set. For example, the trend analysis engine 406 can identify images associated with documents recently accessed by a specific user. Alternatively, the trend analysis engine 406 can identify images associated with documents recently accessed by a user group. Or, the trend analysis engine 406 can identify images associated with documents recently shared between user groups, and so on.

[0069] The index update engine 408 utilizes the item name identifier system 102 during the process of updating the index used by the search engine. Given a search query submitted by a user, the search engine uses the index to locate relevant documents. More specifically, the index update engine 408 can determine the item name associated with each individual electronic document or each individual image it processes. The index update engine 408 can then add the item name information to the index entry associated with that electronic document or individual image.

[0070] The consistency check engine 410 can determine whether a specific item name is consistent with other information submitted by an entity. For example, an advertiser may submit a bundle of information including textual information about a product and an image associated with the product. The consistency check engine 410 can use the item name identifier system 102 to determine the item name associated with the input image. The consistency check engine 410 can then determine whether the identified item name is consistent with the text information. For example, the consistency check engine 410 may mark a discrepancy when the image includes the logo NIKE for the item name while the text information identifies the product as being related to the item name ADDIDAS.

[0071] Again, the example application 402 above is illustrated in the spirit of illustration and not limitation. Other applications may use the item name identifier system 102. In each case, the application invokes the services of the item name identifier system 102 in response to a triggering event.

[0072] A.3. Individual Components of the Item Name Identifier System

[0073] This section explains about the... Figure 1 Descriptive details of the individual components used in the item name identifier system 102. First, Figure 7 The convolutional neural network (CNN) 134 used in the region analysis branch 116 is shown. Figure 7 In the diagram, CNN 134 illustrates the analysis performed in the pipeline stage. One or more convolutional components 702 perform convolution operations on the input image 704. Figure 1 In the context of the example, the input image 704 may correspond to a candidate region identified by the region proposal component 122. One or more pooling components 706 perform a downsampling operation. One or more fully connected components 708 provide one or more fully connected neural networks, each comprising any number of layers. More specifically, the CNN 134 may distribute the three components described above in any order. For example, the CNN 134 may include two or more convolutional components interleaved with the pooling components.

[0074] In each convolution operation, the convolutional component moves an n×m kernel across the input image (where "input image" in this general context refers to any image fed to the convolutional component). In some implementations, at each location of the kernel, the convolutional component generates a dot product of the kernel value and the underlying pixel values ​​of the image. The convolutional component stores this dot product as the output value in the output image at the location corresponding to the current location of the kernel. More specifically, the convolutional component can perform the above operation on different sets of kernels with different machine learning kernel values. Each kernel corresponds to a different pattern. In earlier layers of processing, the convolutional component can apply kernels to identify relatively primitive patterns in the image (such as edges, corners, etc.). In later layers, the convolutional component can apply kernels to find more complex shapes (such as shapes associated with a specific type of object in each candidate region being analyzed).

[0075] In each pooling operation, the pooling component moves a window of a predetermined size across the entire input image (where the input image corresponds to any image fed to the pooling component). The pooling component then performs some kind of aggregation / summary operation on the values ​​of the input image surrounded by the window, such as identifying and storing the maximum value in the window, generating and storing the average value in the window, and so on. Pooling operations can also be called downsampling operations. Although not shown, the corresponding upsampling component can expand the input image into a larger output image, for example, by copying the values ​​in the input image within the output image.

[0076] Fully connected components are typically preceded by flattened components ( Figure 7(Not shown in the image). The flattening component compresses one or more input images into a single input vector. This task can be performed by concatenating rows or columns of one or more input images to form a single input vector. The fully connected component then processes the input vector using a fully connected neural network. To compute the output value of any particular neuron in a particular layer of the fully connected network, the neuron generates a weighted sum of the values ​​from the previous layer, adds a bias value to the sum, and then applies an activation function (such as ReLU or hyperbolic tangent) to the result.

[0077] The final fully connected layer of CNN 134 provides the final representation of the features associated with the input image 704. In the terminology used herein, these features collectively correspond to region feature information. The classification component can manipulate this feature information to generate an output conclusion. For example, CNN 134 may include a softmax output operation, a support vector machine (SVM) classifier, etc. Specifically, the classification component of CNN 134 determines whether candidate regions include item name information such as labels.

[0078] Figure 8 A first implementation of the region proposal component 122 is shown. The region proposal component 122 typically identifies candidate regions in the input image that may contain objects, and is not limited to markers. Figure 8 In its implementation, the hierarchical grouping component 802 uses a hierarchical grouping algorithm to identify candidate regions. In this method, the hierarchical grouping component 802 iteratively merges image regions in the input image that meet a specified similarity test, initially starting with relatively small image regions. The hierarchical grouping component 802 can evaluate similarity based on any combination of features associated with the input image, such as color, brightness, hue, texture, etc. Image 804 shows an illustrative output of the hierarchical grouping component 802. At the end of this iterative process, the box generation component 806 draws bounding boxes around the identified regions. Background information on a non-restrictive hierarchical grouping technique is provided in "Selective Search for Object Recognition" by Uijlings et al. (International Journal of Computer Vision, 104(2), 2013, pp. 154-171).

[0079] Figure 9 A second implementation of the region proposal component 122 is shown. The region identification component 902 can define a grid of analysis points across the entire input image 904. At each analysis point, the region identification component 902 identifies multiple candidate boxes. The candidate boxes can have different sizes and different aspect ratios. For example, Figure 9An illustrative set 906 of candidate bounding boxes at a specific analysis location in the input image 904 is shown. For each such candidate bounding box, the region identification component 902 can use a neural network to determine whether the candidate bounding box is likely to contain an object. The region trimming component 908 identifies a representative region for each case where multiple candidate bounding boxes at least partially overlap with the same object of interest. The region trimming component 908 can perform this task using the well-known non-maximum suppression (NMS) algorithm.

[0080] The region proposal technique described above is presented in an illustrative rather than restrictive spirit. Other methods may also be used to identify candidate regions in the input image.

[0081] Figure 10 An implementation of a text encoder neural network (referred to as "text encoder" for brevity) 140 is shown. The text encoder 140 operates by mapping external text 112 to encoded textual information. From a high-level perspective, the text encoder 140 employs a transformer-based architecture. Background information on the general topic of transformer-based architectures can be found in Devlin et al., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (arXiv:1810.04805v2[cs.CL], May 24, 2019, p. 16) and Vaswani et al., "Attention is All You Need" (arXiv:1706.03762v5[cs.CL], December 6, 2017, p. 15). Other implementations can use text encoders employing other types of architectures. For example, another implementation could use a CNN or RNN to map external text 112 to encoded contextual information.

[0082] from Figure 10 Starting at the bottom, the language embedding mechanism 1002 transforms the tokens in the external text 112 into an input embedding set, also referred to herein as input vectors. The language embedding mechanism 1002 can use different techniques to perform this task. In one approach, the language embedding mechanism 1002 can convert each word in the external text 112 into a vector representation, for example, using a lookup table, a neural network, etc. The language embedding mechanism 1002 can also optionally add a special classification token "[cls]" to the beginning of this series of input embeddings.

[0083] Next, addition mechanism 1004 adds positional information to each input embedding. Positional information describes the position of a token (associated with a specific input embedding) among a series of tokens constituting the external text 112. For example, suppose the external text 112 includes the title “hybrid Ford hatchback 2020”. Addition mechanism 1004 adds positional information to the input embedding associated with “hybrid”, indicating that the word “hybrid” is the first token in the external text 112. In summary, text encoder 202 adds positional information to the input embeddings to inform its self-attention mechanism (described below) of the positional context of each token considered within the external text 112. Addition mechanism 1004 can encode positional information in different ways, such as by mapping token indices to positional information using one or more sine functions, or by mapping token indices to positional information using a machine-trained function. In summary, addition mechanism 1004 produces positionally modified embeddings.

[0084] The transformation mechanism 1006 then modifies the embedding set at that position to map it to the transformer output vector. The transformation mechanism 1006 further includes a chain of one or more transformation units, including a representative transformation unit 1008 and one or more other transformation units 1010. The representative transformation unit 1008 includes a series of layers, including a self-attention mechanism 1012, an addition and normalization mechanism 1014, a feedforward neural network 1016, and another addition and normalization mechanism 1018.

[0085] The self-attention mechanism 1012 determines the importance of each token in the external text 112 to each other token. For example, suppose the given external text 112 comprises a set of linguistic tokens, optionally preceded by the classification token "[cls]". When processing each specific word in the external text 112, the self-attention mechanism 1012 considers the relevance of each other word in the external text 112 to that specific word. For example, suppose the external text is "What is the median sales price of this product in the city of Billings, MT?" When processing the word "Billings", the self-attention mechanism 216 can determine that the tokens "city" and "MT" are most useful in correctly eliminating the ambiguity of the term "Billing", for example, because these contextual terms strongly suggest that "Billings" refers to a physical location rather than a financial term.

[0086] The self-attention mechanism 1012 can determine the aforementioned cross-term correlations by packaging the position-modified embeddings into a single matrix X. Then, the self-attention mechanism 1012 linearly projects this matrix X into three matrices Q, K, and V, corresponding to the query matrix, key matrix, and value matrix, respectively, where d k These are the dimensions of the query and the key in Q and K, respectively. The dot product mechanism calculates attention based on the following equation:

[0087]

[0088] The addition and normalization mechanism 1014 adds the input to the self-attention mechanism 1012 (i.e., the position-modified input embedding) to the output of the self-attention mechanism 1012 and then performs layer normalization on the sum. The feedforward network 1016 transforms the output of the addition and normalization mechanism 1014 into the output using a fully connected (FC) feedforward neural network with any number of layers. In one implementation, the feedforward network 1016 can use a linear transformation with scattered activations (e.g., ReLU activations). Finally, another addition and normalization mechanism 1018 adds the input fed to the feedforward network 1016 to the output of the feedforward network 1016 and then normalizes the sum.

[0089] In one implementation, the text encoder 140 may use the transformed counterpart of the [CLS] notation (in the final output layer) as encoding context information. In another implementation, the text encoder 140 uses the output of its entire final layer as encoding context information. In the special case where the electronic document 110 does not contain external text 112, the text encoder 140 may provide default context information that conveys this fact.

[0090] Figure 11A training framework 1102 for generating models associated with any of the aforementioned neural networks is shown. Example generation system 1104 receives images from one or more image sources 1106. In some cases, the images have been annotated to indicate the location of information related to item names within them. In other cases, example generation system 1104 may rely on user groups to manually add these labels. Example generation system 1104 may associate each labeled image with textual information in different ways. In some cases, the image originates from an electronic document including a title and / or title bar, etc., associated with the image. Example generation system 1104 may treat this information as external text associated with the image, if it exists. Alternatively or additionally, example generation system 1104 may retrieve text related to the objects depicted in the image (if known) from an online encyclopedia (e.g., Open Encyclopedia). Example generation system 1104 may treat this information as external text associated with the image. In any case, example generation system 1104 may store all such information it acquires in a data storage 1108. This information constitutes a training example set. One or more training systems 1110 generate training models based on training examples, for example, using stochastic gradient descent or any other training technique(s).

[0091] B. Explanatory Process

[0092] Figures 12 to 15 The process of explaining the operation of the item name identifier system 102 in Section A is illustrated in flowchart form. Since the basic principles of the operation of the item name identifier system 102 have already been described in Section A, some operations will be handled in a generalized manner in this section. As stated in the preface to the detailed embodiments, each flowchart represents a series of operations performed in a specific order. However, the order of these operations is only representative and can be changed in any way.

[0093] More specifically, Figure 12 and Figure 13 Together, we illustrate an overview of a process 1202 that provides an illustrative mode of operation for an item name identifier system 102. Process 1202 may be executed at least in part using one or more neural networks. At box 1204, in response to a triggering event, the item name identifier system 102 receives instructions to manipulate an electronic document, which includes at least an input image and external text appearing in the electronic document outside the input image. At box 1206, the item name identifier system 102 parses the electronic document to identify the input image and external text. At box 1208, the item name identifier system 102 identifies one or more candidate regions in the input image, each candidate region containing an object. At box 1302… Figure 13In this process, item name identifier system 102 determines one or more target regions from one or more candidate regions. Each target region contains an object of interest and is associated with an item name. In some cases, the item name also directly and / or indirectly conveys the entity to which the target region belongs. Box 1302 also relates to generating one or more instances of region feature information associated with one or more target regions respectively. In box 1304, item name identifier system 102 converts in-image text appearing in the input image into in-image text information. In box 1306, item name identifier system 102 converts external text appearing outside the input image into encoded context information. In box 1308, item name identifier system 102 determines the item name associated with a given target region based on the given region feature information, in-image text information, and encoded context information to provide an identified item name.

[0094] Different applications can utilize the Figure 12 and Figure 13 The process 1202 identifies the item name. For example, in Figure 5 In operation 5.5, the item name identifier system 102 identifies supplementary content items associated with the identified item name. Figure 5 In operation 5.6, the item name identifier system 102 sends supplementary content items to the user's computing device via a computer network.

[0095] Figure 14 The description for execution is shown. Figure 13 The process 1402 of the first technique for determining the operation in box 1308. In box 1404, the item name identifier system 102 fuses in-image text information with encoded context information to generate text fusion information. In box 1406, the item name identifier system 102 fuses the text fusion information with given region feature information associated with a given target region to generate combined fusion information. In box 1408, the item name identifier system 102 determines the identified item name based on the combined fusion information.

[0096] Figure 15 The description for execution is shown. Figure 13 The process 1502 of the second technique for determining the operation in box 1308. In box 1504, the item name identifier system 102 determines a first evaluation of the identified item name based on given region feature information. In box 1506, the item name identifier system 102 determines a second evaluation of the identified item name based on in-image text information. In box 1508, the item name identifier system determines a third evaluation of the identified item name based on encoding context information. In box 1510, the item name identifier system 102 determines a final evaluation of the identified item name based on the first, second, and third evaluations.

[0097] C. Representative calculation function

[0098] Figure 16 A computing device 1602 is shown that can be used to implement any aspect of the mechanisms illustrated in the above figures. For example, refer to... Figure 3 , Figure 16 The computing device 1602 of the type shown can be used to implement any user computing device and any server. In all cases, computing device 1602 represents a physical and tangible processing mechanism.

[0099] The computing device 1602 may include one or more hardware processors 1604. The hardware processors 1604 may include, but are not limited to, one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs) and / or one or more application-specific integrated circuits (ASICs). More generally, any hardware processor may correspond to a general-purpose processing unit or a dedicated processor unit.

[0100] The computing device 1602 may also include a computer-readable storage medium 1606 corresponding to one or more computer-readable medium hardware units. The computer-readable storage medium 1606 retains information 1608 of any kind, such as machine-readable instructions, settings, data, etc. Without limitation, for example, the computer-readable storage medium 1606 may include one or more solid-state devices, one or more magnetic hard disks, one or more optical disks, magnetic tapes, etc. Any instance of the computer-readable storage medium 1606 may use any technology for storing and retrieving information. Furthermore, any instance of the computer-readable storage medium 1606 may represent a fixed or movable unit of the computing device 1602. Additionally, any instance of the computer-readable storage medium 1606 may provide volatile or non-volatile retention of information.

[0101] Computing device 1602 may utilize any instance of computer-readable storage medium 1606 in different ways. For example, any instance of computer-readable storage medium 1606 may represent a hardware memory unit (such as random access memory (RAM)) for storing transient information during the execution of a program on computing device 1602 and / or a hardware storage unit (such as a hard disk) for more permanently retaining / archiving information. In the latter case, computing device 1602 also includes one or more drive mechanisms 1610 (such as hard disk drive mechanisms) for storing and retrieving information from the instance of computer-readable storage medium 1606.

[0102] The computing device 1602 can perform any of the functions described above when the hardware processor(s) 1604 executes computer-readable instructions stored in any instance of the computer-readable storage medium 1606. For example, the computing device 1602 can execute computer-readable instructions to perform each process block described in Section B.

[0103] Alternatively or additionally, computing device 1602 may rely on one or more other hardware logic units 1612 to perform operations using a task-specific set of logic gates. For example, the hardware logic units 1612 may include a fixed configuration of hardware logic gates, such as hardware logic gates created and set at manufacturing time and which cannot be changed thereafter. Alternatively or additionally, the other hardware logic units 1612 may include a set of programmable hardware logic gates that can be configured to perform different application-specific tasks. This latter type of device includes, but is not limited to, programmable array logic devices (PALs), general-purpose array logic devices (GALs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), etc.

[0104] Figure 16 The hardware logic circuit system 1614 generally indicates that it includes any combination of hardware processor(s) 1604, computer-readable storage medium(s) 1606, and / or other hardware logic units(s) 1612. The computing device 1602 may employ any combination of hardware processor(s) 1604 that execute machine-readable instructions provided in the computer-readable storage medium(s) 1606 and / or one or more other hardware logic units(s) 1612 that perform operations using a fixed and / or programmable set of hardware logic gates. More generally, the hardware logic circuit system 1614 corresponds to one or more hardware logic units(s) of any type that perform operations based on logic stored and / or otherwise embodied in the hardware logic units(s).

[0105] In some cases (e.g., where computing device 1602 represents a user computing device), computing device 1602 also includes an input / output interface 1616 for receiving various inputs (via input device 1618) and providing various outputs (via output device 1620). Exemplary input devices include keyboard devices, mouse input devices, touchscreen input devices, digitizers, one or more still image cameras, one or more video cameras, one or more depth camera systems, one or more microphones, voice recognition mechanisms, any motion detection mechanisms (e.g., accelerometers, gyroscopes, etc.), etc. A particular output mechanism may include display device 1622 and an associated graphical user interface (GUI) presentation 1624. Display device 1622 may correspond to liquid crystal display devices, light-emitting diode display (LED) devices, cathode ray tube devices, projection mechanisms, etc. Other output devices include printers, one or more speakers, haptic output mechanisms, file mechanisms (for storing output information), etc. Computing device 1602 may also include one or more network interfaces 1626 for exchanging data with other devices via one or more communication channels 1628. One or more communication buses 1630 couple the above units together for communication.

[0106] The multiple communication channels 1628 can be implemented in any way, such as via a local area network, a wide area computer network (e.g., the Internet), a peer-to-peer connection, or any combination thereof. The multiple communication channels 1628 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., managed by any protocol or combination of protocols.

[0107] Figure 16 The computing device 1602 is shown as a discrete set of separate units. In some cases, the set of units may correspond to discrete hardware units provided in a computing device chassis of any form factor. Figure 16 An illustrative shape factor is shown at its bottom. In other cases, the computing device 1602 may include integrated... Figure 1 The hardware logic unit that combines the functions of two or more units shown. For example, computing device 1602 may include a hardware logic unit that incorporates the functions of two or more units. Figure 16 The integrated circuits corresponding to the functions of two or more units in the shown unit are system-on-chip (SoC or SOC).

[0108] The following overview provides a non-exhaustive set of illustrative examples of the techniques described in this article.

[0109] According to a first example, a computer-implemented method for processing an input image is described, the method being performed at least in part using one or more neural networks. The method includes: receiving, in response to a triggering event, an instruction to manipulate an electronic document, the electronic document including at least an input image and external text, the external text appearing in the electronic document outside the input image; parsing the electronic document to identify the input image and the external text; identifying one or more candidate regions in the input image, each candidate region containing an object; determining one or more target regions from the one or more candidate regions using a first neural network, each target region containing an object of interest and associated with an item name, the operation of determining the one or more target regions including generating one or more instances of region feature information respectively associated with the one or more target regions; converting in-image text appearing in the input image into in-image text information using optical character recognition; converting external text appearing outside the input image into encoded context information using a second neural network; determining a specific item name associated with a given target region based on the given region feature information associated with the given target region, the in-image text information, and the encoded context information; identifying supplementary content items associated with the specific item name; and sending the supplementary content items to a user computing device via a computer network.

[0110] According to the second example, a specific item name is associated with a specific brand, and a given target area includes a logo associated with the specific brand.

[0111] According to the third example, for another case, the received electronic document does not include a target region, and / or does not include in-image text, and / or does not include external text, and the method includes generating default region feature information in the absence of a target region, and / or generating default in-image text information in the absence of in-image text, and / or generating default encoding context information in the absence of external text.

[0112] According to the fourth example, the triggering event is either an indication that the user has accessed the electronic document using the user's computing device, or a confirmation that the electronic document will be sent to the user's computing device. Supplementary content items are presented to the user's computing device as part of the electronic document.

[0113] According to the fifth example, optical character recognition produces OCR output results, wherein the conversion of in-image text to in-image text information also includes encoding the OCR output results into in-image text information.

[0114] According to the sixth example, the operation of determining the specific item name includes: fusing text information within the image with encoding context information to generate text fusion information; fusing the text fusion information with given region feature information associated with a given target region to generate combined fusion information; and determining the specific item name based on the combined fusion information.

[0115] According to the seventh example related to the sixth example, the fusion of text information within the image and the encoding context information is performed by the third neural network, and the fusion of text fusion information and the feature information of the given region is performed by the fourth neural network.

[0116] According to Example 8, the operation of determining a specific item name includes: determining a first evaluation of the specific item name based on given region feature information; determining a second evaluation of the specific item name based on in-image text information; determining a third evaluation of the specific item name based on encoding context information; and determining a final evaluation of the specific item name based on the first evaluation, the second evaluation, and the third evaluation.

[0117] According to the ninth example, one or more computing devices for processing input images are described. The computing devices include: a hardware logic circuit system implementing at least one or more neural networks, the hardware logic circuit system being configured to implement a method comprising: receiving, in response to a triggering event, an instruction to operate on an electronic document, the electronic document including at least an input image and external text, the external text appearing in the electronic document outside the input image; parsing the electronic document to identify the input image and the external text; identifying one or more candidate regions in the input image, each candidate region containing an object; determining one or more target regions from the one or more candidate regions using a first neural network provided by the hardware logic circuit system, each target region containing an object of interest and associated with an item name, the operation of determining the one or more target regions including generating one or more instances of region feature information respectively associated with the one or more target regions; converting in-image text appearing in the input image into in-image text information; converting external text appearing outside the input image into encoded context information using a second neural network provided by the hardware logic circuit system; and determining a specific item name associated with a given target region based on the given region feature information associated with the given target region and the in-image text information.

[0118] According to the tenth example related to the ninth example, the triggering event is an indication that the user has used the user's computing device to access the electronic document, or the triggering event is a determination that the electronic document will be sent to the user's computing device. The operation also includes: identifying supplementary content items associated with a specific item name; and sending the supplementary content items to the user's computing device, whereby the supplementary content items are presented as part of the electronic document.

[0119] According to the eleventh example related to the ninth example, an electronic document is a member of an electronic document set, and the triggering event is the identifier of the electronic document group to be processed, and the operation also includes the distribution of the names of items within the electronic document set.

[0120] According to Example Twelve, which is related to Example Nine, the triggering event is the submission of an electronic document by a submitting entity, and the operation includes determining whether a specific item name is consistent with other information presented in the electronic document.

[0121] According to Example 13, which is related to Example 9, a specific item name is associated with a specific brand, and a given target area includes a logo associated with the specific brand.

[0122] According to the fourteenth example related to the ninth example, the conversion from in-image text to in-image text information is performed at least in part by an optical character recognition (OCR) component, which is implemented by a hardware logic circuit system and generates an OCR output. Furthermore, the conversion from external text to in-image text information also includes encoding the OCR output into in-image text information.

[0123] According to Example Fifteen, which is related to Example Nine, determining the specific item name includes: fusing text information within the image with encoding context information to generate text fusion information; fusing the text fusion information with feature information of a given region to generate combined fusion information; and determining the specific item name based on the combined fusion information.

[0124] According to the sixteenth example related to the fifteenth example, the fusion of text information within the image and the encoding context information is performed by a third neural network, and the fusion of text fusion information and the feature information of a given region is performed by a fourth neural network, wherein the third neural network and the fourth neural network are implemented by a hardware logic circuit system.

[0125] According to Example 17, which is related to Example 9, the determination of a specific item name includes: determining a first evaluation of the specific item name based on given region feature information; determining a second evaluation of the specific item name based on in-image text information; determining a third evaluation of the specific item name based on encoding context information; and determining a final evaluation of the specific item name based on the first evaluation, the second evaluation, and the third evaluation.

[0126] According to Example 18, a computer-readable storage medium for storing computer-readable instructions is described. The computer-readable instructions, when executed by one or more hardware processors, perform a method comprising: receiving, in response to a triggering event, instructions to operate on an electronic document, the electronic document including at least an input image and external text appearing outside the input image; parsing the electronic document to identify the input image and the external text; identifying one or more candidate regions in the input image, each candidate region containing an object; determining one or more target regions from the one or more candidate regions, each target region containing an object of interest and associated with an item name, the operation of determining the one or more target regions including generating one or more instances of region feature information respectively associated with the one or more target regions; converting in-image text appearing in the input image into in-image text information; converting external text appearing outside the input image into encoded context information; fusing the in-image text information with the encoded context information to generate text fusion information; fusing the text fusion information with given region feature information associated with a given target region to generate combined fusion information; and determining a specific item name associated with the given target region based on the combined fusion information.

[0127] According to Example 19, which is related to Example 18, the triggering event is an indication that the user has accessed the electronic document using the user's computing device, or the triggering event is a determination that the electronic document will be sent to the user's computing device.

[0128] According to the twentieth example related to the eighteenth example, the method is executed using one or more neural networks implemented with computer-readable instructions.

[0129] The twenty-first aspect corresponds to any combination of the first to twentieth examples above (e.g., any logically consistent permutation or subset).

[0130] The twenty-second aspect corresponds to any method counterpart, device counterpart, system counterpart, apparatus plus function counterpart, computer-readable storage medium counterpart, data structure counterpart, article counterpart, graphical user interface presentation counterpart, etc., related to the first to twenty-first examples.

[0131] Finally, the features described in this paper can employ various mechanisms to ensure that any user data is processed in a manner that complies with applicable laws, social norms, and the expectations and preferences of individual users. For example, the feature can allow users to explicitly opt in (and then explicitly opt out) of the feature's terms. The feature can also provide appropriate security mechanisms to ensure the privacy of user data (e.g., data sanitization, encryption, password protection, etc.).

[0132] Furthermore, the description has already set forth various concepts in the context of the illustrative challenge or problem. This interpretation is not intended to imply that others have understood and / or clarified the challenge or problem in the manner specified herein. Moreover, this interpretation does not mean that the subject matter recited in the claims is limited to solving the identified challenge or problem; that is, the subject matter in the claims can be applied to the context of other challenges or problems beyond those described herein.

[0133] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary forms for implementing the claims.

Claims

1. A computer-implemented method for processing an input image, the method being performed at least in part using one or more neural networks, the method comprising: In response to a triggering event, an instruction to operate on an electronic document is received, the electronic document including at least an input image and external text, the external text appearing in the electronic document outside the input image; The electronic document is parsed to identify the input image and the external text; Identify one or more candidate regions in the input image, each candidate region containing an object; A first neural network is used to determine one or more target regions from the one or more candidate regions, each target region containing an object of interest and associated with an item name. The determination of the one or more target regions includes: generating one or more instances of regional feature information associated with the one or more target regions respectively; Optical character recognition is used to convert in-image text appearing in the input image into in-image text information; A second neural network is used to convert the external text appearing outside the input image into encoded context information; and Based on the given region feature information associated with a given target region, the in-image text information, and the encoding context information, a specific item name associated with the given target region corresponding to one of the one or more target regions is determined. The determination of the specific item name includes: The text information within the image is fused with the encoded context information to generate text fusion information; The text fusion information is fused with the given region feature information associated with the given target region to generate combined fusion information; and The specific item name is determined based on the combined and fused information.

2. The method of claim 1, wherein the specific item name is associated with a specific brand, and wherein the given target area includes a logo associated with the specific brand.

3. The method of claim 1, wherein, for another case, the received electronic document does not include the target area, and / or does not include text within an image, and / or does not include external text, and the method comprises: Generate default region feature information in the absence of a target region, and / or generate default in-image text information in the absence of in-image text, and / or generate default encoding context information in the absence of external text.

4. The method according to claim 1, The triggering event is an indication that the user has accessed the electronic document using the user's computing device, or a determination that the electronic document will be sent to the user's computing device. The supplementary content items are presented to the user's computing device as part of the electronic document.

5. The method of claim 1, wherein optical character recognition generates an OCR output result, and wherein converting the text within the image into text information within the image further comprises: The OCR output is encoded into text information within the image.

6. The method according to claim 1, The fusion of the in-image text information and the encoded context information is performed by a third neural network, and The fusion of the text fusion information and the given region feature information is performed by a fourth neural network.

7. The method according to claim 1, further comprising: Identify supplementary content items associated with the specific item name; as well as The supplementary content items are sent to the user's computing device via a computer network.

8. One or more computing devices, the one or more computing devices being configured to process an input image, comprising: A hardware logic circuit system implementing at least one or more neural networks, the hardware logic circuit system being configured to implement a method comprising: In response to a triggering event, an instruction to operate on an electronic document is received, the electronic document including at least an input image and external text, the external text appearing in the electronic document outside the input image; The electronic document is parsed to identify the input image and the external text; Identify one or more candidate regions in the input image, each candidate region containing an object; One or more target regions are determined from the one or more candidate regions using a first neural network provided by the hardware logic circuit system. Each target region contains an object of interest and is associated with an item name. The determination of the one or more target regions includes: generating one or more instances of regional feature information associated with the one or more target regions respectively; Convert the in-image text appearing in the input image into in-image text information; The external text appearing outside the input image is converted into encoded context information using a second neural network provided by the hardware logic circuit system; and Based on the given region feature information associated with a given target region, the in-image text information, and the encoding context information, a specific item name associated with the given target region corresponding to one of the one or more target regions is determined. The determination of the specific item name includes: The text information within the image is fused with the encoded context information to generate text fusion information; The text fusion information is fused with the given region feature information associated with the given target region to generate combined fusion information; and The specific item name is determined based on the combined and fused information.

9. One or more computing devices according to claim 8, wherein the triggering event is an indication that a user has used the user computing device to access the electronic document, or the triggering event is a determination that the electronic document will be sent to the user computing device, and wherein the operation further comprises: Identify supplementary content items associated with the specific item name; as well as The supplementary content item is sent to the user computing device, and the supplementary content item is presented to the user computing device as part of the electronic document.

10. The computing device of claim 8, wherein the electronic document is a member of an electronic document set, and wherein the triggering event is an identifier of the electronic document group to be processed, and wherein the operation further comprises: The distribution of item names within the electronic document set is identified.

11. One or more computing devices according to claim 8, wherein the triggering event is a submission of the electronic document by a submitting entity, and wherein the operation includes: Determine whether the specific item name is consistent with other information presented in the electronic document.

12. One or more computing devices according to claim 8, wherein the specific item name is associated with a specific brand, and wherein the given target area includes a logo associated with the specific brand.

13. One or more computing devices according to claim 8, The conversion from in-image text to in-image text information is performed at least in part by an optical character recognition (OCR) component, which is implemented by the hardware logic circuit system and generates an OCR output. The conversion from in-image text to in-image text information further includes: The OCR output is encoded into text information within the image.

14. One or more computing devices according to claim 8, The fusion of the in-image text information and the encoded context information is performed by a third neural network. The fusion of the text fusion information and the given region feature information is performed by a fourth neural network. The third neural network and the fourth neural network are implemented by the hardware logic circuit system.

15. A computer-readable storage medium for storing computer-readable instructions, said computer-readable instructions executing a method when executed by one or more hardware processors, said method comprising: In response to a triggering event, an instruction to operate on an electronic document is received, the electronic document including at least an input image and external text, the external text appearing outside the input image; The electronic document is parsed to identify the input image and the external text; Identify one or more candidate regions in the input image, each candidate region containing an object; One or more target regions are determined from the one or more candidate regions, each target region containing an object of interest and associated with an item name. The determination of the one or more target regions includes: generating one or more instances of regional feature information associated with the one or more target regions respectively; Convert the in-image text appearing in the input image into in-image text information; The external text appearing outside the input image is converted into encoded context information; The text information within the image is fused with the encoded context information to generate text fusion information; The text fusion information is fused with given region feature information associated with a given target region, corresponding to one of the one or more target regions, to generate combined fusion information; and The specific item name associated with the given target region is determined based on the combined and fused information.

16. The computer-readable storage medium of claim 15, wherein the triggering event is an indication that a user has accessed the electronic document using a user computing device, or the triggering event is a determination that the electronic document will be sent to the user computing device.

17. The computer-readable storage medium of claim 15, wherein the method is performed using one or more neural networks implemented by the computer-readable instructions.

Images