Object detection in images

Abstract

In an implementation of object detection in images, a heterogeneous training dataset is used to train an object detector. A first training dataset is used to train an image labeling network to determine an attention map of an input image for a target concept. A second training dataset is used to train a conditional detection network that accepts an attention map and a word embedding of the target concept as conditional input. Although the conditional detection network is trained using a training dataset with a small number of visible classes (e.g., classes in the training dataset), it generalizes to novel, unseen classes through concept conditioning because the target concept propagates through the conditional detection network via the conditional input, thereby affecting classification and region proposals. Thus, the classes of objects that can be detected are expanded without having to expand the training database to include additional classes.

Description

Background Art

[0001] An object detector detects objects in an image, such as using an adaptive model (e.g., a machine learning model, a neural network, etc.) that has been trained with a dataset of images. The performance of an object detector is typically limited to the detection of objects belonging to classes included in the dataset of images used to train the adaptive model, referred to as "visible" classes or categories. Thus, an object detector may be able to detect a dog in an image, but not a tree in an image, when the training dataset includes a dog class but not a tree class.

[0002] In some cases, an object detector biases its detection results toward a visible class, such as the object class of the training data set that is closest to the target class. For example, an object detector may detect a fox in an image as a dog when the training data set includes a dog class but not a fox class.

[0003] In addition, some object detectors, such as zero-shot detectors, can attempt to transfer knowledge from object categories of the training data set (e.g., visible classes) to object categories that are not included in the training data set (e.g., unseen classes). However, these object detectors are limited to transferring knowledge from visible classes to unseen classes for classification purposes rather than object detection and region proposal purposes. As a result, these object detectors often fail to detect image regions of objects of unseen classes.

[0004] To overcome these shortcomings of object detectors, the training dataset of images can be scaled to include additional categories (e.g., tens of thousands of visible classes). However, scaling the training dataset of images may be prohibitive in terms of cost and time. For example, the images need to be annotated to include the additional categories, and the objects in the images corresponding to the additional categories need to be determined, requiring a lot of manual user effort. In addition, the ambiguity of certain class labels exacerbates the difficulty of scaling the training dataset to include other classes or labels, such as assigning a "banana" label to a bunch of bananas or strictly to a single banana. Therefore, the performance of object detectors is still limited by the visible classes of the training dataset used to train the object detector. Summary of the invention

[0005] Techniques, systems, and devices for detecting objects in images with an object detector are described. The object detector is trained using a heterogeneous training dataset of images and can detect objects in images of visible and invisible classes. A first training dataset includes a large-scale image label dataset with image-level annotations (e.g., each image includes multiple labels from a larger label vocabulary) and is used to train a labeling network of an object detector. Based on an input condition including a word-based concept, the labeling network generates an attention map indicating the presence value of the word-based concept within the input image, such as indicating for each pixel of the input image the corresponding presence of the word-based concept having a presence value within a specified range. Word embeddings for the word-based concept are determined, which describe the relationship between the word-based concept and different word-based concepts such as labels from a vocabulary. The attention map generated by the labeling network pre-trained on the first dataset and the word embeddings generated from the word-based concept are provided as conditional inputs to a conditional detection network of the object detector. The conditional detection network is trained using a second training dataset of images and the conditional input generated from the pre-trained labeling network. The second training dataset includes images with a small number of visible classes (e.g., hundreds of classes) annotated with corresponding bounding boxes. By conditioning the attention map and word embeddings on a target concept (e.g., a word-based concept), the conditional detection network is image label agnostic and acts as a concept-specific object detector. For example, even if the conditional detection network is trained using a training data set with a small number of visible classes, it generalizes to novel, unseen classes through concept conditioning because the word-based concepts propagate through the conditional detection network (via word embeddings and attention maps) and thus influence the classification and region proposals of the conditional detection network. Therefore, the object classes detectable by the object detector are scaled by the techniques, systems, and devices described herein without scaling the training database to include additional classes, thereby saving time and money.

[0006] This Summary introduces a selection of concepts in a simplified form that are further described in the Detailed Description below. Therefore, this Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the figure in which the reference number first appears. The use of the same reference number in different instances in the specification and the drawings may indicate similar or identical items. An entity represented in a drawing may refer to one or more entities, and thus single or multiple forms of an entity may be referred to interchangeably in the discussion.

[0008] Figure 1A digital media environment in an example implementation that is operable to employ the techniques described herein is shown.

[0009] Figure 2 The generation of an exemplary attention map according to one or more aspects of the present disclosure is shown.

[0010] Figure 3 An exemplary condition detection network according to one or more aspects of the present disclosure is shown.

[0011] Figure 4 An exemplary training of an object detection system according to one or more aspects of the present disclosure is shown.

[0012] Figure 5 An example system in accordance with one or more aspects of the present disclosure is shown.

[0013] Figure 6 An example system in accordance with one or more aspects of the present disclosure is shown.

[0014] Figure 7 A flowchart depicting an example process in accordance with one or more aspects of the present disclosure is shown.

[0015] Figure 8 A flowchart depicting an example process in accordance with one or more aspects of the present disclosure is shown.

[0016] Fig. 9 A flowchart depicting an example process in accordance with one or more aspects of the present disclosure is shown.

[0017] Fig.10 An example system is shown that includes various components of an example device that can be implemented as a reference Figure 1-9 Any type of computing device described and / or utilized to implement aspects of the techniques described herein. DETAILED DESCRIPTION

[0018] Overview

[0019] Object detectors detect objects in images and can be used in various situations where image processing is required, such as image editing applications for user devices, guidance systems for vehicles (e.g., self-driving cars, drones, robots, etc.), image capture systems (e.g., focusing systems for digital cameras), color pickers, etc. However, the performance of object detectors is typically limited by the object categories (e.g., visible object classes) of the training data set used to train the object detector. Therefore, when the training data set includes a dog class but not a fox class, the object detector may be able to detect dogs in the image, but not foxes in the image, or bias the detection results by detecting foxes as dogs. In addition, some object detectors, such as zero-shot detectors, transfer knowledge from visible classes to invisible classes for classification purposes but not for object detection and region proposal purposes. Therefore, these object detectors often fail to detect image regions of objects of invisible classes.

[0020] In order to improve the detection results of an object detector, the number of visible classes in the training dataset can be scaled (e.g., increased). However, scaling the training dataset of images may be prohibitive in terms of cost and time, especially for a large number of classes (e.g., tens of thousands of classes) and ambiguous classes.

[0021] Thus, the present disclosure describes systems, techniques, and apparatus for detecting objects in images using object detectors for visible and invisible classes without scaling the training database by adding new additional classes to the training database. The object detector is trained using a heterogeneous training data set of images. A first training data set is used to train a labeled network, and a second training set is used together with the output of a pre-trained labeled network to train a conditional detection network.

[0022] The first training data set includes a large-scale image label data set with image-level annotations (e.g., each image includes multiple labels of image features from a larger phrase label), and is used to train a labeling network of an object detector. In one example, the first training data set is collected from a stock photo web service such as Adobe Stock. Based on input conditions including word-based concepts, the labeling network generates an attention map specific to the word-based concepts. For example, the attention map can indicate the presence value of the word-based concept spatially within the input image, such as indicating the corresponding presence value of the word-based concept in the range [-1,1] for each pixel of the input image.

[0023] Based on input conditions including word-based concepts, word embeddings for word-based concepts are generated. The word embeddings describe the relationship between the word-based concepts and different word-based concepts, such as tags from a vocabulary. In one example, word embeddings are generated based on mutual information by constructing a matrix from co-occurrence probabilities of the word-based concepts and different word-based concepts or tags, and forming vectors from eigendecomposition of the matrix.

[0024] The attention map generated by the labeling network pre-trained by the first data set and the word embedding generated from the word-based concept are provided as conditional input to the conditional detection network of the object detector. The conditional detection network may include any suitable network for detecting objects in an image. In one example, the conditional detection network of the object detector includes a backbone network that accepts the conditional input and a convolutional neural network that proposes object regions in the image.

[0025] A conditional detection network of an object detector is trained using a second training dataset of images as input images. The conditional input of the conditional detection network is generated from a pre-trained labeling network (pre-trained with the first training database). The pre-trained labeling network also supplies images from the second training dataset to generate conditional inputs. The second training dataset includes images with a small number of visible classes (e.g., hundreds of classes) annotated with corresponding bounding boxes. In an example, the second training dataset includes an open public dataset for image labeling and classification.

[0026] Additionally or alternatively, the conditional inputs (e.g., word embeddings and attention maps) of the conditional detection network can be zeroed so that the conditional detection network can operate and be trained for traditional multi-class object detection rather than being conditioned on word-based concepts. For example, the presence value of the attention map can be set to 1, and the value of the word embedding can be set to zero.

[0027] In one example, the conditional detection network includes a binary classifier that assigns a positive training label to detection outputs of the conditional detection network that substantially overlap with a truth bounding box of a word-based concept. The binary classifier assigns a negative training label to other detection outputs of the conditional detection network that are not assigned a positive training label.

[0028] Additionally or alternatively, the conditional detection network can be trained using negative classes about the target class. For example, a negative class based on a concept of a word can be determined, such as by selecting a negative class from multiple classes that are most similar to the target class according to a metric (e.g., cosine similarity). As an example, for a target class of "dog", the negative class may include "cat", "rabbit", "fox", and "bear". The selected negative class is provided to the conditional detection network and a negative training label is assigned to the detection output of the conditional detection network, which substantially overlaps with the truth bounding box corresponding to the target class.

[0029] The attention map and word embedding are provided as conditional inputs to the conditional detection network, so that the concept based on the word is propagated through the conditional detection network and affects the detection results of the conditional detection network. In one example, the word embedding is provided to two layers of the backbone network of the conditional detection network, and the attention map is provided to the third layer of the backbone network of the conditional detection network. The third layer can be located between the two layers of the backbone network of the conditional detection network. For example, the 1st and 3rd layers of the fourth level of the backbone network can receive the word embedding, and the 2nd layer of the 4th level of the backbone network can receive the attention map.

[0030] By conditioning on target concepts (e.g., word-based concepts) with attention maps and word embeddings, the conditional detection network is image label agnostic and acts as a concept-specific object detector. For example, even if the conditional detection network is trained with a training data set with a small number of visible classes, it generalizes to novel, unseen classes through concept conditioning. Thus, the object classes that can be detected by the object detector are scaled to include unseen classes without the need to scale the training database to include unseen classes, thereby saving the user effort, time, and money.

[0031] In the following discussion, an example digital media environment is described in which the techniques described herein may be employed. Example implementation details and processes are then described that may be performed in the example digital media environment as well as other environments. Thus, performance of the example processes is not limited to the example environment, and the example environment is not limited to performance of the example processes.

[0032] Sample Digital Media Environment

[0033] Figure 1 1 is an illustration of a digital media environment 100 operable to employ an example implementation of the techniques described herein. As used herein, the term "digital media environment" refers to various computing devices and resources that can be used to implement the techniques described herein. The illustrated digital media environment 100 includes a user 102 having at least one computing device. Figure 1In the example of FIG. 1 , user 102 is shown as having three computing devices, computing device 104-1, computing device 104-2, and computing device 104-3 (collectively, computing devices 104). For example, computing device 104-1 depicts a desktop computer, computing device 104-2 depicts a smart phone, and computing device 104-3 depicts a pair of glasses, such as smart goggles (“goggles”). Computing devices 104 are example computing devices, and any suitable computing device is contemplated, such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a gaming device, goggles, glasses, a camera, a digital assistant, a wearable device (e.g., a watch, an armband, an adhesive patch, etc.), an echo device, an image editor, a non-linear editor, a digital audio workstation, a copier, a scanner, etc., which may include an application to detect objects in an image. Furthermore, discussion of one of computing devices 104 is not limited to that computing device, but applies generally to each of computing devices 104. Furthermore, computing devices 104 may range from full resource devices with large amounts of memory and processor resources (eg, personal computers, game consoles) to low resource devices with limited memory or processing resources (eg, mobile devices).

[0034] Various types of input devices and input infrastructure may be used to provide input to computing device 104. For example, computing device 104 may recognize input as mouse input, stylus input, touch input, input provided through a natural user interface, user gestures on a touch screen, combinations thereof, and the like. Thus, computing device 104 may recognize multiple types of gestures, including touch gestures and gestures provided through a natural user interface. In one example, computing device 104 includes speech recognition, identification and synthesis capabilities, a microphone, and a speaker, which allows computing device 104 to communicate with user 102 in a conversation. In addition, computing device 104 may include an image capture device (e.g., a camera) configured to capture images and video streams.

[0035] Additionally, computing device 104 may represent one or more different devices, such as one or more devices connected to a network that perform operations “through the cloud,” such as with respect to Fig.10 In one example, computing device 104 utilizes a low-power wireless communication standard (e.g. Protocol) to be communicatively coupled to each other. For example, computing device 104-1 can wirelessly communicate with computing device 104-2. Thus, an image processed on one device (e.g., computing device 104-1) can be transmitted to another device (e.g., computing device 104-2) and displayed on the other device (e.g., computing device 104-2).

[0036] The computing device 104 is also coupled to a network 106. The network 106 communicatively couples the computing device 104 with a server 108. For clarity, Figure 1 1 is shown as coupled to network 106, although computing device 104-2 and computing device 104-3 may also be coupled to server 108 via network 106. Network 106 may include various networks, such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a personal area network (PAN), a cellular network, a terrestrial network, a satellite network, a combination of networks, etc., and thus may be wired, wireless, or a combination thereof.

[0037] Servers 108 may include one or more servers or service providers that provide services, resources, or a combination thereof to computing devices 104. In one example, resources provided by servers 108 may be licensed, purchased, or available for free, (e.g., without authentication, license, or account-based access). Resources may include any suitable combination of services and content, such as made available by one or more providers over network 106. Some examples of services include, but are not limited to, online shopping services, photo editing services, image database services (e.g., services that provide training images from a database), web development and management services, collaboration services, social networking services, messaging services, advertising services, graphic design services, image storage services (including storage and access to photos, documents, records, files, etc.), etc. Content may include a variety of asset combinations, including videos, advertisements, audio, multimedia streams, animations, images, reference images, web documents, web pages, applications, device applications, text documents, drawings, presentations, stock photos, user profiles, user preferences, user data (e.g., images stored in a gallery), etc.

[0038] exist Figure 1 In the example of , server 108 includes object detection system 110, which includes object detection application 112 (discussed in more detail below), and computing device 104 includes image system 114. Object detection system 110 and image system 114 can work together to detect objects in an image. For example, object detection system 110 can train an adaptive model, such as one or more neural networks of an object detector, to detect objects in an image, and provide the trained model to image system 114 of computing device 104 via network 106. One or more of computing devices 104 can use the pre-trained adaptive model provided by object detection system 110 with image system 114 to detect objects in an image.

[0039] For example, Figure 1In the example of FIG. 1 , user 102 obtains input image 116 via computing device 104-2. Input image 116 may be any suitable input image, such as a picture, a frame of a video sequence, an animation, a drawing, a cartoon, a diagram, a document, a combination thereof, etc. Input image 116 shows Figure 1 . The user 102 provides input condition 118 by speaking the word-based concept “wheel”. Based on the word-based concept “wheel” of input condition 118, the image system 114 generates an image 120, which is exposed in the user interface on the computing device 104-3. Image 120 includes an image of a car from input image 116 and a detection result 122. The detection result 122 includes a bounding box, as shown by the dashed rectangle, representing an area of ​​the image 120 that includes an object corresponding to the word-based concept “wheel”. Therefore, the image system 114 uses a pre-trained adaptive model, such as a neural network obtained from the object detection system 110 of the server 108, to detect image objects conditioned on the word-based concept. The word-based concept does not need to be part of the training data set, and the adaptive model is trained based on the training data set (for example, the word-based concept is invisible relative to the training data set).

[0040] The object detection system 110 includes a display 124. The display 124 may be any suitable type of display, such as a liquid crystal display, a plasma display, a head mounted display, a projector and screen, a touch screen that recognizes user gestures (e.g., touch gestures), etc. The touch screen of the display 124 may include any suitable type of touch screen, such as a capacitive touch screen, a resistive touch screen, a surface acoustic wave touch screen, an infrared touch screen, an optical imaging touch screen, an acoustic pulse recognition touch screen, a combination thereof, etc. In addition, the display 124 may display any suitable interface.

[0041] The object detection system 110 also includes a processor 126. The processor 126 may include any suitable type and number of processors. Thus, the object detection system 110 may be implemented at least in part by executing instructions stored on the storage device 128 on the processor 126. For example, the processor 126 may execute a portion of the object detection application 112.

[0042] The storage device 128 may be any suitable type of storage device accessible to or included in the object detection system 110. The storage device 128 stores and provides access to and from memory included in the storage device 128 for any suitable type of data. For example, the storage device 128 includes image data 130, including data associated with an image, such as an input image, a training image, an image of a training database, an image region corresponding to an object, annotations of the image (e.g., annotations including bounding boxes, feature descriptions, etc.), a source of the image (e.g., an indicator of a database or image library from which the image was obtained, a descriptor of a capture device used to capture the image, a name of a photographer who captured the image, an online store from which the image is available, combinations thereof, etc.), metadata of the image (e.g., settings of a capture device used to capture the image, such as aperture, shutter speed, focal length, ISO (a measure of light sensitivity of a camera sensor), global positioning system (GPS) coordinates of a location where the image was captured, a size of the image (e.g., a resolution in number of pixels), a date when the image was captured, etc.), a format of the image (e.g., a file format), an identifier of the image in a series of images, such as a video sequence, a thumbnail image, combinations thereof, etc.

[0043] The storage device 128 also includes word data 132, including data about input conditions, such as word-based concepts, word embeddings, vocabularies, records of verbal input conditions, conversions of input conditions (e.g., to different languages), transcriptions of input conditions (e.g., from records to text documents), identification of an algorithm used to generate word embeddings for word-based concepts, parameters of word embeddings, such as vector length, vocabulary size, etc., an indication of whether the word-based concept is visible or invisible (e.g., whether the word-based concept has a corresponding label in an image in a training data set used to train an adaptive model), a language for the word-based concept, parameters of a broadcast function used to reshape the word embedding, combinations thereof, and the like.

[0044] The storage device 128 also includes attention map data 134, including data about the attention map generated by the object detection system 110, such as feature maps, spatial responses, presence values, ranges of presence values ​​(e.g., a range of [-1, 1]), pixel values ​​(e.g., integers from 0 to 255), weights of an image labeling network, number of channels of a neural network, cosine similarity scores, combinations thereof, and the like.

[0045] The storage device 128 also includes detection data 136, including data about detection results generated by the object detection system 110, such as bounding boxes, region proposals, identifiers of classes, the number of bounding boxes identified for target concepts, output images, confidence scores for region proposals, outputs of binary classifiers indicating positive or negative labels, user preferences for detection results (e.g., color or line selection selected by the user to display bounding boxes), indications of layers or levels of the network configured to receive conditional inputs, indications of whether the conditional inputs are used or zeroed, the length of time required to generate a detection result, the difference between a current detection result and a previous detection result (e.g., the difference in detection results from one frame of a video sequence to another frame of the video sequence), combinations thereof, and the like.

[0046] The storage device 128 also includes training data 138, including data about the training of components of the object detection system 110, such as a first training database for training an image labeling network, a second training database for training a conditional detection network, weights of a network (e.g., an image labeling network, a backbone network, a convolutional neural network, etc.), training losses calculated while training the network, sources of training data sets, indicators of images in the training data sets used to train the network, indicators of images in the training data sets that were skipped when training the network, indicators of ignored, positive and negative regions detected by the object detector in training, combinations thereof, and the like.

[0047] In addition, the object detection system 110 includes a transceiver module 140. The transceiver module 140 represents functionality configured to send and receive data using any suitable type and number of communication protocols. For example, data within the object detection system 110 can be sent to the server 108 using the transceiver module 140. In addition, data can be received from the server 108 using the transceiver module 140. The transceiver module 140 can also send data to and receive data from the computing device 104. In one example, the transceiver module 140 includes a low-power wireless communication standard (e.g., a wireless communication protocol) for transmitting data between computing devices. protocol).

[0048] Object detection system 110 also includes asset 142. In one example, asset 142 is stored in storage device 128. Asset 142 may include any suitable asset used or generated by object detection system 110. In one example, asset 142 includes an adaptive model, such as a neural network, a machine learning model, etc., that has been trained by object detection system 110. Thus, the pre-trained model of asset 142 may be provided to computing device 104 from server 108 via network 106 and used in any suitable application for detecting objects in an image, such as image 120.

[0049] The object detection system 110 also includes an object detection application 112. The object detection application 112 includes an image module 144, an input condition module 146, an attention map module 148, a word embedding module 150, a detection module 152, and a training module 154. These modules work in conjunction with each other to generate detection results including detected objects in an image.

[0050] The image module 144 represents a function configured to obtain an image. The image module 144 can obtain any suitable representation of the image in any suitable manner. In one example, the image module 144 obtains the image from an image database, such as a database including a training data set of images suitable for training an object detector to detect objects in the image. For example, the image module 144 can obtain a first training data set including a large-scale image label data set with image-level annotations (e.g., each image includes multiple labels from a larger label vocabulary), and a second training data set including images with a small number of visible classes (e.g., hundreds of classes), the visible classes are annotated with corresponding bounding boxes. In one example, the first training data set is used to train one or more networks (e.g., an image labeling network) of the attention map module 148, and the second training data set is used to train one or more networks (e.g., a conditional detection network, such as a backbone network and a convolutional neural network) of the detection module 152.

[0051] Additionally or alternatively, image module 144 may obtain an image from image data 130 of storage device 128. For example, image module 144 may obtain an input image that is not a training image, such as an image captured by user 102 using a camera on one of computing devices 104, and object detection system 110 may detect one or more objects in the input image captured by user 102.

[0052] In one example, the image module 144 obtains the image by receiving a user-provided image. For example, the user 102 may supply the image 116 to the object detection system 110. Thus, the image module 144 may obtain the image by, for example, extracting the image from the image data 130, receiving a user-provided image, or a combination thereof.

[0053] The images obtained by the image module 144 may be any suitable type of image, such as a stand-alone image (e.g., an image that is not related to other images), an image in a sequence of images (e.g., an animation sequence, a video, a page in a chapter of a book, a slide in a slide show, a training sequence of images, etc.), or a combination thereof. In one example, the images obtained by the image module 144 are extracted from an asset that contains other types of media in addition to images, such as a web page containing images and text.

[0054] The images obtained by the image module 144 may include any suitable metadata, such as annotations indicating labels from a vocabulary that correspond to objects in the image belonging to predetermined classes; coordinates of bounding boxes of objects in the image; labels determined to be similar to target labels; data describing the source of the image; settings of the capture device used to capture the image; the size of the image (e.g., resolution in terms of number of pixels); the date the image was captured, etc.; the format of the image (e.g., file format); an image identifier in a sequence of images, such as a video sequence or a training data set; a thumbnail version of the image, such as can be displayed in a user interface to select and expand the image in the user interface, combinations thereof, and the like.

[0055] The images obtained by the image module 144 are stored in the image data 130 of the storage device 128 and are available to the modules of the object detection application 112, along with any suitable information used and calculated by the image module 144, such as the source location of the image; the file format of the image; an indication of whether the image is related to other images, such as a sequence number in an animation sequence or training data set; image metadata (e.g., information about objects of a predetermined class or category in the image, such as labels, bounding boxes, etc.); a thumbnail version of the image, etc. In one example, the image module 144 obtains an input image, such as a training image of a training data set suitable for training one or more networks of the object detection system 110, and provides the input image to the attention map module 148 and the detection module 152. Additionally or alternatively, the image module 144 can provide the input image to the training module 154 so that the training module 154 can generate training updates based on the input image and provide the training updates to the detection module 152 to update the weights of the one or more networks.

[0056] The input condition module 146 represents a function configured to obtain input conditions, such as word-based concepts. The input condition module 146 can obtain the input conditions in any suitable manner. In one example, the input condition module 146 obtains the input conditions via user input, such as a verbal command or typed text (e.g., the user can speak into a microphone or type into a keyboard, or both). Additionally or alternatively, the input condition module 146 can obtain the input conditions from a file, such as a text file. For example, the input condition module 146 can obtain a file containing a list of words, tags, or word-based concepts, and select entries of the list as input conditions as part of a training process for one or more networks of the object detection system 110. In one example, the file is obtained from an image, such as an image in a training database of images.

[0057] The input conditions obtained by the input condition module 146 can represent any suitable categories, such as classes describing tangible objects (e.g., grass, water, rocks, wheels, etc.), classes describing visual properties (e.g., color, reflectivity, surface texture, shape, etc.), and classes describing parts of objects (e.g., a person's leg or a part of furniture, the lens of a camera, the trigger of a gun, the handle of a coffee cup, etc.).

[0058] In addition, the input condition module 146 can record any suitable representation of the input condition. In one example, the input condition module 146 generates an audio recording of the voice input condition by sampling the output signal from the microphone and recording the audio sample in a file. Additionally or alternatively, the input condition module 146 can generate a text transcription of the input condition, such as by transcribing the verbal input condition (e.g., the user 102 says "round") to a text file. In one example, the input condition module 146 converts the input condition from a first language to a second language. For example, the input condition module 146 can obtain an input condition including text entered in Chinese (e.g., written in Chinese characters), and generate a converted version of the text entered in English (e.g., written in Latin characters).

[0059] The input conditions obtained by the input condition module 146 and any suitable information used by the input condition module 146 or calculated by the input condition module 146, such as the source of the input conditions (e.g., user identification, file name, image identifier, etc.), the recording of the input conditions (e.g., audio file), the transcription or conversion of the input conditions (e.g., text file), metadata of the input conditions (e.g., the language of the input conditions, the sampling rate used to record the voice input conditions, the identifier of the training database of images with annotations corresponding to the input conditions, etc.), etc., are stored in the word data 132 of the storage device 128 and are available to the modules of the object detection application 112. In one example, the input condition module 146 obtains input conditions including word-based concepts and provides word-based concept representations to the attention map module 148 and the word embedding module 150.

[0060] The attention map module 148 represents functionality configured to generate an attention map for a word-based concept from an input image. When provided to the detection module 152, the attention map generated by the attention map module 148 provides concept-specific cues to the conditional detection network of the detection module 152 to better determine objects related to the word-based concept. Therefore, the attention map generated by the attention map module 148 can be any suitable attention map. In one example, the attention map includes pixels indicating the presence value of the word-based concept within the input image, the presence value being greater than or equal to -1 and less than or equal to 1. For example, based on the corresponding presence value within the range [-1,1], each pixel of the attention map can be assigned an integer between 0 and 255. Therefore, each of the presence values ​​indicated by the pixels in the attention map represents the corresponding presence of the word-based concept at the corresponding pixel of the input image.

[0061] The attention map module 148 may generate an attention map based on the input image and the word-based concepts in any suitable manner. For example, the attention map module 148 may include any suitable large-scale classifier to extract features from the image and generate an attention map specific to the input condition from the extracted features. In one example, Figure 2 An exemplary attention map according to one or more aspects of the present disclosure is shown generally at 200. Figure 2 In the example of FIG. 1 , the image labeling network 202 and the class activation map 204 are used to generate an attention map based on an input image. The image labeling network 202 and the class activation map 204 are Figure 1 An example of modules included in the attention map module 148 in .

[0062] exist Figure 2 In the example of , an input image 206 is provided to an image labeling network 202. The image labeling network 202 may include any suitable labeling network to generate any suitable number of feature maps based on the input image 206. For example, the image labeling network may generate a three-dimensional (3D) feature map based on the input image 206.

[0063] In one example, the image labeling network 202 includes a labeling network based on ResNet-50, as described in "Deep residual learning for image recognition" by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, CVPR, pp. 770-778, 2016, the disclosure of which is incorporated herein by reference in its entirety. To support image labeling, the final fully connected (FC) classification layer of ResNet-50 can be replaced with three embedding FC layers. Between the two embedding FC layers is a batch normalization layer and a ReLU activation layer, as described in "Batchnormalization: accelerating deep network training by reducing internal covariance shift" by Sergey Ioffe and Christian Szegedy, ICML, pp. 448-456, and "Deep sparse rectifier neural networks" by Xavier Glorot, Antoine Bordes and Yoshua Bengio, AISTATS, pp. 315-323, 2011, respectively, the disclosure of which is incorporated herein by reference in its entirety.

[0064] The image labeling network 202 may be trained based on soft topic embedding and minimizing the cosine similarity loss, as described below with respect to Figure 6 As discussed. Once trained, the image labeling network 202 generates a feature map from the input image 206 and provides the feature map for the class activation map 204. The class activation map 204 obtains a concept-specific attention map based on the class activation map, such as an attention map 208 corresponding to the word-based concept "wheel" received by the class activation map 204. The global average pooling (GAP) layer of the ResNet-50 in the image labeling network 202 is removed, and the embedding FC layer is convolved by converting the embedding FC weights to a 1×1 convolution filter. The soft topic word embedding representation for the input image can be converted to a 1×1 convolution filter, which is applied to the feature map provided by the image labeling network 202 to obtain a spatially preserved response (e.g., an attention map) for each concept. Because the cosine similarity distance metric is used between the image and its soft topic word embedding, the value of the attention map determined by the class activation map 204 falls within the range [-1, 1].

[0065] exist Figure 2In the example of , the word-based concept "wheel" is provided to the class activation map 204, which generates an attention map 208 accordingly. The attention map 208 indicates the corresponding existence value of each pixel of the input image 206. For example, each pixel of the output attention map 208 can be assigned an integer from 0 to 255 based on the corresponding existence value in the range of [-1, 1]. Therefore, the attention map 208 visually shows areas with a higher probability of including objects corresponding to the "wheel" class as brighter (e.g., white), and areas with a lower probability of including objects corresponding to the "wheel" class as darker (e.g., black). Therefore, the attention map 208 can provide concept-specific cues to the conditional detection network of the detection module 152 to better determine objects related to the word-based concept.

[0066] Return to Figure 1 , the attention map generated by the attention map module 148 and any suitable information used or calculated by the attention map module 148, such as cosine similarity scores, feature maps, presence values, soft topic word embeddings of the image, etc., are stored in the attention map data 134 of the storage device 128 and are available to the modules of the object detection application 112. In one example, the attention map module 148 generates an attention map for a concept from an input image and provides the attention map to the detection module 152 as a conditional input, for example, by providing the attention map to a layer of a conditional detection network of the detection module 152.

[0067] The word embedding module 150 represents functionality configured to generate word embeddings based on term-based concepts of input conditions. The word embedding module 150 can generate any suitable word embeddings.

[0068] The word embeddings describe the relationship between a word-based concept (e.g., "wheel") and different word-based concepts, such as labels from a vocabulary. In one example, the word embeddings generated by the word embedding module 150 model the relationship between the different concepts. For example, the word embeddings may describe the mutual information between the word-based concepts obtained by the input condition module 146 and the labels from a vocabulary, such as the vocabulary stored in the word data 132 of the storage device 128. The word embeddings may be generated based on the mutual information by constructing a matrix from the co-occurrence probabilities of the word-based concepts and different word-based concepts or labels, and forming vectors from the eigendecomposition of the matrix.

[0069] In one example, word embeddings are determined from an available text corpus by selecting ready-made word vectors as word embeddings for word-based concepts. Additionally or alternatively, word embeddings may be derived from label annotations of images. For example, a first training dataset such as a stock photo web service (e.g., Adobe Stock) used to train a labeled network for the attention map module 148 may be used to generate word embeddings from image labels in the dataset.

[0070] In one example, the word embedding module 150 determines the word embedding derived from the label annotation of the image based on the point-wise mutual information (PMI) between the labels. For example, a PMI matrix M is generated, where the (i, j)th element is

[0071]

[0072] Where p(w i , w j ) indicates label w i and w j The co-occurrence probability of , and p(w i )、p(w j ) represent the labels w i and w j The label frequency of . Therefore, the PMI matrix M has size V×V, where V represents the vocabulary size. In one example, V is about 18,000.

[0073] M can be rewritten according to any suitable decomposition, for example by rewriting M as M = USU T To decompose the PMI matrix M. In one example, the PMI matrix M is decomposed via eigendecomposition. For example, the matrix U contains eigenvectors as its columns, and the matrix S contains corresponding eigenvalues ​​on its main diagonal.

[0074] Based on the decomposition of the PMI matrix M, construct the matrix Truncate each row of the submatrix E from the columns, or for some integer K of e i =E t，1：K , generating word embeddings. In one example, K is set to 4096, so that each word embedding is represented by a vector of length 4096. The word embedding for a particular term-based concept can be retrieved from the generated length K vector by the word embedding module 150.

[0075] The word embeddings determined by the word embedding module 150 and any suitable information used by or calculated by the word embedding module 150, such as feature values, feature vectors, vocabulary parameters, term vectors, etc., are stored in the word data 132 of the storage device 128 and are available to the modules of the object detection application 112. In one example, the word embedding module 150 determines word embeddings for word-based concepts and provides the word embeddings to the detection module 152 as conditional inputs, such as by providing the word embeddings to two layers of a conditional detection network of the detection module 152.

[0076] The detection module 152 represents functionality configured to detect objects in an image. In one example, the detection module 152 receives conditional inputs from the object detection system 110, such as an attention map from the attention map module 148 and a word embedding from the word embedding module 150. Thus, the detection module 152 can detect objects in an image based on the conditional inputs. For example, the detection module 152 can detect at least one region of an input image based on word embeddings and attention maps, which includes an object corresponding to a word-based concept used to generate the word embeddings and attention maps. Thus, the detection module 152 may include a conditional detection network. In one example, the detection module 152 includes a conditional detection network trained using a second data set of a second image, the second image including annotations with corresponding bounding boxes of corresponding objects in the second image.

[0077] The detection module 152 may include any suitable conditional detection network configured to receive conditional inputs, such as attention maps and word embeddings, and detect objects in an image based on the conditional inputs. Figure 3 An example of a conditional detection network of the detection module 152 is shown in FIG. Figure 3 An exemplary condition detection network 300 is shown in accordance with one or more aspects of the present disclosure.

[0078] The condition detection network 300 includes a backbone network 302 and a convolutional neural network (CNN) 304. The backbone network 302 can be any suitable backbone network. Figure 3 In the example of , backbone network 302 is based on VGG-16, as described in “Very deep convolutional networks for large-scale image recognition” by Karen Simonyan and Andrew Zisserman, ICLR, 2015, the disclosure of which is incorporated herein by reference in its entirety.

[0079] Convolutional neural network (CNN) 304 may be any suitable convolutional neural network. Figure 3In the example, CNN304 is based on Faster R-CNN, as described in S. Ren, K. He, R. Girshick, and J. Sun in “Faster r-cnn: toward real-time object detection with region proposal networks”, NIPS, 2015 and S. Ren, K. He, R. Girshick, and J. Sun in “Faster r-cnn: toward real-time object detection with region proposal networks”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 6, pp. 1137-1149, 2017, the disclosures of which are incorporated herein by reference in their entirety.

[0080] The conditional detection network 300 receives conditional inputs, including an attention map from the attention map module 148 and a word embedding from the word embedding module 150. These conditional inputs are provided to the convolution level four of the backbone network 302, after the convolution levels one to three 306. The word embedding is provided as the conditional input of the layer one 308 and layer three 312 of the convolution level four, and the attention map is provided as the conditional input of the layer two 310 of the convolution level four. Level four is selected to insert the conditional input because it is quite deep in the hierarchy of VGG-16 that constitutes the backbone network 302, so it can process relatively high-level features (which are semantically high-level) that are compatible with word embeddings and attention maps. In addition, the insertion point of the convolution level four is before the convolution level five 314, which is more suitable for learning nonlinear mappings to combine visual features with conditional inputs. In addition, the layers of the convolution level four of the backbone network 302 are before the regional proposal network (RPN) 316 of the CNN304. Therefore, the input conditions inserted into the layers of the fourth level of the backbone network 302 can directly affect the RPN 316, the region proposal 318, and the final classification with the FC layer 320, which feeds the binary classifier 322 and the bounding box regression 324 of the CNN 304.

[0081] The word embeddings from the word embedding module 150 are provided to the convolutional layer four layers 1 308 and 312. In one example, the word embeddings are vectors of length 4096, or To insert the word embedding as a conditional input, l is set to the layer index of layer one 308 or layer three 312 of convolutional level four. The output of this layer is given by:

[0082]

[0083] Where * represents convolution, z l-1 represents the output of the previous layer, β l represents the bias term, and and are the weights of traditional convolution and word embedding processing, respectively. In one example, N is set to 512. The weight W e Embed the word into vector e inp Mapped to an N-dimensional vector. Due to shape incompatibility, when added to the convolution feature Previously, the broadcast function was used to e e inp In one example, weights are not shared between layer 1 308 and layer 3 312 of convolutional level 4. inp The dimension of is greater than dimension N, so the word embedding vector e inp Provided to the convolutional layer 4 layer 1 308 and layer 3 312 to reduce the e The information loss caused by the dimensionality reduction caused by the mapping.

[0084] The attention map is provided to layer 2 310 of stage 4 of the backbone network 302. In order to incorporate the information of the attention map into the backbone network 302, the attention map is regarded as a mask of the convolutional features. Let l be the layer index of layer 2 310 of stage 4 of the backbone network 302. The output of layer 2 310 is

[0085]

[0086] in represents the convolutional layer weights, Concat(·) represents the connection function that connects features along the channel dimension, and is the attention map of the target concept. The attention map is upsampled to match the spatial size (H×W) of z via bilinear interpolation. The operator ⊙ represents element-wise multiplication. The masked features (A inp ⊙z l-1 ) provides rough knowledge of the region corresponding to the target concept. However, the unmasked feature (z l-1 ) allows information at all locations to be considered, and thus may help preserve information in cases where attention maps are inaccurate or unreliable.

[0087] By inserting the input conditions into the layer of level 4 of the backbone network 302, such as Figure 3 As shown and described above, attention maps and word embeddings can directly influence the region proposal and classification of CNN 304. Figure 3In the example in , the attention map of the input image 206 is provided together with the word embedding of the concept "wheel" based on the word. In response, the conditional detection network 300 generates an output image 326. The output image 326 includes a detection result 328, showing a bounding box on the wheel of the car in the image. Therefore, the conditional detection network 300 has correctly detected the object corresponding to the input condition "wheel" in the input image 206.

[0088] Due to the ability of the conditional detection network 300 to generalize to images not seen during training and the class agnosticism of the network, the conditional detection network 300 can scale and generalize to word embeddings and attention maps of unseen concepts when trained using only a second training dataset of images with a small number of classes (e.g., hundreds of classes).

[0089] Return to Figure 1 , the objects detected by the detection module 152 and any suitable information used and calculated by the detection module 152, such as detection results, bounding boxes, positive and negative labels, the number of objects matching the target concept, etc., are stored in the detection data 136 of the storage device 128 and are available to the modules of the object detection application 112. In one example, the detection module 152 exposes the detection results in a user interface of a display such as the display 124. Additionally or alternatively, the detection module 152 can provide the detection results to the training module 154, which can use the detection results to generate training updates for updating the weights of the network of the detection module 152.

[0090] Training module 154 represents functionality configured to train one or more networks of object detection system 110. Training module 154 may train any suitable network in any suitable manner. In one example, training module 154 trains an image labeling network of attention map module 148, such as Figure 2 The image labeling network 202 in FIG. 1 and the conditional detection network of the detection module 152 such as Figure 3 The conditional detection network 300 in FIG. The training module 154 can train the image labeling network of the attention map module 148 and the conditional detection network of the detection module 152 using different heterogeneous training data sets.

[0091] In one example, the training module 154 trains the image labeling network of the attention map module 148 using a first dataset of first images, the first image including annotations of bounding boxes of objects that are not in the first image, and trains the conditional detection of the detection module 152 using a second dataset of second images, the second image including corresponding bounding boxes of corresponding objects in the second image. The conditional inputs (e.g., word embeddings and attention maps) for the conditional detection of the detection module 152 can be obtained from the image labeling network of the attention map module 148 that has been pre-trained with the first image dataset.

[0092] The first dataset may include a stock-18 dataset, which includes six million or more images with image-level annotations. Each image in the first dataset may be labeled with multiple labels (e.g., 10-20 labels) from a vocabulary of 18,000 classes. In one example, the stock-18 dataset is obtained from a stock photo web service. However, the second dataset may include images from an open image dataset, such as a publicly available dataset, such as the open images (v3) described in “Openimages: a public dataset for large-scale multi-label and multi-class image classification” by I. Krasin et al., available online, 2017, the disclosure of which is incorporated herein by reference in its entirety. In one example, the second dataset includes fewer images than the first dataset, such as approximately 1.5 million (compared to 6 million). Additionally or alternatively, the images of the second dataset may include annotations (e.g., bounding boxes) of concepts from a vocabulary containing 545 trainable classes. In one example, of the 545 trainable classes, only those classes that intersect with the classes of the first dataset are maintained in the second dataset. For example, of the 545 trainable classes in Open Images (v3), only 448 classes may be retained in the second dataset used to train the conditional detection network of the detection module 152. Additionally or alternatively, the second dataset may be constructed by selecting some but not all images from the Open Images dataset, such as selecting 295,000 images from the 1.5 million images in Open Images (v3).

[0093] The training module 154 can train any suitable network according to any suitable loss function. In order to make the conditional detection network label agnostic, a binary loss can be used. Figure 3In the conditional detection network 300 of , the CNN 304 includes a binary classifier 322. In the binary classifier 322, a binary sigmoid cross entropy loss is used instead of the softmax cross entropy loss of Faster R-CNN. A binary classifier such as the binary classifier 322 can assign a positive label (e.g., a positive training label) to a detection output of the conditional detection network that substantially overlaps with a truth bounding box corresponding to a given word-based concept; and can assign a negative label (e.g., a negative training label) to other detection outputs that are not assigned a positive label. Additionally or alternatively, a binary classifier can be used to train a conditional detection network for negative class inputs. For example, a negative class of a word-based concept can be provided to the conditional detection network, and a negative label can be assigned to a detection output that substantially overlaps with a truth bounding box corresponding to a word-based concept by a binary classifier. Figure 4 Training with positive and negative training labels and a negative input class is shown.

[0094] Figure 4 An example training of an object detection system according to one or more aspects of the present disclosure is shown generally at 400. Figure 4 In FIG. 4 , object detection system 402 and object detection system 404 are trained with input image 406. Object detection system 402 and object detection system 404 are examples of object detectors included in detection module 152, such as Figure 3 The condition detection network 300 in FIG.

[0095] exist Figure 4 , object detection system 402 and object detection system 404 are provided with corresponding input conditions together with input image 406. For example, object detection system 402 is provided with target condition "dog". In one example, the target condition is selected by randomly selecting one of a plurality of classes identified by a label included in input image 406.

[0096] The object detection system 404 is provided with a negative condition, such as a negative class, relative to the target condition provided to the object detection system 402. A negative class is a class that may provide similar word embeddings and similar attention maps as the target class, despite being semantically disjoint with the target condition. For example, "dog" and "bear" are examples of classes that may have similar word embeddings and attention maps to each other, but are semantically disjoint.

[0097] The negative class relative to the target class can be determined in any suitable manner. In one example, the training module 154 identifies multiple negative classes that are most similar to the target class from the classes in the training data set based on the cosine similarity measure of the corresponding word embeddings of the classes. Additionally or alternatively, the negative class can be constrained so that it is not a hyponym or hypernym of the target class. The training module 154 can randomly select one of the multiple negative classes (e.g., one of the ten identified negative classes) and provide the selected negative class to the object detection system 404. For example, the object detection system 404 is provided with the negative condition "fox" relative to the target condition "dog".

[0098] During training, object detection system 402 and object detection system 404 may share weights. Based on input image 406 and corresponding input conditions, object detection system 402 generates output image 408, and object detection system 404 generates output image 410. Output image 408 and output image 410 include detection results represented by bounding boxes, and the bounding boxes are classified by binary classifiers, as indicated by their line types according to key 412. For example, key 412 indicates that white corresponds to the truth, black corresponds to positive training labels, long dashed gray and white correspond to negative training labels, and short dashed black and white correspond to ignored detection results.

[0099] In output image 408 and output image 410, bounding box 414 is a truth bounding box. For example, bounding box 414 is included in input image 406, such as in a label corresponding to class "dog". In output image 408 generated by object detection system 402 conditioned on target condition "dog", bounding boxes 416 are assigned positive training labels because they substantially overlap with truth bounding box 414. Overlap can be determined in any suitable manner, such as based on the amount of overlapping area, the ratio of overlapping area to non-overlapping area of ​​the detection results, the centroid of overlap, a combination thereof, etc. However, bounding boxes 418 of output image 408 are assigned negative training labels because they do not substantially overlap with truth bounding box 414. Instead, they correspond to cats in the image rather than dogs.

[0100] In the output image 410 produced by the object detection system 404 conditioned on the negative condition "fox", bounding boxes 420 are assigned negative training labels because they substantially overlap with the truth bounding box 414. Thus, the object detection system 404 is trained to distinguish between dogs and foxes. Bounding box 422 in the output image 410 is ignored.

[0101] In one example, for each detection result of the object detection system 404 that is conditioned on a negative condition that substantially overlaps with a truth bounding box (e.g., a bounding box 420 that overlaps with the truth bounding box 414 in the output image 410), the training module 154 computes two scores. The first score s c Using the features conditioned on the target condition, the second score s n The training module 154 can train the object detector by minimizing a loss determined from the two scores, such as a marginal rank loss L neg =max(0,s n -s c +γ) to encourage s c Greater than s n , with a margin according to the parameter γ.

[0102] Return to Figure 1 , the training module 154 can also be configured for traditional training of the network for multi-class learning without conditional input. For example, the training module 154 can be configured to zero the word embeddings and attention maps, which are provided as conditional inputs to the conditional detection network trained by the training module 154. To zero the conditional inputs, the training module 154 can set the presence value of the attention map to 1 and the value of the word embedding to 0. When the conditional inputs are zeroed, the conditional detection network can detect multiple objects of different classes in the vocabulary (e.g., word-based concepts). When the conditional inputs are zeroed for multi-class learning, the conditional detection network can be trained with a loss λ·L conditional +(1-λ)·L multi-class , where L conditional and L multi-class They are conditional (e.g. conditional input is not zeroed) and multi-class (e.g. conditional input is zeroed) detection losses respectively.

[0103] The training data of the training module 154, as well as any suitable information used by the training module 154 or calculated by the training module 154, such as detection results, bounding boxes, positive and negative training labels, the number of objects matching the target concept, scores, training losses, training data sets (e.g., heterogeneous data sets used to train the networks of the attention map module 148 and the detection module 152), etc., are stored in the training data 138 of the storage device 128 and are available to the modules of the object detection application 112. In one example, the training module 154 generates training updates for the conditional detection network of the detection module 152, such as updates to the weights of the neural network, to train the conditional detection network. Additionally or alternatively, the training module 154 can generate training updates for the image labeling network of the attention map module 148, such as updates to the weights of the neural network, to train the image labeling network.

[0104] exist Figure 1 In the example of , object detection system 110 trains an object detector, such as an object detector including object detection application 112, and provides the trained object detector to computing devices 104 for use in client applications. Thus, each of computing devices 104 includes an image system 114. Image system 114 includes applications 156, which may include any suitable application, such as an application configured to be executed by one or more computing devices 104. Applications 156 include detection application 158. Detection application 158 may be any application configured to detect objects in an image, such as an image editing application, a vehicle application (e.g., a guidance system in an autonomous vehicle, an onboard control system for lighting or entertainment, a noise cancellation system, etc.), a control system for a robot or drone, an image cataloging application, etc.

[0105] Image system 114 also includes assets 160. Assets 160 may include any suitable assets used by image system 114, such as object detectors, training databases, word embeddings, vocabularies, combinations thereof, etc. provided by server 108. Image system 114 also includes a copy of object detection system 110 of server 108. Figure 1 In the example of , object detection system 110 of server 108 is described as training an object detector and providing it to one of computing devices 104, which may additionally or alternatively train an object detector. An object detector trained by image system 114 (e.g., using a copy of object detection system 110) may be stored in asset 160 and may be used in any suitable application, such as detection application 158.

[0106] Having considered an example digital media environment, consider now a discussion of an example system in accordance with one or more aspects of the present disclosure.

[0107] Example Object Detection System

[0108] Figure 5 An example system 500 according to one or more aspects of the present disclosure is shown. In this implementation, the system 500 includes: Figure 1 144, such as image module 144, input condition module 146, attention map module 148, word embedding module 150, detection module 152, and training module 154. System 500 is an example of an object detection system 110 that can be constructed using the modules of object detection application 112. For example, signals can be redefined, and modules can be modified, combined, divided, added, or removed to form a modified system without changing the functionality of system 500. Therefore, such modified systems are considered to be within the scope of the present disclosure.

[0109] In addition, for simplicity, the system 500 is limited to the description of the modules of the object detection application 112 and some of their interconnections. However, the system 500 may include any suitable signals and communications between modules that are omitted for simplicity. These signals may include system clocks, counters, image indicators, figure indicators, image identification numbers, reset signals, etc. In one example, the system 500 can operate in real time (e.g., without noticeable delay to the user) to generate an output image, such as an output image 502 including detection results (e.g., a bounding box). Therefore, signals can be calculated by the modules of the system 500 and communicated between the modules of the system 500 without significant delay. In one example, the system 500 trains a network, such as a neural network of the detection module 152. Additionally or alternatively, the system 500 can generate an output image with detection results for an input image provided by a user, such as an image that is not part of a training data set used to train the network of the system 500.

[0110] In addition, system 500 can be implemented on any suitable device or devices. In one example, system 500 is implemented on a computing device (e.g., server 108 or Figure 1 In another example, system 500 is implemented on more than one computing device. For example, a portion of system 500 may be implemented by a first computing device, such as Figure 1 1 or server 108, and other portions of system 500 may be implemented by one or more additional computing devices, such as computing device 104-2. In one example, a server implements portions of system 500, such as Figure 1 The server 108 in the first computing device. The server can be remote, for example because it is not collocated with the first computing device. The server can be configured to receive signals of the system 500 from a computing device (e.g., one or more of the computing devices 104), process the received signals, such as using the object detection system 110, and send the results of the processing back to the computing device. Thus, Figure 1 The object detection system 110 of the server 108 in the embodiment may include the system 500. In one example, the system 500 is used to train an adaptive model (e.g., a neural network, a machine learning model, etc.) of an object detector by a first computing device such as the server 108, and the trained adaptive model is supplied by the first computing device to a different computing device, such as one of the computing devices 104.

[0111] Additionally or alternatively, portions of system 500 may be implemented by additional computing devices collocated with the first computing device. For example, one of computing devices 104 may be a first computing device, and another of computing devices 104 may be an additional collocated computing device. The first computing device and the additional computing device may be operated by one or more users. Thus, system 500 provides multiple users within an environment with sharing data included in system 500. For example, a training database may be obtained by a first computing device that is operated by a first user who selects digital images to train an object detector, and the selected images may be provided to a second computing device operated by a second user. The second user may use the selected images from the first user to train an object detector on the second computing device, and provide the trained object detector back to the first user on the first computing device. The first user may then use the trained object detector to detect objects in the image and share the detection results with other users, such as the second user on the second computing device.

[0112] Image module 144 obtains one or more images. Image module 144 can obtain images of any suitable type in any suitable manner. In one example, image module 144 obtains images from an image database, such as a database of training data sets that can be used to train an adaptive model (e.g., a neural network) of an object detector. Additionally or alternatively, image module 144 can obtain user-supplied images, such as images from a user provided as input to an image editing application. In one example, image module 144 captures images to be processed by system 500, such as by capturing images while the vehicle is moving and detecting objects in the images in real time to control a guidance system of the vehicle.

[0113] In one example, the image module 144 obtains images from a first training dataset, such as a first dataset of first images, which includes annotations without bounding boxes of objects in the first images. The first training dataset can be used to train the labeling network of the attention map module 148. Additionally or alternatively, the image module 144 can obtain images from a second dataset of second images, which includes annotations with corresponding bounding boxes of corresponding objects in the second images. The second training dataset can be used to train the conditional detection network of the detection module 152.

[0114] The image module 144 provides one or more images as input images to the attention map module 148 and the detection module 152.

[0115] The input condition module 146 obtains one or more input conditions, such as input conditions including word-based concepts. The input condition module 146 can obtain any suitable input conditions in any suitable manner, such as input conditions including word-based concepts that describe the category or class of objects to be detected in the image. The word-based concepts can describe any suitable classes, such as classes that describe tangible objects (e.g., grass, water, rocks, wheels, etc.), classes that describe visual attributes (e.g., color, reflectivity, surface texture, shape, etc.), classes that describe parts of objects (e.g., a person's leg or a furniture leg, a camera lens, a machine's gear, a coffee cup's handle, etc.), etc.

[0116] The input condition module 146 can obtain input conditions in any suitable manner, such as from words spoken by the user (for example, the user can say "turn" into the microphone), text (for example, the user can type the input conditions, the input conditions can be read from a text file, the input conditions can be extracted from the metadata of the image, such as annotations describing objects in the image, etc.), combinations thereof, etc.

[0117] Additionally or alternatively, the input condition module 146 can process the input conditions in any suitable manner, such as converting the input conditions from one language to another language, transcribing the input conditions (e.g., generating a text file containing a transcription of the spoken utterance), correcting spelling errors in the input conditions (e.g., correcting spelling or grammatical errors, etc.), combinations thereof, and the like.

[0118] The input condition module 146 provides word-based concepts from the input conditions to the attention map module 148 and the word embedding module 150.

[0119] The attention map module 148 receives the input image from the image module 144 and the word-based concepts from the input condition module 146 and generates an attention map. Figure 2 The attention map 208 in FIG. 5 is an example of an attention map generated by the attention map module 148, which corresponds to the word-based concept "wheel" of the input image 206. The attention map module 148 can generate the attention map in any suitable manner. In one example, the attention map module 148 in the system 500 includes an image labeling network that has been pre-trained with the training module 154, as described below with respect to Figure 6 discussed.

[0120] The attention map module 148 may generate any suitable attention map. In one example, the attention map module 148 generates an attention map that indicates the presence values ​​of word-based concepts within the input image. For example, the attention map may indicate the corresponding presence of the word-based concept for each pixel of the input image, where the presence value is within a range such as less than or equal to 1 and greater than or equal to negative 1 (e.g., [-1, 1]).

[0121] The attention map generated by the attention map module 148 can provide concept-specific cues to better detect objects in the image. Therefore, the attention map module 148 provides the attention map generated for the input image to the detection module 152 according to the word-based concepts.

[0122] The word embedding module 150 receives the term-based concepts from the input condition module 146 and generates word embeddings for the term-based concepts. The word embeddings generated by the word embedding module 150 describe the relationship between the term-based concepts and different term-based concepts, tags, or terms.

[0123] The word embedding module 150 can generate word embeddings in any suitable manner. In one example, the word embedding module 150 generates a matrix based on word-based concepts and co-occurrence probabilities of different word-based concepts, tags, or words, and performs eigendecomposition of the matrix. The word embedding for a given concept is obtained from the row corresponding to the given decomposed concept, such as by taking the values ​​of the first 4096 columns of the rows of the matrix formed from the decomposition as a vector representing the word embedding for the given concept.

[0124] The word embeddings of the word-based concepts generated by the word embedding module 150 extend the coverage of the word-based concepts beyond the words of a single tag or vocabulary. Therefore, the word embedding module 150 provides the word embeddings to the detection module 152.

[0125] Detection module 152 receives an input image from image module 144, an attention map from attention map module 148, and a word embedding from word embedding module 150, and generates a detection result, such as the detection result shown in output image 502. In one example, the detection result generated by detection module 152 includes at least one region of the input image based on the word embedding and the attention map, which includes a corresponding object corresponding to the word-based concept. For example, when the word embedding corresponds to the word-based concept "dog", detection module 152 can generate an output image with a detection result, which includes a bounding box around the dog in the input image, such as Figure 5 The output image 502 is shown in FIG.

[0126] The detection module 152 can generate the detection result in any suitable manner. In one example, the detection module 152 includes a conditional detection network, such as Figure 3 The condition detection network 300 in FIG.

[0127] exist Figure 5In an example of , detection module 152 provides detection results to training module 154. Training module 154 also receives training data, such as training images of a training database of images, which include annotations showing ground truth bounding boxes of objects in the images. The input image obtained by image module 144 can be an example of training data received by training module 154. Based on comparing the detection results generated by detection module 152 with the training data, training module 154 generates training updates. Training module 154 can generate any suitable training updates in any suitable manner, such as weights of a neural network used in convolution, updated by stochastic gradient descent, which minimizes any suitable loss function. Training module 154 provides the training updates to detection module 152, and detection module 152 updates the parameters of the object detector based on the training updates to the trained detection module 152. System 500 can process any suitable number of training images to train the object detector of detection module 152.

[0128] In one example, the attention map module 148 of the system 500 includes one or more adaptive models (e.g., image labeling networks) pre-trained using the training module 154, which Figure 6 The system 600 is shown in FIG.

[0129] Figure 6 An example system 600 is shown in accordance with one or more aspects of the present disclosure. Figure 6 In the example of , the attention map module is trained using images of a first training data set. The image module 144 obtains images from the first training data set, such as images that do not include bounding box annotations of objects in the images. As described above, the first training data set of images used to train the attention map module 148 can be heterogeneous with respect to the second training data set used to train the detection module 152.

[0130] An input image from the first training data set is provided by the image module 144 to the attention map module 148 and the training module 154. The attention map module generates any suitable map from the received input image in any suitable manner, such as a feature map of the input image, an attention map generated from a feature map of a given concept, a combination thereof, and the like. Figure 2 The image labeling network 202 and the class activation map 204 in FIG. 1 are examples of modules of the attention map module 148, which can generate feature maps and attention maps for training Figure 6 The attention map module 148 in the system 600. The map based on the input image generated by the attention map module 148 is provided to the training module 154.

[0131] The training module 154 also receives training data, including an input image obtained by the image module 144. The training module 154 generates training updates based on comparing the parameters of the map generated by the attention map module 148 with the parameters of the input image, such as by comparing the features identified in the annotations of the input image with the labels generated in the feature map by the attention map module 148. The training module 154 can generate any suitable training updates in any suitable manner, such as weights of a neural network used in convolutions, updated by stochastic gradient descent, which minimizes any suitable loss function. The training module 154 provides the training updates to the attention map module 148, which updates the parameters of the image labeling network according to the training updates to train the attention map module 148. The system 600 can process any suitable number of training images to train the image labeling network of the attention map module 148.

[0132] The training module 154 can generate training updates in any suitable manner. In one example, the training module 154 generates training updates to minimize the cosine similarity loss between the visual embedding (e.g., feature map) output by the attention map module 148 and the soft topic embedding of the input image. For example, the soft topic embedding of the input image can be constructed by aggregating the word representations of the truth labels included in the image. The soft topic embedding can be constructed according to t=∑ i α i ·e i To calculate the weighted average of all labels, where α i =-log(p(w i )) is the word w i The cosine similarity loss can be calculated based on Construct , where v represents any suitable visual embedding output by the attention map module 148, such as a feature map.

[0133] The system described herein constitutes an improvement to a system that is trained without a heterogeneous data set and is limited to detecting objects of visible classes (e.g., defined classes present in images of a training database). In contrast, the system described herein includes an object detector having an image labeling network and a conditional detection network, which is trained using heterogeneous data sets relative to each other and can detect objects in images of visible and invisible classes. An attention map and word embedding for a target concept are provided as conditional inputs to the conditional detection network, so that the conditional detection network is image label agnostic and concept specific. Therefore, although the conditional detection network is trained using a training data set with a small number of visible classes, it generalizes to novel, invisible classes through concept conditioning because word-based concepts are propagated through the conditional detection network via conditional inputs and affect classification and region proposals. Therefore, the object classes that can be detected by the object detector can be expanded by the system described herein without scaling the training database to include additional classes, thereby saving time and money.

[0134] Having considered an example system, consider now a discussion of an example process for object detection in an image in accordance with one or more aspects of the present disclosure.

[0135] Example Process

[0136] Figure 7 An example process 700 for object detection in an image according to one or more aspects of the present disclosure is shown. Aspects of the process may be implemented in hardware, firmware, or software, or a combination thereof. The process is shown as a collection of blocks that specify operations performed by one or more devices and is not necessarily limited to the order shown for performing the operations of the respective blocks. In at least some aspects, the process may be performed by a suitably configured computing device in a digital media environment, such as Figure 1 One or more of computing devices 104 or servers 108 that utilize an object detection system, such as Figure 5 System 500 or Figure 1 The object detection system 110 in process 700 may be a standalone application already installed on the computing device, a service hosted by a service provider accessible by the computing device, a plug-in module of the computing device, or a combination thereof.

[0137] An input image and a word-based concept are obtained (box 702). In one example, the image module 144 obtains an input image, and the input condition module 146 obtains a word-based concept. Additionally or alternatively, the image module 144 can obtain an input image, and the input condition module 146 can obtain a word-based concept as part of a training process, wherein the network is trained to detect objects in an image. For example, the image module 144 can obtain an input image, and the input condition module 146 can obtain a word-based concept from the input image, such as by selecting one of a plurality of tags in the input image that describes an object in the input image. In one example, the image module 144 obtains an input image by receiving an image supplied by a user. Additionally or alternatively, the image module 144 obtains an input image by retrieving the input image from a storage location, such as from a database of training images at a server, from a storage device 128, or a combination thereof.

[0138] An attention map is generated based on the input image and the word-based concepts (box 704). The attention map includes pixels that indicate presence values ​​of the word-based concepts within the input image. In one example, the image labeling network of the attention map module 148 generates an attention map based on the input image and the word-based concepts, the attention map including pixels indicating presence values ​​of the word-based concepts within the input image. Additionally or alternatively, each of the presence values ​​indicates a corresponding presence of the word-based concept at a corresponding pixel of the input image, such as by indicating the presence value of the attention map in the range [-1,1] with a pixel value between 0 and 255.

[0139] In one example, the attention map is generated from an image labeling network that is trained from a first dataset of first images that include annotations of bounding boxes without objects in the first images. For example, the training module 154 can train the image labeling network with the first dataset of first images that include annotations of bounding boxes without objects in the first images.

[0140] Generate a word embedding based on the word-based concept, the word embedding describing the relationship between the word-based concept and the different word-based concepts (box 706). In one example, the word embedding module 150 generates a word embedding based on the word-based concept, the word embedding describing the relationship between the word-based concept and the different word-based concepts.

[0141] Additionally or alternatively, generating word embeddings may include forming vectors from eigendecomposition of a matrix.The matrix may be constructed from word-based concepts and co-occurrence probabilities of different word-based concepts.

[0142] The word embeddings and the attention maps are provided to the corresponding layers of the conditional detection network (block 708). In one example, the attention map module 148 provides the attention maps, and the word embedding module 150 provides the word embeddings as conditional inputs to a conditional detection network, such as the conditional detection network of the detection module 152. For example, the attention maps may be provided to different layers of the conditional detection network than the word embeddings.

[0143] The conditional detection network detects at least one region of the input image based on the word embedding and the attention map (box 710). The at least one region includes a corresponding object corresponding to the word-based concept, and the at least one region includes a corresponding object corresponding to the word-based concept. In one example, the conditional detection network of the detection module 152 detects at least one region of the input image based on the word embedding and the attention map, and the at least one region includes a corresponding object corresponding to the word-based concept. Additionally or alternatively, the conditional detection network is trained using a second data set of a second image, and the second image includes annotations of corresponding bounding boxes of corresponding objects in the second image. For example, the training module 154 can train the conditional detection network using a second data set of a second image, and the second image includes annotations of corresponding bounding boxes of corresponding objects in the second image.

[0144] In one example, word embeddings are provided to two layers of a conditional detection network, and an attention map is provided to a third layer of the conditional detection network, such as a conditional detection network that detects at least one region of an input image. The third layer can be between the two layers of the conditional detection network. Additionally or alternatively, the two layers and the third layer can be included in the same convolutional level of the conditional detection network. For example, providing word embeddings and attention maps to the conditional detection network can include providing the attention map to a layer of the conditional detection network and providing the word embedding to an additional layer of the conditional detection network, the conditional detection network being located between the additional layers of the conditional detection network.

[0145] In one example, a conditional detection network is trained using an image dataset that includes a corresponding bounding box of a corresponding object in each of the images, the training comprising: assigning positive training labels to detection outputs of the conditional detection network that substantially overlap with a truth in the corresponding bounding box corresponding to a word-based concept; and assigning negative training labels to other detection outputs of the conditional detection network that are not assigned the positive training labels.

[0146] Additionally or alternatively, training the conditional detection network includes determining a negative class for the word-based concept, providing the negative class to the conditional detection network, and assigning a negative training label to a detection output of the conditional detection network for the negative class that substantially overlaps with a truth bounding box corresponding to the word-based concept.

[0147] In one example, training a conditional detection network includes setting presence values ​​of an attention map to 1, setting values ​​of word embeddings to zero, and detecting corresponding regions of an input image for a plurality of objects of different word-based concepts.

[0148] Figure 8 An example process 800 for object detection in an image according to one or more aspects of the present disclosure is shown. Aspects of the process may be implemented in hardware, firmware, or software, or a combination thereof. The process is shown as a collection of blocks that specify operations performed by one or more devices and are not necessarily limited to the order shown for performing the operations of the respective blocks. In at least some aspects, the process may be performed by a suitably configured computing device in a digital media environment, such as Figure 1 One or more of computing devices 104 or servers 108 that utilize an object detection system, such as Figure 5 System 500 or Figure 1 The object detection system 110 in process 800 may be a standalone application installed on a computing device, a service hosted by a service provider accessible by a computing device, a plug-in module of a computing device, or a combination thereof.

[0149] An input image is obtained (block 802). In one example, the image module 144 obtains an input image. The image module 144 may obtain any suitable input image, such as a training image (e.g., an image used to train a network of object detectors), a user-supplied image that is not a training image, an image obtained to control a guidance system of a vehicle, an image obtained to control the environment in the vehicle (e.g., to adjust the temperature based on what the person in the vehicle is wearing, to adjust the volume of an entertainment system at speakers directed to certain occupants of the vehicle, such as elderly occupants or occupants with hearing aids, etc.), combinations thereof, and the like.

[0150] Receive word-based concepts (block 804). In one example, the input condition module 146 receives word-based concepts. For example, a user may speak an input condition including a word-based concept such as "wheel", which is received by the input condition module 146 using a microphone and synthesized into text using a speech recognition system of the input condition module 146.

[0151] An attention map for word-based concepts is generated from the input image, the attention map including pixels indicating presence values, each presence value indicating a corresponding presence of the word-based concept at a corresponding pixel of the input image (block 806). In one example, the attention map module 148 generates an attention map for word-based concepts from the input image, the attention map including pixels indicating presence values, each presence value indicating a corresponding presence of the word-based concept at a corresponding pixel of the input image, such as by setting the brightness of the pixel in the attention map to an integer between 0 and 255 (including 0 and 255) according to the presence value in the range [-1, 1].

[0152] In one example, the image labeling network of the attention map module 148 generates the attention map. The image labeling network can be trained using the training module 154 using a first dataset of first images, the first images including annotations without bounding boxes of objects in the first images.

[0153] Generate a word embedding based on the word-based concept, the word embedding describing the mutual information between the word-based concept and the label from the vocabulary (box 808). In one example, the word embedding module 150 generates a word embedding based on the word-based concept, the word embedding describing the mutual information between the word-based concept and the label from the vocabulary. Additionally or alternatively, the word embedding can be generated by decomposing a matrix and selecting at least a portion of one of the rows of the matrix constructed from the decomposition as a word embedding.

[0154] Detecting at least one region of the input image based on the word embeddings and the attention map, the at least one region including a corresponding object corresponding to the word-based concept (block 810). In one example, the detection module 152 detects at least one region of the input image based on the word embeddings and the attention map, the at least one region including a corresponding object corresponding to the word-based concept.

[0155] In one example, the conditional detection network of the detection module 152 detects at least one region. The conditional detection network can be trained using a second dataset of a second image using a training module (e.g., training module 154), the second image including a corresponding bounding box of a corresponding object in the second image. Additionally or alternatively, the second dataset can include the input image as one of the second images. In one example, the input image is not included in the first dataset or the second dataset. For example, when the object detection system is operating in a client computing device, the input image can be supplied by a user or automatically detected by the object detection system, so that the input image may not be part of the dataset of the training image.

[0156] Additionally or alternatively, the word embedding module 150 provides the word embeddings as conditional inputs to the two layers of the conditional detection network, and the attention map module 148 provides the attention map as an additional conditional input to the third layer of the conditional detection network. In one example, the two layers and the third layer are included in the same convolutional level of the conditional detection network, and the third layer is between the two layers in the same convolutional level.

[0157] In one example, the word embedding and the attention map are provided to corresponding layers of a conditional detection network, and the conditional detection network detects at least one region of an input image. The corresponding layers may be included in the same convolutional level of the conditional detection network. Additionally or alternatively, the attention map is provided to a layer of the conditional detection network between additional layers of the conditional detection network that provide the word embedding.

[0158] Additionally or alternatively, the detection results including at least one region of the input image may be exposed in a user interface, such as by displaying the image with a corresponding bounding box specifying the corresponding detection region.

[0159] Fig. 9 An example process 900 for object detection in an image according to one or more aspects of the present disclosure is shown. Aspects of the process may be implemented in hardware, firmware, or software, or a combination thereof. The process is shown as a collection of blocks that specify operations performed by one or more devices and is not necessarily limited to the order shown for performing the operations of the respective blocks. In at least some aspects, the process may be performed by a suitably configured computing device in a digital media environment, such as Figure 1 One or more of computing devices 104 or servers 108 that utilize an object detection system, such as Figure 5 System 500 or Figure 1 The object detection system 110 in the process 900 may be a standalone application installed on the computing device, a service hosted by a service provider accessible by the computing device, a plug-in module of the computing device, or a combination thereof.

[0160] An input image and a term-based concept are obtained (block 902). In one example, the image module 144 obtains an input image, and the input condition module 146 obtains the term-based concept. The image module 144 may obtain the input image in any suitable manner, such as by receiving a user-supplied image, automatically receiving and without user intervention an image from an object detection system such as a navigation system of an automobile, retrieving an image from a storage device (e.g., the storage device 128), retrieving an image from a database of training images (e.g., from the server 108), a combination thereof, and the like.

[0161] Generate an attention map for word-based concepts from the input image, the attention map including pixels, each pixel of the attention map indicating a corresponding presence value, the corresponding presence value indicating a corresponding presence of the word-based concept at the corresponding pixel of the input image (block 904). In one example, the attention map module 148 generates an attention map for word-based concepts from the input image, including an attention map of pixels, each pixel of the attention map indicating a corresponding presence value, and a corresponding presence value indicating a corresponding presence of the word-based concept at the corresponding pixel of the input image. Additionally or alternatively, the corresponding presence value may be greater than or equal to -1 and less than or equal to 1.

[0162] Generate a word embedding based on the word-based concept, the word embedding describing the relationship between the word-based concept and the different word-based concepts (block 906). In one example, the word embedding module 150 generates a word embedding based on the word-based concept, the word embedding describing the relationship between the word-based concept and the different word-based concepts.

[0163] In one example, generating word embeddings includes forming vectors from eigendecomposition of a matrix constructed of word-based concepts and co-occurrence probabilities of different word-based concepts.

[0164] The attention map and the word embeddings may be provided as conditional inputs to a conditional detection network, with the attention map being provided to a different layer of the conditional detection network instead of the word embeddings. In one example, the attention map module 148 provides the attention map, and the word embedding module 150 provides the word embeddings as conditional inputs to a conditional detection network, such as the conditional detection network of the detection module 152, with the attention map being provided to a different layer of the conditional detection network instead of the word embeddings.

[0165] In one example, the word embedding is provided to two layers of a conditional detection network, and the attention map is provided to a third layer of a conditional detection network, such as a conditional detection network that detects at least one region of an input image. The third layer can be between the two layers of the conditional detection network. Additionally or alternatively, the two layers and the third layer can be included in the same convolutional level of the conditional detection network.

[0166] At least one region of the input image is detected based on the attention map and the word embeddings, the at least one region including a corresponding object corresponding to the word-based concept (block 908). In one example, the detection module 152 detects at least one region of the input image using a conditional detection network based on the attention map and the word embeddings, the at least one region including a corresponding object corresponding to the word-based concept.

[0167] The process described herein constitutes an improvement over a process training system without a heterogeneous dataset and is limited to detecting objects of visible classes (e.g., classes present in a training database). In contrast, the process described herein includes an object detector having an image labeling network and a conditional detection network, which are trained using heterogeneous datasets relative to each other and can detect objects in images of visible and invisible classes. An attention map and word embedding for a target concept are provided as conditional inputs to the conditional detection network, so that the conditional detection network is image label agnostic and concept specific. Therefore, although the conditional detection network is trained using a training dataset with a small number of visible classes, it generalizes to novel, invisible classes through concept conditioning because word-based concepts are propagated through the conditional detection network via the conditional input and affect classification and region proposals. Therefore, the object classes that can be detected by the object detector can be expanded through the process described herein without scaling the training database to include additional classes, thereby saving time and money.

[0168] Having considered example procedures according to one or more implementations, consider now example systems and devices that can be used to practice the inventive principles described herein.

[0169] Example Systems and Devices

[0170] Fig.10 An example system is shown generally at 1000, which includes example computing devices 1002-1, 1002-2, 1002-3, 1002-4, and 1002-5 (collectively, 1002) that represent one or more computing systems and devices that may implement the various techniques described herein. Computing device 1002 may be any suitable computing device (e.g., a user computing device). Computing device 1002 may be, for example, a user computing device (e.g., one of computing devices 104) or a server device (e.g., server 108). Furthermore, computing device 1002 may include a system on a chip, a plurality of computing devices, a combination thereof, or any other suitable computing device or computing system. Thus, Fig.10 The computing device 1002 is shown as one or more of a tablet, a laptop, a smart phone, smart glasses, and a vehicle (e.g., a computing device of a vehicle), but these examples are illustrative and are not meant to limit the type or number of devices included in the computing device 1002.

[0171] In addition, computing device 1002 is coupled to a "cloud" 1004 that includes a platform 1006, which represents one or more computing systems and devices, such as servers, edge servers, global servers, or combinations thereof, that can implement the various techniques described herein. This is illustrated by including object detection application 111, detection application 158, object detection system 110, image system 114, server 108, system 500, and system 600 in the modules of platform 1006, which operate as described above.

[0172] The functionality of computing device 1002 may be implemented in whole or in part through the use of a distributed system, such as through the “cloud” 1004 via platform 1006. In addition, platform 1006 may host data accessible to computing device 1002, and thus computing device 1002 may need to be authenticated to platform 1006.

[0173] The platform 1006 includes a processing system 1008, one or more computer-readable media 1010, and one or more I / O interfaces 1012 that are communicatively coupled to each other. Although not shown, the platform 1006 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and a processor or local bus utilizing any of a variety of bus architectures. Various other examples are also contemplated, such as control and data lines.

[0174] Processing system 1008 represents the functionality of performing one or more operations using hardware. Thus, processing system 1008 is shown to include hardware elements 1014, which may be configured as processors, functional blocks, etc. This may include other logic devices implemented in hardware as application specific integrated circuits or formed using one or more semiconductors. Hardware elements 1014 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may include semiconductors and transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions. Figure 1 Processor 126 in is an example of processing system 1008 .

[0175] Computer-readable media 1010 (eg, computer-readable storage media) is shown including a memory / storage component 1016 . Figure 1The storage device 128 in is an example of a memory / storage device included in the memory / storage component 1016. The memory / storage component 1016 may include volatile media (such as random access memory (RAM)), non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.), or a combination thereof. The memory / storage component 1016 may include fixed media (such as RAM, ROM, fixed hard drives, etc.) and removable media (such as flash memory, removable hard drives, optical disks, etc.). The computer-readable medium 1010 may be configured in various other ways, as further described below.

[0176] Input / output interface 1012 represents functionality that allows a user (e.g., a system administrator of platform 1006) to input commands and information to platform 1006, and also allows information to be presented to the user and other components or devices using various input / output devices. Examples of input devices include keyboards, cursor control devices (e.g., mice), microphones, microphone arrays, scanners, touch functionality (e.g., capacitive or other sensors configured to detect physical touch), cameras (e.g., which may employ visible or invisible wavelengths such as infrared frequencies to recognize motion as gestures that do not involve touch), and the like. Examples of output devices include display devices (e.g., monitors or projectors), speakers, printers, network cards, tactile response devices, and the like. Thus, platform 1006 may be configured in various ways as further described below to support user interaction.

[0177] Platform 1006 also includes applications 1018. Applications 1018 represent any suitable application capable of running on platform 1006, and may include a web browser operable to access various web-based resources (e.g., assets, media clips, images, content, configuration files, services, user profiles, advertisements, coupons, etc.). Applications 1018 include object detection application 112 and detection application 158, as previously described. In addition, applications 1018 include any application that supports object detection system 110, image system 114, system 500, or system 600.

[0178] Cloud 1004 includes and represents platform 1006. Platform 1006 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 1004, and includes resources 1020. Resources 1020 may include applications, data, services, and content that may be utilized when computer processing is executed on a server remote from computing device 1002. Resources 1020 may also include services provided over the Internet, over a subscriber network such as a cellular or Wi-Fi network, or a combination thereof.

[0179] Resources 1020 include detection network storage 1022 that operates to provide one or more networks of an object detection system, such as a network that has been trained using a heterogeneous training data set, to one of computing devices 1002. Resources 1020 also include training data set storage that operates to provide one or more training data sets of images that can be used to train an object detector as described herein.

[0180] Various techniques may be described herein in the general context of software, hardware elements, or program modules. Typically, such modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on a variety of commercial computing platforms with a variety of processors.

[0181] An implementation of the described modules and techniques may be stored on or transmitted via some form of computer-readable media. Computer-readable media may include various media that may be accessed by platform 1006. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media”.

[0182] "Computer-readable storage media" refers to media, devices, or combinations thereof that are capable of supporting persistent or non-temporary storage, as opposed to mere signal transmissions, carrier waves, or signals themselves. Thus, computer-readable storage media do not include the signals themselves or signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media, storage devices, or combinations thereof, implemented in a method or technology suitable for storing information such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data. Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, hard disk, cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing the desired information and accessible by a computer.

[0183] "Computer-readable signal media" refers to signal-bearing media that is configured to send instructions to the hardware of platform 1006, such as via a network. Signal media can generally embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information delivery media. The term "modulated data signal" means a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example, and not limitation, communication media include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0184] As previously described, hardware elements 1014 and computer-readable media 1010 represent modules, programmable device logic, fixed device logic implemented in hardware, or a combination thereof, which may be employed in some aspects to implement at least some aspects of the techniques described herein, such as executing one or more instructions. The hardware may include components of an integrated circuit or system on a chip, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations of silicon or other hardware. In this context, the hardware may operate as: a processing device that performs program tasks defined by instructions, logic embodied by hardware, or a combination thereof, and hardware for storing instructions for execution, such as the computer-readable storage medium described previously.

[0185] Combinations of the foregoing may also be employed to implement the various techniques described herein. Thus, software, hardware, or executable modules may be implemented as one or more instructions, logic embodied on some form of computer-readable storage medium or by one or more hardware elements 1014, or a combination thereof. Platform 1006 may be configured to implement specific instructions and functions corresponding to software and hardware modules. Thus, implementations of modules executable by platform 1006 as software may be implemented, at least in part, in hardware, such as by the use of computer-readable storage media and hardware elements 1014 of processing system 1008. Instructions and functions may be executable / operable by one or more articles (e.g., processing system 1008) to implement the techniques, modules, and examples described herein.

[0186] in conclusion

[0187] In one or more implementations, a digital media environment includes at least one computing device. Systems, techniques, and devices for detecting objects in images using an object detector trained using a heterogeneous training data set are described herein. A first training data set is used to train an image labeling network to determine an attention map of an input image for a target concept, and a second training data set is used to train a conditional detection network that accepts an attention map and a word embedding of the target concept as conditional inputs. Although the conditional detection network is trained using a training data set with a small number of visible classes (e.g., classes belonging to the training data set), it generalizes to novel, unseen classes through concept conditioning because the target concept propagates through the conditional detection network via the conditional input, thereby affecting classification and region proposals. Therefore, the object classes detectable by the object detector can be expanded without scaling the training database to include additional classes, thereby saving time and money.

[0188] Although the invention has been described in language specific to structural features and methodological acts, it should be understood that the invention defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed invention.

Claims

1. An object detection method implemented by a computing device in a digital media environment for detecting an object in an image, the object detection method comprising: obtaining, by the computing device, an input image and a word-based concept; generating, by an image labeling network of the computing device, an attention map based on the input image and the word-based concepts, the attention map comprising pixels indicating presence values ​​of the word-based concepts within the input image; generating, by the computing device, a word embedding based on the word-based concept, the word embedding describing a relationship between the word-based concept and a different word-based concept; Providing the word embedding and the attention map to corresponding layers of a conditional detection network by the computing device; as well as The conditional detection network of the computing device detects at least one region of the input image based on the word embeddings and the attention map, the at least one region including a corresponding object corresponding to the word-based concept. 2 . The object detection method according to claim 1 , wherein each of the presence values ​​indicates a respective presence of the term-based concept at a respective pixel of the input image.

3. The object detection method of claim 1, further comprising training the image labeling network using a first dataset of first images, the first images comprising annotations without bounding boxes of objects in the first images.

4. The object detection method according to claim 3, further comprising: The conditional detection network is trained using a second dataset of second images including annotations with corresponding bounding boxes of corresponding objects in the second images.

5. The object detection method according to claim 1, wherein generating the word embedding comprises: A vector is formed according to eigendecomposition of a matrix constructed from the word-based concepts and the co-occurrence probabilities of the different word-based concepts.

6. The object detection method of claim 1, wherein the corresponding layers comprise the same convolutional level of the conditional detection network.

7. The object detection method according to claim 1, wherein the providing comprises: providing the attention map to a layer of the conditional detection network; as well as The word embeddings are provided to an additional layer of the conditional detection network, the layers of the conditional detection network being between the additional layers of the conditional detection network.

8. The object detection method of claim 1 , further comprising training the conditional detection network using a dataset of images, the images comprising respective bounding boxes of respective objects in each of the images, the training comprising: assigning positive training labels to detection outputs of the conditional detection network that substantially overlap with a truth bounding box in the corresponding bounding boxes corresponding to the term-based concept; as well as Negative training labels are assigned to other detection outputs of the conditional detection network that are not assigned to positive training labels.

9. The object detection method according to claim 1, further comprising training the conditional detection network by: determining a negative class of the word-based concept; providing the negative class to the conditional detection network; and A negative training label is assigned to a detection output of the conditional detection network for the negative class that substantially overlaps with a truth bounding box corresponding to the term-based concept.

10. The object detection method according to claim 1, further comprising training the conditional detection network by: setting the presence value of the attention map to one; setting the value of the word embedding to zero; and Corresponding regions of the input image for a plurality of objects of different word-based concepts are detected.

11. An object detection system in a digital media environment for detecting an object in an image, comprising: an image module implemented at least in part in hardware of a computing device to obtain an input image; an input condition module implemented at least in part in hardware of the computing device to receive a term-based concept; an attention map module implemented at least in part in hardware of the computing device to generate an attention map for the word-based concepts based on the input image, the attention map comprising pixels indicating presence values, each presence value indicating a respective presence of the word-based concept at a respective pixel of the input image; a word embedding module implemented at least in part in hardware of the computing device to generate word embeddings based on the word-based concepts, the word embeddings describing mutual information between the word-based concepts and labels from a vocabulary; as well as A detection module, the detection module being implemented at least in part in hardware of the computing device to detect at least one region of the input image based on the word embeddings and the attention map, the at least one region including a corresponding object corresponding to the word-based concept.

12. The object detection system of claim 11, further comprising a training module, wherein the attention map is generated using an image labeling network of the attention map module, the image labeling network being trained using the training module and using a first dataset of first images, the first images comprising annotations without bounding boxes of objects in the first images.

13. The object detection system of claim 12, wherein the detection module comprises a conditional detection network that detects the at least one region, the conditional detection network being trained using the training module and using a second dataset of second images that include corresponding bounding boxes for corresponding objects in the second images.

14. The object detection system of claim 13, wherein the second data set includes the input image as one of the second images.

15. The object detection system of claim 13, wherein the input image is user supplied and is not included in the first data set or the second data set.

16. An object detection system according to claim 11, wherein the detection module includes a conditional detection network for detecting the at least one region, the word embedding module provides the word embedding as conditional input to two layers of the conditional detection network, and the attention map module provides the attention map as an additional conditional input to the third layer of the conditional detection network.

17. The object detection system of claim 16, wherein the two layers and the third layer comprise a same convolutional level of the conditional detection network, and the third layer is between the two layers in the same convolutional level.

18. An object detection method implemented by a computing device in a digital media environment for detecting an object in an image, the object detection method comprising: Obtaining input images and word-based concepts; generating an attention map for the word-based concept according to the input image, the attention map comprising pixels, each pixel of the attention map indicating a corresponding presence value, the corresponding presence value indicating a corresponding presence of the word-based concept at a corresponding pixel of the input image; generating a word embedding based on the word-based concept, the word embedding describing a relationship between the word-based concept and a different word-based concept; as well as A step for detecting at least one region of the input image based on the attention map and the word embeddings using a conditional detection network, the at least one region including a corresponding object corresponding to the term-based concept.

19. The object detection method according to claim 18, wherein said generating said word embedding comprises: A vector is formed based on an eigendecomposition of a matrix constructed based on the co-occurrence probabilities of the term-based concepts and the different term-based concepts.

20. The object detection method according to claim 18, wherein the corresponding existence value is greater than or equal to -1 and less than or equal to 1.

Images