Processing images to locate novel objects

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for training an optical flow object localization system and a novel object localization system. In a first aspect, the training optical flow object localization system processes an optical flow image to generate object localization data, the object localization data defining the location of an object depicted in a video frame corresponding to the optical flow image. In a second aspect, the training novel object localization system processes a video frame to generate object localization data, the object localization data defining the location of a novel object depicted in the video frame.

Description

[0001] This application is a divisional application of the invention patent application with the application date of August 19, 2019, the application number of 201980074876.1, and the invention name of “Processing images to locate novel objects”

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Patent Application No. 16 / 264,222, filed on January 31, 2019, and claims the benefit of U.S. Patent Application No. 62 / 760,594, entitled “PROCESSING IMAGES TO LOCALIZE NOVEL OBJECTS,” filed on November 13, 2018. The disclosures of the above applications are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] This specification relates to image processing. Background Art

[0005] An object localization system is a system having a plurality of trainable parameters that is configured to process an image to localize (i.e., identify) the position of an object depicted in the image. The object localization system can be trained using a set of training examples, where each training example includes: (i) a training image, and (ii) training object localization data defining the position of an object depicted in the training image. Summary of the invention

[0006] This specification describes a training system implemented as a computer program on one or more computers in one or more locations that can be used to train an object localization system.

[0007] According to a first aspect, a method performed by one or more data processing devices is provided, the method comprising: obtaining, for each video frame of a video comprising a plurality of video frames, object localization data defining a position of an object depicted in the video frame; processing the video to generate a corresponding optical flow image for each video frame in the plurality of video frames, the optical flow image representing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; and training an optical flow object localization system to process the optical flow images to generate object localization data defining the position of an object depicted in the video frames corresponding to the optical flow images, using: (i) the optical flow images, and (ii) the object localization data defining the position of an object depicted in the plurality of video frames corresponding to the optical flow images.

[0008] In some implementations, obtaining object localization data defining the location of an object depicted in a video frame can include processing the video frame using a known object localization system. The known object localization system can be a previously trained object localization system. For example, the known object localization system can be configured to process the video frame to generate data defining the location of an object depicted in the video frame, the object being from a predetermined set of object classes. In some implementations, the known object localization system can include a neural network.

[0009] Additionally or alternatively, obtaining object localization data defining the location of an object depicted in a video frame may include obtaining object localization data defining the location of an object depicted in a video frame that was manually annotated by a person. The object localization data defining the location of an object depicted in a video frame includes data defining a bounding box surrounding the location of the object depicted in the video frame. More generally, the object localization data may include data defining coordinates of corners of a shape surrounding the located object.

[0010] Processing the video to generate a corresponding optical flow image for each of the plurality of video frames may include processing the video using a direct numerical optimization method to generate a corresponding optical flow image for each of the plurality of video frames. The direct numerical optimization method may be a method of fitting an optical flow field to pixel data from a video frame in the time domain or frequency domain, such as using local spatial / temporal derivatives or by region matching. Additionally or alternatively, the optical flow object localization system may include a neural network.

[0011] The method may further include ignoring object localization data generated for objects that are stationary or have less than a threshold degree of motion relative to their background. Thus, the method may further include determining, for each object position defined by the object localization data for a video frame corresponding to the optical flow image, a corresponding variance of optical flow data in the (above) optical flow image at the object position. The method then modifies the object localization data used to train the optical flow object localization system by removing data defining the object position, wherein the variance of the optical flow data in the optical flow image at the object position does not meet a (minimum) threshold.

[0012] In another aspect, a method performed by one or more data processing devices includes: processing a video including a plurality of video frames to generate a corresponding optical flow image for each of the plurality of video frames, the optical flow image representing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; for each optical flow image, processing the optical flow image using an optical flow object localization system to generate object localization data defining a location of an object depicted in the video frame corresponding to the optical flow image; and training a second object localization system, e.g., a novel object localization system, to process the video frames to generate object localization data defining a location of an object depicted in the video frame corresponding to the optical flow image using: (i) the plurality of video frames, and (ii) the object localization data generated by the optical flow object localization system by processing the optical flow images corresponding to the plurality of video frames.

[0013] An optical flow object localization system may be trained using: (i) an optical flow image and (ii) object localization data defining the location of an object depicted in a video frame corresponding to the optical flow image. As previously described, the object localization data defining the location of an object depicted in a video frame corresponding to the optical flow image may be obtained by processing the video frame using a known object localization system.

[0014] Thus, the method may include using a known or trained object localization system that may not be able to detect (localize) a particular object, such as a particular class of objects, and using it to provide object localization data for training an optical flow object localization system. An optical flow object localization system, which works on a different basis, namely optical flow, may learn to localize objects that are missed by a known / trained object localization system. Thus, an optical flow object localization system may be used to generate training data that can be used to train a second object localization system, such as to improve a known or trained object localization system through further training, and / or to train a "novel" object localization system (which is not based on optical flow).

[0015] Thus, in some implementations, training a novel object localization system includes, for one or more of a plurality of video frames: processing the video frames using a known object localization system to generate object localization data defining a location of an object depicted in the video frames, wherein the known object localization system is trained to localize objects from a predetermined set of known object classes; identifying a location of a novel object depicted in the video frames, wherein the location of the novel object: (i) is included in the object localization data generated for the video frames by the optical flow object localization system, and (ii) is not included in the object localization data generated for the video frames by the known object localization system; and training the novel object localization system to process the video frames to generate the object localization data, the object localization data including data defining the location of the novel object depicted in the video frames.

[0016] The novel object localization system can be trained to process video frames to generate object localization data that includes object localization data generated by known object localization systems by processing video frames, in addition to data defining locations of novel objects depicted in the video frames.

[0017] The method may further include processing the image using the trained second object localization system to generate object localization data defining the location of an object depicted in the image, wherein the image is not a video frame. Thus, once trained, the second object localization system may be applied to any type of image.

[0018] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0019] This specification describes a training system that can train an object localization system (referred to in this specification as a "novel object localization system") to locate a "novel" object in an image. A novel object in an image is an object that is missed (i.e., not located) by an existing object localization system (referred to in this specification as a "known object localization system"). By identifying where the known object localization system currently fails, the novel object localization system can be used to improve the accuracy of the known object localization system. In a specific example, the novel objects located by the novel object localization system can be used as training data to improve the accuracy of the known object localization system in locating objects in an image. This is a technical advance in the field of image processing.

[0020] The present specification also describes a training system that can train an object localization system (referred to as an "optical flow object localization system" in this specification) to process an optical flow image to locate an object depicted in a video frame corresponding to the optical flow image. An optical flow object localization system trained in this manner can be used to generate training data for training a novel object localization system. In addition, the training system can efficiently generate training data for training the optical flow object localization system by transferring the object localization data from the video frame to the corresponding optical flow image. For example, the training system can process the video frame using a known object localization system to generate an object bounding box surrounding an object depicted in the video frame, and transfer the generated object bounding box to the optical flow image corresponding to the video frame. In this way, the training system can automatically generate training data for the optical flow object localization system without the need to manually annotate the object localization data on the optical flow image, which is a time-consuming and difficult process. This is also a technological advancement in the field of image processing.

[0021] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram of an example optical flow object localization training system and an example novel object localization training system.

[0023] Figure 2 is an example data stream for localizing novel objects in video frames using known object localization systems and optical flow object localization systems.

[0024] Figure 3 is a flow chart of an example process for training an optical flow object localization system.

[0025] Figure 4 is a flow chart of an example process for training a novel object localization system.

[0026] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0027] An object localization system is a system having one or more trainable parameters (e.g., a neural network) that is configured to process an image (e.g., an RGB image) to localize (i.e., identify) the location of an object depicted in the image. The object localization system can be trained using a set of training examples, where each training example includes: (i) a training image, and (ii) training object localization data defining the location of an object depicted in the training image. The present specification describes a method for training a "novel" object localization system to localize an object that is missed (i.e., not localized) by a current "known" object localization system. In some cases, a known object localization system may not be able to localize an object because the object is rare or unusual (e.g., a space shuttle) and does not fit into any of the predetermined object class sets that the known object localization system is trained to localize. In some other cases, a known object localization system may not be able to localize an object in an image if the object has a different visual appearance in the image than in the training images used to train the known object localization system. For example, if the training images used to train the known object localization system all depict cats from a frontal perspective, the known object localization system may not be able to localize a cat depicted in the image from a top-down perspective. Objects missed by known object localization systems are referred to in this specification as "novel objects".

[0028] Using the training data generated by the optical flow object localization system, the novel object localization system is trained to localize the novel object. The trained optical flow object localization system processes the optical flow images corresponding to the video frames of the video to localize the objects in the video frames. The optical flow images corresponding to the video frames describe the motion of the objects depicted in the video frames, rather than the visual appearance (e.g., color, texture, etc.). Therefore, the optical flow object localization system localizes the objects based on the motion of the objects rather than their visual appearance.

[0029] The optical flow object localization system generates training data for the novel object localization system by processing optical flow images corresponding to the training video frames to localize objects depicted in the training video frames based on motion. Objects depicted in the training video frames that (i) are localized by the optical flow object localization system and (ii) are not localized by a known object localization system can be identified as novel objects. That is, if the object is localized by the optical flow object localization system based on the object's motion, but is not localized by a known object localization system based on the object's visual appearance, then the object can be identified as a novel object. The novel object localization system can be trained to process the training video frames to localize the identified novel objects depicted in the training video frames. After training, the novel object localization system can process images including but not limited to video frames.

[0030] An optical flow object localization system is trained using: (i) an optical flow image corresponding to a video frame, and (ii) training object localization data indicating the location of an object depicted in the video frame. In some cases, the training object localization data may be generated by a person manually annotating the location of an object depicted in the video frame. In some other cases, the training object localization data may be generated by processing the video frame using a known object localization system. The training optical flow object localization system processes the optical flow image corresponding to the video frame to locate the object depicted in the video frame (i.e., as specified by the training object localization data).

[0031] These and other features are described in more detail below.

[0032] Figure 1 is a block diagram of an exemplary novel object localization training system 100 and an exemplary optical flow object localization training system 102. System 100 and system 102 are examples of systems implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0033] To train the optical flow object localization system 104, the system 102 obtains a sequence of video frames 106 from one or more videos. Each of the video frames 106 is an image that can be represented as a two-dimensional (2D) array of pixels, where each pixel is represented as a vector of one or more values. For example, if the video frame is a black and white image, each pixel can be represented as an integer or floating point number representing the brightness of the pixel (i.e., a vector with one component). As another example, if the video frame is a red-green-blue (RGB) image, each pixel can be represented as a vector with three integer or floating point components, representing the intensity of the red, green, and blue of the pixel, respectively. In some other examples, YUV or HSV color channels can be used. The pixels of the video frame can be indexed by (x, y) coordinates, where x and y are integer values.

[0034] The system 102 processes the video frames 106 from each video using the optical flow engine 108 to generate a corresponding optical flow image 110 corresponding to each video frame 106. The optical flow image corresponding to the video frame in the video characterizes the displacement of each pixel of the video frame between the video frame and the subsequent video frame in the video. The displacement of a pixel from a video frame to a subsequent video frame can be represented as a vector having two digital (e.g., integer or floating point) components, wherein the first component represents the x displacement of the pixel and the second component represents the y displacement of the pixel. The optical flow image corresponding to the video frame can be represented as a 2D array of displacements indexed by the same (x, y) coordinates as the video frame. That is, the displacement at a specific (x, y) coordinate in the optical flow image represents the displacement of the pixel at the same (x, y) coordinate in the corresponding video frame. The optical flow image 110 corresponding to the video frame 106 from the video describes the motion of the object depicted in the video frame 106, rather than the visual appearance (e.g., color, texture, etc.).

[0035] System 102 obtains training object localization data 112 for each video frame 106. Training object localization data 112 for a video frame defines the location of an object depicted in the video frame. For example, training object localization data 112 for a video frame 106 may define the location of an object depicted in the video frame 106 by data defining coordinates of a bounding box around the object depicted in the video frame 106. As another example, training object localization data 112 for a video frame 106 may define the location of an object depicted in the video frame 106 by data defining a segmentation of the object depicted in the video frame 106. The segmentation of the object depicted in the video frame can be represented as a list of (x, y) coordinates of pixels in the video frame that are included in the object.

[0036] In some implementations, system 102 obtains training object location data 112 for video frame 106 by processing video frame 106 using known object location system 114. As previously described, known object location system 114 is trained to process images to locate objects depicted in the images. In some other implementations, system 102 obtains training object location data 112 for video frame 106 by accessing (i.e., from a data storage) object location data that has been manually generated for video frame 106 by a human.

[0037] System 102 uses training engine 116 to train optical flow object localization system 104 using: (i) optical flow images 110 and (ii) training object localization data 112. Optical flow object localization system 104 is configured to process optical flow images derived from video frames to generate object localization data defining the location of objects depicted in the video frames. Training engine 116 trains optical flow object localization system 104 to process each optical flow image 110 to generate object localization data that matches training object localization data 112 corresponding to optical flow image 110. Optical flow images corresponding to video frames describe the motion of objects depicted in the video frames, rather than the visual appearance (e.g., color, texture, etc.). Thus, optical flow object localization system 104 locates objects based on the motion of the objects, rather than based on the visual appearance of the objects.

[0038] The training engine 116 may be configured to train the optical flow object localization system 104 using any suitable machine learning training technique (e.g., stochastic gradient descent). For example, the training engine 116 may be configured to train the optical flow object localization system 104 by iteratively (i.e., at each of a plurality of training iterations) adjusting current parameter values ​​of the optical flow object localization system 104. Figure 3 An example process for training the optical flow object localization system 104 is further described.

[0039] The novel object localization training system 100 is configured to train a novel object localization system 118 to process images (e.g., video frames) to generate object localization data that defines the locations of novel objects depicted in the images (i.e., objects missed by the known object localization system 114). As will be described in more detail below, the system 100 trains the novel object localization system 118 using an optical flow object localization system 120. The system 100 may set parameter values ​​of the optical flow object localization system 120 to the trained parameter values ​​122 of the optical flow object localization system 104 trained by the system 102.

[0040] To train the novel object localization system 118, the system 100 obtains a sequence of video frames 124 from one or more videos (which may be the same as the video frames 106 used in training the optical flow object localization system 104). The system 100 processes the video frames 124 from each video using the optical flow engine 108 to generate a respective optical flow image 126 corresponding to each video frame 124. The system 100 processes each optical flow image 126 using the optical flow object localization system 120 to generate training object localization data 128 that defines the location of the object depicted in the video frame 124. Because the optical flow object localization system 120 locates objects based on their motion rather than on their visual appearance, the training object localization data 128 is able to recognize novel objects without training data from an original source that characterizes their visual appearance (e.g., color, texture, etc.).

[0041] System 100 trains novel object localization system 118 using training engine 130 to localize novel objects using: (i) video frames 124 and (ii) training object localization data 128 generated by optical flow object localization system 120. For example, to identify a novel object depicted in video frame 124, system 100 may first process video frame 124 using known object localization system 114. System 100 may then identify the object as a novel object if the object: (i) is included in training object localization data 128 generated by optical flow object localization system 120, and (ii) is not included in object localization data generated by known object localization system 114. That is, if the object is localized by optical flow object localization system 120 based on its motion, but is not localized by known object localization system 114 based on its visual appearance, system 100 may identify the object as a novel object. In this example, the training engine 130 can process each video frame 124 by training the novel object localization system 118 to generate object localization data, which includes novel objects recognized by the system 100 in the video frame 124. In some cases, as will be described with reference to Figure 4 Described in more detail, the system is capable of training the novel object localization system 118 to generate object localization data that additionally includes object localization data generated by the known object localization system 114 for a video frame.

[0042] The training engine 130 may be configured to train the novel object localization system 118 using any suitable machine learning training technique (e.g., stochastic gradient descent). For example, the training engine 130 may be configured to train the novel object localization system 118 by iteratively (i.e., at each of a plurality of training iterations) adjusting current parameter values ​​of the novel object localization system 118. Figure 4 An example process for training the novel object localization system 118 is further described.

[0043] The novel object localization system 118 trained by the system 100 is not limited to processing video frames. In particular, after being trained, the novel object localization system 118 is able to process any image (including but not limited to video frames) to locate novel objects depicted in the image.

[0044] Figure 22 is an example data stream 200 for identifying a novel object 202-A in a video frame 204 using a known object localization system 114 and an optical flow object localization system 120. The known object localization system 114 processes the video frame 204 to generate object localization data 206 that defines the locations of objects 208-A and 208-B. The optical flow object localization system 120 processes an optical flow image 210 corresponding to the video frame 204 to generate object localization data 212 that defines the locations of objects 202-A and 202-B. Object 202-A is able to be identified as a novel object because it is located by the optical flow object localization system 120 but not by the known object localization system 114. Object 208-B is located by the known object localization system 114 but not by the optical flow object localization system 120, which may be because object 208-B is a stationary object that cannot be reliably located from the optical flow image 210 (as described in more detail below).

[0045] Figure 3 is a flow chart of an example process 300 for training an optical flow object localization system. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, an optical flow object localization training system appropriately programmed according to the present specification, such as Figure 1 The optical flow object localization training system 102 can perform the process 300 .

[0046] The system obtains a corresponding sequence of video frames from one or more videos (302). The videos can be captured by a digital camera or generated by a computer. Each video frame is an image that can be represented as a 2D array of pixels, where each pixel is a vector represented as one or more color values ​​(e.g., RGB color values).

[0047] The system obtains corresponding training object location data for each video frame (304). The training object location data for the video frame defines the location of the object depicted in the video frame. For example, the training object location data for the video frame may define the location of the object depicted in the video frame by defining data of coordinates of a corresponding bounding box surrounding the object depicted in the video frame. As another example, the training object location data for the video frame may define the location of the object depicted in the video frame by defining data of corresponding segmentations of the object depicted in the video frame. In some implementations, the system obtains the training object location data for the video frame by processing the video frame using a known object location system. The known object location system may be configured to process the video frame according to training values ​​of known object location system parameters to generate object location data for the video frame. In some other implementations, the system is capable of obtaining training object location data for video frames manually annotated by a person.

[0048] The system generates a corresponding optical flow image corresponding to each video frame (306). The system can generate an optical flow image corresponding to the video frame using any suitable method. For example, the system can use a direct numerical optimization method (e.g., Lucas-Kanade method) to process a video frame from the video and a video frame after the video frame in the video to generate an optical flow image corresponding to the video frame. Therefore, the direct numerical optimization method can be a method of fitting an optical flow defined by, for example, a flow velocity vector to flow data defined by a video frame by a direct numerical optimization method. For example, in the Lucas-Kanade method, it is assumed that the local velocity is constant. More generally, a differential technique can be used to determine the optical flow based on local and / or global first and optional second derivatives of pixel values ​​with respect to time and / or x-displacement and y-displacement.

[0049] As another example, the system can process a video frame from the video and a video frame following the video frame in the video using a neural network configured to generate an output defining an optical flow image corresponding to the video frame.

[0050] The optical flow image may include, for example, a set of x-pixel displacement values ​​and y-pixel displacement values ​​for each pixel location. Alternatively, for example, it may include the x-component and y-component of the vector-mapped motion of pixels from one frame to one or more subsequent frames. Optical flow may be unidirectional, forward or backward, or bidirectional. Optionally, a global, e.g., average, motion estimate component may be subtracted to compensate for camera / sensor motion.

[0051] The system trains an optical flow object localization system (308) using: (i) an optical flow image and (ii) training object localization data defining the location of an object depicted in a video frame corresponding to the optical flow image. More specifically, the system trains the optical flow object localization system to process each optical flow image to generate object localization data that matches the training object localization data, which defines the location of an object depicted in a video frame corresponding to the optical flow image. The system can train the optical flow object localization system by iteratively adjusting parameter values ​​of the optical flow object localization system multiple times using a machine learning training technique (e.g., stochastic gradient descent). In some cases, the optical flow object localization system can be implemented as a neural network. An example of an object localization neural network and a method for training an object localization neural network are described with reference to S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: toward real-time object localization with region proposal networks", Conference on Advances in Neural Information Processing Systems (NIPS), 2015. Any other suitable implementation of an object localization system can also be used.

[0052] In some cases, the training object localization data may define locations of objects in a video frame that the optical flow object localization system will not be able to reliably localize from an optical flow image corresponding to the video frame. For example, the optical flow object localization system may not be able to reliably localize an object depicted in a video frame that is stationary (i.e., not moving) relative to a background depicted in the video frame. In this example, the portion of the optical flow image corresponding to the stationary object will not distinguishably characterize the stationary object because the displacement vectors in that portion of the optical flow image may all be approximately zero (or some other constant value). The system may modify the training object localization data used to train the optical flow object localization system by removing data from the optical flow image that defines the locations of objects that the optical flow object localization system will not be able to reliably localize.

[0053] To determine whether the optical flow object localization system will not be able to reliably locate the object specified by the training object localization data, the system can determine the variance of the portion of the optical flow image that characterizes the object. If the variance does not meet a predetermined (minimum) threshold, the system can determine that the optical flow object localization system will not be able to reliably locate the object, and then avoid training the optical flow object localization system to locate the object.

[0054] Figure 4 4 is a flow chart of an example process 400 for training a novel object localization system. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a novel object localization training system appropriately programmed according to the present specification, e.g., Figure 1 The novel object localization training system 100 is capable of performing the process 400 .

[0055] The system obtains a corresponding sequence of video frames from one or more videos (402). The videos can be captured by a digital camera or generated by a computer. Each video frame is an image that can be represented as a 2D array of pixels, where each pixel is represented as a vector of one or more color values ​​(e.g., RGB color values). In some cases, the obtained video frames can be the same video frames used to train the optical flow object localization system (e.g., see FIG. 4 ). Figure 3 described above).

[0056] The system generates a corresponding optical flow image corresponding to each video frame (404). The system can generate optical flow images corresponding to video frames using any suitable method. Reference 306 describes some example methods for generating optical flow images corresponding to video frames.

[0057] The system processes each optical flow image using a trained optical flow object localization system to generate corresponding trained object localization data (406) that defines the location of an object depicted in a video frame corresponding to the optical flow image. The optical flow object localization system may, for example, be trained by referring to Figure 3 The described process 300 is trained. The training object localization data generated by the optical flow object localization system of the optical flow images may be, for example, data defining corresponding bounding box coordinates surrounding objects depicted in the video frames, or data of corresponding segmentations of objects depicted in the video frames.

[0058] The system trains a novel object localization system (408) using: (i) video frames and (ii) training object localization data generated by the optical flow object localization system by processing optical flow images corresponding to the video frames. For example, to identify a novel object depicted in a video frame, the system may first process the video frame using a known object localization system. Then, if the object: (i) is included in the training object localization data generated by the optical flow object localization system, and (ii) is not included in the object localization data generated by the known object localization system, then the system may identify the object as a novel object. That is, if the object is located by the optical flow object localization system based on the object's motion, but is not located by the known object localization system based on the object's visual appearance, then the system may identify the object as a novel object. In this example, the system is able to train the novel object localization system to process each video frame to generate object localization data that includes novel objects identified by the system in the video frame.

[0059] In some implementations, the system can train the novel object localization system to process each video frame to generate object localization data that includes: (i) novel objects identified by the system in the video frame, and (ii) objects located in the video frame by the known object localization system. The system can further train the novel object localization system to process each video frame to generate corresponding object class labels for objects depicted in the video frame. In these implementations, the system can initialize parameter values ​​of the novel object localization system with parameter values ​​of the known object localization system.

[0060] The system can be configured to train the novel object localization system using any appropriate machine learning training technique (e.g., stochastic gradient descent). For example, the system can be configured to train the novel object localization system by iteratively (i.e., at each of multiple training iterations) adjusting the current parameter values ​​of the novel object localization system. For example, the novel object localization system can be implemented as a neural network. Referring to S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: toward real-time object localization with region proposal networks", Conference on Advances in Neural Information Processing Systems (NIPS), 2015, examples of object localization neural networks and methods of training object localization neural networks are described. Any other suitable implementation of the object localization system can also be used.

[0061] This specification uses the term "configured to" in relation to system and computer program components. For a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0062] Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution or control of operations by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0063] The term "data processing apparatus" refers to data processing hardware, and includes various devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus can also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.

[0064] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; the computer program can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordinated files, such as files storing one or more modules, subroutines or portions of code. A computer program can be deployed to execute on one computer, or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0065] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0066] The processes and logic flows described in this specification can be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., FPGA or ASIC) or by a combination of a special purpose logic circuit and one or more programmed computers.

[0067] The computer suitable for executing a computer program can be based on a general or special microprocessor or both, or based on any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for performing or executing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Typically, the computer will also include or be operably coupled to receive data from one or more large-capacity storage devices for storing data, such as disks, magneto-optical disks or optical disks, or transfer data to one or more large-capacity storage devices or both. However, the computer does not have to have such devices. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0068] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0069] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to a user, and a keyboard and pointing device, such as a mouse or trackball, through which a user can provide input to the computer. Other types of devices can also be used to provide interaction with a user, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from the device used by the user, for example, by responding to a request received from the web browser, sending a web page to a web browser on the user's device. In addition, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user.

[0070] The data processing device used to implement the machine learning model can also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0071] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0072] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with an implementation of the subject matter described in this specification, or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0073] The computing system can include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data such as HTML pages to the user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input therefrom. Data generated at the user device, such as the result of user interaction, can be received from the device on the server.

[0074] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on any invention or the scope of the claimed protection, but should be interpreted as descriptions of features specific to a particular embodiment of a particular invention. Specific features described in this specification in the context of a separate embodiment can also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments or in any suitable sub-combination. In addition, although the features can be described above as working in a specific combination and even initially claimed, in some cases, one or more features in the claimed combination can be deleted from the combination, and the claimed combination can be used for a sub-combination or a variation of the sub-combination.

[0075] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0076] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0077] In a first aspect, a method performed by one or more data processing devices is provided, the method comprising: obtaining, for each video frame of a video comprising a plurality of video frames, object localization data defining a position of an object depicted in the video frame; processing the video to generate, for each video frame in the plurality of video frames, a corresponding optical flow image, the optical flow image representing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; and training an optical flow object localization system to process the optical flow images to generate object localization data defining the position of an object depicted in the video frames corresponding to the optical flow images, using: (i) the optical flow images, and (ii) the object localization data defining the position of an object depicted in the plurality of video frames corresponding to the optical flow images.

[0078] In combination with any of the foregoing aspects, in some implementations, obtaining object localization data defining a position of an object depicted in a video frame includes: processing the video frame using a known object localization system, wherein the known object localization system is configured to process the video frame to generate data defining a position of an object depicted in the video frame, the object being from a predetermined set of object classes.

[0079] In conjunction with any of the previous aspects, in some implementations, the known object localization system includes a neural network.

[0080] In conjunction with any of the foregoing aspects, in some implementations, obtaining object localization data defining locations of objects depicted in video frames includes obtaining object localization data defining locations of objects depicted in video frames that were manually annotated by a human.

[0081] In conjunction with any of the foregoing aspects, in some implementations, the object localization data defining the position of an object depicted in a video frame includes data defining a bounding box surrounding the position of the object depicted in the video frame.

[0082] In conjunction with any of the foregoing aspects, in some implementations, processing the video to generate a corresponding optical flow image for each of the plurality of video frames includes processing the video to generate a corresponding optical flow image for each of the plurality of video frames using a direct numerical optimization method.

[0083] In combination with any of the foregoing aspects, in some implementations, the method further includes: for each optical flow image: for each object position defined by object localization data of a video frame corresponding to the optical flow image, determining a corresponding variance of optical flow data in the optical flow image at the object position; and modifying the object localization data for training the optical flow object localization system by removing data defining the object position, wherein the variance of the optical flow data in the optical flow image at the object position does not satisfy a threshold.

[0084] In combination with any of the previous aspects, in some implementations, the optical flow object localization system includes a neural network.

[0085] In another aspect, a method performed by one or more data processing devices is provided, the method comprising: processing a video comprising a plurality of video frames to generate a corresponding optical flow image for each of the plurality of video frames, the optical flow image representing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; for each optical flow image, processing the optical flow image using an optical flow object localization system to generate object localization data defining a position of an object depicted in the video frame corresponding to the optical flow image; and training a novel object localization system to process the video frames to generate object localization data defining a position of an object depicted in the video frame using: (i) the plurality of video frames, and (ii) the object localization data generated by the optical flow object localization system by processing the optical flow images corresponding to the plurality of video frames.

[0086] In combination with any of the previous aspects, in some implementations, processing the video to generate a corresponding optical flow image for each of the plurality of video frames includes: processing the video using a direct numerical optimization method to generate a corresponding optical flow image for each of the plurality of video frames.

[0087] In combination with any of the previous aspects, in some implementations, an optical flow object localization system is trained using: (i) an optical flow image and (ii) object localization data defining locations of objects depicted in video frames corresponding to the optical flow image.

[0088] In combination with any of the foregoing aspects, in some implementations, object localization data defining a location of an object depicted in a video frame corresponding to an optical flow image is obtained by processing the video frame using a known object localization system, wherein the known object localization system is configured to process the video frame to generate data defining a location of an object depicted in the video frame, the object being from a predetermined set of object classes.

[0089] In conjunction with any of the previous aspects, in some implementations, the known object localization system includes a neural network.

[0090] In combination with any of the foregoing aspects, in some implementations, the object localization data defining the location of an object depicted in a video frame corresponding to the optical flow image includes: data defining a bounding box in the optical flow image surrounding the location of the object depicted in the video frame corresponding to the optical flow image.

[0091] In combination with any of the foregoing aspects, in some implementations, training the novel object localization system includes, for one or more of a plurality of video frames: processing the video frames using a known object localization system to generate object localization data defining a location of an object depicted in the video frames, wherein the known object localization system is trained to localize objects from a predetermined set of known object classes; identifying a location of a novel object depicted in the video frames, wherein the location of the novel object: (i) is included in the object localization data generated for the video frames by the optical flow object localization system, and (ii) is not included in the object localization data generated for the video frames by the known object localization system; and training the novel object localization system to process the video frames to generate the object localization data, the object localization data including data defining the location of the novel object depicted in the video frames.

[0092] In combination with any of the foregoing aspects, in some implementations, the method further includes training the novel object localization system to process video frames to generate object localization data, wherein the object localization data includes object localization data generated by a known object localization system by processing the video frames in addition to data defining the location of the novel object depicted in the video frames.

[0093] In conjunction with any of the previous aspects, in some implementations, the method further includes processing the image using the trained novel object localization system to generate object localization data defining a location of an object depicted in the image, wherein the image is not a video frame.

[0094] In conjunction with any of the previous aspects, in some implementations, the novel object localization system includes a neural network.

Claims

1. A method performed by one or more data processing devices, the method comprising: Processing a video including a plurality of video frames to generate a corresponding optical flow image for each of the plurality of video frames, the optical flow image representing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; for each optical flow image, processing the optical flow image using an optical flow object localization system to generate object localization data defining a location of an object depicted in a video frame corresponding to the optical flow image; and Training a visual object localization system to process the video frames to generate object localization data defining locations of objects depicted in the video frames using: (i) a plurality of video frames, and (ii) object localization data generated by an optical flow object localization system by processing optical flow images corresponding to the plurality of video frames, wherein training the visual object localization system comprises performing a multi-step process of: determining training object localization data for the video frame based on object localization data generated by processing optical flow images corresponding to the video frame using an optical flow object localization system; and A visual object localization system is trained to process the video frames to generate object localization data for the video frames that matches the training object localization data for the video frames.

2. The method according to claim 1, wherein: Determining training object localization data for a video frame based on object localization data generated by processing an optical flow image corresponding to the video frame using an optical flow object localization system includes: processing the video frames using a known object localization system to generate object localization data defining locations of objects depicted in the video frames, wherein the known object localization system is trained to localize objects from a predetermined set of known object classes; identifying a location of a novel object depicted in a video frame, wherein the location of the novel object: (i) is included in object localization data generated for the video frame by an optical flow object localization system, and (ii) is not included in object localization data generated for the video frame by a known object localization system; and Included in the training object localization data is data defining the locations of novel objects.

3. The method according to claim 1, wherein: The training object localization data includes object localization data generated by processing optical flow images corresponding to video frames using an optical flow object localization system.

4. The method according to claim 1, wherein: Processing the video to generate a corresponding optical flow image for each of the plurality of video frames includes: The video is processed using a direct numerical optimization method to generate a corresponding optical flow image for each of a plurality of video frames.

5. The method according to claim 1, wherein: An optical flow object localization system is trained using: (i) an optical flow image, and (ii) object localization data defining locations of objects depicted in video frames corresponding to the optical flow image.

6. The method according to claim 5, wherein: Object localization data defining the location of an object depicted in the video frame is obtained by processing the video frame corresponding to the optical flow image using a known object localization system, wherein the known object localization system is configured to process the video frame to generate data defining the location of an object depicted in the video frame, the object being from a predetermined set of object classes.

7. The method according to claim 6, wherein: The known object localization system comprises a neural network.

8. The method according to claim 1, wherein: The object localization data defining the position of an object depicted in a video frame corresponding to the optical flow image includes: Data defining a bounding box in the optical flow image surrounding the location of an object depicted in a video frame corresponding to the optical flow image.

9. The method according to claim 1, further comprising: An image is processed using a trained visual object localization system to generate object localization data defining a location of an object depicted in the image, wherein the image is not a video frame.

10. The method according to claim 1, wherein: The visual object localization system includes a neural network.

11. A system for image processing, comprising: one or more computers; as well as One or more storage devices, communicatively coupled to one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause at least one or more computers to perform operations, including: Processing a video including a plurality of video frames to generate a corresponding optical flow image for each of the plurality of video frames, the optical flow image representing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; for each optical flow image, processing the optical flow image using an optical flow object localization system to generate object localization data defining a location of an object depicted in a video frame corresponding to the optical flow image; and Training a visual object localization system to process the video frames to generate object localization data defining locations of objects depicted in the video frames using: (i) a plurality of video frames, and (ii) object localization data generated by an optical flow object localization system by processing optical flow images corresponding to the plurality of video frames, wherein training the visual object localization system comprises performing a multi-step process of: determining training object localization data for the video frame based on object localization data generated by processing optical flow images corresponding to the video frame using an optical flow object localization system; and A visual object localization system is trained to process the video frames to generate object localization data for the video frames that matches the training object localization data for the video frames.

12. The system according to claim 11, wherein: Determining training object localization data for a video frame based on object localization data generated by processing an optical flow image corresponding to the video frame using an optical flow object localization system includes: processing the video frames using a known object localization system to generate object localization data defining locations of objects depicted in the video frames, wherein the known object localization system is trained to localize objects from a predetermined set of known object classes; identifying a location of a novel object depicted in a video frame, wherein the location of the novel object: (i) is included in object localization data generated for the video frame by an optical flow object localization system, and (ii) is not included in object localization data generated for the video frame by a known object localization system; and Included in the training object localization data is data defining the locations of novel objects.

13. The system according to claim 11, wherein: The training object localization data includes object localization data generated by processing optical flow images corresponding to video frames using an optical flow target localization system.

14. The system according to claim 11, wherein: Processing the video to generate a corresponding optical flow image for each of the plurality of video frames includes: The video is processed using a direct numerical optimization method to generate a corresponding optical flow image for each of a plurality of video frames.

15. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: Processing a video including a plurality of video frames to generate a corresponding optical flow image for each of the plurality of video frames, the optical flow image representing a displacement of each pixel of the video frame between the video frame and a subsequent video frame in the video; for each optical flow image, processing the optical flow image using an optical flow object localization system to generate object localization data defining a location of an object depicted in a video frame corresponding to the optical flow image; and Training a visual object localization system to process the video frames to generate object localization data defining locations of objects depicted in the video frames using: (i) a plurality of video frames, and (ii) object localization data generated by an optical flow object localization system by processing optical flow images corresponding to the plurality of video frames, wherein training the visual object localization system comprises performing a multi-step process of: determining training object localization data for the video frame based on object localization data generated by processing optical flow images corresponding to the video frame using an optical flow object localization system; and A visual object localization system is trained to process the video frames to generate object localization data for the video frames that matches the training object localization data for the video frames.

16. The non-transitory computer storage medium of claim 15, wherein: Determining training object localization data for a video frame based on object localization data generated by processing an optical flow image corresponding to the video frame using an optical flow object localization system includes: processing the video frames using a known object localization system to generate object localization data defining locations of objects depicted in the video frames, wherein the known object localization system is trained to localize objects from a predetermined set of known object classes; identifying a location of a novel object depicted in a video frame, wherein the location of the novel object: (i) is included in object localization data generated for the video frame by an optical flow object localization system, and (ii) is not included in object localization data generated for the video frame by a known object localization system; and Included in the training object localization data is data defining the locations of novel objects.

17. The non-transitory computer storage medium of claim 15, wherein: The training object localization data includes object localization data generated by processing optical flow images corresponding to video frames using an optical flow target localization system.

Images