Systems and methods for object detection using image tiling

Abstract

A computational system for detecting objects in an image is capable of performing operations including generating an image pyramid comprising a first level corresponding to an image at a first resolution and a second level corresponding to an image at a second resolution. The operations may include: tiling the first and second levels by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; inputting the first and second plurality of tiles into a machine learning object detection model; receiving object detection data as the output of the machine learning object detection model, the object detection data including bounding boxes defined with respect to individual tiles in the first and second plurality of tiles respectively; and generating an image object detection output by mapping the object detection data onto the image space of the image.

Description

Technical Field

[0001] This disclosure generally relates to machine learning. More specifically, this disclosure relates to systems and methods for object detection using image tiling. Background Technology

[0002] Machine learning models can be trained to detect objects within images. However, detecting both small and large objects within the same image can be challenging for such models. More specifically, small objects are often missed or misidentified by machine learning models that have not yet been trained to recognize them. Therefore, improved machine learning object detection and / or recognition models, along with methods for training them, are highly desirable in this field. Summary of the Invention

[0003] Aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0004] One aspect of this disclosure relates to a computing system including at least one processor and a machine learning object detection model configured to receive a plurality of tiles and, in response to receiving the plurality of tiles, output object detection data for the plurality of tiles. The object detection data can include a plurality of bounding boxes defined with respect to individual tiles in the plurality of tiles. The computing system can include at least one tangible, non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform an operation. The operation can include generating an image pyramid based on an image having an image space. The image pyramid can include a first level corresponding to an image of a first resolution and a second level corresponding to an image of a second resolution different from the first resolution. The operation can include: tiling the first level and the second level by dividing the first level into a first plurality of tiles and dividing the second level into a second plurality of tiles; inputting the first plurality of tiles and the second plurality of tiles into the machine learning object detection model; receiving object detection data as the output of the machine learning object detection model, the object detection data including a plurality of bounding boxes defined with respect to individual tiles in the first plurality of tiles and the second plurality of tiles; and generating an image object detection output by mapping the object detection data onto the image space of the image.

[0005] Another aspect of this disclosure relates to a method for training a machine learning object detection model. The method may include: for each of a plurality of training images, performing a series of steps, including generating an image pyramid by one or more computing devices based on the corresponding training image having a corresponding image space, the image pyramid including a first level corresponding to the corresponding training image at a first resolution and a second level corresponding to the corresponding training image at a second resolution different from the first resolution. The steps may include: tiling the first and second levels by one or more computing devices by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; inputting the first plurality of tiles and the second plurality of tiles into a machine learning object detection model by one or more computing devices; receiving object detection data as the output of the machine learning object detection model by one or more computing devices, the object detection data including a plurality of bounding boxes defined with respect to individual tiles in the first plurality of tiles and the second plurality of tiles respectively; generating an image object detection output by one or more computing devices by mapping the object detection data onto the corresponding image space of the corresponding training image; and adjusting the parameters of the machine learning object detection model by one or more computing devices based on a comparison between the image object detection output and ground truth object location data corresponding to the corresponding training image in the plurality of training images.

[0006] Another aspect of this disclosure relates to a method for detecting the location of objects in an image. The method can include generating an image pyramid based on an image having an image space by one or more computing devices. The image pyramid can include a first level corresponding to an image of a first resolution and a second level corresponding to an image of a second resolution different from the first resolution. The method can include: tiling the first level and the second level by the one or more computing devices by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; inputting the first plurality of tiles and the second plurality of tiles into a machine learning object detection model by the one or more computing devices; receiving object detection data as the output of the machine learning object detection model by the one or more computing devices, the object detection data including a plurality of bounding boxes defined with respect to individual tiles in the first plurality of tiles and the second plurality of tiles respectively; and generating an image object detection output by the one or more computing devices by mapping the object detection data onto the image space of the image.

[0007] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0008] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0009] A detailed discussion of embodiments for those skilled in the art is set forth in the specification with reference to the accompanying drawings, wherein:

[0010] Figure 1A A block diagram of an example computational system for detecting the location of an object in an image, according to an example embodiment of the present disclosure, is depicted.

[0011] Figure 1B A block diagram of an example computational system for detecting the location of an object in an image, according to an example embodiment of the present disclosure, is depicted.

[0012] Figure 1C A block diagram of an example computing system for generating suggested actions and providing suggested actions to a user of the computing system, according to an example embodiment of the present disclosure, is depicted.

[0013] Figure 2A A block diagram depicts an example machine learning object detection model according to an example embodiment of the present disclosure.

[0014] Figure 2B A block diagram of an example preliminary machine learning object detection model and a machine learning object detection model according to an example embodiment of the present disclosure is depicted.

[0015] Figure 3 This is a simplified flowchart of a method for detecting the position of an object in an image according to an example embodiment of the present disclosure.

[0016] Figure 4 This is a simplified flowchart of a method for detecting the position of an object in an image according to an example embodiment of the present disclosure.

[0017] Figure 5A yes Figure 4 A simplified diagram of the first level of the image pyramid, showing the first multiple tiles.

[0018] Figure 5B yes Figure 4 A simplified diagram of the second level of the image pyramid, showing the second and third multiple tiles.

[0019] Figure 5C The illustration shows the coverage Figure 5B On the second or more tiles Figure 5A The first multiple tiles.

[0020] Figure 6 A simplified version of an image pyramid comprising multiple bounding boxes is depicted according to an exemplary embodiment of the present invention.

[0021] Figure 7A flowchart is provided illustrating a method for detecting the location of an object in an image according to aspects of this disclosure.

[0022] Figure 8 A flowchart is depicted illustrating a method for training a machine learning object detection model according to aspects of this disclosure.

[0023] Figure 9 Experimental detection accuracy for recall mapping is depicted, where objects are identified in images normalized to 1024×1024.

[0024] Figure 10 Experimental detection accuracy was depicted on the first dataset for recall rendering, where training images were tiled to 256×256, input images were set to 529×529, and the model was evaluated on test images of size 1024×2048 without tiling.

[0025] Figure 11 Experimental detection accuracy was depicted on the first dataset for recall rendering, where training images were tiled to 256×256, input images were set to 529×529, the model was evaluated on test images tiled to 1024×2048, and redundant object recognition data was removed using nonmaximum suppression.

[0026] Figure 12 Experimental detection accuracy for the first dataset drawn for recall is depicted, where redundant bounding boxes are removed using nonmaximum suppression, and some detection bounding boxes are removed or "trimmed". Training images are tiled to 256×256, input images are set to 529×529, and the model is evaluated on test images with a tiled size of 1024×2048.

[0027] Figure 13 Experimental detection accuracy for a second dataset with recall rendering is depicted, where an AutoML backend model is used and the input images are set to 1024×1024.

[0028] Figure 14 Experimental detection accuracy for a second dataset with recall rendering is depicted, where an AutoML backend model is used, training images are tiled to 640×640, input images are set to 1024×1024, and nonmaximum suppression is used to remove redundant object recognition data.

[0029] Figure 15 Experimental detection accuracy for a second dataset drawn with recall is depicted, where an AutoML backend model is used, training images are tiled to 640×640, input images are set to 1024×1024, redundant object recognition data is removed using nonmaximum suppression, and some detection bounding boxes are also removed or "trimmed".

[0030] The repeated reference numerals in the various figures are intended to identify the same features in the different embodiments. Detailed Implementation

[0031] Overview

[0032] Generally, this disclosure relates to systems and methods for detecting the location of objects in an image and training a model for detection. The locations of both very small and very large objects in the same image can be detected using the embodiments disclosed herein. For example, an image pyramid comprising two or more levels can be generated based on the image. The levels of the image pyramid can correspond to images of different resolutions. The levels can be divided into tiles (e.g., “tiled”), and object detection can be performed on each tile using a machine learning model to generate object detection data for each tile. By performing object detection on each tile, small objects depicted in the image can be detected more effectively. The object detection data can include bounding boxes (or other data) describing the location of objects displayed in corresponding tiles at multiple levels of the image pyramid. The object detection data can be mapped onto the image space of the image to generate image object detection output. In this way, bounding boxes identifying the locations of small objects in the image can be mapped back to the image space, such that the image object detection output identifies such locations of small objects.

[0033] As indicated above, an image pyramid can be generated based on an image. The image pyramid can include a first level corresponding to an image of a first resolution and a second level corresponding to an image of a second resolution different from the first resolution. Both the first and second levels can be tiled by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles.

[0034] In some implementations, the first plurality of tiles can overlap with respect to the second plurality of tiles, such that each object depicted in the image is completely contained within at least one of the tiles. Additionally, tiles within a given layer of the image pyramid (e.g., within the first plurality of tiles) can overlap with adjacent tiles within the same given layer. This ensures that the trimmed bounding boxes described above do not unintentionally remove all bounding boxes corresponding to a particular image. In one example, each of the first plurality of tiles can overlap with at least two of the second plurality of tiles. For example, each of the first plurality of tiles can overlap by 50% with adjacent tiles in the first plurality of tiles (within the same image pyramid layer) and / or the second plurality of tiles (in different image pyramid layers).

[0035] Machine learning object detection models can be used to detect objects within various types of tiles. More specifically, a first set of tiles and a second set of tiles can be input into the machine learning object detection model. Object detection data can be received as the output of the machine learning object detection model, which includes multiple bounding boxes defined with respect to individual tiles in the first and second sets of tiles, respectively. Therefore, machine learning object detection models can be used to generate bounding boxes that identify the locations of objects within various types of tiles.

[0036] As described above, object detection data can be mapped onto the image space of an image to generate image object detection output describing the location of detected objects in the image. In other words, the bounding boxes that identify objects located within various tiles can be mapped back to image space, enabling them to identify objects located within the image.

[0037] In some implementations, the image object detection output can be "trimmed" to improve its quality. Bounding boxes in the image object detection output can be selected or "trimmed" from the object detection data to generate the output. More specifically, bounding boxes that intersect with one or more boundaries of a tile can be identified and removed. Removing such bounding boxes can improve the quality of the image object detection output by reducing and / or removing partial and / or duplicate detections. As used herein, "intersection" can mean that the boundary of the bounding box is within a threshold distance of the tile boundary. For example, if a bounding box is within a threshold K pixels of the tile boundary, the boundary of the bounding box can be considered to "intersect" the tile boundary, where K can be equal to 2 or less. However, in some embodiments, the threshold can be equal to 4 or less, and in some embodiments, the threshold can be equal to 10 or less.

[0038] As an example, it's possible to remove bounding boxes that intersect one or more boundaries of tiles that define bounding boxes within them. Intersection with boundaries indicates a bounding box corresponding to "partial" detection, where the detected object spans multiple tiles across the corresponding level of the image pyramid. In other words, such intersections indicate that the detected object is not entirely contained within the bounding box. By removing these bounding boxes, partial detections can be removed from the image object detection output, thereby improving the quality of the image object detection output.

[0039] As another example, it is possible to identify and remove bounding boxes that span the corresponding tiles within which the bounding boxes are defined, such that the bounding boxes intersect the (first) boundary of the corresponding tile and a (second) opposing boundary parallel to the (first) boundary. This arrangement can indicate that detected objects extend beyond the tiles.

[0040] As another example, it is possible to identify and remove bounding boxes that intersect with both the boundary of the tile within which the bounding box is defined and the edge of the corresponding level of the image pyramid.

[0041] In some implementations, each bounding box that intersects with the boundary of a tile in which it is defined can be identified and removed. As a result, the image object detection output can have no bounding boxes that intersect with any boundary of the corresponding tile in which they are defined. As a result, bounding boxes corresponding to partial detections can be removed, such that only bounding boxes corresponding to complete detections are retained.

[0042] In some implementations, redundant or duplicate bounding boxes can be removed from the image object detection output. For example, using nonmaximum suppression or a variant thereof, bounding boxes with low detection scores or confidence can be removed.

[0043] In some implementations, preliminary image processing can be performed to generate one or more intermediate feature representations that can be used to generate an image pyramid. For example, a preliminary machine learning object detection model can be used to generate intermediate feature representations. The image pyramid can then be generated based on these intermediate feature representations. More specifically, the image can be tiled into multiple preliminary tiles. These preliminary tiles can be input into a preliminary machine learning object detection model, and multiple intermediate feature representations corresponding to the preliminary tiles can be received as the output of the preliminary machine learning object detection model. Levels of the image pyramid can be generated based on these multiple intermediate feature representations.

[0044] For example, a preliminary image patch can correspond to a preliminary image pyramid with multiple levels. One level of the preliminary image pyramid can be used to generate multiple levels of the image pyramid. More specifically, each level of the preliminary image pyramid can be scaled down to a different resolution corresponding to the multiple levels of the image pyramid. The remaining steps of the method can be performed on the image pyramid to detect objects depicted in the image as described herein. Generating intermediate feature representations as described herein can reduce the computational resources required to identify objects in an image.

[0045] This disclosure also relates to methods for training one or more machine learning models to detect the location of objects in an image. For example, the method can include performing the following steps for each of a plurality of training images: generating an image pyramid based on corresponding training images in the plurality of training images; tiling the levels of the image pyramid; inputting a plurality of tiles into a machine learning object detection model; receiving object detection data as the output of the machine learning object detection model; and generating an image object detection output by mapping the object detection data onto the image space of the image. The method for training the machine learning model can include adjusting the parameters of the machine learning object detection model based on a comparison between the image object detection output and ground-value object location data corresponding to the corresponding training images in the plurality of training images. The ground-value object location data can describe the location of objects in a set of ground-value images.

[0046] In some implementations, reweighting and / or resampling can be used to rebalance the distribution of object size and / or class in the training data. When training images are tiled, small objects may become more dominant in the resulting tiled training set. Additionally, smaller object classes can typically become dominant for the same reason. Therefore, reweighting and / or resampling of the training data can be used to correct for this dominance of small objects.

[0047] As an example, the systems and methods of this disclosure can be included or otherwise used in the context of an application, browser plugin, or other context. Therefore, in some embodiments, the models of this disclosure can be included in or otherwise stored and implemented by a user computing device such as a laptop computer, tablet computer, or smartphone. As yet another example, the models can be included in or otherwise stored and implemented by a server computing device that communicates with the user computing device according to a client-server relationship. For example, the models can be implemented by the server computing device as part of a web service (e.g., a web email service).

[0048] The systems and methods disclosed herein offer numerous technical effects and benefits. More specifically, they enable improved detection of small objects in large images. Furthermore, "pruning" bounding boxes improves the quality of object recognition data by removing bounding boxes corresponding to partial detections of objects spanning multiple tiles. The improved object recognition data is more accurate, requires less storage space, and / or requires fewer computational resources for subsequent processing. Moreover, as mentioned above, using preliminary models and preliminary tiles reduces the computational resources required for object recognition in images (e.g., storage space, processing power, number of triggers, size of random access memory, etc.).

[0049] Exemplary embodiments of this disclosure will now be discussed in more detail with reference to the accompanying drawings.

[0050] Example devices and systems

[0051] Figure 1A A block diagram of an example computing system 100 for detecting the location of objects in an image and training a model to perform the same operation, according to an example embodiment of the present disclosure, is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0052] User computing device 102 can be any type of computing device, such as personal computing devices (e.g., laptop computers or desktop computers), mobile computing devices (e.g., smartphones or tablet computers), game consoles or controllers, wearable computing devices, embedded computing devices, or any other type of computing device.

[0053] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 executed by processor 112 to cause user computing device 102 to perform operations.

[0054] User computing device 102 can store or include one or more machine learning object detection models 120. For example, the machine learning object detection model 120 can be or can include various machine learning models, such as neural networks (e.g., deep neural networks) or other multi-layer nonlinear models. Neural networks can include recurrent neural networks (e.g., long short-term memory recurrent neural networks), feedforward neural networks, or other forms of neural networks. Reference Figure 2A and Figure 2B Example machine learning object detection model 120 is discussed.

[0055] In some implementations, one or more machine learning object detection models 120 can be received from server computing system 130 via network 180, stored in user computing device memory 114, and used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 can implement multiple parallel instances of a single machine learning object detection model 120 (e.g., performing parallel object detection across multiple instances of model 120).

[0056] Additionally or alternatively, one or more machine learning object detection models 140 can be included in, stored in, or implemented by the server computing system 130, which communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning object detection model 140 can be implemented by the server computing system 130 as part of a web service (e.g., an image processing service). Thus, one or more object detection models 120 can be stored and implemented at the user computing device 102, and / or one or more object detection models 140 can be stored and implemented at the server computing system 130.

[0057] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other tools that enable the user to communicate through their input.

[0058] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138 that are executed by processor 132 to cause server computing system 130 to perform operations.

[0059] In some embodiments, the server computing system 130 includes one or more server computing devices or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices are capable of operating according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0060] As described above, the server computing system 130 can store or include one or more machine learning object detection models 140. For example, model 140 can be, or can include, various machine learning models, such as neural networks (e.g., deep recurrent neural networks) or other multi-layer nonlinear models. (Reference) Figure 2A and Figure 2B Example model 140 was discussed.

[0061] Server computing system 130 can train model 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 can be separate from server computing system 130, or it can be part of server computing system 130.

[0062] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes one or more server computing devices or is implemented by one or more server computing devices.

[0063] Training computing system 150 may include model trainer 160, which is capable of training machine learning model 140 stored at server computing system 130, for example, as shown in the following reference. Figure 8 The method described above (e.g., using various training or learning techniques, such as, for example, backpropagation of error). In some implementations, performing backpropagation of error can include performing truncated backpropagation over time. The model trainer 160 is capable of performing various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0064] Specifically, the model trainer 160 can train the object detection model 140 based on the set of training data 142. The training data 142 can include a set of training images and / or ground truth object location data describing the locations of objects in the set of ground truth images.

[0065] In some implementations, if the user has already provided consent, the training examples can be provided by the user computing device 102 (e.g., based on communications previously provided by the user of the user computing device 102). Therefore, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific communication data received from the user computing device 102. In some cases, this process can be referred to as a personalized model.

[0066] Model trainer 160 includes computer logic for providing desired functionality. Model trainer 160 can be implemented with hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium such as a RAM hard disk or an optical or magnetic medium.

[0067] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Typically, communication over network 180 can be carried over any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0068] Figure 1A The illustration shows an example computing system that can be used to implement this disclosure. Other computing systems can also be used. For example, in some embodiments, user computing device 102 can include a model trainer 160 and a training dataset 162. In such embodiments, model 120 can be trained and used locally at user computing device 102. In some such embodiments, user computing device 102 can implement model trainer 160 to personalize model 120 based on user-specific data.

[0069] Figure 1B A block diagram depicts an example computing device 10 implemented according to an exemplary embodiment of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0070] The computing device 10 includes multiple applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0071] like Figure 1B As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0072] Figure 1C A block diagram depicts an example computing device 50 implemented according to an example embodiment of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0073] Computing device 50 includes multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application is able to communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0074] The central intelligence layer comprises multiple machine learning models. For example, such as... Figure 1C As shown, a corresponding machine learning model (e.g., a model) can be provided for each application and managed by a central intelligent layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligent layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligent layer is included within or implemented by the operating system of computing device 50.

[0075] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data warehouse for computing device 50. For example... Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0076] Example model layout

[0077] Figure 2A A block diagram of an example machine learning object detection model 200 according to an exemplary embodiment of the present disclosure is depicted. In some embodiments, the machine learning object detection model 200 is trained to receive a plurality of tiles 202 and outputs object detection data 204 as a result of receiving the plurality of tiles 202. For example, see reference below. Figures 3 to 8 As described, multiple tiles 202 can correspond to corresponding levels in an image pyramid. For example, see the following reference... Figures 3 to 8 As described, object detection data 204 can include bounding boxes (or other data) describing the location of objects displayed in corresponding tiles 202 at multiple levels of the image pyramid.

[0078] Figure 2BBlock diagrams depict example preliminary machine learning object detection models 250 and 252 according to exemplary embodiments of the present disclosure. Preliminary object detection model 250 can be used for preliminary image processing to generate one or more intermediate feature representations 254 that can be used to generate image pyramid 256. More specifically, image 251 can be tiled into multiple preliminary tiles 253. Multiple preliminary tiles 253 can be input to preliminary machine learning object detection model 250. Intermediate feature representations 254 can be received as outputs of preliminary machine learning object detection model 250 and can correspond to the multiple preliminary tiles 253 respectively.

[0079] The levels of the image pyramid 256 can be generated based on multiple intermediate feature representations 254. For example, a preliminary patch 253 can correspond to a preliminary image pyramid with multiple levels. One level of the preliminary image pyramid can be used to generate multiple levels of the image pyramid 256. The corresponding levels of the preliminary image pyramid can be scaled down to different resolutions corresponding to the multiple levels of the image pyramid 256, respectively. In some embodiments, the preliminary patches 253 can be non-overlapping (e.g., within a given pyramid layer), and the patches of the image pyramid 256 can overlap (e.g., corresponding patches within a given image pyramid layer can overlap adjacent patches within the same image pyramid layer), making it possible to generate multiple patches of the image pyramid 256 from a single preliminary patch 253.

[0080] For example, as mentioned above... Figure 2A As described, the levels of the image pyramid 256 can be tiled and input into the object detection model 252. For example, as mentioned above... Figure 2B As described, it is capable of receiving object detection data 258 as the output of object detection model 252.

[0081] Example Method

[0082] Figure 3This is a simplified flowchart of a method 300 for detecting the location of objects in an image according to an example embodiment of the present disclosure. As described above, it is possible to detect the location of both very small and very large images in the same image. An image pyramid 304, also known as an "image tensor," can be generated based on image 302. The levels of the image pyramid can correspond to images 302 at different resolutions. The levels can be divided into tiles 306 (e.g., "tiled"), and object detection can be performed on each tile 306 at (308) using one or more machine learning models 310 to produce object detection data 312 for each tile 306. By performing object detection on each tile, small objects depicted in the image can be detected more effectively. The object detection data 312 can include bounding boxes (or other data) describing the location of objects displayed in the corresponding tiles 306 of the multiple levels of the image pyramid 304. At (314), the object detection data 312 can be mapped onto the image space of image 302 to generate an image object detection output 316.

[0083] The bounding boxes of object detection data 312 can identify the location of objects in image 302. When object detection is performed on a single tile 306 of a level of image pyramid 304, the location of small objects in image 302 can be detected better than when object detection is performed on the entire image 302.

[0084] Figure 4 This is a simplified diagram of a method 400 for detecting the location of an object in an image according to an example embodiment of the present disclosure. As indicated above, an image pyramid 402 can be generated based on image 404. Image pyramid 402 can include a first level 406 corresponding to image 404 at a first resolution and a second level 408 corresponding to image 404 at a second resolution different from the first resolution. Both the first level 406 and the second level 408 can be tiled by dividing the first level 406 into a first plurality of tiles and dividing the second level 408 into a second plurality of tiles. Image pyramid 402 can include one or more additional levels 410 corresponding to image 404 at an additional resolution. Levels 406, 408, and 410 can be divided or “tiled” into tiles 411. Tiles 411 are also represented as boxes 412 overlaid on image 404 and as boxes 414 defined in the respective levels 406, 408, and 410 of image pyramid 402. Patch 411 can be input into a machine learning object detection model, and, for example, as referenced Figure 2AThe described method is capable of receiving object detection data describing the location of detected objects within a corresponding tile as the output of a machine learning object detection model. The object detection data can include bounding boxes 416 identifying the location of detected objects 420 in tile 411. The object detection data including bounding boxes 416 can be mapped onto the image space of image 404 to generate an image object detection output 418 describing the location of detected objects 420 in image 404.

[0085] Figure 5A yes Figure 4 A simplified illustration of the first level 406 of the image pyramid 402, and the first multiple tiles 500. Figure 5B yes Figure 4 A simplified illustration of the second multiple tiles 502 of the second level 408 of the image pyramid 402. Figure 5C The first plurality of tiles 500 are shown, overlaid on the second plurality of tiles 502. (See example...) Figure 5C As shown in the optimal diagram, the first plurality of patches 500 can overlap with respect to the second plurality of patches 502, such that image 404 ( Figure 4 Each object depicted in the image can be completely located within at least one of tiles 500, 502. In one example, each of the first plurality of tiles 500 can overlap with at least two of the second plurality of tiles 502. For example, each of the first plurality of tiles 500 can overlap with at least 50% of the adjacent tiles of the second plurality of tiles 502. In some embodiments, each of the first plurality of tiles 500 can overlap with 50% of the adjacent tiles of the second plurality of tiles 502. Additionally, in some embodiments, a corresponding tile within a given image pyramid layer can overlap with an adjacent tile within the given image pyramid layer. For example, a corresponding tile in the first plurality of tiles 500 can overlap with an adjacent tile within the first plurality of tiles 500. As another example, a corresponding tile in the second plurality of tiles 502 can overlap with an adjacent tile within the second plurality of tiles 502.

[0086] Figure 6 An image pyramid 402 comprising a plurality of bounding boxes 416 is depicted according to an exemplary embodiment of the present disclosure. Figure 4 The simplified level 600 of the image object detection output is described. In some implementations, the image object detection output can be "trimmed" to improve its quality. Bounding boxes in the image object detection output can be removed or "trimmed" from the object detection data to improve its quality. More specifically, bounding boxes 602 intersecting one or more boundaries 604 of patch 606 can be identified and removed. Removing such bounding boxes 602 by removing at least some of the bounding boxes 602 corresponding to partial and / or repeated detections improves the quality of the image object detection output.

[0087] As an example, bounding box 602 can be removed if it intersects with one or more boundaries 604 of the tile 606 in which bounding box 602 is defined. Intersection with boundary 604 indicates that bounding box 602 corresponds to “partial” detection, in which detected objects span image pyramid 402 (…). Figure 4 Multiple tiles 606 at the corresponding level of the bounding box 602. In other words, such intersection can indicate that the detected object is not completely contained within the bounding box 602.

[0088] As another example, bounding box 615 can be identified and removed, intersecting both the boundary 616 of tile 610 and the edge 618 of the corresponding level 600 of the image pyramid, wherein a corresponding bounding box 614 is defined within the boundary 616 of tile 610. This indicates that the detected object extends beyond the corresponding tile 610.

[0089] In some implementations, all bounding boxes 602 intersecting with the boundary 604 of tile 606 can be identified and removed. As a result, the image object detection output can be free of bounding boxes 602 that intersect with any boundary of the corresponding tile 606 within which they are defined. Consequently, all bounding boxes 602 corresponding to partial detections can be removed, such that only the bounding boxes 620 corresponding to complete detections remain in the image object detection output.

[0090] In some implementations, redundant or duplicate bounding boxes can be removed from the image object detection output. Bounding boxes with low detection scores or confidence can be removed, for example, using non-maximum suppression (NMS) or its variants. For example, NMS and / or Soft-NMS, variants of NMS, can be applied using algorithms such as:

[0091] Input: B = {b1,…b2} N}, S={S1,…S N},N t

[0092] B is the list of initial detection boxes.

[0093] S contains the corresponding detection score.

[0094] N t It is the NMS threshold.

[0095]

[0096] Figure 7 A flowchart depicts an example method 700 for detecting the location of an object in an image according to aspects of this disclosure. Although... Figure 7The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specific order or arrangement shown. The various steps of method 700 can be omitted, rearranged, combined, and / or adjusted in various ways without departing from the scope of this disclosure.

[0097] At point 702, the computational system is able to generate an image pyramid based on an image with image space. For example, as referenced above. Figure 3 and Figure 4 As described, the image pyramid can include a first level corresponding to an image of a first resolution and a second level corresponding to an image of a second resolution different from the first resolution.

[0098] At position 704, for example, as referenced above. Figure 3 and Figure 4 As described, the computational system can tile the first and second levels of an image pyramid by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles. As an example, the first level can be uniformly divided into the first plurality of tiles with respect to its height and / or width. As a result, the plurality of tiles can form a grid such that some or all of the first plurality of tiles have the same size and shape (e.g., rectangles, squares, parallelograms, etc.). The second plurality of tiles can have the same size, shape, arrangement, etc., as the first plurality of tiles. However, in some embodiments, the size, shape, arrangement, etc., of the tiles can vary between levels. For example, as the resolution of the level increases, the size of the tiles can decrease or increase. In other embodiments, the first plurality of tiles and / or the second plurality of tiles can be irregular shapes and / or form other shapes, such as triangles.

[0099] At position 706, for example, as referenced above. Figure 2A As described, the computing system is able to input first and second patches into a machine learning object detection model.

[0100] At point 708, the computing system is able to receive object detection data as the output of a machine learning object detection model. For example, as referenced above. Figure 2A As described, object detection data can include multiple bounding boxes defined with respect to individual tiles in a first plurality of tiles and a second plurality of tiles, respectively.

[0101] At 708, for example, as referenced above. Figure 3 and Figure 4As described, the computational system is capable of generating image object detection output by mapping object detection data onto the image space of an image. As discussed above, in some embodiments, some bounding boxes in the image object detection output corresponding to partial object detection can be removed. Bounding boxes intersecting the boundaries of tiles with corresponding bounding boxes within them can be identified and removed from the image object detection output.

[0102] Figure 8 A flowchart depicts an example method 800 for training a machine learning object detection model according to aspects of this disclosure. Although... Figure 8 The steps performed in a specific order are depicted for illustrative and discussion purposes, but the method of this disclosure is not limited to the specific order or arrangement shown. The various steps of method 800 can be omitted, rearranged, combined, and / or modified in various ways without departing from the scope of this disclosure.

[0103] Method 800 can include performing each set of steps described below for each of a plurality of training images. More specifically, method 800 can include generating an image pyramid at (802) based on the corresponding training images having corresponding image spaces. The image pyramid can include a first level corresponding to the corresponding training images at a first resolution and a second level corresponding to the corresponding training images at a second resolution different from the first resolution. Method 800 can include tiling the first and second levels at (804) by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles. Method 800 can include inputting the first plurality of tiles and the second plurality of tiles into a machine learning object detection model at (806), and receiving object detection data as the output of the machine learning object detection model at (808). The object detection data can include multiple bounding boxes defined with respect to individual tiles in the first plurality of tiles and the second plurality of tiles, respectively. Method 800 can include, for example, as referenced above at (810). Figures 2A to 4 As described, image object detection output is generated by mapping object detection data to the corresponding image space of the corresponding training images.

[0104] Method 800 can also include, at (812), adjusting the parameters of a machine learning object detection model based on a comparison between the image object detection output and ground truth object location data corresponding to the corresponding training images in a plurality of training images. For example, a loss function describing the comparison can be evaluated. The gradient of the loss function can be calculated, and, for example, the parameters of the machine learning model can be adjusted via backpropagation of the errors of the layers of the machine learning object detection model.

[0105] As described above, in some embodiments, for example, as referenced above... Figure 2BThe described computational system can include a preliminary machine learning model. In such an implementation, the parameters of the preliminary machine learning model can be adjusted based on comparisons and / or a loss function. Errors can be backpropagated through each of the machine learning model and the preliminary machine learning model. The parameters of one or both models can be adjusted. Therefore, models can be trained in an end-to-end manner.

[0106] Additionally, in some embodiments, method 800 may also include, for example, as referenced above. Figures 5A to 6 As described, the bounding boxes corresponding to partial object detections in the image object detection output are removed.

[0107] It should be understood that various aspects of this disclosure can be implemented during training, inference, or both. For example, tiling can be used to train the model as described above and / or, for example, in reference to... Figure 7 During the inference period, tiling can be used either during training or during inference alone. Similarly, redundant bounding boxes can be removed and / or portions of detection boxes can be removed or "trimmed" during training, inference, or both.

[0108] Additional Disclosure

[0109] This paper discusses technical reference servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functions between and within components. For example, the processes discussed in this paper can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0110] refer to Figure 9 Identify objects in an image normalized to 1024×1024. Target recall rendering accuracy.

[0111] Figures 10 to 12 The illustration shows the data collected experimentally from a first dataset containing images of urban landscapes. (Reference) Figure 10 The training images were tiled to 256×256 pixels, and the input images were set to 529×529 pixels. Without tiling, the model was evaluated on a test image of size 1024×2048. (Reference) Figure 11 The training images were tiled to 256×256 pixels, and the input images were set to 529×529 pixels. The model was evaluated on test images with a tiled size of 1024×2048 pixels. Non-maximum suppression was used to remove redundant object recognition data. (Reference) Figure 12As described in this paper, nonmaximum suppression is used to remove redundant bounding boxes, and some detected bounding boxes are removed or "trimmed". More specifically, the training images are tiled to 256×256, and the input images are set to 529×529. The model is evaluated on test images with a tiled size of 1024×2048.

[0112] Figures 13 to 15 The illustration shows the data collected for the experiment on the second dataset. (Reference) Figure 13 An AutoML backend model was used. The input data was set to 1024×1024. (Reference) Figure 14 An AutoML backend model was used. Training images were tiled to 640×640, and input images were set to 1024×1024. Non-maximum suppression was used to remove redundant object recognition data. (Reference) Figure 15 The model uses an AutoML backend. Training images are tiled to 640×640, and input images are set to 1024×1024. Non-maximum suppression is used to remove redundant object recognition data. (See also: [link to related documentation]). Figures 5A to 6 As described, some detection bounding boxes were also removed or "trimmed".

[0113] While the subject matter has been described in detail with respect to various specific example embodiments, each example is provided by way of explanation rather than limitation. Changes, variations, and equivalents of these embodiments will be readily apparent to those skilled in the art upon gaining an understanding of the foregoing. Therefore, this disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, which will be readily understood by those skilled in the art. For example, features shown or described as part of an embodiment can be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such changes, variations, and equivalents.

Claims

1. A computing system, comprising: At least one processor; A preliminary machine learning object detection model is configured to receive images and, in response to the received images, output intermediate feature representations. A machine learning object detection model is configured to receive multiple tiles and, in response to receiving multiple tiles, output object detection data for multiple tiles, the object detection data including multiple bounding boxes defined for each individual tile in the multiple tiles; as well as At least one tangible, non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause at least one processor to perform operations, said operations including: An image pyramid is generated based on an image with image space. The image pyramid includes a first level corresponding to an image with a first resolution and a second level corresponding to an image with a second resolution different from the first resolution. The image pyramid generation based on the image includes: The image is input into a preliminary machine learning object detection model; Receive intermediate feature representations as the output of a preliminary machine learning object detection model; and The first and second levels of the image pyramid are generated based on intermediate feature representations; The first and second levels are tiled by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; The first and second sets of maps are input into the machine learning object detection model; Receives object detection data as output from a machine learning object detection model, the object detection data including multiple bounding boxes defined respectively with respect to individual tiles in a first plurality of tiles and a second plurality of tiles; and Image object detection output is generated by mapping object detection data onto the image space of the image.

2. The computing system according to any one of the preceding claims, wherein, The operation also includes: Based on the image object detection output, at least one bounding box intersects with the boundaries of one or more of the first or second plurality of map tiles, and the at least one bounding box is identified; and Remove at least one bounding box from the image object detection output.

3. The computing system according to claim 2, wherein, The at least one bounding box is identified based on the fact that it spans one or more of the first plurality of tiles or the second plurality of tiles, such that the at least one bounding box intersects with the boundaries of one or more of the first plurality of tiles or the second plurality of tiles and an opposite boundary parallel to the boundary.

4. The computing system according to claim 2, wherein, The at least one bounding box is identified based on the fact that at least one bounding box intersects the boundaries of one or more of the first plurality of tiles or the second plurality of tiles and intersects the edges of the corresponding level of the image pyramid.

5. The computing system according to claim 2, wherein, Removing at least one bounding box from the image object detection output includes removing each bounding box that intersects with any of the boundaries of a first plurality of tiles or a second plurality of tiles.

6. A method for training a machine learning object detection model, the method comprising: For each of the multiple training images: An image pyramid is generated by one or more computing devices based on corresponding training images having a corresponding image space. The image pyramid includes a first level corresponding to a corresponding training image at a first resolution and a second level corresponding to a corresponding training image at a second resolution different from the first resolution. Generating the image pyramid based on the training images includes: The training images are input into the initial machine learning object detection model; Receive intermediate feature representations as the output of a preliminary machine learning object detection model; and The first and second levels of the image pyramid are generated based on intermediate feature representations; One or more computing devices tile the first level and the second level by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; One or more computing devices input a first set of tiles and a second set of tiles into a machine learning object detection model; The object detection data, which is the output of a machine learning object detection model, is received by one or more computing devices. The object detection data includes multiple bounding boxes defined with respect to individual tiles in a first plurality of tiles and a second plurality of tiles, respectively. The one or more computing devices generate image object detection output by mapping object detection data to the corresponding image space of the corresponding training images; and The parameters of a machine learning object detection model are adjusted by one or more computing devices based on the image object detection output and the comparison of ground truth object location data with the corresponding training images in a plurality of training images.

7. The method according to claim 6, further comprising: The at least one bounding box, output by one or more computing devices based on image object detection, intersects with the boundaries of one or more of a first plurality of tiles or a second plurality of tiles, thereby identifying the at least one bounding box; as well as The at least one bounding box is removed from the image object detection output by one or more computing devices.

8. The method according to claim 7, wherein, The at least one bounding box is identified by one or more computing devices based on at least one bounding box spanning one or more of the first plurality of tiles or the second plurality of tiles, such that the at least one bounding box intersects with the boundaries of one or more of the first plurality of tiles or the second plurality of tiles and an opposite boundary parallel to said boundary.

9. The method according to claim 7, wherein, The at least one bounding box is identified by one or more computing devices based on the fact that at least one bounding box intersects the boundaries of one or more of the first plurality of tiles or the second plurality of tiles and intersects the edges of the corresponding level of the image pyramid.

10. The method according to claim 7, wherein, Removing at least one bounding box from an image object detection output by one or more computing devices includes: removing each bounding box that intersects with any boundary of a plurality of boundaries of a first plurality of tiles or a second plurality of tiles by one or more computing devices.

11. The method according to any one of claims 6 to 10, wherein: Inputting the training images into the initial machine learning object detection model includes: The corresponding training images are tiled into multiple preliminary patches; Multiple preliminary map patches are input into a preliminary machine learning object detection model, which is configured to receive multiple preliminary map patches and, in response to receiving multiple preliminary map patches, output multiple intermediate feature representations corresponding to the multiple preliminary map patches respectively. Generating an image pyramid includes: Receives multiple intermediate feature representations, each corresponding to a plurality of preliminary map tiles, as output from a preliminary machine learning object detection model; and The method generates a first and second level of an image pyramid based on multiple intermediate feature representations; and the method includes adjusting the parameters of a preliminary machine learning object detection model by one or more computing devices based on a comparison of the corresponding image object detection output and ground truth object location data corresponding to the corresponding training images in a plurality of training images.

12. The method according to claim 11, wherein, Multiple preliminary tiles correspond to a first preliminary level of a first resolution, and wherein generating an image pyramid based on the corresponding training images includes scaling down the first preliminary level to generate a second preliminary level of a second resolution before tiling the first and second levels.

13. A method for detecting the location of an object in an image, the method comprising: An image pyramid is generated by one or more computing devices based on an image having an image space. The image pyramid includes a first level corresponding to an image with a first resolution and a second level corresponding to an image with a second resolution different from the first resolution. Generating the image pyramid based on the image includes: The image is input into a preliminary machine learning object detection model; Receive intermediate feature representations as the output of a preliminary machine learning object detection model; and The first and second levels of the image pyramid are generated based on intermediate feature representations; One or more computing devices tile the first level and the second level by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; One or more computing devices input a first set of tiles and a second set of tiles into a machine learning object detection model; Object detection data, received by one or more computing devices as the output of a machine learning object detection model, includes multiple bounding boxes defined respectively with respect to individual tiles in a first plurality of tiles and a second plurality of tiles; and Image object detection output is generated by one or more computing devices by mapping object detection data onto the image space of the image.

14. A computing system, comprising: At least one processor; A machine learning object detection model is configured to receive multiple tiles and, in response to receiving multiple tiles, output object detection data for multiple tiles, the object detection data including multiple bounding boxes defined for each individual tile in the multiple tiles; as well as At least one tangible, non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause at least one processor to perform operations, said operations including: An image pyramid is generated based on an image with image space, the image pyramid including a first level corresponding to an image with a first resolution and a second level corresponding to an image with a second resolution different from the first resolution; The first and second levels are tiled by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; The first and second sets of maps are input into the machine learning object detection model; Receive object detection data as the output of a machine learning object detection model, the object detection data including multiple bounding boxes defined with respect to individual tiles in a first plurality of tiles and a second plurality of tiles respectively; Image object detection output is generated by mapping object detection data onto the image space of the image; Based on the image object detection output, at least one bounding box intersects with the boundaries of one or more of the first or second plurality of map tiles, and the at least one bounding box is identified; and Remove the at least one bounding box from the image object detection output; The at least one bounding box is identified based on the fact that at least one bounding box intersects the boundaries of one or more of the first plurality of tiles or the second plurality of tiles and intersects the edges of the corresponding level of the image pyramid.

15. A computing system, comprising: At least one processor; A preliminary machine learning object detection model is configured to receive multiple preliminary map patches and, in response to receiving multiple preliminary map patches, output multiple intermediate feature representations corresponding to the multiple preliminary map patches respectively; A machine learning object detection model is configured to receive multiple tiles and, in response to receiving multiple tiles, output object detection data for multiple tiles, the object detection data including multiple bounding boxes defined for each individual tile in the multiple tiles; as well as At least one tangible, non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause at least one processor to perform operations, said operations including: An image pyramid is generated based on an image with image space. The image pyramid includes a first level corresponding to an image with a first resolution and a second level corresponding to an image with a second resolution different from the first resolution. The image pyramid generation based on the image includes: Tile the image into multiple preliminary tiles; Multiple preliminary map tiles are input into a preliminary machine learning object detection model; Receives multiple intermediate feature representations, each corresponding to a plurality of preliminary map tiles, as output from a preliminary machine learning object detection model; and The first and second levels of the image pyramid are generated based on multiple intermediate feature representations; The first and second levels are tiled by dividing the first level into a first plurality of tiles and the second level into a second plurality of tiles; The first and second sets of maps are input into the machine learning object detection model; Receives object detection data as output from a machine learning object detection model, the object detection data including multiple bounding boxes defined respectively with respect to individual tiles in a first plurality of tiles and a second plurality of tiles; and Image object detection output is generated by mapping object detection data onto the image space of the image.

16. The computing system according to claim 15, wherein, Multiple preliminary tiles correspond to a first preliminary level of a first resolution, and wherein generating an image pyramid based on an image includes scaling down the first preliminary level to generate a second preliminary level of a second resolution before tiling the first and second levels.

Images