Method and system for generating a detector for process monitoring

Abstract

A method and system for generating a detector for process monitoring. A method of generating a detector is disclosed herein. The method includes: obtaining a first training data set including a first tagged image set identifying a first object; and obtaining a second training dataset comprising a second set of tagged images identifying a second object. A first part-level detector is trained based on the first training data set, and a second part-level detector is trained based on the second training data set. A unified detector is trained based on the first training data set and the second training data set.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for generating detector models, and more particularly, to a detector model for process monitoring. Background Art

[0002] Machine learning is the process of analyzing to determine a model that maps input data to output data. One type of machine learning is supervised learning, in which a model is trained on a dataset of sufficient input data to include known output data. Once a model is trained, it can be deployed, i.e., applied to new input data to predict expected outputs.

[0003] Machine learning can be applied to regression problems (where the output data is a numerical value, such as voltage, pressure, number of cycles) and classification problems (where the output data is a label, class, and / or category, such as pass-fail, failure type, etc.). For both types of problems, a wide range of machine learning algorithms are available, and new algorithms are actively being researched. However, generating datasets to train models can be very labor intensive. Additionally, the quality of the datasets provided for training can greatly impact the model's ability to predict expected outputs from new datasets.

[0004] Therefore, there is a need to improve the input dataset used to train the model and the expected output predicted by the model. Summary of the invention

[0005] A method for generating a detector is disclosed herein. The method includes: obtaining a first training data set including a first set of labeled images identifying a first object; and obtaining a second training data set including a second set of labeled images identifying a second object. A first part-level detector is trained based on the first training data set, and a second part-level detector is trained based on the second training data set. A unified detector is trained based on the first training data set and the second training data set.

[0006] In one or more embodiments of the method, a first training data set is created by the following steps: receiving a first image sequence, identifying a first object in at least one image of the first image sequence; tracking the first object identified in at least one image in the first image sequence; marking a region of interest in each image in which the first object is tracked in the first image sequence; and creating the first training data set by collecting the region of interest from each image in which the first object is tracked in the first image sequence.

[0007] In one or more embodiments, the method includes obtaining the first training data set includes eliminating false negative markings of the first object by verifying the presence of the first object in the first training data set against a ground truth timeline of the first object.

[0008] In one or more embodiments, the method includes: creating a second training data set by receiving a second image sequence, in which a second object is identified in at least one image of the second image sequence. The method includes: tracking the second object identified in at least one image in the second image sequence. The method also includes: marking a region of interest in each image in which the second object is tracked in the second image sequence; and creating the second training data set by collecting the region of interest from each image in which the second object is tracked in the second image sequence.

[0009] In one or more embodiments of the method, the second object includes the first object in a different configuration.

[0010] In one or more embodiments, the method includes obtaining the second training data set includes eliminating false negative markings of the second object by verifying the presence of the second object against a ground truth timeline of the second object.

[0011] In one or more embodiments, the method includes: training the unified detector based on the first training dataset includes using a second part-level detector to mark a region of interest for tracking a second object in each image of the first training dataset.

[0012] In one or more embodiments, the method includes: training the unified detector based on the second training dataset includes using the first part-level detector to mark a region of interest for tracking the first object in each image of the second training dataset.

[0013] In one or more embodiments, the method includes training the unified detector includes eliminating false negative markings of first objects in the second training dataset by verifying the presence of the first objects against a ground truth timeline of the first objects in the second training dataset.

[0014] In one or more embodiments, the method includes training the unified detector includes eliminating false negative markings of a second object in the first training dataset by verifying the presence of the second object against a ground truth timeline of the second object in the first training dataset.

[0015] In one or more embodiments, the method includes: obtaining multiple additional training data sets, each of the multiple additional training data sets including a set of labeled images identifying a corresponding one of multiple additional objects; training multiple additional part-level detectors based on the multiple additional training data sets; and training a unified detector based on the first training data set, the second training data set, and the multiple additional training data sets.

[0016] In one or more embodiments, the method includes: obtaining a plurality of additional training data sets, each of the plurality of additional training data sets including a set of labeled images identifying a corresponding one of the plurality of additional objects. The method includes: training a single additional part-level detector based on the plurality of additional training data sets when the ground truth timelines of the plurality of additional objects do not overlap. The method also includes: training a unified detector based on the first training data set, the second training data set, and the plurality of additional training data sets.

[0017] In one or more embodiments of the method, the plurality of additional objects includes a single object in a plurality of different configurations.

[0018] In one or more embodiments, the method includes: training a single additional part-level detector based on the plurality of additional training data sets includes eliminating false negative markings of the plurality of additional objects by verifying the presence of each of the plurality of additional objects in the plurality of additional training data sets against a ground truth timeline for each of the plurality of additional objects.

[0019] In one or more embodiments of the method, the first and second sets of labeled images include labels that provide object identifiers, locations of bounding boxes highlighting the corresponding objects, and sizes of the bounding boxes.

[0020] A system for detecting an object is disclosed herein. The system includes at least one camera configured to capture a plurality of images and a controller. The controller is configured to obtain a first training data set including a first set of labeled images identifying a first object, and obtain a second training data set including a second set of labeled images identifying a second object. The controller is further configured to train a first part-level detector based on the first training data set and train a second part-level detector based on the second training data set, so as to train a unified detector based on the first training data set and the second training data set.

[0021] In one or more embodiments of the system, a first training data set is created by: receiving a first image sequence, identifying a first object in at least one image of the first image sequence; and tracking the first object identified in at least one image in the first image sequence. The first training data set is further created by: marking a region of interest in each image in which the first object is tracked in the first image sequence; and creating the first training data set by collecting the region of interest from each image in which the first object is tracked in the first image sequence.

[0022] In one or more embodiments of the system, the second training data set is created by: receiving a second image sequence, identifying a second object in at least one image of the second image sequence; and tracking the second object identified in at least one image in the second image sequence. The second training data set is further created by: marking a region of interest in each image in the second image sequence in which the second object is tracked; and creating the second training data set by collecting the region of interest from each image in the second image sequence in which the second object is tracked.

[0023] A non-transitory computer-readable medium containing programmed instructions is disclosed herein, which when executed by a processor is operable to perform a method. The method includes: obtaining a first training data set including a first set of labeled images identifying a first object; and obtaining a second training data set including a second set of labeled images identifying a second object. The method includes: training a first part-level detector based on the first training data set; and training a second part-level detector based on the second training data set. The method also includes: training a unified detector based on the first training data set and the second training data set.

[0024] In one or more embodiments of the method, the first training data set and the second training data set are created by the following steps: receiving a first image sequence, identifying a first object in at least one image of the first image sequence; and receiving a second image sequence, identifying a second object in at least one image of the second image sequence. The method includes: tracking the first object identified in at least one image in the first image sequence; and tracking the second object identified in at least one image in the second image sequence. The method also includes: marking a region of interest in each image in which the first object is tracked in the first image sequence; and marking a region of interest in each image in which the second object is tracked in the second image sequence. The method also includes: creating the first training data set by collecting the region of interest from each image in which the first object is tracked in the first image sequence; and creating the second training data set by collecting the region of interest from each image in which the second object is tracked in the second image sequence.

[0025] The above summary is not intended to represent every possible embodiment or every aspect of the present disclosure. Instead, the above summary is intended to illustrate some of the novel aspects and features disclosed herein. The features, functions and advantages of the present disclosure can be independently implemented in various embodiments, or can be combined in other embodiments, further details of which can be seen with reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Some embodiments of the present disclosure will now be described by way of example only with reference to the accompanying drawings. The same reference numbers represent the same element or the same type of element on all drawings.

[0027] Figure 1 An example camera system and computer system for use in conjunction with an assembly process are shown.

[0028] Figure 2 An example method for generating a training dataset of objects for a part-level detector is shown.

[0029] Figure 3 An example method for training a unified detector is shown. DETAILED DESCRIPTION

[0030] The accompanying drawings and the following description illustrate specific exemplary embodiments of the present disclosure. Those of ordinary skill in the art will be able to design various arrangements that, although not explicitly described or shown herein, specifically implement the principles of the present disclosure and are included within the scope of the present disclosure. In addition, any examples described herein are intended to help understand the scope of the present disclosure and are to be interpreted as not being limited to these specifically described examples and conditions. As a result, the present disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0031] Figure 1 An example camera system 20 is shown having a first camera 20-1 and a second camera 20-2 for monitoring an assembly process (e.g., assembly of an aircraft 22 on an assembly line). The camera system 20 communicates with a computer system 30 that receives images from the camera system 20. Although the illustrated examples of the camera system 20 and the computer system 30 are shown with respect to an assembly process, the present disclosure is applicable to other fields where it is desirable to perform object detection.

[0032] Although for simplicity of illustration, Figure 1 The computer system 30 is depicted as a single computer module, but the computer system 30 may be physically implemented as one or more processing nodes, which have a non-transitory computer-readable storage medium 34 (i.e., sufficient memory for application) and associated hardware and software (such as but not limited to high-speed clocks, timers, input / output circuits, buffer circuits, etc.). The computer-readable storage medium 34 may include sufficient read-only memory, such as magnetic or optical memory. Computer-readable code or instructions that specifically implement the methods described below may be executed during the operation of the computer system 30. To this end, the computer system 30 may include one or more processors 32, such as logic circuits, application-specific integrated circuits (ASICs), central processing units, microprocessors, and / or other necessary hardware required to provide the programming functions described herein. The display screen 36 may be connected or communicated with the computer-readable medium 34 and the processor 32 to facilitate the implementation of the graphical user interface of the present disclosure described below.

[0033] In the example shown, cameras 20-1 and 20-2 are mounted above an aircraft assembly line 24 on opposite lateral sides of the aircraft 22 for monitoring the process of assembling the aircraft 22. Depending on the process being monitored, the cameras 20-1 and 20-2 may capture images at 4K resolution and up to 30 or more frames per second. Alternatively, the cameras 20-1, 20-2 may capture images at a lower resolution (e.g., 960×540) and at a rate of one image every 15 seconds. Variations in camera resolution and image capture rate depend on the size of the object being detected and the length of time the object remains in the field of view of the cameras 20-1, 20-2. In addition, collecting images at a lower resolution and rate reduces the amount of data that the computer system 30 needs to store and process.

[0034] Additionally, the present disclosure is applicable to camera systems 20 that use only a single camera or more than two cameras to capture images of a particular area. The number of cameras used for process monitoring depends on the field of view required to include the objects to be inspected, as some cameras may have obstructed view of one or more objects.

[0035] Figure 2 An example method 100 is shown for generating a training data set for a specific object for training a detection model of a detector (e.g., a part-level detector). The method 100 is implemented for each desired object to be tracked from an image sequence that includes at least one image having the desired object. The method 100 begins at box 102, and from box 102, the method 100 receives a sequence of images (box 104) at box 106. At box 106, the desired object is highlighted in at least one image from the sequence in box 104. In one example, an image from the sequence at or near the first appearance of the object is used to highlight the object. The object can be highlighted by a user of the computer system 30. In one example, the user highlights the object in the image 38 using a bounding box 40 surrounding the object (see Figure 1 ). A user may create a bounding box by placing a rectangle around the desired object in image 38. The location in the image and the size of the rectangle are recorded or labeled with the image and used at block 108 to track the object in other images in the sequence of images.

[0036] When tracking an object between the image sequences at block 108, the tracking algorithm uses the location of the bounding box in the particular image from block 106 and the size of the rectangle identified by the user to "track" or follow the object of interest throughout the image sequence. For each image in the image sequence, the tracker will attempt to locate the same object of interest and record information about the object of interest in association with each image in which the object was identified in the memory 34 of the computer system 30. The information tagged to each frame may include the location and size of the rectangle, a label identifying the object (e.g., the name or class of the object of interest), or a cropped image region of interest (ROI) defining the object by the rectangle.

[0037] If the tracking algorithm determines that the object of interest is no longer present in the image or is "lost" (e.g., which may be indicated by a confidence score below a set confidence threshold), the object tracking process may be interrupted at block 110. Additionally, the user may intervene due to errors in the tracking results as determined by the user (e.g., when the tracker has locked onto an object other than the one initially selected by the user).

[0038] At block 112, method 100 may return to block 106 to modify the bounding box identifying the object in the initial image to again improve tracking of the object of interest. Alternatively, when returning to block 106, the user may select an image in the sequence where the tracking algorithm lost the object, and highlight the correct object of interest in that image to allow the object to be tracked again at block 108. The ability to restart the tracking algorithm where it failed and continue without having to re-evaluate the same image from the image sequence again reduces the time required to evaluate the image sequence.

[0039] At block 114, the position and size of the rectangle, the label identifying the object, and the cropped image region for each image in the sequence of images are collected. A user may review the region of interest from each image in the sequence at block 116 to confirm that the correct object of interest is identified in the region of interest as part of finalizing the collected images and associated information. If any region of interest does not identify the correct object of interest, those regions of interest may be deleted from the finalized images.

[0040] In one example, the user finalizes the images by viewing them on the display 36 of the computer system 30. Once the user confirms that the correct region of interest is identified in the image, the finalized images are collected at block 118 and the method may end at block 120. The finalized images may be stored in the memory 34 of the computer system 30 or in another location such as in the cloud. The finalized images form a training data set for the part-level or first-level detector for objects identified in the image sequence as described below.

[0041] Method 100 may be repeated for the same or a different image sequence showing another object of interest or object configuration to form a training data set for that particular object or configuration. Since user involvement is limited to initiating tracking of an object of interest, method 100 no longer requires the user to utilize image tracking throughout the image sequence to identify the object of interest from each image in the image sequence. This reduces the time and effort required to create or update a training data set for an object.

[0042] The tracking of the object of interest performed at least in blocks 106 and 108 can be accomplished by many different available tracking algorithms, such as simple trackers based on cross-correlation coefficients. These trackers can be used in situations where the appearance and position of the object of interest remain relatively stable. However, other tracker algorithms may be used depending on the object being tracked.

[0043] One feature of the present disclosure is that the tracking process of objects of interest is simplified by treating a single object as a separate object to be tracked as the same object moves between different configurations or classes. Thus, if the objects of interest are in different "configurations," they are defined as different "objects" for the purpose of object tracking in method 100. For the example of objects associated with assembling an aircraft 22, aircraft wings 26 appear in several different configurations, such as when they are first towed to the factory floor ("delivered"), when they are lifted by a crane and moved to a support frame on the side of the fuselage (referred to as "loaded"), and when they are mated to the fuselage ("attached"). Thus, a single object, i.e., wing 26, will create three different configurations or classes, and the corresponding image sequences will be evaluated by method 100 to create different training data sets for each configuration.

[0044] In one example embodiment, at least one of the cameras 20-1, 20-2 captures an image of the assembly shop every 15 seconds. This reduces the burden of storing and processing large amounts of image data when the object of interest is not moving quickly or frequently. In prior art methods, it is difficult to maintain high accuracy in tracking the movement of an object in discontinuous image frames. However, the present disclosure takes advantage of the final state in which the objects of interest (in this example, the wings) are located, rather than how they are moved from one place to another during the assembly process. Therefore, an object of interest being in a "configuration" refers to an object that is in a specific assembly state, such as "left wing delivered", "left wing loaded", etc.

[0045] Furthermore, an object of interest may be in a different configuration even if its position has not changed. An example of such an object is a cradle 28 supporting the front end of the fuselage of an aircraft 22. During the assembly process, the cradle 28 will transition from a configuration supporting the weight of the fuselage ("cradle support plane") to a configuration not supporting the fuselage ("cradle non-support plane") so that the fuselage is supported by the front landing gear. This is called "Downjack" and is an important milestone in the assembly process of most aircraft 22, which can be identified by the detector model disclosed herein.

[0046] In order to train the object detection model, a positive training data set is created based on the results of method 100. The positive training data set includes: images containing positive training samples of an object, the object is marked with a label identifying the object at the location of the object; and the size of each object of interest to be detected in each image. In addition, image data sets without positive training samples are useful for training object detection models.

[0047] The positive training data set may be generated from the method 100 discussed above, and the positive training samples cannot use images of the same assembly plant floor before the object of interest is loaded into the workspace and before any parts to be inspected are present. This ensures that the samples will not include false negatives.

[0048] With respect to labeled images, the label tracking process of method 100 generates training data sets, which are generated one object at a time for each (time-stamped) image in the image sequence that forms the training data set. When more than one object of interest is identified using the same image sequence, the labels of different objects in the same image are merged into a single file for the purpose of training the detection model. In addition, if the labeling process is not exhaustive (i.e., if the process fails to label some objects in some images, or some labeling results are accidentally deleted), the training data set may inadvertently create negative training examples from positive images (i.e., false negative identifications). That is, if the label tracking process of method 100 fails to label the object of interest in some images, a situation is created where the object exists but is not labeled, in fact causing negative samples to be used for training (i.e., telling the training algorithm of the detection model that the object does not exist, when in fact the object does). This can cause damage to the training and reduce the performance of the resulting detection model.

[0049] In order to avoid inadvertently introducing negative training examples, an automated verification process is created to eliminate these negative examples. As described below, this process requires creating a ground truth timeline file by examining each image sequence and identifying and recording the time when each object first appears and disappears, or generally using the term "generalized object detection", when a configuration becomes true for the relevant object and when the configuration is no longer true. This ground truth timeline is stored in a "ground truth" file. In the example shown, each image sequence in the training data set will have a ground truth file. The ground truth timeline provides a verification process to eliminate false negatives from the training data set, and negative training data can be easily created using images with timestamps including time intervals without objects of interest.

[0050] For traditional object detection training methods, once the labeling process and the validation process are complete, all labeled training images in the training dataset are used to train the detector, minus those training images that were excluded from the validation process described above. However, even if we use the semi-automatic labeling process of method 100, doing so places a heavy burden on the labeling process because it requires labeling every object of interest in every training image sequence used. Since more training data will generally lead to better performing detection models, there is an inherent conflict between the need to have more training data and the exponential growth in resources (time and labor) required to label objects in the training data.

[0051] The present disclosure reduces the burden of labeling objects in training images by implementing a two-stage detector training process that utilizes individual part-level detectors and a unified or second-stage detector that can detect individual objects from the individual part-level detectors. The individual part-level detectors are first trained, each of which contains a small number of objects and configurations of interest, and then a second-stage or unified detector is trained that includes all objects of interest and corresponding configurations into a single detector.

[0052] For part-level detector training, a small set of objects (often only one or two objects) and corresponding configurations are selected. For example, two configurations of the tail cone (e.g., "tail cone delivered" and "tail cone attached") are combined to train a single part-level detector. In this example, only one actual object of interest is involved, the tail cone, and the two configurations represent two stages in the assembly process to be monitored. A second example is four states "left wing delivered", "right wing delivered", "left wing loaded", and "right wing loaded" combined into a single part-level detector, which has two objects of interest, left wing and right wing, each with two configurations.

[0053] For the part-level detector’s training data validation, a timeline of ground truth files is created for each training sequence as described above to eliminate any false negatives. Additionally, a collection of images that do not contain any objects detected in the part-level detector helps in training the part-level detector.

[0054] For the second stage of two-stage detector training, a unified detector is trained. The unified detector includes all objects of interest and corresponding configurations from the part-level detectors. The unified detector can then be deployed to generate assembly process monitoring data.

[0055] The two-stage detector training process allows part-level detectors to be trained with less training data. This is because each part-level detector contains only a small number of objects of interest (e.g., one or two objects of interest) and corresponding configurations of objects of interest, and only a subset of the training data is most relevant to these objects or configurations to be labeled. One feature of this is that the labeling requirements are reduced compared to having to label all objects of interest and corresponding configurations in all training datasets. In one example, only two to four image sequences are labeled to create the training dataset for each part-level detector.

[0056] The two-stage detector training process is also faster because the part-level detectors contain fewer objects of interest and configurations, making it faster to train and achieve high detection performance from a smaller training dataset. In addition, the part-level detectors can be managed separately from each other and from the second-stage unified detector. This allows incremental improvements in system performance by independently modifying and improving the individual part-level detectors and then retraining the unified detector from the modified part-level detectors.

[0057] also, Figure 3 An example method 200 for training a second level or unified detector is shown in FIG. Method 200 allows for an automated process for training a unified detector. Once the part level detectors 208, 210, 212 are trained, they are used to process all of the training data 202, 204, 206 to generate “labels” of all objects of interest and corresponding configurations required to train the unified detector (block 210).

[0058] For method 200, the classes or configurations of the unified detector are the union of all classes / configurations in the part-level detectors. Thus, objects and configurations or classes included in the various part-level detectors are combined into a single unified detector that will be able to detect various object corresponding configurations.

[0059] Additionally, the training data used to train the part-level detector is augmented at block 201 prior to being used to train the unified detector. This is because image labeling for a particular training data set is performed only for the objects for which the part-level detector is being trained. Therefore, if an image contains a first object that is not detected by the part-level detector, the image cannot be used to train the unified detector unless the first object is also identified in the image. Otherwise, as described above, the first object in the image will create a false negative example for training. In order to eliminate false negative examples for training the unified detector, the part-level detector is used to generate labels for all image sequences used to generate each individual part-level detector.

[0060] like Figure 3 As shown, box 201 applies the trained part-level detector to all training data (initially collected for part-level detector training) and detects all objects in each image sequence. This allows for additional training data that was not previously used (e.g., those we set aside for testing the part-level detector) when necessary. This step automatically generates labels for the training images. Part-level detectors trained in the above method are very effective at detecting objects in the images they are trained on as well as new images they are not trained on. This provides consistent and high-quality "labels" in terms of class labels and object bounding boxes (i.e., "labels") associated with each image.

[0061] The method 200 validates the training data against the expanded (timeline) ground truth from block 214 (block 216). For training part-level detectors, the ground truth timeline includes only the objects contained in the specific part-level detector. However, for a unified detector, the ground truth is expanded to include all objects of the unified detector, including all objects in all part-level detectors. As with the part-level detectors, the validation process generates a set of images (with labels generated by the part-level detectors) that are used as training data and a set of images that can be used as negative training samples.

[0062] The amount of training data created at block 216 may be several times greater than the sum of the training data used in part-level detector training, which results in a higher quality (unified) detector. Using the training data developed at block 216, a unified or second-level detector may be trained at block 218 to create a detector model at block 220 for use in process monitoring of an assembly line on a factory floor or any other location.

[0063] The following terms provide for the use of Figure 1 The system 20 generates an example configuration of systems and methods for detectors for process monitoring.

[0064] Item 1: A method for generating a detector, the method comprising: obtaining a first training data set including a first set of labeled images identifying a first object; obtaining a second training data set including a second set of labeled images identifying a second object; training a first part-level detector based on the first training data set; training a second part-level detector based on the second training data set; and training a unified detector based on the first training data set and the second training data set.

[0065] Clause 2: A method according to clause 1, wherein the first training data set is created by the following steps: receiving a first image sequence, identifying a first object in at least one image of the first image sequence; tracking the first object identified in at least one image in the first image sequence; marking a region of interest in each image in which the first object is tracked in the first image sequence; and creating the first training data set by collecting the region of interest from each image in which the first object is tracked in the first image sequence.

[0066] Clause 3: The method of clauses 1 to 2, wherein the step of obtaining a first training dataset comprises eliminating false negative markings of the first object by verifying the presence of the first object in the first training dataset against a ground truth timeline of the first object.

[0067] Clause 4: A method according to clauses 1 to 3, wherein the second training data set is created by the following steps: receiving a second image sequence, identifying a second object in at least one image of the second image sequence; tracking the second object identified in at least one image in the second image sequence; marking a region of interest in each image in which the second object is tracked in the second image sequence; and creating the second training data set by collecting the region of interest from each image in which the second object is tracked in the second image sequence.

[0068] Clause 5: The method of clauses 1 to 4, wherein the second object comprises the first object in a different configuration.

[0069] Clause 6: The method of clauses 1 to 5, wherein the step of obtaining a second training data set comprises eliminating false negative markings of the second object by verifying the presence of the second object against a ground truth timeline of the second object.

[0070] Clause 7: The method of clauses 1 to 6, wherein the step of training a unified detector based on the first training dataset comprises using a second part-level detector to mark regions of interest for tracking the second object in each image of the first training dataset.

[0071] Clause 8: The method of clauses 1 to 7, wherein the step of training the unified detector based on the second training dataset comprises using the first part-level detector to mark regions of interest for tracking the first object in each image of the second training dataset.

[0072] Clause 9: A method according to clauses 1 to 8, wherein the step of training the unified detector includes eliminating false negative markings of the first object in the second training dataset by verifying the presence of the first object against the ground truth timeline of the first object in the second training dataset.

[0073] Clause 10: A method according to clauses 1 to 9, wherein the step of training the unified detector includes eliminating false negative markings of the second object in the first training dataset by verifying the presence of the second object against a ground truth timeline of the second object in the first training dataset.

[0074] Clause 11: The method according to clauses 1 to 10, comprising: obtaining multiple additional training data sets, each of the multiple additional training data sets comprising a set of labeled images identifying a corresponding one of multiple additional objects; training multiple additional part-level detectors based on the multiple additional training data sets; and training a unified detector based on the first training data set, the second training data set and the multiple additional training data sets.

[0075] Clause 12: The method according to clauses 1 to 11, comprising: obtaining multiple additional training data sets, each of the multiple additional training data sets comprising a set of labeled images identifying a corresponding one of multiple additional objects; training a single additional part-level detector based on the multiple additional training data sets when the ground truth timelines of the multiple additional objects do not overlap; and training a unified detector based on the first training data set, the second training data set and the multiple additional training data sets.

[0076] Clause 13: The method of clauses 1 to 12, wherein the plurality of additional objects comprises a single object in a plurality of different configurations.

[0077] Clause 14: A method according to clauses 1 to 13, wherein the step of training a single additional part-level detector based on multiple additional training data sets includes eliminating false negative markings of multiple additional objects by verifying the presence of each of the multiple additional objects in the multiple additional training data sets against a ground truth timeline of each of the multiple additional objects.

[0078] Clause 15: The method of clauses 1 to 14, wherein the first and second sets of labeled images include labels that provide an object identifier, a location of a bounding box highlighting the corresponding object, and a size of the bounding box.

[0079] Item 16. A system for detecting an object, the system comprising: at least one camera configured to capture multiple images; and a controller configured to: obtain a first training data set comprising a first set of labeled images identifying a first object; obtain a second training data set comprising a second set of labeled images identifying a second object; train a first part-level detector based on the first training data set; train a second part-level detector based on the second training data set; and train a unified detector based on the first training data set and the second training data set.

[0080] Clause 17: A system according to clause 16, wherein the first training data set is created by the following steps: receiving a first image sequence, identifying a first object in at least one image of the first image sequence; tracking the first object identified in at least one image in the first image sequence; marking a region of interest in each image in which the first object is tracked in the first image sequence; and creating the first training data set by collecting the region of interest from each image in which the first object is tracked in the first image sequence.

[0081] Clause 18: A system according to clauses 16 to 17, wherein the second training data set is created by the following steps: receiving a second image sequence, identifying a second object in at least one image of the second image sequence; tracking the second object identified in at least one image in the second image sequence; marking a region of interest in each image in which the second object is tracked in the second image sequence; and creating the second training data set by collecting the region of interest from each image in which the second object is tracked in the second image sequence.

[0082] Item 19: A non-transitory computer-readable medium comprising programmed instructions, which when executed by a processor are operable to perform a method comprising: obtaining a first training data set comprising a first set of labeled images identifying a first object; obtaining a second training data set comprising a second set of labeled images identifying a second object; training a first part-level detector based on the first training data set; training a second part-level detector based on the second training data set; and training a unified detector based on the first training data set and the second training data set.

[0083] Clause 20: A non-transitory computer-readable medium according to clause 19, wherein the first training data set and the second training data set are created by the following steps: receiving a first image sequence and identifying a first object in at least one image of the first image sequence; receiving a second image sequence and identifying a second object in at least one image of the second image sequence; tracking the first object identified in at least one image in the first image sequence; tracking the second object identified in at least one image in the second image sequence; marking a region of interest in each image in which the first object is tracked in the first image sequence; marking a region of interest in each image in which the second object is tracked in the second image sequence; creating the first training data set by collecting the region of interest from each image in which the first object is tracked in the first image sequence; and creating the second training data set by collecting the region of interest from each image in which the second object is tracked in the second image sequence.

[0084] Although various embodiments are described, the description is intended to be exemplary, not limiting. It will be apparent to those of ordinary skill in the art that more embodiments and implementations are possible, which are within the scope of the embodiments. Unless specifically limited, any feature of any embodiment may be combined with or used in place of any other feature or element in any other embodiment. Therefore, the embodiments are not limited except in consideration of the appended claims and their equivalents. In addition, various modifications and changes may be made within the scope of the appended claims.

Claims

1. A method (200) for generating a detector, the method (100, 200) comprising the following steps: Obtaining a first training data set (202) comprising a first set of labeled images identifying a first object; obtaining a second training data set (204) comprising a second set of labeled images identifying a second object; Training a first part-level detector (208) based on the first training data set (202); Training a second part-level detector based on the second training data set (204); as well as A unified detector (220) is trained based on the first training data set (202) and the second training data set (204).

2. The method (200) according to claim 1, wherein: The first training data set (202) is created by the following steps: receiving a first image sequence (104), identifying the first object in at least one image (38) of the first image sequence; tracking the first object identified in the at least one image (38) in the first sequence of images; marking a region of interest (40) in each image (38) in the first sequence of images in which the first object is tracked; as well as The first training data set (202) is created by collecting the region of interest (40) from each image (38) in the first sequence of images in which the first object is tracked.

3. The method (200) according to claim 2, wherein: The step of obtaining the first training data set (202) includes eliminating false negative markings of the first subject by verifying the presence of the first subject in the first training data set (202) against a ground truth timeline of the first subject.

4. The method (200) according to claim 3, wherein: The second training data set (204) is created by the following steps: receiving a second image sequence, identifying the second object in at least one image (38) of the second image sequence; tracking the second object identified in the at least one image (38) in the second sequence of images; marking a region of interest (40) in each image (38) in the second sequence of images in which the second object is tracked; as well as The second training data set (204) is created by collecting the region of interest from respective images (38) in the second image sequence in which the second object is tracked.

5. The method (200) according to claim 4, wherein: The second object includes the first object in a different configuration.

6. The method (200) of claim 4, wherein: The step of obtaining the second training data set (204) includes eliminating false negative markings of the second object by verifying the presence of the second object against a ground truth timeline of the second object.

7. The method (200) of claim 6, wherein: The step of training the unified detector based on the first training dataset (202) includes using the second part-level detector to mark a region of interest for tracking the second object in each image (38) of the first training dataset (202).

8. The method (200) of claim 7, wherein: The step of training the unified detector based on the second training dataset (204) includes using the first part-level detector to mark regions of interest for tracking the first object in each image (38) of the second training dataset (204).

9. The method (200) of claim 8, wherein: The step of training the unified detector (220) includes eliminating false negative markings of the first object in the second training dataset (204) by verifying the presence of the first object against a ground truth timeline of the first object in the second training dataset (204).

10. The method (200) according to claim 9, wherein: The step of training the unified detector (220) includes eliminating false negative markings of the second object in the first training dataset (202) by verifying the presence of the second object in the first training dataset (202) against a ground truth timeline of the second object.

11. The method (200) according to claim 1, comprising the steps of: obtaining a plurality of additional training data sets (206), each of the plurality of additional training data sets (206) comprising a set of labeled images identifying a corresponding one of a plurality of additional objects; training a plurality of additional part-level detectors (206) based on the plurality of additional training data sets (206); as well as The unified detector (220) is trained based on the first training data set (202), the second training data set (204), and the plurality of additional training data sets (206).

12. The method (200) according to claim 1, comprising the steps of: obtaining a plurality of additional training data sets (206), each of the plurality of additional training data sets (206) comprising a set of labeled images identifying a corresponding one of a plurality of additional objects; training a single additional part-level detector based on the plurality of additional training data sets (206) when the ground truth timelines of the plurality of additional objects do not overlap; as well as The unified detector (220) is trained based on the first training data set (202), the second training data set (204), and the plurality of additional training data sets (206).

13. The method (200) of claim 12, wherein: The plurality of additional objects includes a single object in a plurality of different configurations.

14. The method (200) of claim 13, wherein: The step of training the single additional part-level detector (212) based on the plurality of additional training data sets (206) includes eliminating false negative markings of the plurality of additional objects by verifying the presence of each of the plurality of additional objects in the plurality of additional training data sets (206) against a ground truth timeline for each of the plurality of additional objects.

15. The method (200) of claim 1, wherein: The first and second sets of tagged images include tags that provide an object identifier, the location of a bounding box (40) highlighting the corresponding object, and the size of the bounding box (40).

16. A system (20) for detecting an object, the system (20) comprising: at least one camera (20-1, 20-2), the at least one camera configured to capture a plurality of images (38); as well as A controller (32), the controller being configured to: Obtaining a first training data set (202) comprising a first set of labeled images identifying a first object; obtaining a second training data set (204) comprising a second set of labeled images identifying a second object; Training a first part-level detector (208) based on the first training data set (202); Training a second part-level detector (210) based on the second training data set (204); as well as A unified detector (220) is trained based on the first training data set (202) and the second training data set (204).

17. The system (20) of claim 16, wherein: The first training data set (202) is created by the following steps: receiving a first image sequence, identifying the first object in at least one image (38) of the first image sequence; tracking the first object identified in the at least one image (38) in the first sequence of images; marking a region of interest (40) in each image (38) in the first sequence of images in which the first object is tracked; as well as The first training data set (202) is created by collecting the region of interest from respective images (38) in the first sequence of images in which the first object is tracked.

18. The system (20) of claim 17, wherein: The second training data set (204) is created by the following steps: receiving a second image sequence, identifying the second object in at least one image (38) of the second image sequence; tracking the second object identified in the at least one image (38) in the second sequence of images; marking a region of interest (40) in each image (38) in the second sequence of images in which the second object is tracked; as well as The second training data set (204) is created by collecting the region of interest from respective images (38) in the second image sequence in which the second object is tracked.

19. A non-transitory computer readable medium (34) containing programmed instructions, the instructions being operable when executed by a processor to perform a method (200), the method (200) comprising the steps of: Obtaining a first training data set (202) comprising a first set of labeled images identifying a first object; obtaining a second training data set (204) comprising a second set of labeled images identifying a second object; Training a first part-level detector based on the first training data set (202); Training a second part-level detector based on the second training data set (204); as well as A unified detector (220) is trained based on the first training data set (202) and the second training data set (204).

20. The non-transitory computer readable medium (34) of claim 19, wherein: The first training data set (202) and the second training data set (204) are created by the following steps: receiving a first image sequence, identifying the first object in at least one image (38) of the first image sequence; receiving a second image sequence, identifying the second object in at least one image (38) of the second image sequence; tracking the first object identified in the at least one image (38) in the first sequence of images; tracking the second object identified in the at least one image (38) in the second sequence of images; marking a region of interest (40) in each image (38) in the first sequence of images in which the first object is tracked; marking a region of interest (40) in each image (38) in the second sequence of images in which the second object is tracked; Creating the first training data set (202) by collecting the region of interest from each image (38) in the first image sequence tracking the first object; as well as The second training data set (204) is created by collecting the region of interest from respective images (38) in the second image sequence in which the second object is tracked.

Images