Method and system for process monitoring

Abstract

Methods and systems for process monitoring. A method of monitoring an assembly process is disclosed herein. The method includes obtaining an event model of each of a plurality of objects in an assembly process, where the event model of each of the plurality of objects includes a predetermined time range in which a change occurs. The method comprises: collecting a sequence of images of an assembly process for monitoring; and identifying, using the detector model, whether a change in presence of each of the plurality of objects has occurred. The method further includes viewing the event model for each of the plurality of objects to determine whether a predetermined time range elapses without identifying a change in presence of a corresponding one of the plurality of objects; and issuing an alarm if the predetermined time range elapses without identifying the presence of a corresponding one of the plurality of objects.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for generating detector models, and more particularly, to a detector model for process monitoring to determine delays in an assembly process. Background Art

[0002] Machine learning is the process of analyzing to determine a model that maps input data to output data. One type of machine learning is supervised learning, in which a model is trained on a dataset of sufficient input data to include known output data. Once a model is trained, it can be deployed, i.e., applied to new input data to predict expected outputs.

[0003] Machine learning can be applied to regression problems (where the output data is a numerical value, such as voltage, pressure, number of cycles) and classification problems (where the output data is a label, class, and / or category, such as pass-fail, failure type, etc.). For both types of problems, a wide range of machine learning algorithms are available, and new algorithms are actively being researched. However, generating datasets to train models can be very labor intensive. Additionally, the quality of the datasets provided for training can greatly impact the model's ability to predict expected outputs from new datasets.

[0004] Therefore, the input datasets used to train the models and the expected outputs predicted by the models need to be improved to determine the delays in the assembly process. Summary of the invention

[0005] A method for monitoring an assembly process is disclosed herein. The method includes: obtaining an event model for each of a plurality of objects in the assembly process, wherein the event model for each of the plurality of objects includes a predetermined time range in which a change in existence occurs. The method includes: collecting an image sequence of the assembly process for monitoring; and identifying, using a detector model, whether a change in existence of each of the plurality of objects occurs. The method also includes: checking the event model for each of the plurality of objects to determine whether a predetermined time range has elapsed without identifying a change in existence of a corresponding one of the plurality of objects; and if the predetermined time range has elapsed without identifying the existence of the corresponding one of the plurality of objects, issuing an alarm.

[0006] In one or more embodiments of the method, a state machine is used to filter results from a detector model that identifies changes in the presence of each of a plurality of objects based on whether the object was detected in at least one preceding or succeeding image.

[0007] In one or more embodiments of the method, the step of identifying a change in the presence of each of the plurality of objects includes determining a time at which each of the plurality of objects appears or disappears from the sequence of images.

[0008] In one or more embodiments of the method, the time is based on a relative time set by a change in presence of one of the plurality of objects.

[0009] In one or more embodiments of the method, the plurality of objects includes multiple configurations of a single object.

[0010] In one or more embodiments of the method, the step of collecting a sequence of images of the assembly process is performed in real time.

[0011] In one or more embodiments of the method, an event model for each of the plurality of objects is generated by: receiving at least one training image sequence showing multiple occurrences of an assembly process of each of the plurality of objects; and using a detector model to identify the presence of each of the plurality of objects in the at least one training image sequence. In addition, an event model corresponding to each of the plurality of objects is generated based on a statistical prediction of when the object should appear in the training image sequence.

[0012] In one or more embodiments of the method, the statistical prediction includes a predetermined time range within which a corresponding one of the plurality of objects exists.

[0013] In one or more embodiments of the method, the predetermined time range is based on three times a standard deviation of a mean occurrence time present for a corresponding one of the plurality of subjects.

[0014] In one or more embodiments of the method, the predetermined time range includes a relative time based on a presence of one of the plurality of objects.

[0015] In one or more embodiments of the method, the plurality of occurrences of the assembly process comprises a historical occurrence of the assembly process.

[0016] In one or more embodiments of the method, the detector model is trained by the following steps: obtaining a separate training data set corresponding to each of the plurality of objects, each separate training data set comprising a set of labeled images identifying a corresponding one of the plurality of objects. The detector model is also trained by the following steps: training a part-level detector based on each separate training data set; and training the detector model based on each part-level detector.

[0017] In one or more embodiments of the method, a separate training dataset is created by: receiving a separate image sequence for each of the plurality of objects, identifying a corresponding one of the plurality of objects in at least one image of the separate image sequence; and tracking each of the plurality of objects identified in the at least one image in the corresponding one of the separate image sequences. The separate training dataset is also created by: marking a corresponding one of the regions of interest for tracking each of the plurality of objects in each of the images in the separate image sequence; and creating a separate training dataset for each of the plurality of objects by collecting the regions of interest for tracking the corresponding object from each of the images in each of the separate image sequences.

[0018] In one or more embodiments of the method, the step of obtaining a separate training data set corresponding to each of the plurality of subjects includes eliminating false negative labels by verifying the presence of each of the plurality of subjects in each of the plurality of training data sets against a ground truth timeline for each of the plurality of subjects.

[0019] Disclosed herein is a system for detecting objects in an assembly process. The system includes at least one camera configured to capture multiple images and a controller. The controller is configured to: obtain an event model for each of a plurality of objects in the assembly process, wherein the event model for each of the plurality of objects includes a predetermined time range for occurrence of an existence change; and collect an image sequence of the assembly process for monitoring. The controller is also configured to: identify, using a detector model, whether an existence change occurs for each of the plurality of objects; and check the event model for each of the plurality of objects to determine whether a predetermined time range has passed without identifying an existence change for a corresponding one of the plurality of objects. The controller is also configured to: sound an alarm if a predetermined time range has passed without identifying the existence of a corresponding one of the plurality of objects.

[0020] In one or more embodiments of the system, a state machine is used to filter results from a detector model that identifies changes in the presence of each of a plurality of objects based on whether the object was detected in at least one preceding or succeeding image.

[0021] In one or more embodiments of the system, an event model for each of a plurality of objects is generated by: receiving at least one training image sequence showing multiple occurrences of an assembly process for each of the plurality of objects; identifying the presence of each of the plurality of objects in the at least one training image sequence using a detector model; and generating an event model corresponding to each of the plurality of objects based on statistical predictions of when the object should appear in the training image sequence.

[0022] Disclosed herein is a non-transitory computer-readable storage medium containing programmed instructions, which when executed by a processor are operable to perform a method. The method includes: obtaining an event model for each of a plurality of objects in an assembly process, wherein the event model for each of the plurality of objects includes a predetermined time range for occurrence of a presence change; and collecting a sequence of images of the assembly process for monitoring. The method includes: using a detector model to identify whether a presence change for each of the plurality of objects occurs; and viewing the event model for each of the plurality of objects to determine whether a predetermined time range has passed without identifying a presence change for a corresponding one of the plurality of objects. The method also includes: issuing an alarm if a predetermined time range has passed without identifying the presence of a corresponding one of the plurality of objects.

[0023] In one or more embodiments of the method, an event model for each of a plurality of objects is generated by: receiving at least one training image sequence showing multiple occurrences of an assembly process for each of the plurality of objects; identifying the presence of each of the plurality of objects in at least one image sequence using a detector model; and generating an event model corresponding to each of the plurality of objects based on statistical predictions of when the object should appear in the training image sequence.

[0024] In one or more embodiments of the method, the detector model is trained by the following steps: obtaining a separate training data set corresponding to each of a plurality of objects, each separate training data set comprising a set of labeled images identifying a corresponding one of the plurality of objects; training a part-level detector based on each separate training data set; and training the detector model based on each part-level detector.

[0025] The above summary is not intended to represent every possible embodiment or every aspect of the present disclosure. Instead, the above summary is intended to illustrate some of the novel aspects and features disclosed herein. The features, functions and advantages of the present disclosure can be independently implemented in various embodiments, or can be combined in other embodiments, further details of which can be seen with reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Some embodiments of the present disclosure will now be described by way of example only with reference to the accompanying drawings. The same reference numbers represent the same element or the same type of element on all drawings.

[0027] Figure 1 An example camera system and computer system are shown used in conjunction in an assembly line.

[0028] Figure 2 An example method for generating a training dataset of objects for a detector model is shown.

[0029] Figure 3 An example method for training a detector model is shown.

[0030] Figure 4 A method for performing event modeling and anomaly detection is shown.

[0031] Figure 5 Table showing example configuration from the assembly process.

[0032] Figure 6 Shown for Figure 5 Each configuration shown represents a diagram of the assembly activities by linking the start time and the end time.

[0033] Figure 7 Shows the Figure 6 The assembly activity plots the relative timing of events.

[0034] Figure 8 against Figure 7 The events plotted against time show anomalies. DETAILED DESCRIPTION

[0035] The accompanying drawings and the following description illustrate specific exemplary embodiments of the present disclosure. Those of ordinary skill in the art will be able to design various arrangements that, although not explicitly described or shown herein, specifically implement the principles of the present disclosure and are included within the scope of the present disclosure. In addition, any examples described herein are intended to help understand the scope of the present disclosure and are to be interpreted as not being limited to these specifically described examples and conditions. As a result, the present disclosure is not limited to the specific embodiments or examples described below, but is limited by the claims and their equivalents.

[0036] Figure 1 An example camera system 20 is shown having a first camera 20-1 and a second camera 20-2 for monitoring an assembly process (e.g., assembly of an aircraft 22 on an assembly line). The camera system 20 communicates with a computer system 30 that receives images from the camera system 20. Although the illustrated examples of the camera system 20 and the computer system 30 are shown with respect to an assembly process, the present disclosure is applicable to other fields where it is desirable to perform object detection.

[0037] Although for simplicity of illustration, Figure 1The computer system 30 is depicted as a single computer module, but the computer system 30 may be physically implemented as one or more processing nodes, which have a non-transitory computer-readable storage medium 34 (i.e., sufficient memory for application) and associated hardware and software (such as but not limited to high-speed clocks, timers, input / output circuits, buffer circuits, etc.). The computer-readable storage medium 34 may include sufficient read-only memory, such as magnetic or optical memory. Computer-readable code or instructions that specifically implement the methods described below may be executed during the operation of the computer system 30. To this end, the computer system 30 may include one or more processors 32, such as logic circuits, application-specific integrated circuits (ASICs), central processing units, microprocessors, and / or other necessary hardware required to provide the programming functions described herein. The display screen 36 may be connected or communicated with the computer-readable medium 34 and the processor 32 to facilitate the implementation of the graphical user interface of the present disclosure described below.

[0038] In the example shown, cameras 20-1 and 20-2 are mounted on laterally opposite sides of an aircraft 22 above an aircraft assembly line 24 for monitoring the process of assembling the aircraft 22. Depending on the process being monitored, the cameras 20-1 and 20-2 may capture images at 4K resolution and up to 30 or more frames per second. Alternatively, the cameras 20-1, 20-2 may capture images at a lower resolution (e.g., 960×540) and at a rate of one image every 15 seconds. The variation in camera resolution and image capture rate depends on the size of the object being detected and the length of time the object remains in the field of view of the cameras 20-1, 20-2. In addition, collecting images at a lower resolution and rate reduces the amount of data that the computer system 30 needs to store and process.

[0039] Additionally, the present disclosure is applicable to camera systems 20 that use only a single camera or more than two cameras to capture images of a particular area. The number of cameras used for process monitoring depends on the field of view required to include the objects to be inspected, as some cameras may have obstructed view of one or more objects.

[0040] Figure 2An example method 100 is shown for generating a training data set for a specific object for training a detection model of a detector (e.g., a part-level detector). The method 100 is implemented for each desired object to be tracked from an image sequence that includes at least one image having the desired object. The method 100 begins at box 102, and from box 102, the method 100 receives a sequence of images (box 104) at box 106. At box 106, the desired object is highlighted in at least one image from the sequence in box 104. In one example, an image from the sequence at or near the first appearance of the object is used to highlight the object. The object may be highlighted by a user of the computer system 30. In one example, the user highlights the object in the image 38 using a bounding box 40 surrounding the object (see Figure 1 ). A user may create a bounding box by placing a rectangle around the desired object in image 38. The location in the image and the size of the rectangle are recorded or marked with the image and used at block 108 to track the object in other images in the sequence of images.

[0041] When tracking an object between the sequence of images at block 108, the tracking algorithm uses the location of the bounding box in the particular image from block 106 and the size of the rectangle identified by the user to "track" or follow the object of interest through the sequence of images. For each image in the sequence of images, the tracker will attempt to locate the same object of interest and record information about the object of interest in association with each image in which the object was identified in the memory 34 of the computer system 30. The information tagged to each frame may include the location and size of the rectangle, a label identifying the object (e.g., the name or class of the object of interest), or a cropped image region of interest (ROI) defining the object by the rectangle.

[0042] If the tracking algorithm determines that the object of interest is no longer present in the image or is "lost" (e.g., which may be indicated by a confidence score below a set confidence threshold), the object tracking process may be interrupted at block 110. Additionally, the user may intervene due to errors in the tracking results as determined by the user (e.g., when the tracker has locked onto an object other than the one initially selected by the user).

[0043] At block 112, method 100 may return to block 106 to modify the bounding box identifying the object in the initial image to again improve tracking of the object of interest. Alternatively, when returning to block 106, the user may select an image in the sequence where the tracking algorithm lost the object, and highlight the correct object of interest in that image to allow the object to be tracked again at block 108. The ability to restart the tracking algorithm where it failed and continue without having to re-evaluate the same image from the image sequence again reduces the time required to evaluate the image sequence.

[0044] At block 114, the position and size of the rectangle, the label identifying the object, and the cropped image region for each image in the sequence of images are collected. A user may review the region of interest from each image in the sequence at block 116 to confirm that the correct object of interest is identified in the region of interest as part of finalizing the collected images and associated information. If any region of interest does not identify the correct object of interest, those regions of interest may be deleted from the finalized images.

[0045] In one example, the user finalizes the images by viewing them on the display 36 of the computer system 30. Once the user confirms that the correct region of interest is identified in the image, the finalized images are collected at block 118 and the method may end at block 120. The finalized images may be stored in the memory 34 of the computer system 30 or in another location such as in the cloud. The finalized images form a training data set for the part-level or first-level detector for objects identified in the image sequence as described below.

[0046] Method 100 may be repeated for the same or a different image sequence showing another object of interest or object configuration to form a training data set for that particular object or configuration. Since user involvement is limited to initiating tracking of an object of interest, method 100 no longer requires a user to identify an object of interest from each image in the image sequence using image tracking throughout the image sequence. This reduces the time and effort required to create or update a training data set for an object.

[0047] The tracking of the object of interest performed at least in blocks 106 and 108 can be accomplished by many different available tracking algorithms, such as simple trackers based on cross-correlation coefficients. These trackers can be used in situations where the appearance and position of the object of interest remain relatively stable. However, other tracker algorithms may be used depending on the object being tracked.

[0048] One feature of the present disclosure is to simplify the tracking process of objects of interest by treating a single object as a separate object to be tracked as the same object moves between different configurations or classes. Thus, if the objects of interest are in different "configurations," they are defined as different "objects" to facilitate object tracking in method 100. For the example of objects associated with assembling an aircraft 22, aircraft wings 26 appear in several different configurations, such as when they are first towed to the factory floor ("delivered"), when they are lifted by a crane and moved to a support frame on the side of the fuselage (referred to as "loaded"), and when they are mated to the fuselage ("attached"). Thus, a single object, i.e., wing 26, will create three different configurations or classes, and the corresponding image sequences will be evaluated by method 100 to create different training data sets for each configuration.

[0049] In one example embodiment, at least one of the cameras 20-1, 20-2 captures an image of the assembly shop every 15 seconds. This reduces the burden of storing and processing large amounts of image data when the object of interest is not moving quickly or frequently. In prior art methods, it is difficult to maintain high accuracy by tracking the movement of objects through discontinuous image frames. However, the present disclosure utilizes the final state in which the objects of interest (in this example, the wings) are located, rather than how they are moved from one place to another during the assembly process. Therefore, the object of interest being in a "configuration" refers to an object in a specific assembly state, such as "left wing delivered", "left wing loaded", etc.

[0050] Furthermore, an object of interest may be in a different configuration even if its position has not changed. An example of such an object is the cradle 28 that supports the front end of the fuselage of an aircraft 22. During the assembly process, the cradle 28 will transition from a configuration that supports the weight of the fuselage ("cradle support plane") to a configuration that does not support the fuselage ("cradle non-support plane"), such that the fuselage is supported by the front landing gear. This is called "Down jack" and is an important milestone in the assembly process of most aircraft 22, which can be identified by the detector model disclosed herein.

[0051] In order to perform object detection model training, a positive training data set is created based on the results of method 100. The positive training data set includes images containing positive training samples of objects, the objects are marked with labels identifying the objects at their locations and the size of each object of interest to be detected in each image. In addition, image data sets without positive training samples are useful for training object detection models.

[0052] The positive training data set may be generated from the method 100 discussed above, and the positive training samples cannot use images of the same assembly plant floor before the object of interest is loaded into the workspace and before any parts to be inspected are present. This ensures that the samples will not include false negatives.

[0053] With respect to labeled images, the label tracking process of method 100 generates training data sets, which are generated one object at a time for each (time-stamped) image in the image sequence that forms the training data set. When more than one object of interest is identified using the same image sequence, the labels of different objects in the same image are merged into a single file for the purpose of training the detection model. In addition, if the labeling process is not exhaustive (i.e., if the process fails to label some objects in some images, or some labeling results are accidentally deleted), the training data set may inadvertently create negative training examples from positive images (i.e., false negative identifications). That is, if the label tracking process of method 100 fails to label the object of interest in some images, a situation is created where the object exists but is not labeled, in fact causing negative samples to be used for training (i.e., telling the training algorithm of the detection model that the object does not exist, when in fact the object does). This can cause damage to the training and reduce the performance of the resulting detection model.

[0054] In order to avoid inadvertently introducing negative training examples, an automated verification process is created to eliminate these negative examples. As described below, this process requires creating a ground truth timeline file by examining each image sequence and identifying and recording the time when each object first appears and disappears, or generally using the term "generalized object detection", when a configuration becomes true for the relevant object and when the configuration is no longer true. This ground truth timeline is stored in a "ground truth" file. In the example shown, each image sequence in the training data set will have a ground truth file. The ground truth timeline provides a verification process to eliminate false negatives from the training data set, and negative training data can be easily created using images with timestamps including time intervals without objects of interest.

[0055] For traditional object detection training methods, once the labeling process and the validation process are complete, all labeled training images in the training data set are used to train the detector, minus those data excluded from the validation process described above. However, even if we use the semi-automatic labeling process of method 100, doing so places a heavy burden on the labeling process because it requires labeling every object of interest in every training image sequence used. Since more training data will generally lead to better performing detection models, there is an inherent conflict between the need to have more training data and the exponential growth in resources (time and labor) required to label objects in the training data.

[0056] The present disclosure reduces the burden of labeling objects in training images by implementing a two-stage detector training process that utilizes separate part-level detectors and a unified or second-stage detector that can detect each object from each separate part-level detector. The separate part-level detectors are first trained, each of which contains a small number of objects and configurations of interest, and then the second-stage or unified detector is trained to include all objects of interest and corresponding configurations into a single detector.

[0057] For part-level detector training, a small set of objects (often only one or two objects) and corresponding configurations are selected. For example, two configurations of the tail cone (e.g., "tail cone delivered" and "tail cone attached") are combined to train a single part-level detector. In this example, only one actual object of interest is involved, the tail cone, and the two configurations represent two stages in the assembly process to be monitored. A second example is four states "left wing delivered", "right wing delivered", "left wing loaded", and "right wing loaded" combined into a single part-level detector, which has two objects of interest, left wing and right wing, each with two configurations.

[0058] For the part-level detector’s training data validation, a timeline of ground truth files is created for each training sequence as described above to eliminate any false negatives. Additionally, a collection of images that do not contain any objects detected in the part-level detector helps in training the part-level detector.

[0059] For the second stage of two-stage detector training, a unified detector is trained. The unified detector includes all objects of interest and corresponding configurations from the part-level detectors. The unified detector can then be deployed to generate data for assembly process monitoring.

[0060] The two-stage detector training process allows part-level detectors to be trained with less training data. This is because each part-level detector contains only a small number of objects of interest (e.g., one or two objects of interest) and corresponding configurations of objects of interest, and only a subset of the training data is most relevant to these objects or configurations to be labeled. One feature of this is that the labeling requirements are reduced compared to having to label all objects of interest and corresponding configurations in all training datasets. In one example, only two to four image sequences are labeled to create the training dataset for each part-level detector.

[0061] The two-stage detector training process is also faster because the part-level detectors contain fewer objects of interest and configurations, making it faster to train and achieve high detection performance from a smaller training dataset. In addition, the part-level detectors can be managed separately from each other and from the second-stage unified detector. This allows incremental improvements in system performance by independently modifying and improving the individual part-level detectors and then retraining the unified detector from the modified part-level detectors.

[0062] also, Figure 3 An example method 200 for training a second level or unified detector is shown in FIG. Method 200 allows for an automated process for training a unified detector. Once the part level detectors 208, 210, 212 are trained, they are used to process all of the training data 202, 204, 206 to generate “labels” of all objects of interest and corresponding configurations required to train the unified detector (block 210).

[0063] For method 200, the classes or configurations of the unified detector are the union of all classes / configurations in the part-level detectors. Thus, objects and configurations or classes included in the various part-level detectors are combined into a single unified detector that will be able to detect various object corresponding configurations.

[0064] Additionally, the training data used to train the part-level detector is augmented at block 201 prior to being used to train the unified detector. This is because image labeling for a particular training data set is performed only for the objects for which the part-level detector is being trained. Therefore, if an image contains a first object that is not detected by the part-level detector, the image cannot be used to train the unified detector unless the first object is also identified in the image. Otherwise, as described above, the first object in the image will create a false negative example for training. In order to eliminate false negative examples for training the unified detector, the part-level detector is used to generate labels for all image sequences used to generate each individual part-level detector.

[0065] like Figure 3 As shown, box 201 applies the trained part-level detector to all training data (initially collected for part-level detector training) and detects all objects in each image sequence. This allows for additional training data that was not previously used (e.g., those we set aside for testing the part-level detector) when necessary. This step automatically generates labels for the training images. Part-level detectors trained in the above method are very effective at detecting objects in the images they are trained on as well as new images they are not trained on. This provides consistent and high-quality "labels" in terms of class labels and object bounding boxes (i.e., "labels") associated with each image.

[0066] The method 200 validates the training data against the expanded (timeline) ground truth from block 214 (block 216). For training part-level detectors, the ground truth timeline includes only the objects contained in the specific part-level detector. However, for a unified detector, the ground truth is expanded to include all objects of the unified detector, including all objects in all part-level detectors. As with the part-level detectors, the validation process generates a set of images (with labels generated by the part-level detectors) that are used as training data and a set of images that can be used as negative training samples.

[0067] The amount of training data created at block 216 may be several times greater than the sum of the training data used in part-level detector training, which results in a higher quality (unified) detector. Using the training data developed at block 216, a unified or second-level detector may be trained at block 218 to create a detector model at block 220 for use in process monitoring of an assembly line on a factory floor or any other location.

[0068] Figure 4 An example method 300 for monitoring an assembly process is shown. The method 300 monitors an assembly process by performing event modeling and anomaly detection of the assembly process. In one example, the assembly process is an aircraft 22 on an assembly line 24, and monitoring is performed using a detector model such as the unified detector described above. However, the method 300 can be used to monitor processes for assembling other components. The method 300 uses data provided by the detector model to detect events related to abnormal assembly activities based on history or program priority in the time domain or spatial (location) domain.

[0069] like Figure 4 As shown, at box 302, event modeling and anomaly detection begins with collecting an input image sequence. The input image sequence may include real-time images or previously collected stored images. When modeling an assembly process to determine a temporal sequence of events of the assembly process, the image sequence may include images captured over an extended period of time. The extended period of time depends on the length of time required for the assembly process to complete at least one build cycle and the number of cycles required to create a model of the assembly process. When method 300 considers a larger number of build cycles, method 300 can generate a more accurate model of the assembly process.

[0070] The detector model (block 304), such as the unified detector described above, performs at least one of object detection, identification, or tracking to generate detection results at block 308. The detection results generated at block 306 by the detector model analyzing the image sequence may include a marker identifying the detected object plus the size and location of the detected object for each image in the input image sequence. The detection results shown in block 308 may also include objects highlighted in individual specific images 38 in the image sequence, such as by a bounding box 40, to allow a user to view the detected objects in the individual images in the image sequence.

[0071] The method 300 may use the above information from the detection results in block 308 to perform post-processing analysis of at least one of event modeling or anomaly detection at block 310. In order to perform anomaly detection, an event model that determines the normal assembly process of each object must be generated. In addition to object identifiers, the size and location of the detected and assembled objects, the detection results from block 308 generated by the detector model also include time series timestamps. The time series data is used to build an event model of the assembly process by correlating the detection of objects with events in the time domain, which will allow anomaly detection, as discussed further below.

[0072] Figure 5 An example table of configurations of various objects for an example assembly process for aircraft 22 is provided. Figure 5 The table in identifies only a few configurations to be tracked and associated with events in the assembly process of the aircraft 22, but the present disclosure is not limited to the configurations or assemblies of the aircraft listed in the table, but rather any assembly process that can be used in conjunction with object detection. For example, Figure 5 The table in tracks fuselage, tail cone delivery, tail cone attachment, right wing delivery, left wing delivery, right wing loading, and left wing loading. The image sequence from block 302 provided to method 300 when creating the event model may include multiple assembly processes of the aircraft 22 to create a larger data sample size based on the event model.

[0073] As the detector model from block 304 analyzes the image sequence for use in generating the event model, the method 300 will generate various detections that reflect the progress of the assembly process. For example, when a fuselage is loaded into a station, the fuselage will be detected as Figure 5 Similarly, when the tail cone is delivered to the station, "Tail Cone Delivery" will be detected. At any time during the assembly process, Figure 5 The subset of configurations shown may describe assembly states within the scope of an application setting.

[0074] Figure 5One property of the configurations listed in is persistence. When the various configurations become true based on the detector results from block 308, they will persist for an unspecified period of time determined by the physical properties (i.e., "tail cone delivered" will continue to be true until someone moves the tail cone). Vice versa.

[0075] That is, if a configuration changes (from true) to false, it will remain false for an unspecified period of time. However, due to the variation in the output of the detection model, the output results from the detection model may not always be continuously true or false at every time step of the output results. That is, the output of the detector model may drop after an initial determination that an object has been detected, or vice versa, which may not reflect the real world situation.

[0076] In order to filter out these detection errors from the detection model, we combine Figure 5 The configuration listed in uses a state machine. The state machine is designed to filter out accidental rejections and false positives (detections that do not correspond to real objects) from the detection results. Figure 5 When the discrete outputs of the configurations listed in are fed into the state machine, the output of the state machine is a cleaned version of the raw output from the detector model.

[0077] Formula (1) represents an example implementation of the function of the state machine. For the following formula (1), C(t)∈{true, false} represents Figure 5 The input to the state machine of any configuration C listed in , where t represents a discrete time step corresponding to the time of an image in the image sequence, and T>0 is the time step threshold (persistence threshold) of the state machine.

[0078]

[0079] In the above formula, S C (·) is the output of the state machine, and N is the length of the input image sequence. In other words, the output of the state machine is true only when the input C(t) from the current and past T-1 time steps is true; otherwise, if the input C(t) from the current and past T-1 time steps is false, the output of the state machine is false. In all other cases, the state machine output remains unchanged. The state machines shown in equation (1) filter independently of each other. Figure 5 Output of all objects or configurations shown, and will output S C (·) for the next step. An example persistence threshold is 5 time steps. However, the persistence threshold may include more or fewer time steps.

[0080] Figure 6 Shown by Figure 5 A diagram representing assembly activities is created by linking the start and end times of the various configurations shown. In other words, the lines span the duration when the object or configuration identified on the vertical axis is true.

[0081] When assemblies are performed one at a time, as when assembling an aircraft one at a time from the fuselage and wings to the vertical tail and tail cone, etc., the inspection sequence can be represented by similar units. After the inspection sequence from the inspection model is filtered as described, the sequence is organized into "segments" S, each segment starting shortly after the previous assembly at an assembly station has been moved to the next station, and ending immediately after the assembly itself has moved. In addition, by recording Figure 5 The start and end times of the various configurations are shown, and the detection can be presented in a compressed form (due to the persistence discussed earlier).

[0082] In the previous discussion above, whenever an event (e.g. Figure 5 When referring to the times when the object or configuration in the image becomes true), it will be understood that they are discrete times when the corresponding images were taken. In order to model the assembly process, it is convenient that these discrete times have a common reference to facilitate comparison and analysis. Within each segment, a reference start time is selected that corresponds to when the reference object or configuration is loaded (e.g., time = 0). The start and end times of all other events relative to the reference start time are then expressed relative to the start time of the reference object or configuration.

[0083] For the following discussion, unless otherwise stated, the word "time" always refers to the relative time of events within a segment. Figure 7 Illustrate the concept of relative time of events in the example activity diagram. Because time is relative, the time can become negative, as in the case of "Right Wing Delivered" and "Left Wing Delivered" because they become true before the fuselage is loaded and becomes true.

[0084] In one example, a unified detector may be used to collect an event model representing a normal assembly process over an extended period of time (e.g., over a period of several months). With information collected from the unified detector, the assembly period for each component may be divided into sequences (e.g., each aircraft assembly process from fuselage loading to line movement) and the event model may be calculated. Figure 5 The (relative) time of all events in the configuration listed in . Equation (2) represents the statistical analysis used to determine the relative time and when collecting Figure 5 The formula for the time statistics of each characteristic in .

[0085] In formula (2-3), T C is the (relative) time of the event representing the start time of configuration C, using a MAD (median absolute deviation) based method to estimate T C Time distribution N(mc,σ c ), as follows:

[0086]

[0087] σ c = 1.486 *MAD (4)

[0088] In the above formula (2), median(·) is a function that calculates the median of a set of values ​​in an independent variable. Figure 7 Show Figure 5 A graphical representation of the distribution model of the start time of objects or configurations in m c and ±σ c (Equation 4) is shown as a cross. After removing the outliers (by removing c The deviation is greater than 3*σ c After those) C The min / max boundaries are shown as boxes.

[0089] For Figure 5 For each event related to one of the configurations shown, a time limit (relative to the fuselage start time) when the configuration should occur is pre-calculated. In one example, this is calculated using the following equation (3).

[0090] T c, max = m c + 3*σ c (5)

[0091] In the case where the above statistical event model is generated from equation (2) and if Figure 7 As shown in the example in , anomalies can be detected when events in a segment do not fall into the normal range described by the event model. For example, if an event occurs whose (relative) time deviates from the mean (m) by more than 3 times the standard deviation (3*σ), then the event is declared an anomaly.

[0092] Figure 8 Shows all abnormal events (shown as "*") and their expected distribution (e.g. Figure 7 In the example shown, only the lagging events are shown, e.g., objects or configurations that have not become true at 3 times the standard deviation. This is because if the deviation from the mean is negative, the event will occur earlier than the mean and may indicate that the object or configuration became true early, which is not as concerning as becoming true later.

[0093] Figure 7 The results shown can be obtained offline when data is collected in advance. The event model is estimated from equations (2-4) for each desired configuration. The event model for each configuration is used to generate an alarm for an abnormal event of the corresponding object or configuration. In one example, Figure 4Alerts are calculated in real time from a live image feed at box 302 in an online or cloud mode, and detection results from box 308 are streamed to box 310. Any anomalies detected at box 310 can then be reported as an alert, or tracked to monitor the assembly process. The following clauses provide Figure 1 System 20 is an example configuration of a system and method for obtaining grazing angle measurements.

[0094] Item 1: A method for monitoring an assembly process, the method comprising: obtaining an event model for each of a plurality of objects in the assembly process, wherein the event model for each of the plurality of objects includes a predetermined time range within which an existence change occurs; collecting an image sequence of the assembly process for monitoring; identifying whether an existence change occurs for each of the plurality of objects using a detector model; checking the event model for each of the plurality of objects to determine whether a predetermined time range has elapsed without identifying an existence change for a corresponding one of the plurality of objects; and issuing an alarm if the predetermined time range has elapsed without identifying the existence of the corresponding one of the plurality of objects.

[0095] Clause 2: The method of clause 1, wherein the results from the detector model identifying the presence changes for each of the plurality of objects are filtered using a state machine based on whether the object was detected in at least one previous or subsequent image.

[0096] Clause 3: The method of clauses 1 to 2, wherein the step of identifying a change in the presence of each of the plurality of objects comprises determining a time at which each of the plurality of objects appears or disappears from the sequence of images.

[0097] Clause 4: The method of clauses 1 to 3, wherein the time is based on a relative time set by a change in the presence of one of the plurality of objects.

[0098] Clause 5: The method of clauses 1 to 4, wherein the plurality of objects comprises multiple configurations of a single object.

[0099] Clause 6: A method according to clauses 1 to 5, wherein the step of collecting a sequence of images of the assembly process is performed in real time.

[0100] Clause 7: A method according to clauses 1 to 6, wherein an event model for each of the multiple objects is generated by the following steps: receiving at least one training image sequence showing multiple occurrences of an assembly process for each of the multiple objects; using a detector model to identify the presence of each of the multiple objects in the at least one training image sequence; and generating an event model corresponding to each of the multiple objects based on statistical predictions of when the object should appear in the training image sequence.

[0101] Clause 8: The method of clauses 1 to 7, wherein the statistical prediction includes a predetermined time range within which a corresponding one of the plurality of objects exists.

[0102] Clause 9: The method of clauses 1 to 8, wherein the predetermined time range is based on three times the standard deviation of a mean occurrence time of a corresponding one of the plurality of objects.

[0103] Clause 10: The method of clauses 1 to 9, wherein the predetermined time range comprises a relative time based on a presence of one of the plurality of objects.

[0104] Clause 11: The method of clauses 1 to 10, wherein the multiple occurrences of the assembly process include historical occurrences of the assembly process.

[0105] Clause 12: A method according to clauses 1 to 11, wherein the detector model is trained by the following steps: obtaining a separate training data set corresponding to each of a plurality of objects, each separate training data set comprising a set of labeled images identifying a corresponding one of the plurality of objects; training a part-level detector based on each separate training data set; and training the detector model based on each part-level detector.

[0106] Clause 13: A method according to clauses 1 to 12, wherein a separate training dataset is created by the following steps: receiving a separate image sequence for each of a plurality of objects, identifying a corresponding one of the plurality of objects in at least one image of the separate image sequence; tracking each of the plurality of objects identified in at least one image in a corresponding one of the separate image sequences; marking a corresponding one of the regions of interest for tracking each of the plurality of objects in respective images in the separate image sequences; and creating a separate training dataset for each of the plurality of objects by collecting the regions of interest for tracking the corresponding object from respective images in respective images in respective separate image sequences.

[0107] Clause 14: A method according to clauses 1 to 13, wherein the step of obtaining a separate training data set corresponding to each of the multiple objects includes eliminating false negative labels by verifying the presence of each of the multiple objects in each of the multiple training data sets against a ground truth timeline for each of the multiple objects.

[0108] Item 15: A system for detecting objects in an assembly process, the system comprising: at least one camera configured to capture multiple images; and a controller configured to: obtain an event model for each of a plurality of objects in the assembly process, wherein the event model for each of the plurality of objects includes a predetermined time range within which a change in existence occurs; collect a sequence of images of the assembly process for monitoring; use a detector model to identify whether a change in existence occurs for each of the plurality of objects; check the event model for each of the plurality of objects to determine whether a predetermined time range has elapsed without identifying a change in existence for a corresponding one of the plurality of objects; and issue an alarm if the predetermined time range has elapsed without identifying the existence of the corresponding one of the plurality of objects.

[0109] Clause 16: The system of clause 15, wherein the results from the detector model identifying the presence change for each of the plurality of objects are filtered using a state machine based on whether the object was detected in at least one preceding or succeeding image.

[0110] Clause 17: A system as described in clauses 15 to 16, wherein an event model for each of the multiple objects is generated by the following steps: receiving at least one training image sequence showing multiple occurrences of an assembly process for each of the multiple objects; using a detector model to identify the presence of each of the multiple objects in the at least one training image sequence; and generating an event model corresponding to each of the multiple objects based on statistical predictions of when the object should appear in the training image sequence.

[0111] Item 18: A non-transitory computer-readable storage medium comprising programmed instructions, which when executed by a processor are operable to perform a method comprising: obtaining an event model for each of a plurality of objects in an assembly process, wherein the event model for each of the plurality of objects includes a predetermined time range within which a change in existence occurs; collecting a sequence of images of the assembly process for monitoring; identifying, using a detector model, whether a change in existence occurs for each of the plurality of objects; reviewing the event model for each of the plurality of objects to determine whether a predetermined time range has elapsed without identifying a change in existence for a corresponding one of the plurality of objects; and issuing an alarm if the predetermined time range has elapsed without identifying the existence of the corresponding one of the plurality of objects.

[0112] Clause 19: A non-transitory computer-readable storage medium as described in clause 18, wherein an event model for each of a plurality of objects is generated by the following steps: receiving at least one training image sequence showing multiple occurrences of an assembly process for each of the plurality of objects; using a detector model to identify the presence of each of the plurality of objects in at least one image sequence; and generating an event model corresponding to each of the plurality of objects based on statistical predictions of when the object should appear in the training image sequence.

[0113] Clause 20. A non-transitory computer-readable storage medium according to clauses 18 to 19, wherein the detector model is trained by the following steps: obtaining a separate training data set corresponding to each of a plurality of objects, each separate training data set comprising a set of labeled images identifying a corresponding one of the plurality of objects; training a part-level detector based on each separate training data set; and training the detector model based on each part-level detector.

[0114] Although various embodiments are described, the description is intended to be exemplary, not limiting. It will be apparent to those of ordinary skill in the art that more embodiments and implementations are possible, which are within the scope of the embodiments. Unless specifically limited, any feature of any embodiment may be combined with or used in place of any other feature or element in any other embodiment. Therefore, the embodiments are not limited except in consideration of the appended claims and their equivalents. In addition, various modifications and changes may be made within the scope of the appended claims.

Claims

1. A method (300) for monitoring an assembly process (24), the method (300) comprising the following steps: obtaining an event model (214) for each of a plurality of objects in the assembly process (24), wherein the event model for each of the plurality of objects includes a predetermined time range within which an existence change occurs; collecting a sequence of images (38) of the assembly process (24) for monitoring (302); identifying, using a detector model, whether a presence change of each of the plurality of objects occurs (304); reviewing the event model (308) for each of the plurality of objects to determine whether the predetermined time range has elapsed without identifying a change in the presence of a corresponding one of the plurality of objects; and If the predetermined time frame elapses without identifying the presence of a corresponding one of the plurality of objects, an alarm is issued (310).

2. The method (300) according to claim 1, wherein: Results from the detector model identifying the presence changes for each of the plurality of objects are filtered using a state machine based on whether the object was detected in at least one previous or subsequent image (38).

3. The method (300) of claim 1, wherein: The step of identifying the change in presence (304) of each of the plurality of objects includes determining when each of the plurality of objects appears or disappears from the sequence of images (38).

4. The method (300) according to claim 3, wherein: The time is based on a relative time set by the change in presence of one of the plurality of objects.

5. The method (300) of claim 1, wherein: The plurality of objects includes multiple configurations of a single object.

6. The method (300) of claim 1, wherein: The step of collecting said sequence of images (38) of said assembly process is performed in real time.

7. The method (300) of claim 1, wherein: The event model of each of the plurality of objects is generated by: receiving a sequence of at least one training image (38) showing multiple occurrences of the assembly process (24) for each of the plurality of objects; utilizing the detector model (304) to identify the presence of each of the plurality of objects in the at least one sequence of training images (38); as well as The event model (304) corresponding to each of the plurality of objects is generated based on statistical predictions of when the object should appear in the sequence of training images (38).

8. The method (300) of claim 7, wherein: The statistical prediction includes a predetermined time range within which a corresponding one of the plurality of objects exists.

9. The method (300) of claim 8, wherein: The predetermined time range is based on three times the standard deviation of a mean occurrence time of a corresponding one of the plurality of objects.

10. The method (300) of claim 8, wherein: The predetermined time range includes a relative time based on a presence of one of the plurality of objects.

11. The method (300) of claim 8, wherein: The multiple occurrences of the assembly process include historical occurrences of the assembly process.

12. The method (300) of claim 7, wherein: The detector model is trained by the following steps: obtaining a separate set of training data (202, 204, 206) corresponding to each of the plurality of objects, each of the separate sets of training data (202, 204, 206) comprising a set of labeled images identifying a corresponding one of the plurality of objects; Training a part-level detector (201) based on each of the individual training data sets (202, 204, 206); as well as The detector model (220) is trained based on each of the part-level detectors.

13. The method (300) of claim 12, wherein: The separate training data sets (202, 204, 206) are created by the following steps: receiving a sequence of individual images (38) of each of the plurality of objects, identifying a corresponding one of the plurality of objects in at least one image (38) of the sequence of individual images (38); tracking each of the plurality of objects identified in the at least one image (38) in a corresponding one of the sequences of individual images (38); marking a region of interest for tracking a corresponding one of each of the plurality of objects in each image (38) of the sequence of individual images (38); as well as The separate set of training data (202, 204, 206) is created for each of the plurality of objects by collecting the region of interest for tracking the corresponding object from each image (38) in each of the sequences of separate images (38).

14. The method (300) of claim 13, wherein: The step of obtaining the separate training data (202, 204, 206) set corresponding to each of the plurality of subjects includes eliminating false negative labels by verifying the presence of each of the plurality of subjects in each of the plurality of training data (202, 204, 206) sets against a ground truth timeline for each of the plurality of subjects.

15. A system (20) for detecting an object during an assembly process, the system (20) comprising: at least one camera (20-1, 20-2), the at least one camera configured to capture a plurality of images (38); as well as A controller (32), the controller being configured to: obtaining an event model for each of a plurality of objects in the assembly process (24), wherein the event model for each of the plurality of objects includes a predetermined time range within which an existence change occurs; collecting a sequence of images (38) of the assembly process (24) for monitoring (302); identifying, using a detector model, whether a presence change of each of the plurality of objects occurs (304); reviewing the event model (308) for each of the plurality of objects to determine whether the predetermined time range has elapsed without identifying a change in the presence of a corresponding one of the plurality of objects; and If the predetermined time frame elapses without identifying the presence of a corresponding one of the plurality of objects, an alarm is issued (310).

16. The system (20) of claim 15, wherein: Results from the detector model identifying the presence changes for each of the plurality of objects are filtered using a state machine based on whether the object was detected in at least one preceding or succeeding image (38).

17. The system (20) of claim 15, wherein: The event model of each of the plurality of objects is generated by: receiving a sequence of at least one training image (38) showing multiple occurrences of the assembly process (24) for each of the plurality of objects; utilizing the detector model (304) to identify the presence of each of the plurality of objects in the at least one sequence of images (38); as well as The event model (304) corresponding to each of the plurality of objects is generated based on statistical predictions of when the object should appear in the sequence of training images (38).

18. A non-transitory computer-readable storage medium containing programmed instructions, the instructions being operable when executed by a processor to perform a method (100, 200, 300), the method comprising the steps of: obtaining an event model (214) for each of a plurality of objects in an assembly process, wherein the event model for each of the plurality of objects includes a predetermined time range within which an existence change occurs; collecting a sequence of images (38) of the assembly process for monitoring; identifying, using a detector model, whether a presence change of each of the plurality of objects occurs (304); reviewing the event model (308) for each of the plurality of objects to determine whether the predetermined time range has elapsed without identifying a change in the presence of a corresponding one of the plurality of objects; and If the predetermined time frame elapses without identifying the presence of a corresponding one of the plurality of objects, an alarm is issued (310).

19. The non-transitory computer-readable storage medium of claim 18, wherein: The event model of each of the plurality of objects is generated by: receiving a sequence of at least one training image (38) showing multiple occurrences of the assembly process for each of the plurality of objects; utilizing the detector model to identify the presence of each of the plurality of objects in the at least one sequence of images (38); as well as The event model corresponding to each of the plurality of objects is generated based on statistical predictions of when the object should appear in the sequence of training images (38).

20. The non-transitory computer-readable storage medium of claim 19, wherein: The detector model is trained by the following steps: obtaining a separate set of training data (202, 204, 206) corresponding to each of the plurality of objects, each of the separate sets of training data (202, 204, 206) comprising a set of labeled images identifying a corresponding one of the plurality of objects; Training a part-level detector based on each of the individual training data sets (202, 204, 206); The detector model (220) is trained based on each of the part-level detectors.

Images