Multi-model object detection

Abstract

The present disclosure relates to multi-model detection of objects. An object detection system is disclosed, which is configured to detect objects of interest within an image of a scene using multiple object detection models that generate sets of object detection conclusions. The object detection system is configured to implement a series of functions to reconcile any differences in its multiple sets of object detection conclusions so as to generate a set of conclusions for each perceived object of interest within a given image.

Description

Technical Field

[0001] The present disclosure relates to object detection, and more particularly, to multi-model detection of an object. Background Art

[0002] Computers have been developed to perform a variety of computer vision tasks. One such computer vision task that is particularly important in many aspects of daily life is object detection, which typically involves a computer processing a given image to identify and locate certain objects of interest within the given image.

[0003] Recently, systems capable of performing computer vision tasks have been developed to assist human screeners (e.g., security personnel at airports, museums, etc.) in identifying objects of interest (e.g., potential security threats, contraband, etc.) present on a person and / or contained in a piece of luggage and other locations where objects of interest may be present from an image of a scene at a certain location. Such systems improve upon object recognition traditionally performed by humans.

[0004] As described above, humans are traditionally tasked with identifying objects of interest within scene images, such as TSA security screeners determining whether images of scanned baggage items contain any objects deemed to be security risks (e.g., guns, knives, etc.). However, humans typically perform poorly at performing such sporadic visual searches, at least in part due to cognitive fatigue. As a result, humans typically fail to identify objects of interest in scanned scene images, which can lead to a variety of negative consequences, such as guns entering passenger trains, flights, buses, etc.

[0005] Therefore, computer systems have been developed to perform object detection on behalf of humans. In general, object detection refers to the following functions: (i) identifying potential objects of interest in a scene image, (ii) classifying the object of interest (e.g., by determining that the object belongs to the class "gun" or "sharp object"), (iii) determining the confidence of the object classification, and (iv) determining the location description of the object within the image (e.g., by defining a bounding box for the object within the image).

[0006] Recently, computer systems have become relatively good at performing object detection on "simple" scene images. However, as scene images become more "complex," existing computer systems become increasingly poor at performing object detection. For example, some existing systems may accurately perform object detection on objects of interest within an image of a scanned baggage item that contains a small number of objects arranged in a minimally overlapping manner. On the other hand, the accuracy of existing systems decreases as the number of objects within the scanned baggage item increases, the arrangement of the objects becomes more crowded, and / or the number of target object classes of interest increases.

[0007] To help address these and other issues with existing object detection systems, examples disclosed herein provide an object detection system that utilizes multiple object detection models to detect objects of interest within a scene image. Such a system provides a technical improvement over existing systems that rely on a single model to perform object detection and existing systems that rely solely on multiple models to perform the classification function of object detection (which is inherently less complex than object detection as a whole). In this regard, the new multi-model object detection system of the present disclosure can provide various technical advantages over existing systems, such as improved object detection accuracy, improved detection rates, reduced false positives, an increase in the number of object classifications that can be supported, and / or facilitating continued system enhancements.

[0008] In addition to providing technical improvements over existing object detection systems, multi-model object detection systems also present their own additional technical challenges, which are addressed by the example embodiments disclosed herein. For example, for a given image, each model of the multi-model system generates its own set of object detection conclusions (i.e., "inferences") about a particular object of interest within the given image, which may include (i) a classification "label" assigned to the particular object of interest (which may constitute all or part of a "classification conclusion"), (ii) the confidence of the classification of the particular object of interest (which may constitute all or part of a "confidence conclusion"), and (iii) a description of the location of the particular object of interest within the given image (e.g., the size and / or position of the bounding box defined by the object of interest) (which may form all or part of a "location conclusion"). A set of object detection conclusions may also include other types of conclusions, which may depend on the nature of a given object detection model.

[0009] In practice, a set of object detection conclusions from a first model of a multi-model system may at least sometimes differ from a set of object detection conclusions from a second model of the multi-model system. For example, the first model may classify a particular object of interest as a "sharp object" while the second model may classify the particular object of interest as a "gun." As another example, the first model may define a bounding box for the particular object of interest that defines an area within a given image that is different from the area of ​​the bounding box defined by the second model. Thus, a multi-model system is presented in which there is a challenge of determining whether there are multiple objects of interest within a given image and / or whether two or more competing conclusions point to the same object of interest within a given image.

[0010] These challenges may be associated with one or more "downstream" actions that may be performed by the multi-modal system based on multiple sets of object detection conclusions. For example, the multi-modal system may have to select which, if any, of multiple competing object detection conclusions to use as a basis for presenting the object detection results to a display of a security screener workstation, etc. As another example, the multi-modal system may have to select whether a scanned baggage item should be allowed to pass through a screening system, or blocked for further inspection by a baggage handler, etc., based on multiple competing object detection conclusions.

[0011] Therefore, multi-model systems are often faced with the challenge of reconciling the differences in their multiple sets of object detection conclusions in order to generate a set of conclusions for each perceived object of interest within a given image. The new multi-model object detection system of the present disclosure helps provide a technical solution to this reconciliation challenge. Summary of the invention

[0012] In an example embodiment, the multi-model object detection system is configured so that it can be coupled to one or more detection devices, such as an X-ray machine, a metal detector, an MRI scanner, a CT scanner, a millimeter wave scanner, or other types of scanning devices. Each detection device is configured to scan a scene (e.g., an item of luggage passing through a screening system, one or more individuals within a given space, etc.), and then provide such scanned images to the multi-model object detection system.

[0013] In some implementations, the multi-model object detection system is configured to be communicatively coupled to some or each of the one or more detection devices via a wide area network (WAN) such as the Internet. For example, the multi-model object detection system can be cloud-based. In other implementations, the multi-model object detection system is configured to be communicatively coupled to some or each of the one or more detection devices via a local area network (LAN). For example, the multi-model object detection system can be local to a screening system. In some cases, all or part of the multi-model object detection system can be physically coupled to some or each of the one or more detection devices, or can be integrated therein. For example, in some aspects, the multi-model object detection system can be physically attached to a screening system.

[0014] In any case, the multi-model object detection system is configured to receive images of a scanned scene from one or more detection devices and perform multi-model object detection on the received images. In some implementations, performing multi-model object detection may involve the multi-model object detection system executing two or more object detection models on each given image it receives to generate two or more sets of object detection conclusions. In other implementations, performing multi-model object detection may additionally or alternatively involve: the multi-model object detection system receives two or more sets of object detection conclusions for a given image from one or more other systems configured to execute object detection models, possibly after the multi-model object detection system first transmits the given image to one or more other systems. There are other possibilities.

[0015] In some example embodiments, the multi-model object detection system may be configured to first select two or more object detection models (e.g., where there are more than two available object detection models) that will generate an object detection conclusion for a given image, which the multi-model object detection system may implement in a variety of ways. As one possibility, the multi-model object detection system may select two or more object detection models based on a variety of factors that inform the multi-model object detection system about which two or more object detection models will be most suitable for performing object detection for a given image, such as the vendor, manufacturer, etc. that designed the model, and / or the type of training data used to train the model, among other factors. As another possibility, the multi-model object detection system may select two or more object detection models based on the output of a machine learning model (e.g., at least one neural network) that receives a given image as input and outputs an indication that informs the multi-model object detection system about which one or more object detection models will be most suitable for performing object detection for a given image, such as indicating one or more features of the image (e.g., the image is considered "complex", the image includes high-density areas, etc.). In some cases, such machine learning models may have been trained or updated based on a "continuous learning" function that will be discussed later.

[0016] In any case, each of the two or more object detection models is trained (e.g., using supervised or unsupervised learning techniques) based on the training image data to identify one or more specific objects of interest in the image. As a result of such training, each object detection model is configured to receive a given image as input and then output a set of object detection conclusions about the given image, which may include some or all of the classification conclusions, confidence conclusions, and localization conclusions, as well as other possible conclusions.

[0017] Each specific object detection model can take a variety of forms. For example, some or each of the two or more object detection models can take the form of a machine learning object detection model (e.g., a neural network-based object detection model), a rule-based object detection model (e.g., an explosive detection system model), or some other computer vision-based object detection model, as well as other forms of object detection models. In some cases, some or each of the two or more object detection models may include or otherwise take the form of one or more single-stage or multi-stage object detection models, which may also include one or more neural network-based object detection models (e.g., YOLO, SSD, AlexNet, RetinaNet, VGGNET, FPN, R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, etc.), as well as other possibilities.

[0018] It is worth noting that at least one feature of two or more object detection models may be different. For example, two or more object detection models may have been trained to identify different objects of interest (e.g., a first model is trained to identify "guns" and a second model is trained to identify "sharp objects"). As another example, two or more object detection models may have been trained to identify the same object of interest, but are trained differently, such as by using different training sets (e.g., a first model is trained for adversarial image data and a second model is trained for general image data) and / or training parameters, as well as other possibilities. As yet another example, two or more object detection models may have been trained to identify the same object of interest, but are designed by different suppliers, manufacturers, etc. As a further example, two or more object detection models may have been trained to identify the same object of interest, but have different architectures (e.g., a first model with a YOLO architecture and a second model with a FasterR-CNN architecture). It is also feasible that other example features of the two models are different.

[0019] Because two or more object detection models differ in some aspects, the two or more sets of object detection conclusions output by the two or more object detection models may include at least one conclusion that is different between the two models. For example, the classification conclusion of the first model may include a "gun" label, while the classification conclusion of the second model may include a "knife" label. As another example, the positioning conclusion of the first model may define a bounding box for the object of interest, and the area of ​​the bounding box is larger than the bounding box defined by the second model. Other examples are also feasible, some of which are discussed below.

[0020] In any case, after a multi-model object detection system identifies two or more sets of object detection conclusions for a given image, in the event that there are discrepancies between the two or more sets of conclusions, the function of the multi-model object detection system is to perform a "reconciliation" process that generally involves resolving any discrepancies between the multiple sets of object detection conclusions to arrive at a single set of conclusions for a given perceived object of interest within the given image. In practice, the multi-model object detection system may perform this process in a variety of ways.

[0021] As one possibility, the multi-model object detection system can be configured to start by determining whether two or more localization conclusions from two or more sets of object detection conclusions correspond to the same or substantially similar locations within a given image. In an example embodiment, the multi-model object detection system can perform this function by determining whether the localization descriptions from the two or more sets of object detection conclusions are comparable (e.g., by determining whether the bounding boxes defined by the two or more object detection models overlap or are adjacent to each other). Other ways of determining whether a localization conclusion corresponds to the same or substantially similar location within a given image are also possible.

[0022] When it is determined that two or more localization conclusions are the same or substantially similar, the multi-model object detection system can infer that the two or more conclusions correspond to the same perceived object of interest within a given image. For example, when multiple bounding boxes overlap (or are within a threshold distance of each other), the multi-model object detection system is configured to conclude that the multiple sets of object detection conclusions corresponding to these bounding boxes refer to the same perceived object of interest within a given image. The multi-model object detection system can also infer that two or more sets of object detection conclusions correspond to the same perceived object of interest within a given image in other ways (e.g., based on similarities between classification conclusions).

[0023] In some cases, if the multi-model object detection system determines that two or more localization conclusions from two or more sets of object detection conclusions correspond to the same or substantially similar locations within a given image (e.g., if the bounding boxes do overlap), the multi-model object detection system may determine whether one of the confidence conclusions from the two or more sets of object detection conclusions is above a predetermined threshold. If so, the multi-model object detection system may select a corresponding set of object detection conclusions over one or more other sets of object detection conclusions. Otherwise (and in some embodiments, even when the confidence conclusion is greater than a predetermined threshold), the multi-model object detection system may reconcile. In some implementations, in the event that two or more confidence conclusions are above a predetermined threshold, the multi-model object detection system may select a set of object detection conclusions corresponding to the highest confidence, or the multi-model object detection system may reconcile.

[0024] In any case, the multi-model object detection system can continue the reconciliation process by determining the degree of similarity between two or more localization conclusions, which the multi-model object detection system can determine in a variety of ways. As one possibility, the multi-model object detection system can first identify a specific point (e.g., a "vertex") in each localization conclusion, such as by finding a center point within a bounding box defined by each localization conclusion.

[0025] If two localization conclusions are substantially similar (e.g., they overlap or are close to each other), the multi-model object detection system can define a connection between the two localization conclusions. For example, the multi-model object detection system can generate a line (e.g., an "edge") connecting the identified vertices of the overlapping bounding boxes.

[0026] The multi-model object detection system can then quantify the similarity of the localization conclusions. For example, the multi-model object detection system can assign a value to each generated line connecting two bounding boxes, where the assigned value quantifies how much the bounding boxes overlap each other, for example, by determining an intersection over union (IoU) value of the bounding boxes.

[0027] Next, the multi-model object detection system can define one or more partitions (e.g., "clusters") based on the localization conclusions, such that each partition (i) contains no more than one localization conclusion from any given object detection model and (ii) maximizes the similarity between the localization conclusions. The multi-model object detection system can define one or more partitions in various ways.

[0028] As one possible approach, the multi-model object detection system can first generate an initial partition for each vertex (and therefore each bounding box). For each given initial partition, the multi-model object detection system can then determine whether the given partition should be merged with one or more other partitions.

[0029] In one particular implementation, the multi-model object detection system may perform a merge function by (i) identifying any generated lines connecting vertices of a given partition to vertices of another partition, and (ii) excluding all identified lines connecting vertices of the given partition to vertices of another partition that correspond to a bounding box defined by the same object detection model that defines the bounding box corresponding to the vertices of the given partition. Next, the multi-model object detection system may evaluate the assigned weights of any remaining identified lines and identify a merge vertex corresponding to the endpoint of the remaining line with the largest assigned weight. The multi-model object detection system may then merge the given partition with the partition corresponding to the identified merge vertex.

[0030] In operation, the multi-model object detection system may evaluate any remaining initial partitions for potential merging, but each merged partition includes its original vertices as well as any merged vertices. In this way, the multi-model object detection system defines one or more partitions, each of which refers to the same perceived object of interest within a given image.

[0031] After the multi-model object detection system defines one or more partitions based on the localization conclusions, it then selects a representative partition for a given perceived object of interest in a given image, which partition forms the basis for the multi-model object detection system to reach a single set of object detection conclusions for the given perceived object. The multi-model object detection system can select representative partitions in a variety of ways.

[0032] As one possibility, a multi-model object detection system could utilize a “voting” scheme to select representative partitions, where each object detection model provides a “vote” for each partition that contains vertices corresponding to the given model.

[0033] In an example implementation utilizing a "voting" scheme, the weight (e.g., value) of a "vote" can be the same for each object detection model (e.g., each model's "vote" has a value) or can vary between object detection models. Moreover, the weight of a "vote" for a given object detection model can be constant (i.e., predetermined) or can also change dynamically over time. In practice, the weight of a "vote" for a given object detection model can be determined based on: (i) based on various factors (e.g., the vendor, manufacturer, etc. that designed the model, the type of training data used to train the model, the architecture type of the model, etc.), (ii) based on the output of a machine learning model that receives a given image and at least a portion of the object detection conclusions of each model as input and outputs an indication of how to weight the "vote" of each model, and / or (iii) based on the results of the "continuous learning" function discussed below, among other possibilities.

[0034] After the multi-model object detection system applies the applicable weights of the "votes" of each object detection model to one or more partitions, the multi-model object detection system can then select a representative partition based on the relative weights of the one or more partitions. As an example, the multi-model object detection system can select a partition that receives a predetermined threshold amount of "votes", such as a 50% threshold, or a partition that receives the most "votes" from one or more partitions, among other possibilities.

[0035] In any case, after the multi-model object detection system selects a representative partition for a given perceived object of interest in a given image, the multi-model object detection system can complete the reconciliation process by generating and outputting a single reconciled set of conclusions for the given object of interest in the given image. In practice, the multi-model object detection system that generates a reconciled set of conclusions based on the representative partition may involve determining (i) a representative classification label, (ii) a representative confidence value, and / or (iii) a representative localization description of the given perceived object of interest in the given image (e.g., the size and / or location of a representative bounding box of the object of interest in the given image).

[0036] In the event that the multi-model object detection system determines that a given image contains any other perceived objects of interest, it will then repeat the above process for each other perceived object of interest. In any case, one of ordinary skill in the art will appreciate that the multi-model object detection system described herein more accurately detects objects of interest and has a lower false alarm rate, among other advantages, compared to existing systems.

[0037] In an example embodiment, the multi-model object detection system may be configured to perform one or more functions based on: (i) the output of one or more functions performed during the reconciliation and / or (ii) the final output of the reconciliation process as a whole (i.e., the reconciled set of object detection conclusions).

[0038] As one possibility, the multi-model object detection system may present one or more indications of one or more representative conclusions from the reconciled set of object detection conclusions on a computer display, etc. For example, the multi-model object detection system may display a graphical visualization of bounding boxes and / or labels for a given object of interest at a workstation of a security screener at a screening checkpoint, such as an airport.

[0039] Similarly, the multi-model object detection system may display one or more notifications related to the initially scanned scene on a computer display, etc., based on one or more representative conclusions in the reconciled set of object detection conclusions. For example, based on the reconciled set of object detection conclusions, the multi-model object detection system may cause a visualization of the scanned baggage item to be displayed on a display of a security screener workstation only if an object of interest within the baggage item identified in the reconciled set of object detection conclusions is deemed security-related. Otherwise, the multi-model object detection system may cause the display to display any visualization of the scanned baggage item.

[0040] As another possibility, the multi-model object detection system can cause a notification or the like to be presented at a particular computing device (e.g., a wearable computing device, a mobile phone, etc.) that is part of or associated with the screening system based on one or more representative conclusions from the harmonized set of object detection conclusions, such as sending a text message or application notification to a computing device assigned to a baggage handler at a screening checkpoint.

[0041] As another possibility, the multi-model object detection system can enable an operation to be performed in a screening system, such as a “pass” operation, based on one or more representative conclusions from a reconciled set of object detection conclusions.

[0042] For example, many screening systems include a conveyor belt on which an item of baggage, etc., moves past one or more detection devices, which then capture one or more images of the item of baggage. In conventional screening systems, a security screener's workstation includes a user interface (e.g., buttons, etc.) that controls the conveyor belt so that the security screener manually controls the flow of the item of baggage through the system. Typically, by default, the conveyor belt will only move, and thus the item of baggage will only pass through the system, when the security screener provides input to cause the conveyor belt to move. When the security screener provides such input, the item of baggage (i) passes through one or more detection devices that scan the item of baggage, and (ii) passes through a scanning area of ​​the system to a "main" channel, where the owner of the item of baggage retrieves the item. On the other hand, when a security screener identifies a potential security concern in a scanned image of a baggage item, depending on the nature and / or sophistication of the screening system, the security screener will typically either (i) stop a conveyor or otherwise refuse to pass the baggage item to a “main” lane and manually move the baggage item to a “search” lane where a baggage handler retrieves the baggage item and searches its contents, or (ii) provide an input on a user interface (e.g., a button on a screening system workstation including a “smart” lane, etc.) that diverts the baggage item to a second conveyor where a baggage handler retrieves the baggage item and searches its contents.

[0043] In an example embodiment, the multi-model object detection system, based on a coordinated set of object detection conclusions for a given image of a baggage item (or similar item), causes a pass-through operation to occur at the screening system, which may involve causing the screening system to automatically cause the baggage item to no longer pass through a "main" channel, but instead enter a "search" channel, for example, by causing an actuator or the like to automatically divert the baggage item into the "search" channel, or by diverting the baggage item onto an auxiliary conveyor belt (for example, when the screening system is equipped with or otherwise includes a "smart" channel). In this way, the multi-model object detection system can promote high throughput and improved object detection accuracy of the screening system, and can also help reduce the number of operators required at a given screening checkpoint.

[0044] Likewise, the multi-model object detection system's coordinated set of object detection conclusions based on a given image of a baggage item (or similar item) such that a pass-through operation occurs at the screening system may involve causing the screening system to automatically start or stop a conveyor belt of the screening system without control input from a human screener. For example, when the coordinated set of object detection conclusions indicates that there is no object of interest in the baggage item, the multi-model object detection system may cause the conveyor belt to automatically start (or continue to move), thereby overriding the default setting of the screening system and passing the baggage item through the system to a "main" channel for retrieval by the owner of the baggage item. On the other hand, in some security systems that, by default, have a conveyor belt that continuously transports passing baggage items until the system receives a "stop" input from a human screener, when the coordinated set of object detection conclusions indicates that there is an object of interest that is considered security-related in the baggage item, the multi-model object detection system may cause the conveyor belt to automatically stop, thereby overriding the default setting of the screening system.

[0045] It is noteworthy that in the case where the multi-model object detection system is configured to stop the conveyor belt of the screening system, the multi-model object detection system may also configure the screening system so that the ability of the human screener to continue moving the conveyor belt may be disabled until other events occur (i.e., in addition to the system receiving input at the user interface to start / stop the conveyor belt), such as the human screener providing input indicating that object detection by the multi-model object detection system has been confirmed, the human screener or his / her supervisor providing certain credentials in the screening system, etc.

[0046] Based on one or more representative conclusions from the reconciled set of object detection conclusions, the multi-model object detection system may cause various other operations to occur in the screening system.

[0047] As described above, in example embodiments, the multi-model object detection system may be configured to receive feedback data based on actions occurring at a screening system or the like, such as a screening system that scans a scene and outputs an indication of multi-model object detections by the multi-model object detection system. For example, a person at a screening system may provide input at a workstation or the like that (i) provides an indication of whether the multi-model object detections by the object detection system are correct (in whole or in part), (ii) provides an indication that the multi-model object detection system failed to detect an object of interest within an image but was detected by an operator, or (iii) provides an indication of a human's classification of an object in an image (e.g., a label of an object entered by a human that may or may not be considered security-related), among other possibilities. In practice, such input may take the form of direct feedback to the multi-model object detections by the object detection system (e.g., by a human viewing a visual representation of the multi-model object detections and then using a GUI to "rule out" a false detection, a baggage handler providing input on a computing device confirming that a baggage item contains a security threat, etc.), and / or such input may take the form of indirect feedback (e.g., a conveyor belt action taken by an operator, etc.).

[0048] Based on feedback data from actions occurring at a screening system or the like, the multi-model object detection system can be configured to update the multi-model object detection process in various ways. For example, the feedback data may generate new or updated labeled training data that can be used to update some or all of the two or more object detection models of the multi-model object detection system. As another example, the feedback data may result in modifying the probability threshold of a given object detection model (e.g., the feedback data may indicate that there are relatively frequent false positives for "sharp objects," and as a result, the probability threshold of an object detection model trained to detect "sharp objects" may be increased). Other examples are also possible. In addition, updates to the multi-model object detection process can be performed "on the fly" (e.g., after each image is processed and the applicable output is provided to the screening operator), or updates can be performed periodically (or in "batches"), such as every 24 hours, every week, etc.

[0049] Relatedly, as described above, in example embodiments, the multi-model object detection system may be configured to perform "continuous learning," which generally involves the multi-model object detection system receiving feedback data regarding the performance of two or more object detection models and then updating aspects of the multi-model object detection process based on the feedback data. In practice, such feedback data may be a result of the multi-model object detection process itself, and / or may be received as a result of actions taken by humans in response to the multi-model object detection of the object detection system. In any case, the multi-model object detection system may perform continuous learning in a variety of ways.

[0050] In one aspect, a multi-model object detection system may be configured to evaluate the performance of a given object detection model over time and to modify the weight of a "vote" of the given object detection model based on the evaluation. For example, the multi-model object detection system may determine that a set of conclusions of a first object detection model is considered correct more than a first threshold amount of times (e.g., the representative partitions historically selected by the multi-model object detection system include vertices corresponding to the first object detection model), and therefore, the multi-model object detection system may increase the weight of the "vote" of the first object detection model. On the other hand, the multi-model object detection system may determine that a set of conclusions of a second object detection model is considered correct less than the first threshold amount of times (and may be less than a lower second threshold amount), and therefore, the multi-model object detection system may reduce the weight of the "vote" of the second object detection model.

[0051] In another aspect, a multi-model object detection system may be configured to evaluate the performance of a given object detection model over time, and based on the evaluation, modify whether the given object detection model should be used in conjunction with certain other object detection models. For example, the multi-model object detection system may determine that a first object detection model's detection of a first class of objects of interest (e.g., "liquids") was deemed correct more than a threshold amount of times, and a second object detection model's detection of a second class of objects of interest (e.g., "sharps") was deemed correct more than a threshold amount of times. Thus, the multi-model object detection system may modify its multi-model object detection process such that the first object detection model and the second object detection model are not used together (or the combination is downgraded in other combinations) because each model is relatively good at detecting objects of different classifications. As another example, the multi-model object detection system may determine that detections of a first object detection model within a "side view" image of a scene historically match detections of a second object detection model within a "top view" image of the scene. Thus, the multi-model object detection system may modify its multi-model object detection process so that the first object detection model and the second object detection model are used together in a manner that the first object detection model evaluates a "side view" image of the scene, and the second object detection model evaluates a "top view" image of the scene. Other examples are also possible.

[0052] In another aspect, the multi-model object detection system can be configured to consider not only feedback data regarding the performance of a given object detection model, but also other data related to the object detection of the given object detection model (e.g., image data used in object detection, external contextual data related to the scanned scene, etc.), and based on such evaluation, update aspects of the multi-model object detection process. For example, when evaluating a "complex" image (e.g., an image of a cluttered cargo container) or evaluating an image of a high-density area (e.g., an image of a mail package containing metal), the multi-model object detection system may determine that the first object detection model is historically the most accurate, and therefore, when a given image has such features, the multi-model object detection system may first select the first object detection model and / or increase the "voting" weight of the first object detection model. As another example, when analyzing images scanned at a particular geographic location at a particular time of year, the multi-model object detection system may determine that a first object detection model is historically most accurate, and thus, when external contextual data (e.g., time of year and / or geographic location data) indicates that such conditions exist at the location of the detection device receiving the image, the multi-model object detection system may select the first object detection model first and / or increase the "voting" weight of the first object detection model. Other examples are also possible.

[0053] In some or all of the above examples of "continuous learning", the multi-model object detection system can be configured to use one or more machine learning models that are trained or otherwise updated based on feedback data by applying one or more machine learning techniques (e.g., neural networks, regression, random forests, support vector machines (SVM), naive Bayes, decision trees, dimensionality reduction, k-nearest neighbors (kNN), gradient boosting, etc.). Once trained or otherwise updated, these one or more machine learning models can be configured to receive image data of a scanned scene, as well as external contextual data that may be for the scanned scene, and then output instructions that tell the multi-model object detection system to decide which particular object detection model is most accurate for a given situation, and / or what weight should be applied to the corresponding "vote" of each model.

[0054] Therefore, in one aspect, the present invention discloses a computer-implemented method comprising: (i) receiving a scene image scanned by a detection device of a screening system, wherein the image includes an object of interest; (ii) identifying at least a first set of object detection conclusions for the image generated by a first object detection model, and a second set of object detection conclusions for the image generated by a second object detection model, wherein there is at least one conclusion difference between the first set of object detection conclusions and the second set of object detection conclusions; (iii) resolving at least one conclusion difference by: (a) determining a similarity between a first positioning conclusion from the first set of object detection conclusions and a second positioning conclusion from the second set of object detection conclusions, wherein both the first positioning conclusion and the second positioning conclusion are determined to involve the object of interest; (b) defining a plurality of candidate partitions, wherein at least one of the plurality of candidate partitions is defined based on the similarity between the first positioning conclusion and the second positioning conclusion; and (c) deriving a reconciled set of object detection conclusions for the object of interest from the first set of object detection conclusions and the second set of object detection conclusions based on a weight assigned to at least one candidate partition, thereby resolving at least one conclusion difference; and (iv) determining that the scene image contains the object of interest based on the reconciled set of object detection conclusions.

[0055] On the other hand, a computing system is disclosed herein, comprising at least one processor, a non-transitory computer-readable medium, and program instructions stored on the non-transitory computer-readable medium executable by the at least one processor to cause the computing system to perform one or more functions disclosed herein, including but not limited to the functions of the aforementioned computer-implemented method.

[0056] In another aspect, disclosed herein is a non-transitory computer-readable medium including program instructions stored thereon executable by at least one processor to cause a computing system to perform one or more functions disclosed herein, including but not limited to the functions of the aforementioned computer-implemented method.

[0057] These and many other aspects will be apparent to those of ordinary skill in the art upon reading the following disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Example system configurations are provided in which example embodiments may be implemented.

[0059] Figure 2 A simplified block diagram is provided showing some components that may be included in an object detection system.

[0060] Figure 3 A conceptual diagram of an exemplary single-stage object detection model is provided.

[0061] Figure 4A conceptual diagram of an exemplary two-stage object detection model is provided.

[0062] Figure 5 A flow chart of an exemplary multi-model object detection process is provided.

[0063] Fig. 6A A conceptual diagram of an example image of a scanned scene with example sets of object detection conclusions is provided.

[0064] Figure 6B A conceptual diagram of an example image of a scanned scene with example bounding boxes and example vertices for each bounding box is provided.

[0065] Figure 6C A conceptual diagram of an example image of a scanned scene with example bounding boxes and example generated lines is provided.

[0066] Fig.6D A conceptual diagram of an example image of a scanned scene with example bounding boxes and example candidate partitions is provided.

[0067] Fig. 6E A conceptual diagram of an example image of a scanned scene with example bounding boxes and identification lines for particular candidate partitions is provided.

[0068] Fig. 6F A conceptual diagram of an example image of a scanned scene with example bounding boxes and merged forms of specific candidate partitions is provided.

[0069] Figure 6G A conceptual diagram of an example image of a scanned scene with a reconciled set of object detection conclusions is provided. DETAILED DESCRIPTION

[0070] The disclosure presented herein refers to a number of example embodiments and examples and to the accompanying drawings. It should be understood by those skilled in the art that such references are for clarity and explanation purposes only and are not meant to be limiting. In fact, some or all of the disclosed systems, devices, and / or processes may be rearranged, combined, added, and / or omitted in a variety of ways, each of which is contemplated herein.

[0071] I. Example System Configuration

[0072] Figure 1 An example system configuration 100 is shown in which an example embodiment may be implemented. As shown, the system configuration 100 includes at its core an object detection system 110 coupled to one or more screening systems 120 via one or more links 130 .

[0073] In practice, each screening system 120 is located at a screening checkpoint at a specific geographic location, and the screening checkpoint may include multiple screening systems 120 that are communicatively coupled to each other, such as at an airport, etc. Each screening system 120 includes at least one detection device 122 that is configured to scan or otherwise capture one or more images of a scene. Each detection device 122 can take a variety of forms, including but not limited to an X-ray machine, a metal detector, an MRI scanner, a CT scanner, a millimeter wave scanner, a spectral band scanner, or other types of scanning devices.

[0074] Typically, a scene includes a two-dimensional (2D) area or a three-dimensional (3D) space that can be scanned by at least one detection device 122 at a given point in time (during which the area or space may contain one or more objects). There are a variety of scene examples, such as an area within a screening system where luggage items, cargo containers, mail packages, etc. pass through for scanning, an area within a screening system where humans stand for scanning, or an area within a public place where a group of people are scanned, and many other scene examples.

[0075] Likewise, there may be various examples of objects in the scene, such as luggage items (e.g., a wallet, a briefcase, a piece of luggage, a backpack, etc.), a person, a freight or cargo container, and mail or other packages, among many other examples. In practice, objects in a scan scene often themselves contain one or more objects (e.g., electronic devices, wallets, keys, books, clothing, food, lighters, contraband, etc.), and some objects may be considered objects of interest at a particular screening system 120.

[0076] Generally, an object of concern is an object that has one or more characteristics of an object that is considered to be of a type or classification of concern (e.g., a security threat), and therefore the object should be given special attention. Some examples of objects of concern may include guns, sharp objects (e.g., knives, etc.), liquids, explosives, lighters, certain electronic devices, and objects that are traditionally not of concern but have been modified in some way, as well as many other examples.

[0077] In operation, each screening system 120 is assigned one or more different objects of interest, and those one or more objects of interest may depend on the type, nature, and / or location of the particular screening system 120, as well as other factors. Furthermore, the objects of interest for a particular screening system 120, once defined, may be static or dynamically updateable. For example, the one or more objects of interest for a given screening system 120 may change as the security threat level, etc., of a geographic area near the given screening system 120 changes.

[0078] In any case, the object detection system 110 may receive and store data reflecting the corresponding one or more objects of interest for each particular screening system 120. In an example embodiment, the data may indicate one or more object detection requirements for each particular screening system 120, such as a particular set of objects of interest and / or particular criteria for marking these objects of interest (e.g., minimum confidence, etc.). In practice, the object detection system 110 may receive the data from the screening system 120 itself and / or another system communicatively coupled to the object detection system 110.

[0079] As mentioned above, each detection device 122 is configured to capture one or more images of a scene. In some implementations, detection device 122 can capture a single image of a scene. In other instances, detection device 122 can capture a plurality of different images of a scene. Multiple images can be from identical single viewing angle or from a plurality of different viewing angles (e.g., a top view image and a side view image of a given luggage case). In some cases, one or more images can include a three-dimensional "slice" of a scene, wherein each slice represents the scanning of a scene performed with different scanning depth levels.

[0080] In fact, each detection device 122 is configured to represent one or more captured images thereof using one or more data representations. For example, pixels, voxels, polygons, three-dimensional slices or any other type of data structure that can be used to construct 2D or 3D image data can be used to represent the captured image. Each of the one or more captured images can include a low-level representation of the scanned scene or a high-level representation of the scanned scene, as well as other possibilities. In some cases, the captured image can be represented according to a standard format, such as a secure digital imaging and communications (DICOS) format, as well as other possibilities. In addition, each detection device 122 can be configured to generate metadata for a given captured image describing various aspects of a scene, such as corresponding to part or all of a given captured image's material density, geometric dimensions, atomic number and / or spectral data, as well as other aspects.

[0081] After the detection device 122 captures one or more images of the scene, the device can then output the image data of the captured one or more images to various devices. As a possibility, the detection device 122 can provide the image to one or more output devices 124 of the screening system 120 (the detection device 122 is a part of the system). Typically, each output device 124 of the screening system 120 takes the form of a computing device, which is configured to provide visual, auditory, tactile and / or other sensory outputs in response to one or more functions related to object detection described herein. In this regard, each output device 124 can include at least one of a display, a speaker, a vibration mechanism and some other user output components, as well as other components. The example of the output device 124 includes a fixed computing device (e.g., a computer as a part of a workstation of the screening system 120), and a mobile computing device (e.g., a smart phone, a tablet computer, a wearable computing device) coupled to the screening system 120 via a wireless network communication, and other possibilities.

[0082] As another possibility, the detection device 122 may transmit the image data of the captured one or more images to the object detection system 110 via one or more links 130. In some instances, the object detection system 110 is communicatively coupled to part or all of the screening system 120 via a wide area network (WAN) such as the Internet, and thus, the one or more links 130 may take the form of one or more wired and / or wireless communication paths that facilitate WAN communications. In other instances, the object detection system 110 is communicatively coupled to part or all of the screening system 120 via a local area network (LAN), and thus, the one or more links 130 may take the form of one or more wired and / or wireless communication paths that facilitate LAN communications. In other instances, the object detection system 110 is communicatively coupled to part or all of the screening system 120 via a combination of one or more WANs and one or more LANs.

[0083] In some cases, all or a portion of object detection system 110 is physically coupled to part or all of one or more screening systems 120, such as via a serial bus or other connection mechanism. In this regard, all or a portion of object detection system 110 may be integrated with one or more screening systems 120. In other cases, all or a portion of object detection system 110 is located remotely from one or more screening systems 120. For example, object detection system 110 or a portion thereof may be implemented in the cloud, while each screening system 120 may be located at an airport, a train station, a bus station, a mail sorting facility, a freight center, etc., among other possibilities. In any of the above arrangements, the object detection system 110 can be configured to interface with a screening system 120 that may be manufactured by a third party (i.e., a manufacturer different from the manufacturer of the object detection system 110 itself), such as an astrophysics screening system, a Rapiscan screening system (e.g., current Rapiscan systems include Rapiscan 618XR HP, 620XR HP, 622XR HP, 618XR, 620XR, 620DV, or 622XR), or a Smiths screening system, among other examples.

[0084] II. Example Object Detection System

[0085] Generally, object detection system 110 is configured to perform object detection functions based on scene images from one or more screening systems 120. In example embodiments, object detection may involve some or all of the following functions: (i) identifying an object of interest perceived within an image of a scene, (ii) classifying the perceived object of interest (e.g., by determining that the object belongs to a class of "gun" or "sharp object") (iii) determining a confidence level for the classification of the perceived object of interest, and (iv) determining a localization description of the perceived object of interest within the image (e.g., by defining a bounding box for the perceived object of interest within the image).

[0086] Figure 2 1 is a simplified block diagram illustrating some of the components that may be included in object detection system 110. In practice, object detection system 110 may include one or more computer devices that act together to perform object detection, and these one or more computer devices may collectively include at least one processor 111, data storage 112, at least one graphics processing unit (GPU) 114, and at least one communication interface 115, all of which may be connected via a communication link 116 in the form of a system bus or some other connection mechanism.

[0087] Processor 111 may include one or more processor components, such as one or more general-purpose processors, one or more special-purpose processors, and / or one or more programmable logic devices, as well as other types of processors. In some cases, processor 111 may include one or more central processing units (CPUs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or any combination thereof. Processor 111 may also take various other forms.

[0088] Data storage 112 may include one or more non-transitory computer-readable storage media, such as volatile storage media (e.g., random access memory, registers, cache, etc.) and non-volatile storage media (e.g., read-only memory, hard drive, solid-state drive, flash memory, optical storage device, etc.), among other examples. Data storage 112 includes software that enables object detection system 110 to perform the functions disclosed herein. In this regard, the software may take the form of one or more sets of program instructions executable by processor 111 to cause object detection system 110 to perform the functions disclosed herein.

[0089] In addition, if Figure 2 As shown, the data storage 112 may be equipped with one or more object detection models 113a, 113b. In an embodiment where the data storage 112 is equipped with two or more object detection models 113a to 113b, the object detection system 110 may be configured as a multi-model object detection system. Figure 2 The object detection system 110 shown in FIG. 1 only includes two object detection models 113 a to 113 b , but this should not be construed as limiting. In fact, it should be understood that the object detection system 110 may include more than two object detection models.

[0090] Typically, each of the object detection models 113a to 113b is configured to receive a given image of a scanned scene, evaluate the given image for one or more objects of interest, and then generate one or more sets of object detection conclusions for the given image. Each set of object detection conclusions may include one or more conclusions (i.e., "inferences") about a given perceived object of interest within a given image, which may include (i) a classification "label" assigned to the given perceived object of interest (which may constitute all or part of a "classification conclusion"), (ii) a confidence in the classification of the given perceived object of interest (which may constitute all or part of a "confidence conclusion"), and / or (iii) a description of the location of the given perceived object of interest within the given image (e.g., the size and / or position of an appropriate bounding box (which may be defined by a set of coordinates, etc.) for the given perceived object of interest) (which may constitute all or part of a "location conclusion"). A set of object detection conclusions may alternatively or additionally include one or more other types of object detection conclusions, which may depend on the nature of the given object detection model that generated the set of object detection conclusions. In practice, a given object detection model 113a-113b will sometimes generate multiple sets of object detection conclusions for the same perceptually interesting object within a given image.

[0091] Each object detection model 113a to 113b can take a variety of forms. For example, some or each of the object detection models 113a to 113b can take the form of a machine learning object detection model, a rule-based object detection model, or some other form of computer vision-based object detection model, as well as other possibilities. In an example implementation, each of the object detection models 113a to 113b can take the form of one or more single-stage or multi-stage object detection models. Similarly, each of the object detection models 113a to 113b can take the form of one or more neural network-based object detection models, each of which is designed according to a specific architecture type (e.g., YOLO, SSD, AlexNet, RetinaNet, VGGNET, FPN, R-CNN, Fast R-CNN, Faster R-CNN, Mask R-CNN, etc.). Additional functional details about the object detection models 113a to 113b are discussed below.

[0092] In practice, two or more object detection models 113a to 113b may differ in at least one feature. For example, two or more object detection models 113a to 113b may have been trained to identify different objects of interest (e.g., object detection model 113a is trained to identify "guns" and object detection model 113b is trained to identify "sharp objects"). As another example, two or more object detection models 113a to 113b may have been trained to identify the same object of interest, but are trained differently, such as by using different training image sets (e.g., object detection model 113a is trained for adversarial image data and object detection model 113b is trained for general image data) and / or training parameters, among other possibilities. As yet another example, two or more object detection models 113a to 113b may have been trained to identify the same object of interest, but are designed by different suppliers, manufacturers, etc. As a further example, two or more object detection models 113a-113b may have been trained to recognize the same object of interest, but with different architectures (e.g., object detection model 113a may have a YOLO architecture, and object detection model 113b may have a Faster R-CNN architecture). Additionally, other example features that may differ between the two models are possible.

[0093] Because two or more object detection models 113a to 113b differ in some respects, the two or more sets of object detection conclusions output by the two or more object detection models 113a to 113b may at least sometimes include one or more conclusions that are different between the two models. For example, the classification conclusion of the object detection model 113a may include a label of "gun", while the classification conclusion of the object detection model 113b may include a label of "knife". As another example, the positioning conclusion of the object detection model 113a may define a bounding box for the object of interest, the area of ​​which is larger than the bounding box defined by the object detection model 113b. Other examples are also possible. As described below, the object detection system 110 is configured to resolve any such differences in multiple sets of object detection conclusions output by two or more object detection models 113a to 113b, and / or multiple sets of object detection conclusions output by a single object detection model.

[0094] The GPU 114 of the object detection system 110 may include one or more graphics processing units, which may be configured alone and / or together with the processor 111 to train the object detection models 113a to 113b and / or execute such trained object detection models 113a to 113b. In an example implementation, the GPU 114 may take the form of a GPU including multiple parallel and pipelined single instruction multiple data (SIMD) units (which may be configured to perform mathematical operations simultaneously). Examples of such GPUs may include GPUs manufactured by AMD, NVIDIA, Intel, etc. In some implementations, the GPU 114 may include one or more processors specifically designed to perform machine learning, such as one or more Google's tensor processing units (TPUs), as well as other examples. The GPU 114 may also take a variety of other forms.

[0095] The communication interface 115 may generally be configured to facilitate communication between the object detection system 110 and a data source (e.g., Figure 1 122) and an output system (e.g., Figure 1 In an example implementation, such data communications may be wireless and / or wired communications. To this end, the communication interface 115 may take any suitable form for facilitating these data communications, such as a serial bus interface (e.g., USB, FireWire, etc.), an Ethernet interface, a wireless chipset and / or antenna configured for wireless communication, and / or a proprietary wireless and / or wired communication interface, among other possibilities.

[0096] In operation, communication interface 115 enables object detection system 110 to output data to various recipients as a result of performing various object detection functions. For example, object detection system 110 may cause output device 124 of a given screening system 120 to display visualizations via one or more links 130 as a result of object detection system 110 performing multi-model object detection to enable human security screeners to analyze a particular baggage item, etc.

[0097] III. Example Object Detection Model Functionality

[0098] As mentioned above, Figure 2 Each object detection model 113a-113b in can take a variety of forms. In an example embodiment, each of the object detection models 113a-113b is defined based on one or more machine learning techniques and is designed according to a specific model architecture that can take a variety of forms. The following discussion provides two example types of object detection models that can be used in example implementations, but many other types are possible.

[0099] Figure 3A conceptual illustration of an exemplary single-stage object detection model 300 in the form of a single-stage neural network is provided. At a high level, the single-stage object detection model 300 is configured to receive image data 301 as input and then locate and classify any objects of interest within the image data (which the single-stage object detection model 300 was previously trained to detect).

[0100] More specifically, the single-stage object detection model 300 includes a series of layers 302a to 302n (collectively referred to as "layers" 302), each of which can take the form of a convolutional layer, a residual layer, and / or a pooling layer. In an example implementation, some or all of the layers 302 are convolutional layers, each of which is configured to (i) receive a set of input values ​​(e.g., an array of pixel values, a set of voxel values, etc.) based on the original input image data 301 (e.g., for layer 302a, a set of input values ​​corresponds to the original input image data 301; for deeper layers (302b to 302n), a set of input values ​​corresponds to the output of the previous layer), and (ii) apply one or more "kernels" to the set of input values ​​(wherein each "kernel" includes a set of weights and operates as a filter) to convolve across the set of input values ​​to generate a "feature map" (also called an "activation map") that provides a representation of certain features in the original input image data 301 identified by a given layer.

[0101] In practice, each layer 302 may have a different associated set of one or more kernels, which may depend on the features that a given layer 302 is trained to recognize. For example, layer 302a may include a first set of one or more kernels to facilitate recognition of low-level features (e.g., simple edges or curves), while a deeper layer may include a second set of one or more kernels to facilitate recognition of more complex features (e.g., blades, etc.).

[0102] Typically, a set of weights for each given kernel is "learned" during a training phase, during which (i) training images labeled as containing objects of a particular classification (also referred to as "ground truth" images) are trained and (ii) corresponding localization descriptions for these objects (e.g., bounding boxes identifying the locations of the objects within the training images) are propagated through the single-stage object detection model 300. The single-stage object detection model 300 typically learns the weights for each given kernel by minimizing a function. As an example of such a function, a loss function can be used that either or both of (i) penalizes incorrectly classified objects and / or incorrectly localized bounding boxes, and (ii) affirmatively reinforces correctly classified objects and / or correctly localized bounding boxes. Other examples of loss functions are also possible.

[0103] As described above, to apply a given kernel, the single-stage object detection model 300 performs convolution on a set of input values ​​input to a given convolutional layer 302. The convolution process generally involves (i) identifying a subset of values ​​(e.g., a subset of neighboring pixel values ​​or feature map values) from a set of values ​​provided as input to a given layer 302, (ii) multiplying the identified subset of values ​​by a given kernel value (e.g., performing element-wise multiplication), (iii) summing the results of these multiplications, and (iv) repeating these steps for each subset of values ​​in the set of values ​​provided as input to the given layer 302, thereby generating a set of summed values ​​referred to as "feature maps" because the output values ​​indicate or otherwise contain information about one or more features detected by the given layer 302. As such, some or each of the convolutional layers 302 may be considered a "feature extractor" or "feature extraction layer."

[0104] In an example implementation, one or more layers of the single-stage object detection model 300 are designed so that the feature map generated by a given convolutional layer is used as the input of a subsequent convolutional layer. In other implementations, one or more layers of the single-stage object detection model 300 are designed so that the feature map generated by a given convolutional layer is used as the input of a classification layer.

[0105] exist Figure 3 In the illustrated case, the feature map generated by feature extraction layer 302a is the input to a subsequent feature extraction layer 302b, which in turn may further apply a convolution operation to the input feature map to generate a feature map indicative of more complex features than the feature map of layer 302. The process of generating a feature map and passing the generated feature map through subsequent feature extraction layers may continue to a final layer 302n.

[0106] The classification layer is typically configured to generate a probability distribution indicating the probability that a particular portion of the original input image 301 defined by a given proposed positioning conclusion (e.g., a candidate bounding box) contains an object belonging to a particular object classification. In practice, a set of one or more candidate bounding boxes (e.g., "anchor boxes" or "priors") guide the search for the original input image 301, such as a set of anchor boxes with corresponding aspect ratios (e.g., 1:1, 1:2, 2:1, and 1:3). For each spatial pixel in a given feature map, these layers can be configured to output corresponding bounding box refinements and corresponding respective classification confidence values ​​for a set of one or more candidate bounding boxes.

[0107] Thereafter, the single-stage object detection model 300 may generate an embedding 304 representing one or more features detected by the layer 302 within the raw input image data 301. The embedding 304 may take the form of a multi-dimensional vector, with each dimension representing a given feature detected within the raw input image data 301. From the embedding 304, the single-stage object detection model 300 may generate a final probability distribution 306 including one or more confidence values, each representing the probability that a portion of the raw input image data 301 corresponding to a given bounding box includes an object belonging to a particular object classification.

[0108] The single-stage object detection model 300 is then able to provide a set of object detection conclusions about a given image, which may include some or all of a localization conclusion, a confidence conclusion, and a classification conclusion, as well as other possible conclusions. For example, (i) a localization conclusion may correspond to a given bounding box, which may be defined by one or more coordinates and / or one or more dimensions within the original input image data 301, (ii) a confidence conclusion may correspond to an output probability (e.g., from a probability distribution 306), indicating the probability that a portion of the original input image data 301 corresponding to a given bounding box includes an object belonging to a specific object classification, and (iii) a classification conclusion may correspond to a classification label, etc., which corresponds to a specific object classification associated with the confidence conclusion.

[0109] Now go to Figure 4 , provides a conceptual diagram of an exemplary two-stage object detection model 400 including at least two neural networks. At a high level, the two-stage object detection model 400 is conceptually similar to Figure 3 The exemplary single-stage object detection model 300 of FIG. 4 is different in that the two-stage object detection model 400 includes an additional neural network (referred to as a region proposal network (RPN)) that performs functions not performed in the single-stage object detection model 300.

[0110] More specifically, the two-stage object detection model 400 includes a first series of layers configured to perform an initial stage of feature extraction. For example, the initial stage of feature extraction 402 receives image data 401 as input, propagates the image data 401 through one or more layers that may take the form of convolutional, residual, and / or pooling layers, and outputs an initial set of one or more sets of feature maps that are provided to an RPN 404 and a region of interest (ROI) pooling layer 406.

[0111] The RPN 404 is configured to receive an initial set of one or more feature maps and identify regions of interest in the original input image data 401 that are most likely to contain an object that the RPN is trained to recognize. The RPN 404 outputs one or more "region candidates" and their accompanying scores, where (i) each region candidate takes the form of a candidate localization conclusion (e.g., a candidate bounding box) that identifies a potential object of interest, and (ii) each accompanying score (i.e., a "region candidate score") takes the form of a value that indicates the probability that the candidate localization conclusion (e.g., a candidate bounding box) contains the object of interest (i.e., a measure of the "confidence" of the RPN 404). In other words, a given region candidate score (e.g., a value between 0 and 1) can represent the degree to which a given region of the image 401 defined by a candidate bounding box (e.g., an anchor box) encloses an object of interest. In practice, these region candidate scores may be relatively agnostic to the enclosed object classification.

[0112] After the RPN 404 identifies one or more region candidates, it passes information about the one or more region candidates to subsequent layers, which helps the two-stage object detection model 400 perform additional feature extraction and classification on the identified one or more region candidates. For example, the ROI pooling layer 406 takes as input (i) information about the one or more region candidates and (ii) an initial set of one or more feature maps from the initial stage 402 of feature extraction, and then reduces the number of region candidates to a more manageable number for further analysis by subsequent layers of the two-stage object detection model 400.

[0113] More specifically, in an example embodiment, the ROI pooling layer 406 may take some or all of the region candidates and reshape each corresponding region candidate into two or more fixed-size pooling portions. Thereafter, the ROI pooling layer 406 may convert (e.g., "pool") the feature map values ​​of each of the two or more fixed-size pooling portions (e.g., using a maximum pooling function) to obtain a representative feature map value for each fixed-size pooling portion, which may take the value of the maximum value in a given pooling portion. In this manner, the ROI pooling layer 406 is used to downsample the feature map values ​​from the initial stage 402 of feature extraction. In an example implementation, the ROI pooling layer 406 may generate a ROI vector by inputting a set of representative feature map values ​​into one or more fully connected layers (not shown).

[0114] The output of the ROI pooling layer 406 then enters a "second" stage of feature extraction 408, which is typically used to (i) determine whether any fixed-size pooled portion contains an object that belongs to the object class (the second stage of feature extraction 408 is trained to recognize), and (ii) perform localization conclusion (e.g., bounding box) regression. In an example implementation, the second stage of feature extraction 408 includes or otherwise takes the form of a support vector machine (SVM), a softmax classifier, or another form of classifier.

[0115] In any case, the second stage of feature extraction 408 can generate one or more of the following as outputs (i) precise localization conclusions 410 (e.g., precise bounding boxes), each corresponding to a determination by the second stage of feature extraction 408 that a given localization description (e.g., a given set of bounding box sizes and / or positions) is likely to contain an object belonging to a particular object classification, and (ii) corresponding confidences 412 (e.g., confidence conclusions), each confidence 412 indicating a measure of the probability that a given precise localization conclusion contains an object belonging to a particular object classification, rather than merely including background imagery. Based on the one or more outputs, the object detection model 400 can identify respective classification labels corresponding to the respective precise localization conclusions, which can form all or part of the classification conclusions of the model 400. In practice, the object detection model 400 can identify classification labels based on an index, etc., each entry of which maps to a given label, such as "sharp object", "knife", "gun", etc.

[0116] Additional details regarding an example object detection model are disclosed in U.S. Patent Application No. 15 / 800,609, the contents of which are incorporated herein by reference in their entirety.

[0117] As previously described, forms of object detection models other than neural network-based object detection models may be utilized. For example, one or more of the two or more object detection models 113a to 113b may take the form of a rule-based object detection model. In a specific example, the rule-based object detection model may receive data corresponding to a scanned scene as input data, determine one or more features of the scanned scene (e.g., density and / or atomic, elemental, and / or chemical structure) from the data, and output one or more object detection conclusions based on (i) one or more determined features and (ii) one or more predefined rules that form the basis of the rule-based object detection model. In this regard, the object detection system 110 may define a rule-based object detection model by identifying one or more corresponding values ​​(e.g., a single value or a range of values) for one or more scanned features, which values ​​indicate the classification of one or more objects of interest, which may then be embodied as a set of one or more rules that form the basis of the rule-based object detection model. For example, one possible example of a rule-based object detection model may be configured to designate a portion of a scanned scene as an object of interest if the density of the portion exceeds a threshold density value (e.g., the portion is considered a "high density" area) and the portion has a particular atomic structure. Other examples are also possible.

[0118] Regardless of the object detection system 110 used Figure 2 Regardless of the type and / or architecture of the object detection models 113a to 113b, each of the object detection models 113a to 113b is configured to receive data corresponding to the scanned scene (e.g., image data) as input, and then output a set of object detection conclusions based on the received data, which may include part or all of classification conclusions, confidence conclusions, and positioning conclusions, as well as other possible conclusions.

[0119] IV. Blending Operation Examples

[0120] As discussed previously, humans are tasked with identifying objects of interest in images of a scene, such as a security screener determining whether an image of a scanned baggage item contains any objects considered to be a security risk (e.g., guns, knives, etc.). However, humans are generally poor at performing such sporadic visual searches, at least in part due to cognitive fatigue, and they often fail to identify objects of interest in images of scanned scenes, which can lead to a variety of negative consequences, such as guns entering passenger trains, flights, buses, etc.

[0121] Therefore, computer systems have been developed to perform object detection on behalf of humans. In fact, in recent times, such computer systems have become relatively good at performing object detection on images of "simple" scenes. However, as scene images become more "complex", existing computer systems have become increasingly poor at performing object detection. For example, some existing systems can accurately detect objects of interest within an image of a scanned scene that contains a small number of objects arranged in a manner with minimal overlap. On the other hand, the accuracy of existing systems decreases as the number of objects within the scanned scene increases, the arrangement of the objects becomes more crowded or chaotic, and / or the number of target object classifications of interest increases.

[0122] To help address these and other issues with existing object detection systems, examples disclosed herein provide an object detection system 110 that utilizes multiple object detection models to detect objects of interest within a scene image. The system provides a technical improvement over existing systems that (i) rely on a single model to perform object detection, and (ii) rely solely on multiple models to perform the classification functions of object detection (which are inherently less complex than object detection as a whole) (e.g., determining a classification label and confidence for a given object of interest). In this regard, the object detection system 110 of the present disclosure can provide various technical advantages over existing systems, such as improved object detection accuracy, improved detection rates, reduced false positives, an increase in the number of supported object classifications of interest, and / or facilitate continued system enhancement, among other advantages.

[0123] In addition to providing technical improvements over existing object detection systems, multi-model object detection systems also present their own additional technical challenges that are addressed by the object detection system 110 disclosed herein. For example, for a given image, each model of the multi-model object detection system generates one or more of its own sets of object detection conclusions about a given perceptually interesting object within the given image, where each such set may include some or all of the following: (i) a classification conclusion, including a classification label assigned to the given perceptually interesting object, (ii) a confidence conclusion, including a confidence in the classification of the given perceptually interesting object, and (iii) a localization conclusion, including a description of the location of the given perceptually interesting object within the given image (e.g., a description of the size and / or position of a bounding box of the given object defined by a set of coordinates), among other possible conclusions.

[0124] In fact, the first model (e.g. Figure 2 A set of object detection conclusions for a given image in the object detection model 113a) of the first model (e.g., Figure 2Therefore, the object detection system 110 is generally faced with the challenge of determining (i) whether there are multiple objects of interest in a given image, and / or (ii) whether two or more competing conclusions point to the same object of interest in a given image. In other words, the object detection system 110 is faced with the challenge of reconciling the differences in its multiple sets of object detection conclusions in order to generate a set of conclusions for each perceived object of interest within a given image. The object detection system 110 of the present disclosure is configured to provide a technical solution to this challenge by performing one or more operations disclosed herein.

[0125] For illustrative purposes only, Figure 1 Example operations that may be performed to resolve object detection discrepancies are described in the context of an example system configuration 100 of FIG. 1 , wherein an object detection system 110 is configured as a multi-model object detection system and is configured to perform example reconciliation operations.

[0126] To assist in describing some of these operations, reference may be made to a flow chart to describe a combination of operations that may be performed. In some implementations, each flow chart block diagram may represent a module or portion of a program code, and the program code includes instructions that may be executed by one or more processors to implement a specific logical function or step in a process. The program code may be stored on any type of computer-readable medium, such as a non-transitory computer-readable medium. In other implementations, each box may represent a circuit that is wired to perform a specific logical function or step in a process. In addition, based on a particular embodiment, the boxes depicted in the flow chart may be rearranged in a different order, combined into fewer boxes, separated into additional boxes, and / or removed.

[0127] Now go to Figure 5500, the object detection system 110 is configured to implement an exemplary multi-model object detection process, which generally involves (i) receiving an image of a scanned scene from a particular screening system at block 502, (ii) identifying two or more sets of object detection conclusions for the received image at block 504, (iii) determining similarity between at least a first localization conclusion and a second localization conclusion (determined to refer to the same object of interest within the received image) at block 506, (iv) defining a plurality of candidate partitions based on at least the first localization conclusion and the second localization conclusion at block 508, (v) selecting a representative partition from the plurality of candidate partitions at block 510, (vi) deriving a reconciled set of object detection conclusions for the object of interest based on the selected representative partition at block 512, (vii) determining whether the received image includes an object of interest for the particular screening system based on the reconciled set of object detection conclusions at block 514, and (viii) performing one or more operations at block 516 in response to at least the derivation of block 512. Each of these functions will be described in further detail below.

[0128] At block 502, object detection system 110 receives a particular image corresponding to a scene scanned by one or more detection devices of a particular screening system. In an example implementation, object detection system 110 receives the particular image in a data transmission from screening system 120 via one or more links 130, which may take the form of one or more wireless and / or wired communication paths. In operation, object detection system 110 may receive data representing the particular image in various forms, such as reference images. Figure 1 A form of discussion.

[0129] At block 504, the object detection system 110 identifies two or more sets of object detection conclusions for the received image. In an example embodiment, the two or more sets of object detection conclusions are generated by two or more object detection models (e.g., Figure 2 In some cases, a single object detection model (e.g., object detection model 113a or 113b) may generate two or more sets of object detection conclusions at a time. Other examples are also possible.

[0130] In some example embodiments, before identifying two or more sets of object detection conclusions for a received image, the object detection system 110 is configured to first select two or more object detection models (e.g., if there are more than two object detection models) that will generate object detection conclusions for the received image. In practice, the object detection system 110 may perform this function in a variety of ways.

[0131] As one possibility, object detection system 110 may select two or more object detection models based on a variety of factors that inform object detection system 110 as to which two or more object detection models will be best suited to perform object detection for a received image, such as the vendor, manufacturer, etc., that designed the models, and / or the type of training data used to train the models, among other factors. For example, a first object detection model may be prioritized over another model because the designer of the first model has historically created more accurate models than the designer of the other model. As another example, a first object detection model may be prioritized over another model because the training data used to train or otherwise update the first model is superior to the training data used to train or otherwise update the other model. Other examples are also possible.

[0132] As another possibility, the object detection system 110 may select two or more object detection models based on the output of a machine learning model (e.g., one or more neural networks) that receives as input a received image and outputs one or more indications that inform the object detection system 110 as to which two or more object detection models are best suited to perform object detection on a particular received image. For example, the one or more indications may provide an indication of one or more features of a particular received image (e.g., the received image is considered "complex", the received image includes a high density area, etc.), which the object detection system 110 may use as a basis for selecting two or more object detection models that generally perform well in object detection for images containing those one or more features. As another example, the one or more indications may be a corresponding probability that an object detection model available to the object detection system 110 will accurately perform object detection on a particular received image. Other examples are also possible. In some cases, such a machine learning model may be trained or otherwise updated based at least in part on a "continuous learning" function discussed later.

[0133] In any case, after the object detection system 110 has identified two or more object detection models, the object detection system 110 can then identify one or more sets of object detection conclusions for the received image from each of these models in various ways. As one possibility, the object detection system 110 itself includes the two or more object detection models and executes each model using the received image to generate two or more sets of object detection conclusions.

[0134] As another possibility, the object detection system 110 may rely on one or more other systems communicatively coupled to the object detection system 110 to perform some or all of two or more object detection models using the received image, and then receive corresponding portions of two or more sets of object detection conclusions therefrom. In some such implementations, the object detection system 110 may first send to one or more other systems: (i) the received image, and (ii) identify some or all of the two or more object detection models that will analyze the received image, which will then facilitate the one or more other systems to generate two or more sets of object detection conclusions. There are other possibilities, such as the object detection system 110 operating in conjunction with another system to generate two or more sets of object detection conclusions.

[0135] In any case, as described above, each of the two or more object detection models is trained to recognize one or more objects of interest and is configured to generate one or more sets of object detection conclusions for a received image. In this regard, at least sometimes, any given object detection model may output multiple sets of object detection conclusions for the same perceived object of interest within a given image.

[0136] It is worth noting that two or more object detection models differ in at least one feature, such as Figure 2 One of the features discussed. As a result, two or more sets of object detection conclusions output by two or more object detection models may include at least one different conclusion.

[0137] To illustrate, Fig. 6A A conceptual illustration of an example image 600 of a scanned scene with four sets of object detection conclusions 610, 620, 630, 640 is provided. In this example, image 600 may be a scan of luggage passing through a TSA screening system at an airport. Here, multiple sets of object detection conclusions 610 and 640 are generated by a first object detection model (e.g., "model A") (represented by the same dashed bounding box), while multiple sets of object detection conclusions 620 and 630 are generated by a second object detection model (e.g., "model B") (represented by the same dashed bounding box). As shown, each of the four sets of object detection conclusions 610 to 640 includes the same classification conclusion (e.g., "knife"), different positioning conclusions (e.g., as indicated by the different positions and sizes of the four bounding boxes), and different confidence conclusions (e.g., 32% vs. 39% vs. 40% vs. 45%). In other words, although there is no difference between the classification conclusions, there is a difference between the positioning conclusions and the confidence conclusions.

[0138] The object detection system 110 operates to resolve the differences between the multiple sets of object detection conclusions to arrive at a single reconciled set of conclusions for each perceived object of interest within the received image. In practice, the object detection system 110 may perform such operation in a variety of ways, examples of which are provided by Figure 5 Boxes 506 to 512 represent.

[0139] In some implementations, the object detection system 110 system may proceed to perform the reconciliation function described below only if two or more classification conclusions from two or more sets of identified object detection conclusions are the same or substantially similar. In other words, the object detection system 110 system may be configured to infer, based on their respective classification conclusions, that the two or more sets of identified object detection conclusions correspond to the same perceived object of interest within the received image. For example, if the first classification label is "knife" and the second classification label is also "knife", the object detection system 110 system may proceed to perform the reconciliation function. As another example, if the first classification label and the second classification label are related (e.g., the first classification label is "knife" and the second classification label is "sharp object"), the object detection system 110 system may proceed to perform the reconciliation function.

[0140] On the other hand, in some implementations, when two or more classification conclusions from two or more sets of identified object detection conclusions are not identical or substantially similar (e.g., the first classification label is "knife" and the second classification label is "gun"), the object detection system 110 system may abandon the conciliation function described below and instead select a set of object detection conclusions that are considered to have the highest threat from the two or more sets of identified object detection conclusions. For example, since the objects of interest in the "gun" class are more threatening than the objects of interest in the "knife" class, a set of object detection conclusions with a classification label of "gun" will be selected as a set of object detection conclusions to be reconciled. There are other possibilities.

[0141] At block 506, the object detection system 110 determines a similarity measure between at least a first positioning conclusion and a second positioning conclusion from two or more sets of identified object detection conclusions, wherein the first positioning conclusion and the second positioning conclusion are determined to refer to the same object of interest in the received image. In practice, the object detection system 110 can perform this function in a variety of ways.

[0142] As one possibility, the object detection system 110 may first determine whether two or more localization conclusions from two or more sets of identified object detection conclusions correspond to the same or substantially similar locations within the received image. In an example embodiment, this determination may involve determining whether the localization descriptions are comparable, for example, by determining whether the defined bounding boxes overlap or are adjacent to each other. Other ways of determining whether the localization conclusions correspond to the same or substantially similar locations within the image are also possible.

[0143] When two or more localization conclusions are determined to be the same or substantially similar, object detection system 110 may be configured to infer that the two or more conclusions correspond to the same perceived object of interest within the received image. For example, when multiple bounding boxes overlap (or are within a threshold distance of each other), object detection system 110 may conclude that the multiple sets of object detection conclusions corresponding to the bounding boxes refer to the same object of interest within the received image. Object detection system 110 may also infer in other ways that multiple sets of object detection conclusions correspond to the same perceived object of interest within the image.

[0144] In some cases, if the object detection system 110 determines that the localization conclusions correspond to the same or substantially similar locations within the received image (e.g., the bounding boxes do overlap), the object detection system 110 may determine whether one of the confidence conclusions from two or more sets of identified object detection conclusions is above a predetermined threshold (e.g., 60%). If so, then the object detection system 110 may select the corresponding set of object detection conclusions as a single reconciled set of object detection conclusions that is higher than one or more other sets of conclusions. Otherwise, the object detection system 110 may continue to perform the reconciliation function. However, in some embodiments, even if the object detection system 110 determines that one of the confidence conclusions from two or more sets of identified object detection conclusions is above a predetermined threshold, the object detection system 110 may continue to perform the reconciliation function.

[0145] In some implementations, when two or more confidence conclusions are above a predetermined threshold, the object detection system 110 may select a set of object detection conclusions corresponding to the highest confidence. In other implementations, despite multiple confidence conclusions being above a predetermined threshold, the object detection system 110 may continue to perform a reconciliation function to output a single reconciled set of object detection conclusions.

[0146] In any case, object detection system 110 may continue to perform the reconciliation function by determining the degree of similarity between two or more localization conclusions, which may be determined in a variety of ways. As one possibility, object detection system 110 may first identify a particular point (e.g., a "vertex") within each localization conclusion, such as by finding a center point within a bounding box defined by each localization conclusion. To illustrate, Figure 6B Provided by Fig. 6A 600, with four bounding boxes from multiple sets of object detection conclusions, and example vertices 611, 621, 631, 641 for each bounding box. Figure 6B In the example of , vertices 611, 621, 631, and 641 correspond to the center points of the four bounding boxes. Figure 6B and subsequent figures omitted Fig. 6A Classification labels and confidence values ​​shown.

[0147] Next, if two localization conclusions are substantially similar (e.g., they overlap or are close to each other), object detection system 110 may define a connection between the two localization conclusions. For example, object detection system 110 may generate a line (e.g., an "edge") connecting the identified vertices of the overlapping bounding boxes. To illustrate, Figure 6C Provided by Fig. 6A 600 with four bounding boxes from multiple sets of object detection conclusions, and example generated lines 623, 624, and 634. As shown, line 623 connects vertex 621 of the bounding box of set of object detection conclusions 620 to vertex 631 of the bounding box of set of object detection conclusions 630, because the bounding boxes overlap. Line 624 connects vertex 621 of the bounding box of set of object detection conclusions 620 to vertex 641 of the bounding box of set of object detection conclusions 640, because the bounding boxes overlap. Line 634 connects vertex 631 of the bounding box of set of object detection conclusions 630 to vertex 641 of the bounding box of set of object detection conclusions 640, because the bounding boxes overlap. Notably, no line connects vertex 611 to any other vertex because the bounding box corresponding to vertex 611 is not near the threshold of any other bounding box.

[0148] Finally, the object detection system 110 can then quantify the similarity between two or more localization conclusions. For example, the object detection system 110 can assign a value to each generated line that quantifies the degree of overlap of bounding boxes connecting vertices through a given generated line, such as by determining an intersection over union (IoU) value for each pair of overlapping bounding boxes. In this way, the object detection system 110 determines a measure of similarity between two or more localization conclusions from two or more sets of identified object detection conclusions.

[0149] To illustrate, return to Figure 6C , table 602 provides an overlap value (OV) for each line 623, 624, 634 connecting two overlapping bounding boxes. In this example, the overlap values ​​(e.g., 0.03, 0.10, and 0.40) represent the IoU values ​​for each pair of overlapping bounding boxes. A person of ordinary skill in the art will appreciate that Figure 6C The conceptual graph shown represents an undirected graph with bidirectional edges.

[0150] return Figure 5 At block 508, the object detection system 110 defines a plurality of candidate partitions (e.g., "clusters"), which may be defined based on at least the first positioning conclusion and the second positioning conclusion. In an example embodiment, each candidate partition may be defined such that a given candidate partition (i) includes no more than one positioning conclusion from any particular object detection model, and (ii) maximizes the similarity between the positioning conclusions. The object detection system 110 may define the candidate partitions in a variety of ways.

[0151] As one possible approach, the object detection system 110 may first generate an initial candidate partition for each vertex of the localization conclusion (and therefore for each bounding box). For illustration, Fig.6D Provided by Fig. 6A Conceptual illustration of an example image 600 with four bounding boxes from multiple sets of object detection conclusions, and example candidate partitions 615, 625, 635, 645 for each vertex.

[0152] The object detection system 110 can then determine whether any of the initial candidate partitions should be merged, which can be done in a variety of ways. As one possibility, for each given initial candidate partition, the object detection system 110 can determine whether the given initial candidate partition should be merged with one or more other initial candidate partitions based on the similarity between the positioning conclusions of these partitions, thereby generating a "merged" candidate partition.

[0153] In one particular implementation, the object detection system 110 may perform a merging function that involves (i) identifying any generated lines that connect a vertex of a particular initial candidate partition to another vertex, and (ii) excluding any identified lines that connect a vertex of the given candidate partition to another vertex that corresponds to a bounding box defined by the same object detection model that defines the bounding box corresponding to the vertex of the given candidate partition. To illustrate, Fig. 6E Provided by Fig. 6A 600 with four bounding boxes from multiple sets of object detection conclusions, and identification lines 624 and 634 for specific candidate partitions. Here, the specific candidate partition is partition 645, whose vertex 641 is connected to vertex 621 by line 624, and to vertex 631 by line 634. Notably, vertices 621 and 631 correspond to an object detection model (e.g., "Model B") that is different from the object detection model corresponding to vertex 641 (e.g., "Model A").

[0154] Next, the object detection system 110 can evaluate the assigned values ​​of any remaining identification lines, which can involve identifying the merged vertex based on the remaining identification lines with the largest assigned values ​​(e.g., by sorting the assigned values ​​in descending order and selecting the highest value). Fig. 6E , table 604 only provides overlap values ​​for lines 624 and 634 .

[0155] The object detection system 110 can then merge the particular candidate partition with the candidate partition of the identified merged vertex, thereby defining a merged candidate partition. For illustration, Fig. 6F Provided by Fig. 6A Conceptual diagram of an example image 600 with four bounding boxes from multiple sets of object detection conclusions and Fig. 6E The merged form 655 of the particular candidate partition 645 of FIG. 655. In this example, line 624 has the largest assigned overlap value (making vertex 621 a merge vertex), and therefore, the corresponding candidate partition 625 is merged with the particular candidate partition 645 to define the candidate partition 655.

[0156] In operation, the object detection system 110 may perform a merging process on any remaining initial candidate partitions, but each merged candidate partition includes its original vertices as well as any merged vertices. In this manner, the object detection system 110 generates partitions that each refer to the same perceived object of interest within the received image. Fig. 6F In the example shown, object detection system 110 can take advantage of the merging process given: (i) the bidirectional nature of line 634 connecting initial candidate partition 635 to merged candidate partition 645, and (ii) the absence of lines connecting initial candidate partition 615 to any other candidate partitions.

[0157] Back to Figure 5 At block 510, object detection system 110 selects a representative partition from the plurality of candidate partitions that informs object detection system 110 how to generate a single reconciled set of object detection conclusions for a given perceived object of interest in a received image. In practice, object detection system 110 may perform this function in a variety of ways.

[0158] As one possibility, object detection system 110 can utilize a "voting" scheme to select representative partitions, wherein each object detection model provides a "vote" for a particular candidate partition containing vertices corresponding to the model. In an example implementation utilizing a "voting" scheme, the weight (e.g., value) of the "vote" can be the same for each object detection model (e.g., each model's "vote" has a value) or can vary across object detection models. Moreover, the weight of a given object detection model's "vote" can be constant (i.e., predetermined) or can also change dynamically over time.

[0159] In practice, the "voting" weights of a given object detection model can be determined from: (i) based on various factors (e.g., the designer of the model, the type of training data used to train the model, the type of architecture of the model, etc.), (ii) based on the output of a machine learning model (e.g., one or more neural networks) that takes as input a received image and at least a portion of the object detection conclusions of each model and outputs an indication of how to weight the "vote" of each model, and / or (iii) based on the results of the "continuous learning" function discussed below. Other ways of determining "voting" weights are also possible.

[0160] After object detection system 110 applies the applicable weights to the "votes" of each object detection model, object detection system 110 can then select a representative partition based on the relative weights that have been assigned to the candidate partitions. As an example, object detection system 110 can select the candidate partition that received a predefined threshold amount of "votes" (e.g., more than 50% of the "votes") or the most "votes" (e.g., the partition with the largest weight), among other possibilities.

[0161] To illustrate, return to Fig. 6F , object detection system 110 selects a representative partition from candidate partition 615, candidate partition 635, and merged candidate partition 655. Using a "voting" scheme that gives equal weight to the "vote" of each object detection model, candidate partition 615 receives one "vote" from model A, candidate partition 635 receives one "vote" from model B, and merged candidate partition 655 receives "votes" from both model A and model B. In this example, merged candidate partition 655 is selected as the representative partition because it is weighted more than the other two candidate partitions.

[0162] exist Figure 5At block 512 of , the object detection system 110 derives a reconciled set of object detection conclusions for a given perceived object of interest in the received image based on the multiple sets of object detection conclusions corresponding to the selected representative partition. In an example embodiment, the object detection system 110 deriving the reconciled set of object detection conclusions may involve determining (i) a representative classification label, (ii) a representative (e.g., aggregate) confidence value, and / or (iii) a representative localization description of the given perceived object of interest within the received image (e.g., the size and / or location of a representative bounding box of the given perceived object of interest in the received image, which may be defined by a set of coordinates, etc.). In practice, the object detection system 110 may make these determinations based on two or more sets of original object detection conclusions corresponding to the selected representative partition (e.g., any set of object detection conclusions whose corresponding vertices are within the selected partition).

[0163] In this regard, the object detection system 110 can derive a representative classification label from two or more original classification conclusions based on one or more rules. For example, if the labels of two or more original classification conclusions are the same, then the representative classification label will be assigned the same label. However, if the two or more original classification conclusions are different, the representative classification label can be assigned based on one or more predefined rules, which can take a variety of forms.

[0164] One possible rule might involve considering the relative threat or danger level associated with each label, and might indicate that the label associated with the highest threat level is selected, (e.g., between a first label of "weapons" and a second label of "sharps," the rule might indicate that "weapons" is selected because it is associated with a class of objects that are considered more dangerous than the "sharps" class) or the rule might indicate the opposite. Another possible rule might indicate that a more specific label (e.g., "knife") is selected over a less specific label (e.g., "sharps"), or the rule might indicate the opposite (i.e., a general label is selected over a more specific label). Other rules are also possible.

[0165] Similarly, object detection system 110 may derive a representative confidence value based on two or more original confidence conclusions (eg, by determining an aggregate confidence value). Fig. 6A, the original confidence conclusions corresponding to the selected candidate partition 655 are 0.32 from the set of object detection conclusions 620 and 0.45 from the set of object detection conclusions 640. These values ​​indicate that the confidence that "Model A" is correctly classified for "knife" is 45%, and the confidence that "Model B" is correctly classified for "knife" is 32%. The object detection system 110 can derive an aggregate confidence value by determining the following terms: (i) the combined probability that both of the original confidence conclusions are wrong (i.e., both models incorrectly classify the perceived object of interest as "knife") by calculating (1-0.45)×(1-0.32) is 0.374, and (ii) the combined probability that the object of interest is correctly classified as "knife" by calculating (1-0.374) is 0.626, which is set as the aggregate confidence value. It is noteworthy that although each of the original confidence conclusions (32% and 45%) is not high enough (for example, the confidence level is above 60%) for the object detection system 110 to conclude that the received image contains a "knife", the representative confidence conclusion (62.6%) is high enough for the object detection system 110 to make this determination.

[0166] Finally, object detection system 110 may derive a representative position description of a given perceived object of interest within a received image based on the two or more raw position conclusions. In practice, object detection system 110 may perform this function in a variety of ways.

[0167] As one possibility, the object detection system 110 may derive a representative positioning description that circumscribes each of two or more original positioning conclusions. For example, the object detection system 110 may define a bounding box that circumscribes each of two or more original bounding boxes. As another possibility, the object detection system 110 may derive a representative positioning description by applying an aggregate function (e.g., averaging, identifying a median, etc.) to two or more original positioning conclusions. For example, the object detection system 110 may define a bounding box by averaging the coordinates of each of the two or more original bounding boxes. As another possibility, the object detection system 110 may derive a representative positioning description by selecting a positioning conclusion of two or more original positioning conclusions with the highest corresponding confidence conclusion. As another possibility, the object detection system 110 may derive a representative positioning description that locates the representative bounding box at a position that is considered to be the midpoint between the vertices corresponding to the two or more original positioning conclusions and defines the size of the representative bounding box as the maximum (or minimum) size defined by the two or more original bounding boxes. Other examples are also possible.

[0168] To illustrate, Figure 6G Provided by Fig. 6A 6, which has a reconciled set of object detection conclusions 660. Consistent with the discussion above, object detection system 110 derives a representative confidence value of approximately 63% from a set of object detection conclusions 620 and 640 and a representative bounding box that encloses the original bounding box.

[0169] After object detection system 110 has concluded a coordinated set of object detections, object detection system 110 completes the coordination process for at least a given perceived object of interest in the received image. To the extent object detection system 110 determines that the received image contains any other perceived objects of interest, it will then repeat the above functions for each additional perceived object of interest (e.g., combining Figure 5 Using the reconciled set of object detection conclusions, object detection system 110 can then perform its multi-model object detection on the received image, which generally involves concluding from the reconciled set of object detection conclusions that the received image contains at least a given perceived object of interest.

[0170] In some implementations, Figure 5 At block 514 of , object detection system 110 may optionally determine whether the received image includes an object of interest for a particular screening system that provided the original image of the scanned scene based on the coordinated set of object detection conclusions. In other words, object detection system 110 may perform specific object detection on the received image for the particular screening system based on the coordinated set of object detection conclusions.

[0171] For example, in an example implementation, one or more screening systems in communication with object detection system 110 may have specific object detection requirements, such as a particular set of objects of concern and / or specific criteria for marking such objects of concern (e.g., minimum confidence level, etc.). Furthermore, in some cases, the object detection requirements of a screening system may change dynamically, which may depend on the time of year, the current security threat level, etc.

[0172] In any case, object detection system 110 can evaluate the reconciled set of object detection conclusions and the specific object detection requirements of the specific screening system to determine whether the received image includes an object that meets the specific object detection requirements so that object detection system 110 can identify it as an object of interest for the specific screening system. Other possibilities exist as well.

[0173] At block 516, object detection system 110 performs one or more operations based on one or both of (i) the reconciled set of object detection conclusions and (ii) the optional determination from block 514. These one or more operations may take a variety of forms.

[0174] As one possibility, object detection system 110 may present one or more indications of one or more conclusions from the reconciled set of object detection conclusions on a computer display or the like. For example, object detection system 110 may cause output device 124 to display, via one or more links 130, a graphical visualization of a classification label for a given perceived object of interest, a graphical visualization of a confidence level for a classification of a given perceived object of interest, and / or a graphical visualization of a representative bounding box for a given perceived object of interest so that security screeners may visualize the analysis of object detection system 110. In some cases, such graphical visualizations may be presented overlaid with (or together with) a visualization of a scanned scene, such as a visualization of a scanned baggage item.

[0175] Similarly, object detection system 110 may present one or more notifications related to the original scanned scene on a computer display, etc., based on one or more conclusions from the reconciled set of object detection conclusions. For example, object detection system 110 may cause output device 124 to display a visualization of the baggage item via one or more links 130 only if object detection system 110 determines, based on the reconciled set of object detection conclusions, that an object of interest within the baggage item is deemed security-relevant for a particular screening system 120. Otherwise, object detection system 110 may forgo causing output device 124 to display any visualization of the scanned baggage item.

[0176] As another possibility, object detection system 110 may provide a notification or the like to a specific output device (e.g., a wearable computing device, a mobile phone, etc.) associated with a specific screening system 120 that provided the original scanned image based on one or more conclusions from the reconciled set of object detection conclusions, such as sending a text message or application notification to a computing device assigned to a baggage handler at a security checkpoint. For example, when object detection system 110 determines that an object of interest within a scanned baggage item is deemed security-relevant for a specific screening system 120 based on the reconciled set of object detection conclusions, object detection system 110 may send a notification to a specific output device 124, which may prompt the baggage handler to physically search the baggage item.

[0177] As another possibility, the object detection system 110 may, based on one or more conclusions from the harmonized set of object detection conclusions, cause the specific operation to occur at the screening system by, for example, sending specific instructions indicating an operation (e.g., a specific pass operation) to the screening system 120 via one or more links 130 through the object detection system 110.

[0178] For example, the screening system 120 may include a conveyor belt on which a baggage item (or the like) moves past one or more detection devices 122, which then capture one or more images of the baggage item. A security screener workstation at the screening system 120 includes a user interface (e.g., buttons, etc.) for controlling the conveyor belt so that a human security screener can manually control the flow of baggage items through the system. By default, the conveyor belt moves, and thus the baggage item passes through the system, when the security screener provides input to cause the conveyor belt to move. When the security screener provides such input, the baggage item (i) passes through one or more detection devices 122 that capture one or more images of the baggage item, and (ii) enters a "main" channel, where the owner of the baggage item retrieves the item. On the other hand, when a security screener identifies a potential security concern in a scanned image of a baggage item, depending on the nature and / or sophistication of the screening system 120, the security screener will typically either (i) stop the conveyor (or refuse to pass the baggage item through the "main" lane) and manually move the baggage item to a "search" lane where a baggage handler retrieves the baggage item and searches it, or (ii) provide input at a user interface (e.g., a user interface at a workstation equipped with "smart" lane technology at the screening system 120) that diverts the baggage item to an auxiliary conveyor where a baggage handler retrieves the baggage item and searches its contents.

[0179] In an example embodiment, the object detection system 110, based on the coordinated set of object detection conclusions, the operation at the screening system for the received image of the baggage item may involve: causing the screening system 120 to automatically cause the baggage item to no longer enter the "main" channel, but to enter the "search" channel, for example, by causing an actuator within the conveyor system of the screening system, etc. to automatically divert the baggage item outside the "main" channel, or by diverting the baggage item to an auxiliary conveyor (for example, when the screening system 120 is configured with "smart" channel technology). In this way, the object detection system 110 can promote the high throughput of the screening system and the improved object detection accuracy, and can also help reduce the number of operators required at a given screening checkpoint.

[0180] Likewise, the object detection system 110 may cause the screening system 120 to automatically start or stop a conveyor belt based on the coordinated set of object detection conclusions for a received image of a baggage item, without control input from a human security screener. For example, when the coordinated set of object detection conclusions indicates that there is no object of interest in the baggage item (or determines that there is no object of interest that is considered security-related in the baggage item), the object detection system 110 may cause the conveyor belt to automatically start (or continue to move), thereby overriding the default setting of the screening system and passing the baggage item through the system to a "main" channel for retrieval by the owner of the baggage item. On the other hand, in some security systems, which, by default, have a conveyor belt that continuously transports passing baggage items until the system receives a "stop" input from a human screener, when the coordinated set of object detection conclusions indicates that there is an object of interest that is considered security-related in the baggage item, the object detection system 110 may cause the conveyor belt to automatically stop, thereby overriding the default setting of the screening system.

[0181] It is noteworthy that in the event that the object detection system 110 is configured to stop the conveyor, the object detection system 110 may also configure the screening system 120 such that the ability of a human security screener to continue moving the conveyor may be disabled until other events occur (in addition to the system receiving input at the user interface to start / stop the conveyor), such as the security screener providing input indicating that object detection by the object detection system 110 has been confirmed, the security screener or their supervisor providing certain credentials in the screening system 120, etc.

[0182] Relatedly, in some embodiments, object detection system 110 may be configured to (i) perform object detection by applying at least one object detection model to a given received image to obtain a set of object detection conclusions, and ii) consider an operator's response to the system's object detection (as indicated by feedback data) as a second set of (potentially conflicting) object detection conclusions. Depending on the object detection by object detection system 110, object detection system 110 may then be able to override the operator's response and / or cause a notification to be output at screening system 120. For example, a security screener at screening system 120 may be compromised such that the security screener intentionally allows a baggage item containing a prohibited item (e.g., a gun, etc.) to pass through screening system 120. Object detection system 110 may be configured such that when object detection system 110 detects an object with a high degree of confidence that a baggage item contains a security threat and object detection system 110 receives an indication (e.g., via feedback data) that security screening personnel have not stopped or diverted the baggage item for further inspection, object detection system 110 may (i) automatically divert the baggage item, (ii) stop a conveyor belt of screening system 120, (iii) sound an alarm in screening system 120, and / or (iii) alert a supervisor of security screening personnel, among other possibilities.

[0183] As described above, in example embodiments, object detection system 110 may be configured to receive feedback data based on actions occurring on screening system 120 (e.g., after outputting an indication of multi-model object detection by object detection system 110), via one or more links 130. For example, an operator of screening system 120 may provide input at a workstation or the like to: (i) provide an indication of whether multi-model object detection was correct (in whole or in part), (ii) provide an indication that object detection system 110 failed to detect an object of interest within an image detected by the operator, or (iii) provide an indication of the operator's classification of an object in an image (e.g., a human-entered label of an object that may or may not be considered safety-related), among other possibilities. In practice, such input may take the form of direct feedback to the multi-model object detections of the object detection system 110 (e.g., by an operator viewing a visual representation of the multi-model object detections on the output device 124 and then “ruling out” erroneous detection results using a GUI or the like, or by baggage handlers providing input at the output device 124 to confirm or refute that the baggage item contains a security threat), and / or such input may take the form of indirect feedback (e.g., conveyor belt actions taken by screening operators, etc.).

[0184] Based on feedback data from actions occurring at the screening systems 120, the object detection system 110 may be configured to update the multi-model object detection process in various ways. For example, the feedback data may generate new or updated labeled training data that may be used to update some or all of the two or more object detection models of the object detection system 110, such as by back-propagating when the feedback data indicates that the object detection system 110 erroneously identified an image as containing an object of interest. For example, the feedback data may indicate that operators at a particular screening system 120 have frequently marked lighters in scanned images as security threats, which is a security threat that the two or more object detection models of the object detection system 110 were not previously trained to detect. Based on such feedback data, the object detection system 110 may train one or more object detection models to detect lighters for future analysis of scanned images from the particular screening system 120.

[0185] In another case, the feedback data may result in modifying the object detection confidence thresholds of one or more object detection systems 110. For example, the feedback data may indicate that there are relatively frequent false alarms for "sharps," such as coin rolls within scanned images that are labeled as "sharps" by the object detection system 110. Based on such feedback data, the object detection system 110 may increase the confidence thresholds for its "sharps" detections, for example, by changing one or more confidence thresholds of at least one object detection model trained to classify "sharps," and possibly retraining one or more such models using the mislabeled images. Other examples are also possible.

[0186] Furthermore, updates to the multi-model object detection process may occur “on the fly” (e.g., after each image has been processed and applicable output provided to a screening operator), or updates may occur periodically (or in “batches”), such as every 24 hours, weekly, etc., among other circumstances.

[0187] Object detection system 110 may perform various other operations based at least in part on the reconciled set of object detection conclusions.

[0188] V. Continuous Learning Operation Example

[0189] As previously described, in example embodiments, object detection system 110 may be configured to perform "continuous learning," which generally involves object detection system 110 receiving feedback data regarding the performance of one or more of its two or more object detection models and then updating aspects of the multi-model object detection process based on the feedback data. In practice, such feedback data may be the result of the multi-model object detection process itself, and / or may be received as a result of human actions regarding multi-model object detection of object detection system 110, as described above. In any case, object detection system 110 may perform "continuous learning" in a variety of ways.

[0190] In one aspect, object detection system 110 can be configured to evaluate the performance of a given object detection model over time, and based on the evaluation, first select a given object detection model for multi-model object detection (e.g., as Figure 5 504) and / or modify the weights of a given object detection model's "votes" (e.g., as part of Figure 5 For example, object detection system 110 may determine that a conclusion from a first object detection model is considered correct more than a first threshold number of times (e.g., the partitions historically selected by object detection system 110 include vertices corresponding to the first object detection model), and thus, object detection system 110 may first select the given object detection model for multi-model object detection, and / or increase the weight of the “vote” of the first object detection model.

[0191] On the other hand, the object detection system 110 may determine that the number of times the conclusion from the second object detection model is considered correct is less than the first threshold number of times (and may be less than a lower second threshold number of times), and therefore, the object detection system 110 may refuse to select the second object detection model for multi-model object detection, reduce the weight of the "vote" of the second object detection model, or suppress the object detection conclusion of the second object detection model. In other words, based on the object detection system 110 evaluating the performance of a given object detection model over time, when the evaluation indicates that the model is consistently inaccurate, the object detection system 110 may suppress the given object detection model.

[0192] In another aspect, the object detection system 110 may be configured to evaluate the performance of a given object detection model over time, and based on the evaluation, modify whether the given object detection model should be used in conjunction with certain other object detection models. For example, the object detection system 110 may determine that the number of times a first object detection model's detection of a first class of objects (e.g., "liquids") is considered correct exceeds a threshold number of times, and the number of times a second object detection model's detection of a second class of objects (e.g., "sharps") is considered correct exceeds a threshold number of times. As a result, assuming that each of these models is good at detecting objects of different classifications, the object detection system 110 may modify its multi-model object detection process such that the first object detection model and the second object detection model are not used together or such a combination is downgraded in priority in other combinations. As another example, the object detection system 110 may determine that detections of a first object detection model within an image viewed from a first angle of a scene (e.g., a "side view" image) have historically matched detections of a second object detection model within an image viewed from a second, different angle of the scene (e.g., a "top view" image). As a result, object detection system 110 may modify its multi-model object detection process such that a first object detection model and a second object detection model are used together in a manner such that the first object detection model evaluates a “side view” image of the scene and the second object detection model evaluates a “top view” image of the scene.

[0193] In another aspect, the object detection system 110 may be configured to consider not only feedback data regarding the performance of a given object detection model, but also other data related to the object detection of the given object detection model (e.g., image data used in object detection, external contextual data related to the scanned scene, etc.), and based on such evaluation, update aspects of the multi-model object detection process. For example, when evaluating "complex" images (e.g., cluttered scene images) or evaluating images with high-density areas (e.g., scene images with high-density materials), the object detection system 110 may determine that the first object detection model is historically the most accurate, and as a result, the object detection system 110 may first select the first object detection model for multi-model object detection, and / or increase the weight of the "vote" of the first object detection model when a given image has such features. As another example, when analyzing images scanned at a particular geographic location at a particular time of year (e.g., due to the types of items that are commonly present in luggage items at a particular time of year, such as summer, at a particular location where the scanning occurs), object detection system 110 may determine that a first object detection model is historically most accurate, and thus, when external contextual data (e.g., time of year and / or geographic location data) indicates that such conditions exist at the location of the detection device receiving the image, object detection system 110 may select the first object detection model first and / or increase the "voting" weight of the first object detection model. Other examples are also possible.

[0194] In some or all of the above examples of “continuous learning,” the object detection system 110 may be configured to utilize one or more machine learning models (e.g., one or more neural networks) that are trained or otherwise updated based at least on feedback data indicating the respective performance of two or more object detection models of the object detection system 110, and possibly other data related to the scanned scene (the two or more object detection models have analyzed the image data of the scanned scene) (e.g., contextual data). These one or more machine learning models may be configured to receive as input the image data and other data related to the scanned scene (e.g., contextual data), and then output (i) an indication informing the object detection system 110 of which particular object detection model is most accurate for a given situation, and / or (ii) an indication informing the object detection system 110 of how much weight should be assigned to the “vote” of a given object detection model for a given situation (e.g., as a “vote” of a given object detection model). Figure 5 There are other possibilities as well.

[0195] VI. Conclusion

[0196] Exemplary embodiments of the disclosed innovation have been described above. However, those skilled in the art will appreciate that changes and modifications may be made to the described embodiments without departing from the true scope and spirit of the invention, which is defined by the claims.

[0197] In addition, references herein to "an embodiment" mean that a particular feature, structure, and / or characteristic described in conjunction with the embodiment may be included in at least one example embodiment of the present invention. Phrases appearing in various places in the specification do not necessarily all refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive. Thus, embodiments described herein may be combined with other embodiments as explicitly and implicitly understood by those of ordinary skill in the art.

[0198] Furthermore, to the extent that the examples described herein involve actions performed or initiated by actors, such as "people," "operators," "users," or other entities, this is for purposes of illustration and explanation only. The claims should not be interpreted as requiring such actions by such actors unless expressly stated in the claim language.

Claims

1. A computing system comprising: at least one processor; non-transitory computer readable medium; as well as program instructions stored on the non-transitory computer-readable medium, the program instructions being executable by the at least one processor to cause the computing system to: receiving an image of a scene scanned by a detection device of a screening system, wherein the image includes an object of interest; At least identifying a first group of object detection conclusions generated by a first object detection model for the image and a second group of object detection conclusions generated by a second object detection model for the image, wherein the first group of object detection conclusions includes a first classification label assigned to the object of interest and a first confidence of the first classification of the object of interest, wherein the second group of object detection conclusions includes a second classification label assigned to the object of interest and a second confidence of the second classification of the object of interest, wherein at least one conclusion difference exists between at least one of the first classification label and the second classification label, the first confidence and the second confidence of the first group of object detection conclusions and the second group of object detection conclusions; wherein the first object detection model is a first type machine learning model and the first group of object detection conclusions are derived from the first type machine learning model, the second object detection model is a second type machine learning model and the second group of object detection conclusions are derived from the second type machine learning model, and the first type machine learning model and the second type machine learning model are different; Resolve at least one of the conclusion differences by: Determining a similarity between a first positioning conclusion from the first set of object detection conclusions and a second positioning conclusion from the second set of object detection conclusions, wherein both the first positioning conclusion and the second positioning conclusion are determined to refer to the object of interest; defining a plurality of candidate partitions, wherein at least one candidate partition among the plurality of candidate partitions is defined based on the similarity between the first positioning conclusion and the second positioning conclusion; and deriving a reconciled set of object detection conclusions for the object of interest from the first set of object detection conclusions and the second set of object detection conclusions based on the weight assigned to the at least one candidate partition, thereby resolving the at least one conclusion difference; and Based on the harmonized set of object detection conclusions, it is determined that the image of the scene contains the object of interest.

2. The computing system of claim 1, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Before at least identifying the first group of object detection conclusions and the second group of object detection conclusions, the first object detection model and the second object detection model are selected from more than three available object detection models, and the first object detection model and the second object detection model will generate object detection conclusions for the image based on one or more of the following: (i) respective designers of the first object detection model and the second object detection model; (ii) types of training data used to respectively train the first object detection model and the second object detection model; and (iii) feedback data on the performance of each of the more than three available object detection models.

3. The computing system of claim 1, wherein: The first positioning conclusion from the first group of object detection conclusions includes a first bounding box of the object of interest, wherein the second positioning conclusion from the second group of object detection conclusions includes a second bounding box of the object of interest, and wherein determining the similarity between the first positioning conclusion from the first group of object detection conclusions and the second positioning conclusion from the second group of object detection conclusions includes: determining the overlap between the first bounding box and the second bounding box.

4. The computing system of claim 1, wherein: Limiting the multiple candidate partitions includes: Defining a first candidate partition according to the first positioning conclusion, and defining a second candidate partition according to the second positioning conclusion; and The at least one candidate partition is limited by merging the first candidate partition with the second candidate partition based on the similarity between the first positioning conclusion and the second positioning conclusion.

5. The computing system of claim 1, wherein: Deriving the reconciled set of object detection conclusions for the object of interest from the first set of object detection conclusions and the second set of object detection conclusions includes: Assigning a weight to each of the plurality of candidate partitions, wherein the weight assigned to a given candidate partition is based on one or more object detection models corresponding to the given candidate partition, and wherein the weight assigned to the at least one candidate partition is based on (i) a first weight assigned to the first object detection model and (ii) a second weight assigned to the second object detection model.

6. The computing system of claim 5, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Based on the reconciled set of object detection conclusions for the object of interest derived from the first set of object detection conclusions and the second set of object detection conclusions, modify (i) the first weight assigned to the first object detection model and / or (ii) the second weight assigned to the second object detection model.

7. The computing system of claim 5, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Based on feedback data about the performance of the first object detection model and / or the second object detection model, (i) the first weight assigned to the first object detection model and / or (ii) the second weight assigned to the second object detection model are modified.

8. The computing system of claim 1, wherein: The program instructions executed by the at least one processor to cause the computing system to determine the image of the scene including the object of interest include: program instructions executed by the at least one processor to cause the computing system to: Based on the reconciled set of object detection conclusions and one or more specific object detection requirements of the screening system, it is determined that the image of the scene contains the object of interest for the screening system.

9. The computing system of claim 1, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Based on determining that the image of the scene contains the object of interest, at least one of the following items is generated: (i) a display screen associated with the screening system displays a visualization effect related to the coordinated set of object detection conclusions, or (ii) a pass operation is performed in the screening system.

10. A tangible, non-transitory computer-readable medium comprising stored program instructions executable by at least one processor to cause a computing system to: receiving an image of a scene scanned by a detection device of a screening system, wherein: The image includes an object of interest; At least identifying a first group of object detection conclusions generated by a first object detection model for the image and a second group of object detection conclusions generated by a second object detection model for the image, wherein the first group of object detection conclusions includes a first classification label assigned to the object of interest and a first confidence of the first classification of the object of interest, wherein the second group of object detection conclusions includes a second classification label assigned to the object of interest and a second confidence of the second classification of the object of interest, wherein at least one conclusion difference exists between at least one of the first classification label and the second classification label, the first confidence and the second confidence between the first group of object detection conclusions and the second group of object detection conclusions; wherein the first object detection model is a first type machine learning model and the first group of object detection conclusions are derived from the first type machine learning model, the second object detection model is a second type machine learning model and the second group of object detection conclusions are derived from the second type machine learning model, and the first type machine learning model and the second type machine learning model are different; Resolve at least one of the conclusion differences by: Determining a similarity between a first positioning conclusion from the first set of object detection conclusions and a second positioning conclusion from the second set of object detection conclusions, wherein both the first positioning conclusion and the second positioning conclusion are determined to refer to the object of interest; defining a plurality of candidate partitions, wherein at least one candidate partition among the plurality of candidate partitions is defined based on the similarity between the first positioning conclusion and the second positioning conclusion; and deriving a reconciled set of object detection conclusions for the object of interest from the first set of object detection conclusions and the second set of object detection conclusions based on the weight assigned to the at least one candidate partition, thereby resolving the at least one conclusion difference; and Based on the harmonized set of object detection conclusions, it is determined that the image of the scene contains the object of interest.

11. The non-transitory computer readable medium of claim 10, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Before at least identifying the first set of object detection conclusions and the second set of object detection conclusions, selecting the first object detection model and the second object detection model from more than three available object detection models, the first object detection model and the second object detection model generating object detection conclusions for the image based on one or more of the following: (i) designers of the first object detection model and the second object detection model respectively; (ii) types of training data for respectively training the first object detection model and the second object detection model; and (iii) feedback data on the performance of each of the three or more available object detection models.

12. The non-transitory computer readable medium of claim 10, wherein: The first positioning conclusion from the first group of object detection conclusions includes a first bounding box of the object of interest, wherein the second positioning conclusion from the second group of object detection conclusions includes a second bounding box of the object of interest, and wherein determining the similarity between the first positioning conclusion from the first group of object detection conclusions and the second positioning conclusion from the second group of object detection conclusions includes: determining the overlap between the first bounding box and the second bounding box.

13. The non-transitory computer readable medium of claim 10, wherein: Limiting the multiple candidate partitions includes: Defining a first candidate partition according to the first positioning conclusion, and defining a second candidate partition according to the second positioning conclusion; and The at least one candidate partition is limited by merging the first candidate partition with the second candidate partition based on the similarity between the first positioning conclusion and the second positioning conclusion.

14. The non-transitory computer readable medium of claim 10, wherein: Deriving the reconciled set of object detection conclusions for the object of interest from the first set of object detection conclusions and the second set of object detection conclusions includes: Assigning a weight to each of the plurality of candidate partitions, wherein the weight assigned to a given candidate partition is based on one or more object detection models corresponding to the given candidate partition, and wherein the weight assigned to the at least one candidate partition is based on (i) a first weight assigned to the first object detection model and (ii) a second weight assigned to the second object detection model.

15. The non-transitory computer readable medium of claim 14, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Based on the reconciled set of object detection conclusions for the object of interest derived from the first set of object detection conclusions and the second set of object detection conclusions, modify (i) the first weight assigned to the first object detection model and / or (ii) the second weight assigned to the second object detection model.

16. The non-transitory computer readable medium of claim 14, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Based on feedback data about the performance of the first object detection model and / or the second object detection model, (i) the first weight assigned to the first object detection model and / or (ii) the second weight assigned to the second object detection model are modified.

17. The non-transitory computer readable medium of claim 10, wherein: The program instructions executable by the at least one processor to cause the computing system to determine the image of the scene including the object of interest include: program instructions executable by the at least one processor to cause the computing system to: Based on the reconciled set of object detection conclusions and one or more specific object detection requirements of the screening system, it is determined that the image of the scene contains the object of interest for the screening system.

18. The non-transitory computer readable medium of claim 10, wherein: The program instructions executable by the at least one processor also include program instructions executed by the at least one processor to cause the computing system to: Based on determining that the image of the scene contains the object of interest, at least one of the following items is generated: (i) a display screen associated with the screening system displays a visualization effect related to the coordinated set of object detection conclusions, or (ii) a pass operation is performed in the screening system.

19. A computer-implemented method, the method comprising: receiving an image of a scene scanned by a detection device of a screening system, wherein the image includes an object of interest; At least identifying a first group of object detection conclusions generated by a first object detection model for the image and a second group of object detection conclusions generated by a second object detection model for the image, wherein the first group of object detection conclusions includes a first classification label assigned to the object of interest and a first confidence of the first classification of the object of interest, wherein the second group of object detection conclusions includes a second classification label assigned to the object of interest and a second confidence of the second classification of the object of interest, wherein at least one conclusion difference exists between at least one of the first classification label and the second classification label, the first confidence and the second confidence between the first group of object detection conclusions and the second group of object detection conclusions; wherein the first object detection model is a first type machine learning model and the first group of object detection conclusions are derived from the first type machine learning model, the second object detection model is a second type machine learning model and the second group of object detection conclusions are derived from the second type machine learning model, and the first type machine learning model and the second type machine learning model are different; Resolve at least one of the conclusion differences by: Determining a similarity between a first positioning conclusion from the first set of object detection conclusions and a second positioning conclusion from the second set of object detection conclusions, wherein both the first positioning conclusion and the second positioning conclusion are determined to refer to the object of interest; defining a plurality of candidate partitions, wherein at least one candidate partition among the plurality of candidate partitions is defined based on the similarity between the first positioning conclusion and the second positioning conclusion; and deriving a reconciled set of object detection conclusions for the object of interest from the first set of object detection conclusions and the second set of object detection conclusions based on the weight assigned to the at least one candidate partition, thereby resolving the at least one conclusion difference; and Based on the harmonized set of object detection conclusions, it is determined that the image of the scene contains the object of interest.

20. The computer-implemented method of claim 19, further comprising: Based on determining that the image of the scene contains the object of interest, at least one of the following items is generated: (i) a display screen associated with the screening system displays a visualization effect related to the coordinated set of object detection conclusions, or (ii) a pass operation is performed in the screening system.

21. The computer-implemented method of claim 19, further comprising: Before at least identifying the first set of object detection conclusions and the second set of object detection conclusions, selecting the first object detection model and the second object detection model from more than three available object detection models, the first object detection model and the second object detection model generating object detection conclusions for the image based on one or more of the following: (i) respective designers of the first object detection model and the second object detection model; (ii) types of training data used to train the first object detection model and the second object detection model, respectively; (iii) feedback data on the performance of each of the three or more available object detection models.

22. The computer-implemented method of claim 19, wherein: Deriving the reconciled set of object detection conclusions for the object of interest from the first set of object detection conclusions and the second set of object detection conclusions includes: Assigning a weight to each of the plurality of candidate partitions, wherein the weight assigned to a given candidate partition is based on one or more object detection models corresponding to the given candidate partition, and wherein the weight assigned to the at least one candidate partition is based on (i) a first weight assigned to the first object detection model and (ii) a second weight assigned to the second object detection model.

23. The computer-implemented method of claim 22, further comprising: Based on feedback data about the performance of the first object detection model and / or the second object detection model, (i) the first weight assigned to the first object detection model and / or (ii) the second weight assigned to the second object detection model are modified.

Images