Image-based anomaly detection based on object-based machine learning analysis

Abstract

An object analysis system is disclosed herein. The object analysis system can receive an input image depicting an object. The object analysis system can determine, using a feature extraction model and from the input image, a first feature output associated with one or more features of the object. The feature extraction model can be trained based on a reference image depicting a reference object that is a type of the object. The object analysis system can determine, using a classification model, that an abnormal state of the object indicates that the object includes an anomaly. The classification model can be trained based on the reference image.

Description

Background Technology

[0001] Quality control involves analyzing and / or reviewing products to ensure they conform to certain quality standards and / or guidelines. For physical products, visual inspection may be necessary to identify anomalies on or associated with the product that could prevent it from meeting certain quality standards and / or guidelines. The form of such anomalies may depend on the product type and / or be unique in terms of various characteristics. Therefore, a system is needed to detect unique anomalies on or associated with a particular product. Summary of the Invention

[0002] In some implementations, a method associated with detecting anomalies related to an object includes: receiving an input image depicting the object; processing the input image using a feature extraction model to indicate one or more features of the object in a first feature output, wherein the feature extraction model is trained based on a reference image associated with the object's type, wherein the reference image depicts one or more non-abnormal objects of the same type as the object's type; determining an anomalous state of the object indicating that the object includes an anomaly based on the one or more features using a classification model, wherein the classification model is configured to determine the anomalous state based on a classification score associated with the first feature output and a classification threshold of the classification model, wherein the classification threshold is determined based on a similarity analysis involving the reference image; determining the location of the anomaly associated with the anomalous state based on a second feature output of the feature extraction model, wherein the location of the anomaly is determined using an anomaly localization model trained based on the reference image; generating anomalous data associated with the anomaly based on the anomalous state and location; and providing the anomalous data to an object management system.

[0003] In some embodiments, a device includes one or more memories and one or more processors coupled to the memories, the processors being configured to: receive an input image depicting an object; process the input image using a feature extraction model to generate a first feature output associated with one or more features of the object, wherein the feature extraction model is trained based on a reference image associated with a type of the object; determine an anomalous state of the object using a classification model based on the first feature output, wherein the classification model is trained to determine the anomalous state based on a similarity analysis involving non-abnormal objects depicted in the reference image; determine the location of the anomalous object in the input image based on a second feature output from the feature extraction model, based on the anomalous state indicating that the input image depicts an object with an anomalous state, wherein the location of the anomalous object is determined using an anomalous localization model trained on the reference image; generate anomalous data associated with the anomalous object based on the anomalous state and location; and perform an action associated with the anomalous data.

[0004] In some embodiments, a tangible machine-readable medium storing a set of instructions includes one or more instructions that, when executed by one or more processors of a device, cause the device to: receive an input image depicting an object; use a convolutional neural network encoder and determine, based on the input image, a first feature output associated with one or more features of the object, wherein the convolutional neural network encoder is trained based on a reference image depicting a reference object as a type of object; use a support vector machine to determine an anomalous state of the object indicating that the object includes an anomalous state, wherein the support vector machine is trained based on the reference image; use a convolutional neural network decoder to determine the location of the anomalous state based on a second feature output of the convolutional neural network encoder, wherein the convolutional neural network decoder is configured to determine the location of the anomalous state based on the second feature output of the convolutional neural network encoder, and wherein the convolutional neural network decoder is trained based on the reference image; and perform an action associated with the location of the anomalous state. Attached Figure Description

[0005] The accompanying drawings (in which the same reference numerals indicate the same or functionally similar elements in all separate views) together with the following detailed description are included herein and form part of the specification, and are used to further illustrate embodiments of the concepts disclosed herein, and to explain the various principles and advantages of those embodiments.

[0006] Figure 1 This is an illustration of an example implementation associated with training a machine learning model for the object analysis system described in this paper.

[0007] Figure 2 This is an illustration of an example implementation related to image-based anomaly detection in the object analysis system described herein.

[0008] Figure 3 This is a diagram illustrating an example implementation of the classification model described in this article.

[0009] Figure 4 This is a diagram illustrating an example implementation of the anomaly localization model described in this article.

[0010] Figure 5 This is an illustration of an example implementation of training and using a machine learning model in conjunction with image-based anomaly detection.

[0011] Figure 6 This is a diagram of an example environment in which the systems and / or methods described herein can be implemented.

[0012] Figure 7 yes Figure 6 An illustration of an example component of one or more devices.

[0013] Figure 8 This is a flowchart of an example process related to image-based anomaly detection in object-based machine learning analysis.

[0014] Those skilled in the art will understand that the elements in the accompanying drawings are shown for simplicity and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the drawings may be exaggerated relative to other elements to aid in understanding the embodiments described herein.

[0015] The elements of the apparatus and method have been indicated in the accompanying drawings by conventional symbols in appropriate places, which show only those specific details relevant to understanding the embodiments described herein, so as not to obscure this disclosure with details that would be obvious to those skilled in the art thanks to the description herein. Detailed Implementation

[0016] The following detailed description of the exemplary embodiments is with reference to the accompanying drawings. The same reference numerals in different drawings may identify the same or similar elements.

[0017] Image-based analytics can be used to perform anomaly detection. For example, image processing models (e.g., computer vision models) can be trained to identify anomalies on objects, such as products, equipment, structures, or other types of physical objects. Image processing models can be trained using images depicting objects with anomalies (which may be referred to as “anomalous objects”), such as scratches, cracks, punctures, discoloration, missing elements, extra elements, or other types of anomalies. However, such image processing models may be relatively inaccurate due to the variety of types of anomalies, the various characteristics of each type (e.g., size, shape, location), and / or the ability of anomaly characteristics to change over time (e.g., due to unknown or unexpected changes in the environment). This can lead to false negatives or false positives in anomaly detection. For example, because the image processing model has not been trained to identify a specific type of anomaly and / or specific characteristics of anomalies, it may not accurately detect that type of anomaly on an object. Therefore, there is a need for an object analysis system capable of robustly and accurately detecting unique and / or unknown anomalies on or associated with objects.

[0018] Some embodiments described herein provide object analysis systems for detecting, classifying, and / or locating anomalies on objects. The object analysis system may include and / or utilize a model arrangement trained on reference images depicting one or more reference objects that do not include or have any anomalies. For example, the object analysis system may include feature extraction models, classification models, and anomaly localization models configured to analyze objects to determine whether an object contains an anomaly. The models may be trained to identify, analyze, and / or detect features of reference objects based on the analysis of reference images depicting one or more non-abnormal objects. In this way, the object analysis system can detect, classify, and / or locate anomalies based on a comparison of the reference object with objects depicted in an input image (e.g., an image depicting the object being analyzed by the object analysis system). In this way, without training using training images depicting a specific anomaly or a specific configuration of anomalies, the object analysis system described herein can robustly and accurately detect, classify, and / or locate anomalies on objects. Furthermore, the model of the object analysis system can be trained with less training data than other systems, thereby saving computational resources (e.g., processing resources, memory resources, and / or storage resources) while maintaining and / or improving the robustness and accuracy of image-based anomaly detection relative to other systems.

[0019] Figure 1 This is an illustration of an example implementation 100 associated with a machine learning model of a training object analysis system. (See illustration.) Figure 1 As shown, Example Implementation 100 includes a reference data structure and an operation management system, the operation management system including a feature extraction model and a classification model. In Example Implementation 100, the object analysis system can train the feature extraction model and the classification model to detect anomalies on objects based on reference images depicting one or more objects. Although Example Implementation 100 can be described as combining the training of the feature extraction model and / or the classification model to detect and / or classify anomalies on or associated with a specific type of object, it is not possible to combine... Figure 1 The described examples can be similarly applied by combining trained feature extraction models and / or classifications to detect and / or classify anomalies on or associated with multiple types of objects.

[0020] like Figure 1As shown and indicated by reference numeral 110, the object analysis system can receive reference image data associated with an object type. For example, the object analysis system can obtain reference image data from a reference image data structure during a training period associated with training a feature extraction model and / or a classification model. The reference image data structure may include storage devices and / or memory devices that receive and / or store images from one or more image sources (e.g., one or more image capture devices, one or more image databases, and / or one or more networks or systems). The reference image data may include reference images depicting object types (e.g., object types to be analyzed by the object analysis system for anomaly detection). In some embodiments, the reference images may depict multiple types of objects (e.g., according to the examples described herein, to allow the feature extraction model and / or classification model to detect anomalies on multiple types of objects).

[0021] As described herein, a reference image can depict non-abnormal objects, thereby allowing feature extraction models to identify features of the objects depicted in the image and / or classify objects as anomalous or non-abnormal based on the identified features. A non-abnormal object can be an object that does not include anomalous features. For example, as used herein, a non-abnormal object can be considered a normal object, a standard object, or an acceptable object relative to standards (e.g., industry standards) or tolerances (e.g., design tolerances and / or manufacturing tolerances).

[0022] like Figure 1 As further illustrated, and indicated by reference numeral 120, the object analysis system extracts features from non-abnormal objects. Non-abnormal objects can be depicted in a reference image that depicts reference objects of the same type. For example, via a feature extraction model, the object analysis system can analyze reference image data (and / or a reference image of the reference image data) to identify features of a type of object. More specifically, the object analysis system can analyze reference image data to identify features of reference objects depicted in the image data. Such features can be a set of features typically depicted in the reference image.

[0023] In some implementations, the feature extraction model may include an image processing model and / or be associated with an image processing model trained to preprocess a reference image to identify and / or extract pixels of the reference image associated with and / or depicting the reference image. For example, the image processing model may include an object detection model, a segmentation model, an edge detection model, and / or other types of models configured to determine bounding boxes associated with the depiction of an object within the reference image, and / or be associated with the object detection model, segmentation model, edge detection model, and / or other types of models configured to determine bounding boxes associated with the depiction of an object within the reference image. Thus, image processing techniques can remove any background and / or noise from the reference image, thereby improving the accuracy and efficiency of identifying features of the reference object. In this way, the feature extraction model can be trained using only portions of the reference image that include or indicate features of the reference object (e.g., to facilitate and / or improve unsupervised learning and / or unsupervised training of the feature extraction model).

[0024] Feature extraction models may include or be associated with machine learning models that are trained (e.g., by an object analysis system and / or another system) and / or used as described elsewhere herein. For example, an object analysis system may train a feature extraction model to analyze the type of an object based on a reference image of reference image data associated with that object type. In some implementations, the feature extraction model may be trained to identify a set of features, thereby providing the feature set to a classification model (e.g., as a first feature output from the output layer of the feature extraction model). In this way, the feature set can be used to train a classification model to determine whether an input image depicts an anomalous or non-anomalous object, as described elsewhere herein.

[0025] like Figure 1 As further illustrated, and indicated by reference numeral 130, the object analysis system trains a classification model based on a reference image. For example, the object analysis system may train the classification model based on a set of features identified and / or extracted by a feature extraction model. The object analysis system can train the classification model to determine the anomalous state of objects depicted in the image. For example, the anomalous state may (e.g., according to a binary classification technique) indicate whether the object is anomalous or not. Additionally or alternatively, the anomalous state may indicate whether the object includes an anomalous element, or a specific type of anomalous element.

[0026] The classification model may include a support vector machine (SVM) and / or be associated with an SVM. For example, an object analysis system (and / or another system) may train an SVM to determine and / or predict the similarity to a non-abnormal object depicted in a reference image from reference image data. More specifically, the classification model may determine a classification score based on a feature set via an SVM. Furthermore, the classification model may be trained to compare the classification score with a threshold to determine whether an object depicted in the input image is an anomalous or non-abnormal object. In some implementations, the threshold may be a fixed threshold, such as a fixed value (e.g., within a fixed range) set independently of the feature set learned from the reference image data. Additionally or alternatively, the threshold may be a customized threshold specific to a feature set of one or more reference objects identified by the feature extraction model. Such a customized threshold may be further refined into one or more classification thresholds to account for relatively small variations or deviations in the features of non-abnormal objects depicted in the reference image (e.g., variations or deviations that would not be considered anomalous according to standards or tolerances).

[0027] like Figure 1 As further shown, and indicated by reference numeral 140 in the accompanying drawing, the object analysis system identifies classification thresholds based on test data. For example, as... Figure 1As shown, the sets of test and training data can be configured and / or arranged based on reference image data. More specifically, the first set of training and test data may include a first image (image_1) from a set of N images used as test data, which is compared with the remaining images (image_2 to image_N) used via similarity analysis to train a classification model in a first iteration. Based on the similarity analysis, the classification model (e.g., via a support vector machine) can determine a first classification score (shown as the support vector machine (SVM) score of s_1) for the first set of training and test data. Similarly, the second set of training and test data may include a second image (image_2) from a set of N images used as test data, which is compared with the remaining images (image_1 and images_3 to image_N) used for training a classification model in a second iteration via similarity analysis. Based on the similarity analysis, the classification model (e.g., via a support vector machine) can determine a second classification score (shown as the SVM score of s_2) for the first set of training and test data, and so on. In this way, N classification scores can be obtained from a set of N test data and training data to determine a classification threshold. For example, the classification threshold can be determined based on a threshold identified or learned from the training classification model by moving a set of features identified by the feature extraction model. In this way, based on similarity analysis involving non-abnormal objects depicted in a reference image, an object analysis system (and / or another system) can train (or refine) a classification model to identify a classification threshold to reduce errors or inaccuracies that might otherwise be caused by relatively small variations or biases in the features of the reference object (e.g., negligible differences in the shape, size, color, or configuration of features).

[0028] Therefore, as described in conjunction with Example Implementation 100, the object analysis system (and / or another system) can train a feature extraction model and / or use the feature extraction model to train a classification model to determine the anomalous state of an object based on a reference image depicting a non-abnormal object. In this way, the feature extraction model and the classification model can be coordinated to robustly and accurately detect various anomalies on an object without prior training to identify specific types or characteristics of the anomaly. The feature extraction model and / or classification model can be trained according to any suitable technique, such as combining... Figure 5 As described.

[0029] As mentioned above, Figure 1 Provided as an example. Other examples may be found with reference to [the relevant source]. Figure 1 The descriptions are different. Figure 1 The number and arrangement of equipment shown are provided as an example. In practice, with Figure 1 Compared to the equipment shown, there may be additional equipment, fewer equipment, different equipment, or equipment arranged differently. Furthermore, Figure 1 The two or more devices shown can be implemented in a single device, or Figure 1 The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, Figure 1 The set of devices shown (e.g., one or more devices) can perform actions described as being performed by Figure 1 The other set of devices shown performs one or more functions.

[0030] Figure 2 This is an illustration of an example implementation 200 associated with image-based anomaly detection in an object analysis system. (See illustration for example.) Figure 2 As shown, Example Implementation 200 includes an object analysis system, an object management system, and a user device. In Example Implementation 200, the object analysis system includes a feature extraction model, a classification model, and an anomaly localization model.

[0031] Feature extraction models and / or classification models can be combined Figure 1 Training can be performed using methods similar to those described herein. In example implementation 200, the feature extraction model may include or be associated with a convolutional neural network encoder. Furthermore, the anomaly localization model may correspond to a convolutional neural network decoder. In this way, the feature extraction model and the anomaly localization model can be arranged as a convolutional neural network autoencoder, as described elsewhere herein. The classification model may include or be associated with one or more support vector machines, as described elsewhere herein.

[0032] In Example Implementation 200, the object management system may include an image capture device (e.g., a camera) and / or be associated with an image capture device configured to capture and / or provide images of objects, as described herein. The object management system may include an assembly or manufacturing system, an inventory management system, and / or a transportation system, etc. An object analysis system may be associated with the object management system to facilitate object processing based on whether the object analysis system detects anomalies (or specific types of anomalies) on the objects. In some implementations, the object analysis system may be configured to indicate and / or identify patterns of detected anomalies, which may indicate problems with one or more components or elements of the object management system (e.g., a faulty part of a manufacturing machine causing the object to contain an anomaly). Therefore, the object analysis system may provide information to the object management system and / or user equipment to allow users of the object management system and / or user equipment to process and / or manage objects (or the object management system) based on whether an object is determined to be an anomalous object.

[0033] like Figure 2As shown and indicated by reference numeral 210, the object analysis system receives an input image. For example, the object analysis system may receive the input image from an image capture device. According to the examples described herein, the image capture device may be configured and / or positioned within the object management system to facilitate image-based anomaly detection. In some embodiments, the image capture device may be configured within the object management system in the same or similar location as one or more image capture devices used to capture the aforementioned reference image in conjunction with the trained feature extraction model and / or classification model in example embodiment 100. In this way, the input image can depict objects processed by the object management system in a manner similar to that depicting non-abnormal objects in the aforementioned reference image.

[0034] like Figure 2 As further illustrated and indicated by reference numeral 220, the object analysis system identifies the depicted features of an object. For example, as shown, the object analysis system may receive image data associated with an input image via a feature extraction model and identify object features based on pixel values ​​of the image data (e.g., pixel values ​​depicting the object in the input image). The convolutional neural network encoder of the feature extraction model may be trained based on a reference image depicting a non-abnormal object, as described elsewhere herein. In some embodiments, the convolutional neural network encoder has multiple layers. For example, the convolutional neural network encoder may include an input layer that receives image data, one or more intermediate layers trained to process the image data based on the identified features depicted in the image, and an output layer that provides feature data to a classification model as a first feature output. In this way, the classification model may receive and / or identify the set of features identified by the feature extraction model in the input image, allowing the classification model to determine the anomalous state of the object.

[0035] In some implementations, based on a received input image, the object analysis system can perform one or more preprocessing techniques on the image to facilitate image-based anomaly detection, as described herein. For example, the object analysis system (e.g., via a feature extraction model) can utilize an image processing model (e.g., a model utilizing object detection, segmentation, and / or edge detection techniques) to locate objects depicted in the input image. More specifically, the preprocessing techniques can identify bounding boxes associated with objects and indicating their perimeter based on pixel-level analysis. Thus, pixels depicting objects in the input image can be extracted from pixels of parts of the input image that are not depicted (e.g., to remove background and / or noise from the input image). Additionally or alternatively, pixels of any part of an object that is not depicted can be set to a fixed value (e.g., zero) so that the feature extraction model (or classification model or anomaly localization model) does not have to waste resources analyzing pixels that do not depict a part of an object. In this way, the image data in Example Implementation 200 can include only pixels depicting objects (or associated with objects), such that the feature extraction model analyzes only pixels of the input image that are part of the object.

[0036] The feature extraction model can provide location data to the anomaly localization model. In some implementations, the feature extraction model provides (or enables the anomaly localization model to obtain) location data based on being triggered by the anomaly localization model to determine the location of the anomaly (e.g., based on determining that the object contains the anomaly). In this way, the feature extraction model may not provide location data to the anomaly localization model until the anomaly needs to be located, as described elsewhere in this document.

[0037] The feature extraction model can provide localization data as a second feature output from intermediate layers of the convolutional neural network encoder. Therefore, the first and second feature outputs can come from different layers of the convolutional neural network (e.g., the convolutional neural network encoder) of the feature extraction model. The feature extraction model can be trained to output localization data from intermediate layers that detect features indicating anomalies and / or intermediate layers that detect unknown features (such as features not learned during the training of the feature extraction model and / or classification model). In this way, the feature extraction model can allow the anomaly localization model to determine the location of anomalies in parallel with the feature extraction model, identifying additional features. Therefore, as described elsewhere in this paper, the anomaly localization model can indicate the location of anomalies relatively quickly compared to cases where localization data (or the same as the feature data) is provided from the output layer and / or the anomaly localization model is arranged in series between the feature extraction model and the classification model. In this way, because the classification model does not have to wait as long to receive location information associated with the anomaly, as mentioned above, the object analysis system can detect and / or classify anomalies based on their location, which is faster than the case where the anomaly localization model receives location data from the output layer of the feature extraction model and / or the anomaly localization model is concatenated between the feature extraction model and the classification model, as described elsewhere in this document.

[0038] like Figure 2 As further illustrated, and indicated by reference numeral 230 in the accompanying drawing, the object analysis system locates anomalous features based on pixel errors. For example, the anomaly localization model of the object analysis system can receive pixel errors within the localization data from the feature extraction model and locate anomalous features based on the localization data. As described above, the anomaly localization model can receive localization data as a second feature output of the feature extraction model from a layer different from the first feature output of the feature extraction model, which includes feature data.

[0039] In some implementations, after a classification model detects that an input image depicts an anomalous object, an anomaly localization model can be triggered (e.g., by the classification model) to determine the location of the anomaly and / or acquire localization data. In this case, the anomaly localization model may not receive or acquire localization data until the classification model detects an anomaly on the object and / or (e.g., according to a binary classification technique) determines that the input image depicts an anomalous object. In this way, the anomaly localization model can avoid processing localization data from the feature extraction model until the classification model detects the anomaly, thus saving computational resources that would otherwise be wasted attempting to locate a (non-existent) anomaly in an input image that depicts a non-anomaly.

[0040] A convolutional neural network encoder for an anomaly localization model can be trained based on a reference image depicting a non-abnormal object. For example, the anomaly localization model can be trained in a manner similar to that of a feature extraction model, as described elsewhere in this document. Pixel errors can be determined based on similarity analysis involving the input image and a reconstructed image of the object from the localization data (e.g., using a structural similarity index measure (SSIM) for each identified pixel error). For example, the reconstructed image can represent pixel values ​​corresponding to a depiction of a non-abnormal object. Therefore, based on the confidence level determined by the anomaly localization model for pixel values ​​in the input image (and / or localization data) corresponding to pixel values ​​in the reconstructed image (e.g., not within a specified range of pixel values), the anomaly localization model can identify the location of anomalies on the object (e.g., as at least in conjunction with...). Figure 4 (Further description).

[0041] Anomaly localization models can indicate the location within anomaly location information. For example, anomaly location information can identify the coordinates of an image and / or its position relative to an object (e.g., using orientation identifiers such as top, bottom, middle, left portion, right portion and / or surface identifiers such as top surface, side surface, bottom surface, etc.). In some implementations, the anomaly localization model can determine the size of the anomaly based on the number of pixel errors, the number of pixels in a cluster determined according to the pixel errors (e.g., a group of pixels within a region or boundary formed by the identified pixel errors), and / or the coordinates of the image including the pixel errors. The anomaly localization model can indicate the size of the anomaly within the anomaly location information to a classification model. In this way, using anomaly localization models, object analysis systems can determine the location of anomalies and / or indicate the location of anomalies to a support vector machine.

[0042] like Figure 2 As further illustrated and indicated by reference numeral 240, the object analysis system determines a classification score based on features. For example, the classification model of the object analysis system may receive feature data and / or anomaly location information, and determine a classification score indicating an abnormal state of the object based on the feature data and / or anomaly location information. In some embodiments, the classification model may determine the classification score based on the first feature output of a feature extraction model (e.g., feature data from the output layer of a convolutional neural network encoder) (e.g., via a support vector machine).

[0043] As described above, the classification model may include a support vector machine (SVM) or associated with an SVM configured to provide a classification score indicating an anomalous state. In some embodiments, the SVM of the classification model may be a single-class SVM specifically trained to analyze the type of objects depicted in an input image, as described elsewhere herein. An anomalous state may indicate whether the input image depicts an object with anomalous features based on a classification score indicating that the object includes anomalous features. An anomalous state may be determined and / or indicated based on a comparison of the classification score and a classification threshold (e.g., a threshold associated with whether the anomalous features indicate an anomalous state). In this case, the classification model may output a binary classification based on the classification threshold (based on whether the classification score meets the classification threshold). Thus, an anomalous state may indicate whether the object includes anomalous features. In some embodiments, if the object includes anomalous features, the anomalous state may indicate certain characteristics of the anomalousness associated with the anomalous features (e.g., type, location, and / or size, etc.).

[0044] In some implementations, the anomaly localization model can provide a binary classification (e.g., indicating whether an object is anomalous or not) to the classification model. In this way, the classification model can combine the binary classification from the anomaly localization model with the support vector machine classification (e.g., a binary classification from a support vector machine) of the classification model to verify that the classification score accurately indicates whether the object is anomalous or not. Therefore, the anomaly localization model can improve the confidence and / or accuracy regarding whether an object is detected (or predicted) to be anomalous. In some implementations, if the binary classification from the anomaly localization model does not verify or prove the support vector machine classification of the classification model (or vice versa), the object analysis system can (e.g., indicate to the user device and / or the object management system) that further processing is required. Additionally or alternatively, the object analysis system can request or obtain another input image depicting the object (and perform analysis based on that other input image), and / or can cause the object management system to reconfigure the object before requesting or obtaining another image depicting the object.

[0045] exist Figure 2In this model, the classification model can utilize multiple support vector machines (SVMs) to classify anomalies identified on an object. For example, the classification model may include: a first SVM trained (e.g., based on a first classification threshold) to determine a binary classification indicating whether an object contains an anomaly; a second SVM trained (e.g., based on a second classification threshold) to determine a binary classification indicating whether the identified anomaly is a scratch; and a third SVM trained (e.g., based on a third classification threshold) to determine whether the anomaly is a discoloration, and so on. Therefore, as shown in the figure, via the first SVM, the classification model can generate anomaly data indicating that the object contains an anomaly and / or that the two identified features of the object correspond to two anomalies (anomaly 1 and anomaly 2). Furthermore, as shown in the figure, via the second SVM, the classification model can generate anomaly data indicating that the first anomaly (anomaly 1) is a scratch. Furthermore, via the third SVM, the classification model can generate anomaly data indicating that the second anomaly (anomaly 2) is a discoloration.

[0046] In some implementations, as shown in the figures, the object analysis system can combine anomaly location information with anomaly classification to generate anomaly data. For example, as shown in the figures, the anomaly data for a first anomaly could indicate that a scratch is located on the object, as depicted at coordinates (x1, y1), and has a size of 3 mm. Furthermore, the anomaly data for a second anomaly could indicate that a discoloration is located on the object, as depicted at coordinates (x2, y2), and has a size of 1 mm. In some implementations, as described elsewhere herein, the anomaly data can be combined with an input image to indicate the location and / or type of anomaly. For example, the object analysis system can generate location indicators (e.g., highlighting, outlines, arrows, and / or overlays, etc.) that indicate the location of anomalies on objects depicted in the input image by overlaying the location indicators on the input image and / or embedding the location indicators within the input image.

[0047] like Figure 2As further illustrated and indicated by reference numeral 250, the object analysis system provides anomalous data to the user device. For example, the object analysis system may provide a notification to the user device including anomalous data and / or a user warning (e.g., via a prompt or message indicator) that objects include anomalous data (e.g., to allow the user to identify and / or address objects within the object management system). In some embodiments, the object analysis system may generate reports based on the anomalous data. For example, the report may be associated with a period of time or a batch of objects analyzed by the object analysis system, as described herein. In some embodiments, the report may indicate statistics associated with detected anomalies (e.g., the number of objects identified as including anomalous data, the number of certain types of anomalies, patterns or trends associated with certain characteristics of the anomalies, etc.). In some embodiments, the object analysis system may base its reporting on detected objects including anomalous data, based on the number of analyzed objects including anomalous data detected within a threshold period, based on a threshold percentage of analyzed objects including anomalous data detected within a specific period, and / or based on a specific trend that has developed in association with a particular characteristic of anomalies on objects within a specific period (e.g., multiple objects are identified as having the same or similar anomalies, which may indicate a problem with one or more components of the object management system causing the anomalies).

[0048] like Figure 2 As shown and indicated by reference numeral 260, the object analysis system facilitates object processing. For example, the object analysis system can provide anomaly data to the object management system to enable the object management system to manage objects. More specifically, the object analysis system can provide anomaly data to the object management system to enable the object management system to control one or more devices to manage objects according to one or more operations. Such operations may include: discarding objects (e.g., by removing objects from processing), marking objects as anomalies (e.g., via a marking mechanism of the object analysis system), and / or routing objects to designated areas for anomaly objects (e.g., areas where the anomaly object can be further inspected and / or repaired), etc. In some embodiments, the object analysis system can control the object management system based on anomaly data to perform one or more operations associated with the object.

[0049] In this way, object analysis systems can leverage robust and accurate image-based anomaly detection models to facilitate object management and / or processing and ensure that analyzed objects meet certain criteria or standards (e.g., are non-abnormal objects) before being output from the object management system, used in the domain, and / or sold to consumers. This reduces or prevents objects that do not meet specific criteria or standards (e.g., anomalous objects) from posing a danger or degrading the consumer experience.

[0050] As mentioned above, Figure 2Provided as an example. Other examples may be found with reference to [the relevant source]. Figure 2 The descriptions are different. Figure 2 The number and arrangement of equipment shown are provided as an example. In practice, with Figure 2 Compared to the equipment shown, there may be additional equipment, fewer equipment, different equipment, or equipment arranged differently. Furthermore, Figure 2 The two or more devices shown can be implemented in a single device, or Figure 2 The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, Figure 2 The set of devices shown (e.g., one or more devices) can perform actions described as being performed by Figure 2 The other set of devices shown performs one or more functions.

[0051] Figure 3 These are illustrations of example implementations associated with the classification model described in this article. Figure 3 As shown, the example implementation includes an example of an arrangement of support vector machines (SVMs) that can be included within, trained by, and / or associated with a classification model described elsewhere herein. This arrangement can include a cascade 300 of multiple SVMs, including a first SVM and subsequent SVMs (sSVM_t) for various types(t) of anomalies. The first and subsequent SVMs can be separate single-class SVMs trained independently for a specific purpose. More specifically, the first SVM can be trained to provide a binary classification of whether a received input image contains an anomaly. The threshold of the SVM can vary and can be determined based on a reference image of a non-anomaly object, as described elsewhere herein. Furthermore, each subsequent SVM in the cascade 300 can provide a binary classification indicating whether the anomaly is of the corresponding type of anomaly detected by the respective subsequent SVM.

[0052] Therefore, as shown by reference numeral 302, the first support vector machine (SVM) can analyze the input image to determine whether the input image depicts a non-abnormal object. If the first SVM determines that the input image depicts a non-abnormal object, then the first SVM indicates that the object is "OK" (e.g., this can represent non-abnormality), as shown by reference numeral 304. If the first SVM determines that the input image depicts an anomalous object, then the first SVM can indicate that an anomalous object has been detected, as shown by reference numeral 306. Furthermore, as shown by reference numeral 308, in the first iteration, subsequent SVMs determine whether the detected anomalous object corresponds to a specific type of anomalous object. If a subsequent SVM determines that the anomalous object is a specific type of anomalous object, then it can indicate that specific type, as shown by reference numeral 310. If a subsequent SVM determines that the anomalous object is not a specific type of anomalous object, then it can indicate that the subsequent SVM is trained to identify whether the cascade should check for another anomalous object, as shown by reference numeral 312. If the subsequent SVM is the last SVM in the cascade, then the classification analysis can end. Otherwise, as shown in Figure 314, the classification model uses another subsequent support vector machine for iterative analysis, which is trained to determine whether the anomaly is another type of anomaly, and so on.

[0053] Subsequent support vector machines (SVMs) can be arranged within the cascade in any suitable manner. For example, the most frequently detected anomaly types can be located in the cascade closest to the first SVM (e.g., to reduce the likelihood of needing to perform multiple iterations through the cascade). Additionally or alternatively, the simplest or least complex SVM (e.g., which can be configured to identify the most easily detectable anomalies among the SVMs) can be located in the cascade closest to the first SVM (e.g., to ensure that anomalies are analyzed first for the most easily detectable anomalies, which may require relatively less processing power).

[0054] Therefore, as Figure 3 As shown, a first support vector machine can be configured within a cascade to output a first binary classification indicating that the object includes anomalous features. Furthermore, a second support vector machine (e.g., one of subsequent support vector machines) can be configured to output a second binary classification indicating whether the anomaly is a specific type of anomaly or not, based on the first binary classification indicating that the object includes anomalous features. Accordingly, the classification model can generate anomalous data based on the second binary classification, the anomalous data including labels indicating whether the anomaly is a specific type of anomaly or not.

[0055] As mentioned above, Figure 3 Provided as an example. Other examples may be found with reference to [the relevant source]. Figure 3 The descriptions are different.

[0056] Figure 4 This is an illustration of an example implementation associated with the anomaly localization model 400. (See illustration for example.) Figure 4 As shown, the anomaly localization model 400 includes a convolutional neural network autoencoder, which includes an encoder (e.g., corresponding to the convolutional neural network decoder described in conjunction with the feature extraction model described above) and a decoder (e.g., corresponding to the convolutional neural network decoder described in conjunction with the anomaly localization model described above) trained according to examples described elsewhere in this document. Furthermore, the anomaly localization model 400 includes a comparator module, a classifier module, and a clustering module.

[0057] As shown in the figure, at reference numeral 402, the convolutional neural network autoencoder receives an input image. The input image is shown as including anomalies. The encoder identifies features in the input image provided to the decoder. The decoder generates a reconstructed image of a reference object based on these features (e.g., representing the object type that the convolutional autoencoder was trained to identify), as shown at reference numeral 404. At reference numeral 406, a comparator module compares the input image and the reconstructed image (e.g., using SSIM analysis of per-pixel error). Based on the comparison of pixel values ​​(and / or the confidence level of pixel values ​​in the input image corresponding to the pixel values ​​in the reconstructed image), pixel errors that indicate anomalies can be detected, and / or the location of the pixel errors can correspond to the location of anomalies. At reference numeral 408, the comparator module can generate an anomaly heatmap, which can be used to indicate the location of anomalies. Additionally or alternatively, at reference numeral 410, a clustering module can perform clustering techniques (e.g., k-means clustering) to determine the regions and / or surroundings of anomalies to be included or indicated in the anomaly location information, as shown at reference numeral 412. In this way, the anomaly localization model can indicate and / or provide the classification model with location information associated with the anomaly and / or that can be used to generate anomaly data associated with the input image and / or the objects depicted in the input image.

[0058] As mentioned above, Figure 4 Provided as an example. Other examples may be found with reference to [the relevant source]. Figure 4 The descriptions are different.

[0059] Figure 5 This is an illustration of example 500 of training and using a machine learning model in conjunction with image-based anomaly detection. The training and use of the machine learning model described herein can be performed using a machine learning system. A machine learning system may include, or may be included in, computing devices, servers, cloud computing environments, etc. (such as object analysis systems described in more detail elsewhere herein).

[0060] As shown by reference numeral 505 in the accompanying drawings, an observation set can be used to train a machine learning model. The observation set can be obtained from training data (e.g., historical data), such as data collected during one or more processes described herein. In some implementations, the machine learning system may receive the observation set (e.g., as input) from a reference data structure and / or an object management system (e.g., from an image capture device of the object management system), as described elsewhere herein.

[0061] As indicated by reference numeral 510 in the accompanying drawings, the observation set includes a feature set. The feature set may include a set of variables, and variables may be referred to as features. A specific observation may include a set of variable values ​​(or feature values) corresponding to the variable set. In some implementations, the machine learning system may determine the variables of the observation set and / or the variable values ​​of a specific observation based on input received from a reference data structure and / or an object management system. For example, the machine learning system may identify a feature set (e.g., one or more features and / or feature values) by: extracting a feature set from structured data (e.g., image data associated with images depicting non-abnormal objects), performing image processing techniques to extract a feature set from unstructured data (e.g., image data associated with images depicting both anomalous and non-abnormal objects), and / or receiving input from an operator.

[0062] As an example, the feature set of an observation set may include first contour data features (e.g., a representation of a physical element or aspect of an object that can be identified in an image of the object), second pixel data features (e.g., the red, green, and blue (RGB) color values ​​of the pixels of the feature), third position data features (e.g., coordinates identifying the location of the feature in the observation), and so on. As shown, for a first observation, the first feature may have the value contour_1 (e.g., an identifier of the feature type), the second feature may have the value RGB_1 (e.g., the value of one or more pixels associated with the feature in the observation), the third feature may have the value (X1, Y1) (e.g., a set of coordinates relative to a reference point of the object and / or coordinates relative to a reference point of an image of the object, such as a reference image or an image captured by an image capture device), and so on. These features and feature values ​​are provided as examples and may differ in other examples. For example, a feature set may include one or more of the following features: size data (e.g., data representing an image region that describes the features of the observation), shape data (e.g., data representing the perimeter of an object), source data (e.g., data identifying the source device associated with the observation), object type data (e.g., data identifying the type of object associated with the observation), object size data (e.g., data identifying the size of an object), and so on.

[0063] As shown by reference numeral 515 in the attached figure, the observation set can be associated with a target variable. The target variable can represent a variable with numerical values, a variable with numerical values ​​falling within a range of values, or a variable with some discrete possible values, a variable that can be selected from one of several options (e.g., one of multiple categories, classifications, or labels), and / or a variable with Boolean values. The target variable can be associated with target variable values, and the target variable values ​​can be observation-specific. In Example 500, the target variable is a feature type that has a value feature_1 for a first observation and a value feature_2 for a second observation. Feature_1 and / or feature_2 can correspond to features of non-abnormal objects associated with an object type (e.g., because a reference image depicting a non-abnormal object can be used to train a machine learning model). Accordingly, the features (feature_1 and feature_2) can be associated with a reference object to which the machine learning model is trained to identify and / or configure based on the observations.

[0064] The target variable can represent the value that a machine learning model is being trained to predict, and the feature set can represent the variables that are input into the trained machine learning model to predict the value of the target variable. The observation set can include the values ​​of the target variable, allowing the machine learning model to be trained to recognize patterns in the feature set that lead to the values ​​of the target variable. A machine learning model trained to predict the value of the target variable can be called a supervised learning model.

[0065] In some implementations, a machine learning model can be trained on an observation set that does not include the target variable. This can be referred to as an unsupervised learning model. In this case, the machine learning model can learn patterns from the observation set without labels or supervision, and can provide an output indicating such patterns, for example, by using clustering and / or association to identify related itemsets within the observation set.

[0066] As shown in the attached figure 520, the machine learning system can use an observation set and one or more machine learning algorithms (such as regression, decision tree, neural network, k-nearest neighbor, support vector machine, etc.) to train a machine learning model. After training, the machine learning system can store the machine learning model as trained machine learning model 525 for analysis of new observations.

[0067] As shown by reference numeral 530 in the figure, the machine learning system can apply the trained machine learning model 525 to a new observation, such as by receiving the new observation and inputting the new observation into the trained computer learning model 525. As an example, as shown, the new observation may include a first feature contour_N, a second feature RGB_N, and a third feature (X... N Y NThe machine learning system can apply a trained machine learning model 525 to new observations to generate outputs (e.g., results). The type of output can depend on the type of machine learning model and / or the type of machine learning task being performed. For example, when using supervised learning, the output may include predicted values ​​for the target variable. Additionally or alternatively, when using unsupervised learning, the output may include information identifying the cluster to which the new observation belongs and / or information indicating the similarity between the new observation and one or more other observations.

[0068] As an example, a trained machine learning model 525 can predict anomaly values ​​for a target variable of a new observation's feature type, as indicated by reference numeral 535. For example, the target variable can indicate anomalies to suggest that the observation is associated with an anomalous object. The trained machine learning model 525 can predict anomaly values ​​when the feature set of a new observation cannot be mapped to the feature type learned during training the machine learning model. Based on this prediction, the machine learning system can provide a first recommendation, provide output for determining the first recommendation, perform a first automated action, and / or enable the first automated action to be performed (e.g., by instructing another device to perform the automated action), etc. The first recommendation may include, for example, a recommendation by a user device to resolve anomalies on a newly observed object and / or a recommendation by an object management system to manage objects based on including anomalies (e.g., sorting or removing objects from non-anomaly objects). The first automated action may include, for example, providing anomaly data to a user device to indicate that a newly observed object includes anomalies and / or enabling the object management system to manage the newly observed object based on including anomalies (e.g., classifying or removing objects from non-anomaly objects).

[0069] As another example, if a machine learning system is to predict the value associated with a learned feature of a target variable of a feature type, the machine learning system can provide a second (e.g., different) recommendation (e.g., a recommendation to use or output an object or a recommendation that enables the use or output of an object) and / or can perform a second (e.g., different) automated action (e.g., enabling the use or output of an object) or cause the execution of a second automated action.

[0070] In some implementations, the trained machine learning model 525 can classify (e.g., cluster) new observations into clusters, as indicated by reference numeral 540. Observations within a cluster may have a threshold similarity. As an example, if the machine learning system classifies a new observation into a first cluster (e.g., a cluster associated with anomalies of a first type), the machine learning system can provide a first recommendation, such as the first recommendation described above. Additionally or alternatively, the machine learning system may perform a first automatic action based on classifying the new observation into the first cluster and / or may (e.g., by instructing another device to perform an automatic action) cause the first automatic action to be performed, such as the first automatic action described above.

[0071] As another example, if a machine learning system is to classify a new observation into a second cluster (e.g., a cluster associated with a second type of anomaly), the machine learning system can provide a second (e.g., different) recommendation (e.g., recommending damaged or recycled objects if the second type of anomaly is an unrepairable type) and / or can perform a second (e.g., different) automated action (such as damaging objects) or cause the execution of the second automated action.

[0072] In some implementations, recommendations and / or automated actions associated with new observations may be based on target variable values ​​with specific labels (e.g., classification or categorization), may be based on whether the target variable value meets one or more thresholds (e.g., the target variable value is greater than a threshold, less than a threshold, equal to a threshold, or falls within a threshold range, etc.), and / or may be based on the cluster into which the new observation is classified.

[0073] In this way, machine learning systems can apply rigorous and automated processes to detect and / or classify anomalies associated with objects. Machine learning systems enable the identification and / or labeling of tens, hundreds, thousands, or millions of features and / or feature values ​​for tens, hundreds, thousands, or millions of observations, thereby improving accuracy and consistency and reducing latency associated with the detection and / or classification of anomalies associated with objects, relative to the latency of manually detecting and / or classifying anomalies associated with objects using features or feature values, which would require allocating computational resources to tens, hundreds, or thousands of operators.

[0074] As mentioned above, Figure 5 Provided as an example. Other examples may be combined. Figure 5 The descriptions are different.

[0075] Figure 6 This is a diagram of an example environment 600 in which the systems and / or methods described herein can be implemented. (See diagram 600 for example environment 600.) Figure 6As shown, environment 600 may include object analysis system 610, reference image data structure 620, object management system 630, user equipment 640, and network 650. The devices in environment 600 can be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0076] Object analysis system 610 includes one or more devices capable of receiving, generating, storing, processing, providing, and / or routing information associated with image-based anomaly detection based on machine learning analytics as described elsewhere herein. Object analysis system 610 may include communication devices and / or computing devices. For example, object analysis system 610 may include servers such as application servers, client servers, web servers, database servers, host servers, proxy servers, virtual servers (e.g., executed on computing hardware), or servers in a cloud computing system. In some embodiments, object analysis system 610 includes computing hardware used in a cloud computing environment.

[0077] The reference image data structure 620 includes one or more devices capable of generating, storing, processing, and / or providing reference image data associated with one or more types of objects to train one or more models, as described elsewhere herein. For example, the reference image data structure 620 may include storage devices and / or memory devices for receiving and / or storing reference images from one or more image sources. Additionally or alternatively, the reference image data structure may include communication devices and / or computing devices for receiving, processing, and / or providing reference image data to the object analysis system 610.

[0078] The object management system 630 includes one or more devices capable of receiving, generating, storing, processing, providing, and / or routing information associated with managed objects, as described elsewhere herein. For example, the object management system 630 may include one or more devices configured to facilitate the assembly, manufacture, sort, allocate, transport, and / or store of one or more objects. The object management system 630 may include communication devices, computing devices, sensors, robotic devices, and / or any other suitable devices for control systems associated with a particular industry (e.g., manufacturing, logistics, transportation, and / or other industries associated with supply chain management).

[0079] User equipment 640 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with monitoring objects analyzed by object analysis system 610 and / or managed by object management system 630, as described elsewhere herein. User equipment 640 may include communication devices and / or computing devices. For example, user equipment 640 may include wireless communication devices, mobile phones, user equipment, laptops, tablets, desktop computers, game consoles, set-top boxes, wearable communication devices (e.g., smartwatches, smart glasses, head-mounted displays, or virtual reality headsets) or similar types of devices.

[0080] Network 650 includes one or more wired and / or wireless networks. For example, network 650 may include a wireless wide area network (e.g., a cellular network or public terrestrial mobile network), a local area network (e.g., a wired local area network or a wireless local area network (WLAN), such as a Wi-Fi network), a personal area network (e.g., a Bluetooth network), a near-field communication network, a telephone network, a private network, the Internet, and / or combinations of these or other types of networks. Network 650 enables communication between devices in environment 600.

[0081] Figure 6 The number and arrangement of devices and networks shown are provided as examples. In practice, with Figure 6 Compared to the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged differently. Furthermore, Figure 6 The two or more devices shown can be implemented in a single device, or Figure 6 The single device shown can be implemented as multiple distributed devices. Additionally or alternatively, a group of devices in environment 600 (e.g., one or more devices) can perform one or more functions, which are described as being performed by another group of devices in environment 600.

[0082] Figure 7 This is an illustration of example components of a device 700 that may correspond to an object analysis system 610, a reference image data structure 620, an object management system 630, and / or a user device 640. In some embodiments, the object analysis system 610, the reference image data structure 620, the object management system 630, and / or the user device 640 may include one or more devices 700 and / or one or more components of device 700. For example... Figure 7 As shown, device 700 may include bus 710, processor 720, memory 730, storage unit 740, input unit 750, output unit 760 and communication unit 770.

[0083] Bus 710 includes components that enable wired and / or wireless communication between parts of device 700. Processor 720 includes a central processing unit, graphics processing unit, microprocessor, controller, microcontroller, digital signal processor, field-programmable gate array, application-specific integrated circuit, and / or another type of processing component. Processor 720 is implemented in hardware, firmware, or a combination of hardware and software. In some embodiments, processor 720 includes one or more processors that can be programmed to perform functions. Memory 730 includes random access memory, read-only memory, and / or another type of memory (e.g., flash memory, magnetic storage, and / or optical storage).

[0084] Storage component 740 stores information and / or software related to the operation of device 700. For example, storage component 740 may include a hard disk drive, disk drive, optical disk drive, solid-state drive, optical disk, digital universal disk, and / or another type of non-transitory computer-readable medium. Input component 750 enables device 700 to receive input, such as user input and / or sensed input. For example, input component 750 may include a touchscreen, keyboard, keypad, mouse, buttons, microphone, switch, sensor, GPS component, accelerometer, gyroscope, and / or actuator. Output component 760 enables device 700 to provide output, such as via a display, speaker, and / or one or more light-emitting diodes. Communication component 770 enables device 700 to communicate with other devices, such as via a wired connection and / or wireless connection. For example, communication component 770 may include a receiver, transmitter, transceiver, modem, network interface card, and / or antenna.

[0085] Device 700 can perform one or more of the processes described herein. For example, a non-transitory computer-readable medium (e.g., memory 730 and / or storage unit 740) can store a set of instructions (e.g., one or more instructions, code, software code, and / or program code) for execution by processor 720. Processor 720 can execute the set of instructions to perform one or more of the processes described herein. In some embodiments, execution of the set of instructions by one or more processors 720 causes one or more processors 720 and / or device 700 to perform one or more of the processes described herein. In some embodiments, hardwired circuitry may be used in place of or in combination with instructions to perform one or more of the processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0086] Figure 7 The number and arrangement of the components shown are provided as an example. Figure 7Compared to the arrangement shown, device 700 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a set of components of device 700 (e.g., one or more components) may perform one or more functions, which are described as being performed by another set of components of device 700.

[0087] Figure 8 This is a flowchart of an example process 800 associated with image-based anomaly detection in object-based machine learning analytics. In some implementations, Figure 8 One or more processing blocks can be executed by an object analysis system (e.g., object analysis system 610). In some implementations, Figure 8 One or more processing blocks may be executed by another device or set of devices, separate from or including the object analysis system, such as an object management system (e.g., object management system 630), and / or a user device (e.g., user device 640). Additionally or alternatively, Figure 8 One or more processing blocks can be executed by one or more components of the device 700 (such as processor 720, memory 730, storage component 740, input component 750, output component 760 and / or communication component 770).

[0088] like Figure 8 As shown, process 800 may include receiving an input image depicting an object (box 810). For example, an object analysis system may receive an input image depicting an object, as described above.

[0089] like Figure 8 As further shown, process 800 may include using a convolutional neural network encoder to determine a first feature output associated with one or more features of an object based on the input image (box 820). For example, an object analysis system may use a convolutional neural network encoder to determine a first feature output associated with one or more features of an object based on the input image, as described above. The convolutional neural network encoder may be associated with and / or included within the feature extraction model of the object analysis system.

[0090] In some implementations, the convolutional neural network encoder is trained based on a reference image depicting a reference object as a type of object. The reference object depicted in the reference image can be a non-abnormal object, as described herein.

[0091] like Figure 8As further shown, process 800 may include using a support vector machine to determine an abnormal state of an object indicating that the object includes an anomaly (box 830). For example, an object analysis system may use a support vector machine to determine an abnormal state of an object indicating that the object includes an anomaly, as described above. In some embodiments, the support vector machine is trained based on a reference image.

[0092] Support vector machines (SVMs) can be trained to determine binary classifications that indicate whether an object contains anomalous features or not. The SVM can be trained to determine a classification threshold, which is used to determine the binary classification based on similarity analysis involving a reference image.

[0093] like Figure 8 As further shown, process 800 may include using a convolutional neural network decoder based on the second feature output of the convolutional neural network encoder to determine the location of an anomaly in the input image (box 840). For example, an object analysis system may use a convolutional neural network decoder based on the second feature output of the convolutional neural network encoder to determine the location of an anomaly in the input image, as described above. The convolutional neural network decoder may be associated with and / or included within the anomaly localization model of the object analysis system.

[0094] In some implementations, the convolutional neural network decoder is configured to determine the location of anomalies based on a second feature output from the convolutional neural network encoder. In some implementations, the convolutional neural network decoder is trained on a reference image. The first feature output may come from the output layer of the convolutional neural network encoder, and the second feature output may come from an intermediate layer of the convolutional neural network encoder. The convolutional neural network encoder and the convolutional neural network decoder may be associated with the same convolutional neural network autoencoder trained on the reference image.

[0095] In some implementations, the object analysis system can generate anomalous data associated with abnormal data. For example, anomalous data can identify the abnormal state of an object (e.g., whether the object includes or does not include an anomaly) and / or the location of the anomaly (if an anomaly is detected). Additionally or alternatively, using an anomaly classification model, anomalous data can be generated to indicate that the anomaly is a specific type of anomaly.

[0096] like Figure 8As further shown, process 800 may include performing an action associated with the location of the anomaly (block 850). For example, the object analysis system may perform an action associated with the location of the anomaly, as described above. In some embodiments, to perform the action, the object analysis system may generate a location indicator identifying the anomaly location, combine the location indicator with an input image to form an anomaly indicator, and provide the anomaly indicator to a user device. Additionally or alternatively, the object analysis system may (e.g., to a user device) provide anomaly data identifying the anomaly location. The anomaly data may include a location indicator identifying the anomaly location and / or a combination of the location indicator and an input image.

[0097] although Figure 8 An example block of process 800 is shown, but in some implementations, it is different from... Figure 8 Compared to the boxes depicted, process 800 may include additional boxes, fewer boxes, different boxes, or boxes arranged differently. Alternatively, two or more boxes in process 800 may be executed in parallel.

[0098] Specific embodiments have been described in the foregoing disclosure. However, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the invention as set forth in the following claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this disclosure. Additionally, the described embodiments / examples / implementations should not be construed as mutually exclusive, but rather as potentially composable if such combinations are permitted in any way. In other words, any feature disclosed in any of the foregoing examples or implementations may be included in any of the other foregoing examples or implementations.

[0099] As used herein, the term "component" is intended to be interpreted broadly as hardware, firmware, and / or a combination of hardware and software. As used herein, each of the terms "tangible machine-readable medium," "non-transitory machine-readable medium," and "machine-readable storage device" is explicitly defined as a storage medium (e.g., a hard disk drive platter, a digital universal disk, an optical disk, flash memory, read-only memory, random access memory, etc.) on which machine-readable instructions (e.g., code in the form of software and / or firmware) may be stored. Instructions may be stored for any suitable duration, such as permanently, for extended periods (e.g., while the program associated with the instructions is executing), or for short periods (e.g., when the instructions are cached, during a buffering process, etc.). Furthermore, as used herein, each of the terms "tangible machine-readable medium," "non-transitory machine-readable medium," and "machine-readable storage device" is explicitly defined to exclude the propagation of signals. That is, as used in any claim herein, "tangible machine-readable medium," "non-transitory machine-readable medium," and "machine-readable storage device," etc., should not be construed as being implemented as a propagation of signals.

[0100] As used in this article, depending on the context, a threshold can refer to a value that is greater than the threshold, greater than or equal to the threshold, less than the threshold, less than or equal to the threshold, equal to the threshold, not equal to the threshold, etc.

[0101] These benefits, advantages, solutions to problems, and any elements(s) that may make any benefit, advantage, or solution occur or become more prominent are not to be construed as key, essential, or necessary features or elements of any or all claims. The claimed invention is defined solely by the appended claims, including any modifications made during the pending examination of this application and all equivalents of those claims in the grant announcement.

[0102] Furthermore, as used herein, relational terms such as first and second, top and bottom, etc., can be used individually to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms “comprises,” “has,” “having,” “includes,” “including,” “contains,” “containing,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes, has, includes, or contains a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element beginning with “comprises—,” “has—,” “includes—,” or “contains—” does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes, has, includes, or contains that element, unless further constraints are imposed.

[0103] The terms “a” and “an” are defined as one or more unless expressly stated otherwise herein. Furthermore, as used herein, the article “the” is intended to include one or more items referenced in combination with the article “the” and may be used interchangeably with “the one or more”. Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, or a combination of related and unrelated items) and may be used interchangeably with “one or more”. The phrase “only one” or similar language is used if intended to indicate only one item. Furthermore, as used herein, the terms “has,” “have,” “having,” or similar terms are intended to be open-ended terms. Furthermore, unless expressly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Furthermore, as used herein, the term “or” is intended to be inclusive when used in a series and may be used interchangeably with “and / or” unless expressly stated otherwise (e.g., if used in combination with “any” or “only one of…”). The terms “basically,” “approximately,” “about,” “approximately,” or any other version of these terms are defined as being as close as understood by those skilled in the art, and in one non-limiting embodiment, these terms are defined as being within 10%, in another within 5%, in yet another within 1%, and in still another within 0.5%. The term “coupled” as used herein is defined as connected, although not necessarily directly connected or mechanically connected. A device or structure “configured” in a certain way is configured at least in this manner, but may also be configured in ways not listed.

[0104] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not a limitation on the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0105] Even if specific combinations of features are described in the claims and / or disclosed in the description, these combinations are not intended to limit the disclosure of various embodiments. In fact, many of these features can be combined in ways not specifically set forth in the claims and / or not disclosed in the description. Although each dependent claim listed below may directly depend on only one claim, the disclosure of various embodiments includes combinations of each dependent claim with each other claim in the claims. As used herein, the phrase “at least one of” in a list of items refers to any combination of these items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical items.

[0106] This abstract is provided to allow the reader to quickly determine the nature of the disclosure. This abstract is submitted with the understanding that it is not intended to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen from the foregoing description, various features have been grouped together in various embodiments for the purpose of making this disclosure holistic. This method of disclosure should not be construed as reflecting an intention to require more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive subject matter may lie in fewer than all the features of a single disclosed embodiment. Therefore, the following claims are thus incorporated into the detailed description, wherein each claim represents itself as a separately claimed subject matter.

Claims

1. A method associated with detecting anomalies related to an object, comprising: The device receives an input image depicting the object; The input image is processed by the device using a feature extraction model to indicate one or more features of the object in a first feature output. The feature extraction model is trained based on a reference image associated with the type of the object. The reference image depicts one or more non-abnormal objects of the same type as the object. The device, and based on the one or more features, uses a classification model to determine the abnormal state of the object, indicating that the object includes an anomaly. The classification model is configured to determine the abnormal state based on a classification score associated with the first feature output and a classification threshold of the classification model. The classification threshold is determined based on similarity analysis involving the reference image. The location of the anomaly associated with the abnormal state is determined by the device and based on the second feature output of the feature extraction model. The location of the anomaly is determined using an anomaly localization model trained based on the reference image. The device generates abnormal data associated with the abnormality based on the abnormal state and the location; and The device provides the abnormal data to the object management system.

2. The method as described in claim 1, characterized in that, The classification model includes a support vector machine, which is configured as follows: The classification score is determined based on the first feature output; and The abnormal state is indicated by comparing the classification score with the classification threshold. The support vector machine is a single-class support vector machine specifically trained to analyze the type of the object. The abnormal state is a binary classification determined based on the comparison and indicating whether the object has abnormal characteristics or does not have abnormal characteristics.

3. The method as described in claim 1, characterized in that, The classification model includes: A first support vector machine is configured to output a first binary classification based on the classification threshold. Wherein, the first binary classification indicates that the object includes anomalous features, and A second support vector machine is configured to output a second binary classification indicating whether the anomaly is a specific type of anomaly or not, based on a first binary classification that indicates the object includes anomalous features. The abnormal data is generated as a label based on the second binary classification, indicating whether the abnormality is an abnormality of the specific type or not.

4. The method as described in claim 1, characterized in that, The first feature output comes from the output layer of the convolutional neural network encoder of the feature extraction model, and The second feature output comes from the intermediate layer of the convolutional neural network encoder.

5. The method as described in claim 1, characterized in that, The anomaly localization model includes a convolutional neural network decoder configured to determine the location of the anomaly.

6. The method as described in claim 5, characterized in that, The second feature output comes from the intermediate layer of the convolutional neural network decoder of the feature extraction model.

7. The method as described in claim 1, characterized in that, The generation of the abnormal data includes: Generate a location indicator that identifies the location of the anomaly; and Combine the location indicator with the input image.

8. An apparatus associated with detecting an anomaly related to an object, comprising: One or more memory units; as well as One or more processors, coupled to the one or more memories, are configured to: Receive the input image depicting the object; The input image is processed using a feature extraction model to generate a first feature output associated with one or more features of the object. The feature extraction model is trained based on a reference image associated with the type of the object. The abnormal state of the object is determined using a classification model based on the output of the first feature. The classification model is trained to determine the anomalous state based on similarity analysis involving non-abnormal objects depicted in the reference image. Based on the abnormal state indication, the input image depicts the object with the abnormality, and the location of the abnormality in the input image is determined based on the second feature output of the feature extraction model. The location of the anomaly is determined using an anomaly localization model trained based on the reference image; Based on the abnormal state and the location, generate abnormal data associated with the abnormality; and Perform the action associated with the abnormal data.

9. The device as described in claim 8, characterized in that, The feature extraction model includes a convolutional neural network encoder.

10. The device as claimed in claim 8, characterized in that, The classification model includes a support vector machine configured to provide a classification score indicating whether the object includes or does not contain anomalous features. The abnormal state is configured to indicate that the input image depicts the object having the abnormality based on the classification score indicating that the object includes abnormal features.

11. The device as claimed in claim 10, characterized in that, The similarity analysis is configured to determine a classification threshold for the support vector machine compared to the classification score, in order to determine a binary classification of the anomalous state associated with the object including or excluding anomalous features.

12. The device as claimed in claim 8, characterized in that, The first feature output and the second feature output come from different layers of the convolutional neural network of the feature extraction model.

13. The device as claimed in claim 8, characterized in that, In order to generate the abnormal data, the one or more processors are configured to: Generate a location indicator that identifies the location of the anomaly; and Combine the location indicator with the input image.

14. The device as claimed in claim 8, characterized in that, In order to perform the action, the one or more processors are configured to be at least one of the following: Transmit the abnormal data to the user equipment, or The object management system is controlled based on the abnormal data to perform operations associated with the object.

15. A tangible machine-readable medium storing an instruction set, the instruction set comprising: One or more instructions, when executed by one or more processors of the device, cause the device to: Receive the input image depicting the object; A convolutional neural network encoder is used to determine a first feature output associated with one or more features of the object based on the input image. The convolutional neural network encoder is trained based on a reference image depicting a reference object that is the type of the object. Using a support vector machine to determine the abnormal state of an object indicates that the object includes an anomaly. The support vector machine is trained based on the reference image. The location of the anomaly is determined using a convolutional neural network decoder based on the second feature output of the convolutional neural network encoder. The convolutional neural network decoder is configured to determine the location of the anomaly based on the second feature output of the convolutional neural network encoder. The convolutional neural network decoder is trained based on the reference image; and Perform the action associated with the location of the anomaly.

16. The tangible machine-readable medium as claimed in claim 15, characterized in that, The reference object depicted in the reference image is a non-abnormal object.

17. The tangible machine-readable medium as claimed in claim 15, characterized in that, The support vector machine is trained to determine a binary classification that indicates whether the object includes anomalous features or does not. The support vector machine is trained to determine a classification threshold, which is used to determine the binary classification based on similarity analysis involving the reference image.

18. The tangible machine-readable medium as claimed in claim 15, characterized in that, The first feature output comes from the output layer of the convolutional neural network encoder, and The second feature output comes from the intermediate layer of the convolutional neural network encoder.

19. The tangible machine-readable medium as claimed in claim 15, characterized in that, The convolutional neural network encoder and the convolutional neural network decoder are associated with the same convolutional neural network autoencoder trained based on the reference image.

20. The tangible machine-readable medium as claimed in claim 15, characterized in that, The one or more instructions that cause the device to perform the action cause the device to: Generate a location indicator that identifies the location of the anomaly; The location indicator is combined with the input image to form an anomaly indicator; as well as Provide the exception indicator to the user equipment.

Images