Method and apparatus for grading images of collected items using image segmentation and image analysis.

Abstract

In some embodiments, the method may include augmenting the set of images of the collectible to generate a set of composite images of the collectible. The method may further include combining the set of images of the collectible and the set of composite images of the collectible to result in a training set. The method may further include training a set of machine learning models based on the training set. Each machine learning model from the set of machine learning models may generate a grade for an image attribute from the set of image attributes. The set of image attributes may include an edge, a corner, a center, or a surface. The method may further include running the set of machine learning models after training to generate a set of grades for images of the collectible not included in the training set.

Description

Technical Field

[0001] Cross - Reference to Related Applications

[0001] This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 150,793, filed on February 18, 2021, entitled "METHODS AND APPARATUS FOR GRADING IMAGES OF COLLECTABLES USING MACHINE LEARNING MODELS", which is incorporated herein by reference in its entirety.

[0002] Technical Field

[0002] The present disclosure relates to image analysis of images representing "real" objects, and more particularly to devices and methods for performing image analysis on one or more segments of an image to grade an image of a collectable.

Background Art

[0003] Background

[0003] Grading an image of a collectable can be useful, for example, in assessing the value of an asset. Grading an image of a collectable can include grading various segments of the image, such as segments representing corners or edges of the collectable. However, known grading methods can require a lot of labor and can be costly. Therefore, there is a need for devices and methods for accurately and efficiently grading collectables.

Summary of the Invention

Means for Solving the Problems

[0004] Summary

[0004] In some embodiments, the method may include receiving a set of images of a collection. Each image in the set of images is associated with at least one defect type label and at least one of a first grade classification label relating to the surface condition of an item from the collection, a second grade classification label relating to the edge condition of an item, a third grade classification label relating to the corner condition of an item, or a fourth grade classification label relating to the centering condition of an item. The method may further include, for each image in the set of images, detecting a boundary defining an item in the image, performing a perspective warp transform on the image from the set of images where the boundary relating to the image does not have a predetermined shape, and removing portions of the image that are not within the boundary defining an item, thereby generating a set of preprocessed images based on the set of images. The method may further include training at least one model based on each preprocessed image from a set of preprocessed images, at least one defect type label associated with that preprocessed image, and at least one of (1) a first grade classification label associated with that preprocessed image, (2) a second grade classification label associated with that preprocessed image, (3) a third grade classification label associated with that preprocessed image, or (4) a fourth grade classification label associated with that preprocessed image. The method may further include applying at least one model to new images of new collectibles not included in the collectible set. The method may further include displaying an output indicating that the new collectible contains defects, the approximate location of the defects, and the type of defects associated with the defects.

[0005]

[0005] In some embodiments, a non-temporary processor-readable medium stores code representing instructions to be executed by the processor. The instructions include code to cause the processor to preprocess images of collectibles to generate a preprocessed image by detecting boundaries defining collectibles in the image, to perform a perspective warp transform to give the boundaries a predetermined shape, and to remove portions of the image that are not within the boundaries defining collectibles. The instructions may further include code to cause the processor to apply a machine learning (ML) model to the preprocessed image to generate a set of defect confidence levels. Each defect confidence level from the set of defect confidence levels (1) relates to an eigenpart of the preprocessed image from an eigenpart of the preprocessed image, and (2) indicates the possibility that there is at least one defect within that eigenpart of the preprocessed image. The instructions may further include code to cause the processor to display the preprocessed image on a display. The instructions may further include code to cause the processor to display each eigenpart of the preprocessed image from an eigenpart relating to a defect confidence level from a set of defect confidence levels outside a predetermined range.

[0006]

[0006] In some embodiments, the device includes memory and a processor operably coupled to the memory. The processor may be configured to expand a set of images of a collection to generate a set of composite images of the collection. The processor may be further configured to combine the set of images of the collection and the set of composite images of the collection to yield a training set. The processor may be further configured to train a set of machine learning models based on the training set. Each machine learning model from the set of machine learning models is configured to produce grades for image attributes from a set of image attributes. The set of image attributes includes at least one of edges, corners, centers, or surfaces. The processor may be further configured to run the set of machine learning models after training to produce a set of grades for images of collections not included in the training set. [Brief explanation of the drawing]

[0007] Brief explanation of the drawing [Figure 1]

[0007] This is a schematic block diagram of a grading apparatus according to one embodiment. [Figure 2]

[0008] This is a flowchart illustrating a training method for a grading device according to one embodiment. [Figure 3]

[0009] This is a flowchart illustrating the method of using a grading device according to one embodiment. [Figure 4]

[0010] This is a flowchart illustrating a training method for a grading device according to one embodiment. [Figure 5]

[0011] This is a schematic description of a machine learning model used for grading, according to one embodiment. [Figure 6]

[0012] This is a flowchart illustrating a method for training and using a model based on a set of pre-processed images, according to one embodiment. [Figure 7]

[0013] This is a flowchart of a method for using a model to generate and use defect confidence levels, according to one embodiment. [Figure 8]

[0014] This is a flowchart of a method for training a model using a training set containing a set of composite images, according to one embodiment. [Modes for carrying out the invention]

[0008] Detailed explanation

[0015] Various aspects of the embodiments and non-limiting examples of modifications are described herein and shown in the accompanying drawings.

[0009]

[0016] The methods and apparatus described herein can generate grading for assets such as trading cards (e.g., sports cards, game cards, etc.), coins, and currencies.

[0010]

[0017] Figure 1 is a schematic block diagram of a grading apparatus 101 according to one embodiment. The grading apparatus 101 (also referred to herein as the “grading apparatus”) may be, or include, a hardware-based computing device and / or multimedia device such as a computer, desktop, laptop, or smartphone. The grading apparatus 101 includes a memory 102, a communication interface 103, and a processor 104. The grading apparatus 101 can operate a set of grader models 105 that can collectively generate grades for images of collectibles (e.g., trading cards, sports cards, collectible cards, coins, currency, works of art, stamps, antiques, comic books, toys, jewelry, etc.).

[0011]

[0018] The memory 102 of the grading device 101 may be, for example, a memory buffer, random access memory (RAM), read-only memory (ROM), a hard drive, a flash drive, etc. The memory 102 may store, for example, a set of images of a collection (for example, a set of images of trading cards, a set of images of collectible cards, a set of images of coins, a set of images of stamps, a set of images of artwork, etc.), a set of grades (for example, a set of numerical values), and / or code (for example, a program written in C, C++, Python, etc.) that includes instructions for the processor 104 to execute one or more processes or functions (for example, a set of grader models 105).

[0012]

[0019] The communication interface 103 of the grading apparatus 101 may be a hardware component of the grading apparatus 101 for facilitating data communication between the grading apparatus 101 and external devices (e.g., a network, a computer, and / or a server, not shown). The communication interface 103 may be operably coupled to and used by the processor 104 and / or memory 102. The communication interface 103 may be, for example, a network interface card (NIC), a Wi-Fi® module, a Bluetooth® module, an optical communication module, and / or any other suitable wired and / or wireless communication interface.

[0013]

[0020] The processor 104 may be a hardware-based integrated circuit (IC) or any other suitable processing unit configured to run or execute, for example, a set of instructions or a set of code. For example, the processor 104 may include a general-purpose processor, a central processing unit (CPU), an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), etc. The processor 104 is operably coupled to memory 102 by a system bus (e.g., an address bus, a data bus, and / or a control bus, not shown). The processor 104 includes a set of grader models 105. Each grader model from the set of grader models 105 may include software stored in memory 102 and executed by the processor 104, which can be configured to grade the attributes of a collectible or a portion of an image from a set of images of the collectible. In some examples, the grader models from the set of grader models 105 may include a collectible and / or a card type predictor (not shown) and / or a tile defect predictor (not shown). Each collectible and / or card type predictor or tile defect predictor may include software stored in memory 102 and executed by processor 104.

[0014] Generating a trained model

[0021] Figure 4 is a flowchart of a training method for a grading apparatus (e.g., grading apparatus 101) according to one embodiment. In some implementations, the method discussed with respect to Figure 4 may be performed by a processor (e.g., processor 104 in Figure 1). In step 1, the training images are preprocessed. Preprocessing may result in the training images and / or parts of the training images (e.g., images of collectibles only) having a standardized format in at least one aspect. Preprocessing may include, for example, cropping, trimming, filtering, transforming, and / or resizing the training images and / or parts thereof. The training images and / or preprocessed training images may be associated with one or more labels and / or grades, such as centering grade, tile defect label, collectible and / or card type label, player information, character information, edge grade, corner grade, etc. Furthermore, as will be discussed in more detail herein, in some examples, composite images may be generated for training.

[0015]

[0022] In step 2, homography values ​​are generated by comparing collectibles (e.g., cards, stamps, artworks, etc.) with various centering grades. In step 7, a centering grade regression model (e.g., a neural network) can be trained using the homography values, centering grade labels, and / or pre-processed training images to generate a trained centering model. In some implementations, homography values ​​and / or pre-processed images can be used as input training data for the centering grade regression model, and centering grade labels can be used as target training data for the centering grade regression model.

[0016]

[0023] In step 3, an NxN surface tile is generated for each of the preprocessed training images. In step 8, a defect classification model can be trained using the surface tile, defect labels that may be associated with the surface tile, labels of the type of collectible and / or card, player information, and / or character information in order to identify defects (e.g., generate a defect confidence level for the surface tile). Then in step 11, a trained defect classification model can be executed using the surface tile, defect labels, type of collectible and / or card, player information, and / or character information to generate a defect confidence level for each surface tile. In step 12, a surface grade regression model can be trained using the preprocessed training images, transfer weights from the defect classification model trained in step 8, and the defect confidence levels generated in step 11.

[0017]

[0024] In step 4, an edge image is generated using the preprocessed training images. The edge image and the label of the edge grade can be used in step 9 to train and generate an edge grade regression model. In some implementations, the edge image can be used as input learning data for the edge grade regression model, and the label of the edge grade can be used as target learning data for the edge grade regression model.

[0018]

[0025] In step 5, a corner image is generated using the preprocessed training images. The corner image and the label of the corner grade can be used in step 10 to train and generate a corner grade regression model. In some implementations, the corner image can be used as input learning data for the corner grade regression model, and the label of the corner grade can be used as target learning data for the corner grade regression model.

[0019]

[0026] In step 6, a classification model for collectibles and / or card types / players / characters can be trained and generated using pre-processed training images (e.g., using computer vision). The classification model for collectibles and / or card types / players / characters can be trained to identify collectible types, card types (e.g., set, year, etc.), stamp types (e.g., year, issue, etc.), coin types, player information, character information, and / or other arbitrary information about collectibles. In some implementations, step 6 is performed before steps 8 and 11, and the trained classification model for collectibles and / or card types / players / characters outputs collectible types, card types, player information, stamp types (e.g., year, issue, etc.), coin types, character information, and / or other information about collectibles used in steps 8 and 11. In some implementations, steps 1-12 can be performed in any order. In some implementations, steps 1-12 can be performed sequentially, in parallel, or in any combination thereof.

[0020] Image preprocessing

[0027] A set of images of collectibles (e.g., images of sports cards, game cards, collector cards, coins, stamps, works of art, etc.) used to train a set of grader models 105, and / or images of collectibles (not included in the set of images of collectibles) used when running the set of grader models 105 after training, can be taken using an imaging device (e.g., a camera, scanner, etc. (not shown)) of a device operably coupled to the grading apparatus or grading apparatus 101. For example, a set of images of collectibles and / or images of collectibles can be taken with a smartphone camera or scanner. Thus, images for processing by the processor 104 of the grading apparatus 101 can be taken at slightly different angles, under various lighting conditions, and / or may include additional background surrounding the actual collectibles (e.g., cards). Thus, image preprocessing can be used to generate preprocessed images based on images used to train a set of grader models 105 (e.g., a set of images of collectibles) and / or images graded by the set of grader models 105 (e.g., to normalize the images of collectibles). In some cases, one or more of the following preprocessing steps can be performed using any appropriate technique to use images of collected items for training and / or grading purposes (e.g., step 1 in Figure 4): 1. Boundary detection of collected items within images of collected items. 2. Perspective warp transform to convert images of collectibles taken from imperfect angles into rectangles. 3. Removal of the background by cropping the outer area surrounding the boundaries of the collected items within the image of the collected items. 4.1 Resize the images of the collected materials to a uniform size and resolution suitable for processing by machine learning models from the set of grader models 105.

[0021]

[0028] In some implementations, a desired shot of the collectible image can be selected from live camera video feeds and / or recordings (e.g., from an imaging device) by applying a boundary detection algorithm in near real-time and selecting the frame with the detected boundary closest to the rectangle. This minimizes the degree of perspective warp transformation and improves image quality and overall grading accuracy. In addition, or alternatively, a desired shot of the collectible image can be selected from live camera video and / or recordings by applying resizing and / or resolution adjustment and selecting the frame with the size and / or resolution closest to the desired size and / or resolution. In some implementations, if multiple images (e.g., frames from video and / or multiple still images) exist for the collectible, the selected desired shot can be used to generate the collectible grade.

[0022]

[0029] In some implementations, images of collectibles with glossy surfaces can undergo additional preprocessing steps to detect and skip video frames containing distracting reflections, saturation in the image, or white spots. Additional machine learning models can be trained to detect distracting reflections, saturation in the image, or white spots and filter out unwanted frames, pixels, etc., from the collectible images. In some implementations, the final image of the collectible can be obtained by stitching together the unaffected portions of the collected collectible images or video frames. For example, if the first image of the collectible in the first frame contains reflections in the upper half of the first image, and the second image of the collectible in the second frame (different from the first image) contains reflections in the lower half of the second image, the lower half of the first image can be combined with the upper half of the second image (e.g., by stitching) to form a final image without distracting reflections (e.g., the upper half does not overlap with the lower half, or the upper half partially overlaps with the lower half).

[0023] Grading

[0030] In some examples, the grading of an asset (e.g., collectibles such as sports cards) may include four scores (or grades) within a numerical range (e.g., on a scale from 1 to 10). The scores may represent the condition of the asset's surface, edges, corners, and / or centering. In some examples, a higher score (or grade) may indicate better condition (e.g., surface, edges, corners, or centering). Each side of the asset (or collectible) may be graded separately and may have its own set of grades. While this example shows four scores, other implementations may identify and use any number of scores for various aspects and / or attributes of the asset.

[0024]

[0031] Asset grading can be achieved by training an ensemble of machine learning models (e.g., artificial neural networks, convolutional neural networks, recurrent neural networks, self-organizing maps, Boltzmann machines, autoencoders, etc.) designed to handle specific types of grades. Each grade may have one or more designated machine learning models (e.g., neural network models). In some implementations, two machine learning models can be used to identify a grade (e.g., a surface grade). In some implementations, each grade may be identified using any number of machine learning models.

[0025]

[0032] In some implementations, a first machine learning model may be configured and / or trained to detect the types of surface defects on a surface image of a set of images of a collectible, which is divided into several nearly equal sub-sections (e.g., tiles). In some implementations, the surface image of the collectible is divided into nearly equal sections, and one or more defect labels can be assigned (e.g., by a human, by a machine learning model, by a computer vision algorithm, etc.) to (1) each section (including those with and without defects), or (2) only to sections with defects. The defect labels may be letters, words, numbers, etc., indicating the presence of a defect and / or the type of defect in a given section (e.g., wrinkles, creases, etc.). The defect labels and various equal sections can be used to train the first machine learning model. For example, each section can be used as input training data for a neural network, and the defect labels associated with that section can be used as output training data for a neural network. A second machine learning model can perform regression of the final grade by using the entire surface image of the collectible and the tile defect information from the first machine learning model regarding that collectible. Machine learning models can be trained using training datasets containing existing images (e.g., thousands) of collectibles that can be pre-graded by expert human graders. For surface grading, for example, the training dataset might include grade labels assigned to each photograph of a collectible, as well as defect type labels assigned to individual tiles.

[0026]

[0033] The input to the training process includes providing a set of images of the collectible, each associated with a first grade classification label for the surface (e.g., an integer value from 1 to 10), a second grade classification label for the edges (e.g., an integer value from 1 to 10), a third grade classification label for the corners (e.g., an integer value from 1 to 10), and / or a fourth grade classification label for centering (e.g., an integer value from 1 to 10). The grade classification labels can be assigned separately to each photograph for the front and back surfaces of each image from the set of images of the collectible.

[0027]

[0034] For surface grading, in addition to grade labels, the training set may include classification labels for various types of defects assigned to individual surface tiles of an image of a collectible outlined by tiles in an NxN grid. Each tile in the grid may have multiple types of defect labels, such as wrinkles, creases, printing defects, stains, and ink. In some implementations, a set of surface defect codes (represented, for example, as letters, numbers, etc.) may represent the type of tile defect. In some examples, a set of surface defect codes may include, for example: C - Wrinkles, creases, bends, crushes, etc. H-holes, pinholes, punches, etc. I - Traces, scratches, etc. M-Misscut P-printing, stains, alignment, etc. S-stain T - Tears, rips, folds, etc. W - Writing, ink, etc. Corner / edge scratches code X-angle(1~4) E-Edge (1~N) A - Wear and tear (curling of paper or surface and / or minor damage) B - Wrinkles or lifting (wrinkles / bends or lifting on corner surfaces) Y-impact (marks, dents, or other indentations not indicated by surface scratch codes)

[0028]

[0035] In some implementations, additional models can be trained to detect collectible and / or card types (e.g., which set they belong to, year of manufacture, manufacturer, etc.), player and / or character information, stamp types (e.g., year, issue, etc.), coin types (e.g., year, coin identification, etc.), and / or other arbitrary information about the collectible. This information can be used in the underlying grading model (e.g., the machine learning model described above) to reduce the number of false positives specific to a particular collectible and / or card type. For example, some cards may have creases on a player's clothing that could be mistakenly identified as defects. Adding card type and / or player / character information (and / or other collectible-specific information) to the input of the grading model can help eliminate such false positives by training the model with collectible-specific exceptions. The additional models could be, for example, machine learning models, artificial intelligence models, analytical models, or mathematical models. In some implementations, additional models can be trained using computer vision to detect collectibles and / or card types, player information, character information, stamp types (e.g., year, issue, etc.), coin types (e.g., year, coin identification, etc.), and / or other arbitrary information about the collectibles. In some implementations, the additional models can be trained using supervised learning. In some implementations, the additional models can be trained using unsupervised learning. In some implementations, the additional models are neural networks (e.g., convolutional neural networks) trained using images of collectibles (e.g., cards) as input training data, and card types, player information, character information, and / or characteristics specific to the collectibles as output training data.

[0029]

[0036] The resulting trained machine learning model can be used to grade the collectibles (as shown in step 301 of Figure 3). During the grading process, the same image preprocessing steps (in step 302) used in the training phase can be applied to input images of collectibles (e.g., new collectibles unrelated to the training data) to generate preprocessed images. The preprocessed images can then be input to a set of grader models 105 (including, for example, a trained surface grade regression model) for predicting grades. The set of grader models 105 can predict card type and / or player / character information (or other information about the collectible) (in step 303). The set of grader models 105 can further predict tile defects (in step 304). The set of grader models 105 can further predict grades (in step 305) based on information about the collectible (e.g., card type, player / character information), tile defects, and / or other grades generated by the set of grader models 105. In some implementations, after a grade is calculated by a set of grader models 105, an additional overlay image can be constructed from the weights of the convolutional layers of the grader models. The overlay image can be used to highlight specific image regions where defects are identified from a set of images of the collectible (e.g., images of cards). In addition, or alternatively, the overlay image can be used to highlight specific image regions where defects are not identified from a set of images of the collectible.

[0030] Grader for surfaces

[0037] A set of grader models 105 may include a surface grader model. In some implementations, the surface grader model may be an ensemble of two separate models, or may include them: • Tile defect classification model • Regression model of surface grade In some implementations, both tile defect classification models and surface grade regression models can be generated using existing machine learning models (e.g., using transfer learning). For example, the machine learning model could be a pre-trained neural network model trained on a massive dataset of general-purpose images (e.g., the ImageNet dataset, a publicly available dataset containing over 14 million images of real-world objects). Using a pre-trained neural network model allows existing knowledge of the shapes of various objects to be added to the machine learning model, making a set of grader models (e.g., a tile defect classification model and / or a surface grade regression model) more effective in distinguishing between known object shapes and surface defects. In some implementations, pre-trained ImageNet-based models such as VGGNet, ResNet, Inception, and Xception can be used.

[0031]

[0038] The tile defect classification model can be trained using a small subset of training images classified by surface defect codes (as shown in steps 3 and 8 of Figure 4). After training the tile defect classification model, it can be used to classify tiles in the training set and generate confidence levels for possible defects on each tile (as shown in step 11 of Figure 4).

[0032]

[0039] Figure 5 shows the layer structure within a tile defect classification model (e.g., a neural network) according to one embodiment. Additional layers responsible for classifying tile defects in images of collectibles can be added to the base model (e.g., trained using generic images). Dropout layers may be used to reduce overfitting of the model and to provide better generalization with respect to the neural network. In some implementations, the size of the final output layer may be determined by the number of defect types supported. For example, in an application of the grading device 101 for a particular type of collectible, there may be N (e.g., 7, 10, 100, etc.) types of defects that can occur for that particular type of collectible. Therefore, the size of the final output layer of the tile defect classification model may be N (e.g., an integer of N). For example, the output of the tile defect classification model may include N confidence levels in the range of 0.0 to 1.0, where N is the number of defect types supported.

[0033]

[0040] The layer structure within a surface grade regression model can be similar to that within a tile defect classification model. The difference in layer structure might be the size of the final output layer in the surface grade regression model. Since a surface grade regression model is a regression model, some implementations have a single output representing a continuous value of the surface grade. Similarly, in contrast to a tile defect classification model where the output is a label, such implementations can represent the grade as a decimal number (e.g., 5.0, 8.5, 9.99).

[0034]

[0041] In addition, weights from the tile defect classification model (Figure 5) can be transferred to the surface grade regression model, allowing the surface grade regression model to recognize defect patterns learned by the tile defect classification model (steps 8 and 12 in Figure 4).

[0035] Graders for edges and corners

[0042] A set of grader models 105 may include dedicated models for edges and corners. In some implementations, the dedicated models for edges and corners may have the same and / or similar layer structure. In some examples, the dedicated models for edges and corners may be similar to the tile defect classification model, the difference being the number of outputs in the final layer (output layer). In some implementations, the dedicated models for edges and corners have one output representing a continuous grade value (e.g., a value from 0 to 10). The grade can represent the condition of the edge and / or corner and can be used to determine whether corrective action should be taken. For example, if the grade is outside a given tolerance range, the edge and / or corner may be indicated as defective.

[0036]

[0043] In some implementations, separate input images for the edge grader model and the corner grader model can be extracted from a pre-processed image. Similar to tile defect classification models and surface grade regression models, the edge grader model and / or corner grader model may be given the ability to generate overlay images to highlight edge and / or corner defects.

[0037] Grader for centering

[0044] A set of grader models 105 may include a dedicated model for the centering of the collectible image to determine how centrally the collectible is positioned. In some implementations, where the collectible is an image (e.g., of a player or character) printed on card stock, the centering grade may indicate how centrally the image is positioned on the card stock. In some implementations, the centering grade of the collectible image may be calculated by a center regression grader model that takes a set of homography matrices as input. (As shown in step 2 of Figure 4) The set of homography matrices may be calculated by comparing a pre-processed image in a training set with several other collectibles (e.g., cards) having different centering grades. Such a method may be analogous to triangulation, where homography distances between different centering grades are considered (e.g., using computer vision). The grade can represent the centering state of the collectible image and may be used to determine whether corrective action should be taken. For example, if the grade falls outside a predetermined tolerance range, the centering of the image of that collectible can be indicated as defective (e.g., by text or any other label indicating that the centering is undesirable).

[0038]

[0045] In some cases, in addition to homography values, the central regression grader model can take specific information about collectibles (e.g., card type and / or player / character information) as input. This ensures that biases specific to collectibles and / or card types are avoided.

[0039] Handling unbalanced training data

[0046] In some cases, the challenging part of making a set of grading models accurate is the problem of overfitting the training set when using a limited, unbalanced training dataset. The grading device 101 in Figure 1 can train an accurate predictive model using a training set that does not cover a large number of samples for each collectible-specific information (e.g., combinations of card type, player / character, and grade). In other words, a set of grader models 105 in the grading device 101 is developed to generalize grade generation based on images of collectibles. Thus, without maintaining an extremely large training set (e.g., billions of images), the same set of grader models trained on the training set can successfully grade images of a wide variety of collectibles (e.g., a wide variety of sports cards and / or player sets, a wide variety of stamps, a wide variety of artworks, etc.) based on a few reusable images within the training set (e.g., thousands of images).

[0040]

[0047] In some implementations, generalization methods to avoid or mitigate some of the problems associated with imbalanced training data may include, for example, one or more of the following: 1. Upsampling and downsampling of the training set. The purpose of this step is to adjust the training set so that the training set has relatively equal variances across several samples across all grades. For grades with a sample count above the mean, the dataset can be reduced by randomly removing excess samples from the dataset (downsampling). For grades with a sample count below the mean, additional synthetic images of collected items can be generated and added to the training set (upsampling). 2. Dropout layer. Using a dropout layer enables a computationally inexpensive and effective regularization method to reduce overfitting of grading models and improve generalization errors. 3. Layer Weight Regularizers. Similar to dropout layers, weight regularizers reduce the possibility of overfitting a machine learning model (e.g., a neural network) by constraining the range of weight values ​​within the network. In some examples, weight regularizers can be added to individual layers of a network, including layers in a base model trained on generic image data. 4. K-fold validation can be used to improve generalization and reduce over-adaptation. 5. Additional image augmentation by generating synthetic training data.

[0041]

[0048] In some examples, the number of dropout layers, the dropout rate, and / or the number of weight regularizers may be determined during the hyperparameter optimization stage. The hyperparameter optimization stage can improve, tune, and / or optimize the hyperparameters of the model (e.g., the model from a set of grader models 105 in Figure 1). Further details regarding hyperparameter optimization are discussed below.

[0042] Synthetic training images

[0049] In some implementations, the grading apparatus 101 can generate a composite image (in addition to the set of images from the collection) to further improve the accuracy of a set of grader models 105 trained on a suitable dataset. In some examples, a set of image augmentation techniques can be randomly applied to a set of images from the collection to extend the training set with the additional composite image. The set of image augmentation techniques may include rotation, vertical and / or horizontal shifting, scaling, brightness and contrast adjustment, vertical and / or horizontal flipping, etc., for one or more images from the set of images from the collection to generate a composite image. The composite image can be used to train or retrain one or more grader models from the set of grader models 105, in addition to the set of images from the collection. In some implementations, the composite image is preprocessed (e.g., perspective warp transformation, resizing, background cropping, etc.) before being used to train one or more grader models from the set of grader models 105.

[0043]

[0050] A set of augmentation techniques can ensure consistent grading accuracy for images of collectibles taken using cameras with different features (e.g., resolution, zoom, filters, depth, etc.) and / or under different lighting conditions (e.g., angle). Using augmentation can also significantly increase the number of samples in the training set and improve the generalization of a set of grader models 105.

[0044] Hyperparameter tuning

[0051] The hyperparameters of a set of grader models 105 can be optimized using one of the following tuning algorithms, namely random search, hyperbanding, or Bayesian optimization. The effectiveness of a particular tuning algorithm may vary depending on the training set and other factors. Therefore, tuning algorithms can be individually evaluated to achieve the best accuracy for a particular model and training set. Adjustable parameters and / or hyperparameters for a set of grader models 105 may include, for example: 1. Neural network parameters Layer size • Number of dropout layers • Dropout rate • Types of weight regularizers • Regularization coefficient • Types of ImageNet-based models 2. Image enhancement parameters • Range of rotation angle, shift, brightness, scaling, and flip 3. Training parameters • Types of optimizers • Learning rate Batch size Number of epochs

[0045] Defect Visualization

[0052] Defects identified by a pair of grader models 105 can be visualized as an overlay of the original collected image. The overlay can be constructed from the weights of the last convolutional layer of a model trained using generic image data. For example, if the model trained using generic image data is a VGGNet (Visual Geometry Group Network) model, the last convolutional layer would be block5_conv3. Larger weight values ​​represent a higher confidence that a defect will be detected in the corresponding pixel or group of pixels.

[0046]

[0053] The range of weight values ​​can be represented using various overlay colors or pixel intensities to effectively create a heatmap representation. Other visual cues can be realized by displaying contours around high-intensity clusters where the weight value exceeds a certain threshold, or by highlighting areas around such clusters. Such visual representations can be presented and / or displayed to the user by user equipment (e.g., grading apparatus 101 and / or equipment operably coupled to the grading apparatus).

[0047]

[0054] In some implementations, the grading device 101 may be operably coupled via a network to a computing device (not shown) and / or a server (not shown) in order to transmit and / or receive data (e.g., images of collectibles) and / or analysis models over the network. In some examples, the computing device and / or server may provide training data to the grading device 101. In some examples, the computing device and / or server may run a trained machine learning model to perform grading of assets, such as collectibles.

[0048]

[0055] Figure 6 is a flowchart of a method 600 for training and using a model based on a set of preprocessed images, according to one embodiment. In some implementations, method 600 may be executed by a processor (e.g., processor 104 in Figure 1). For example, instructions for processor 104 to execute method 600 may be stored in memory 102 in Figure 1.

[0049]

[0056] In 602, a set of images of a collection (e.g., one collection, two collections, three collections, etc.) is received. Each image from the set of images is associated with at least one defect type label and at least one of the following: a first grade classification label relating to the surface condition of the collection from the collection, a second grade classification label relating to the edge condition of the collection, a third grade classification label relating to the corner condition of the collection, or a fourth grade classification label relating to the centering condition of the collection. In some implementations, a collection may include only trading cards, only coins, only currency, only works of art, only stamps, only antiques, only comic books, only toys, only jewelry, or a combination thereof. In some implementations, a set of images may represent a common face (e.g., front) of the collection. In some implementations, a set of images may represent various different faces (e.g., front and back) of the collection. In some implementations, a collection refers to an item of interest to a collector. In some implementations, a collection refers to something that can be collected.

[0050]

[0057] Step 604 generates a set of pre-processed images based on a set of images by detecting the boundaries defining collectibles within each image of the set, performing a perspective warp transformation on the image from the set of images where the boundaries relating to the image do not have a predetermined shape (e.g., square, rectangle, parallelogram, etc.), and removing the portion of the image that is not within the boundaries defining collectibles. In some implementations, step 604 is performed automatically (e.g., without requiring human input) in response to receiving a set of images. In some implementations, generating a set of pre-processed images further includes resizing each image from a set of images that have a size other than a predetermined size to give the image a predetermined size. In some implementations, generating a set of pre-processed images further includes resizing each image from a set of images that have a resolution outside a predetermined resolution range to give the image a resolution within a predetermined resolution range.

[0051]

[0058] In 606, at least one model (e.g., a set of grader models 105 shown in Figure 1) is trained based on each preprocessed image from a set of preprocessed images, at least one defect type label associated with that preprocessed image, and at least one of (1) a first grade classification label associated with that preprocessed image, (2) a second grade classification label associated with that preprocessed image, (3) a third grade classification label associated with that preprocessed image, or (4) a fourth grade classification label associated with that preprocessed image. In some implementations, at least one model includes at least one dropout layer to reduce overfitting. In some implementations, at least one model includes: (1) a first model trained with (a) each preprocessed image from a set of preprocessed images and (b) a first grade classification label associated with that preprocessed image; (2) a second model trained with (a) each preprocessed image from a set of preprocessed images and (b) a second grade classification label associated with that preprocessed image; (3) a third model trained with (a) each preprocessed image from a set of preprocessed images and (b) a third grade classification label associated with that preprocessed image; (4) a fourth model trained with (a) each preprocessed image from a set of preprocessed images and (b) a fourth grade classification label associated with that preprocessed image; and (5) a fifth model trained with (a) each preprocessed image from a set of preprocessed images and (b) at least one defect type label associated with that preprocessed image.

[0052]

[0059] In step 608, at least one model is applied to new images of new collectibles not included in the existing collection. In some implementations, at least one model is automatically applied to new images in response to the receipt of a new image representation (e.g., by processor 104 in Figure 1).

[0053]

[0060] 610 displays an output indicating that a new collectible contains a defect, the approximate location of the defect, and the type of defect associated with it. In some implementations, 610 is performed automatically (e.g., without requiring human input) in response to 608 applying at least one model to the new image. In some implementations, a processor (e.g., processor 104) sends at least one electrical signal to a display (not shown in Figure 1) to display the output, and the display is operablely coupled to the processor via wired and / or wireless connection to cause the display to indicate that a new collectible contains a defect (e.g., by text, symbols, color coding, highlighting, etc.), the approximate location of the defect (e.g., by text, symbols, color coding, highlighting, etc.), and the type of defect associated with it (e.g., bend, wrinkle, etc.) (e.g., by text, symbols, color coding, highlighting, etc.).

[0054]

[0061] In some implementations of Method 600, a first image from a set of images is captured or taken under first lighting conditions, and a second image from the same set of images is captured or taken under second lighting conditions different from the first lighting conditions. The lighting conditions may be, for example, the amount of luminance.

[0055]

[0062] In some implementations of Method 600, a first image from a pair of images is captured or taken at a first angle relative to a first collectible from a collection of collectibles, and a second image from the pair of images is taken at a second angle relative to the first collectible or to one of the second collectibles from a different collection of collectibles. The second angle is different from the first angle. The first and second images can be captured or taken using the same imaging device (e.g., a single common camera) or different imaging devices (e.g., two different cameras).

[0056]

[0063] In some implementations of Method 600, a first image from a set of images is taken with a first background, and a second image from the same set of images is taken with a second background that is different from the first background. For example, the first and second backgrounds may have different colors, textures, patterns, shapes, orientations, landscapes, etc.

[0057]

[0064] In some implementations, Method 600 further includes optimizing and / or improving hyperparameters associated with at least one model using at least one of a random search algorithm, a hyperband algorithm, or a Bayesian optimization algorithm.

[0058]

[0065] Figure 7 is a flowchart of a method 700 for using a model to generate and use a defect confidence level, according to one embodiment. In some implementations, method 700 may be executed by a processor (e.g., processor 104 in Figure 1). For example, instructions for causing processor 104 to execute method 700 may be stored in memory 102 in Figure 1.

[0059]

[0066] In step 702, the image of the collectible is preprocessed to generate a preprocessed image by detecting the boundary defining the collectible within the image, performing a perspective warp transformation to give the boundary a predetermined shape (e.g., rectangle, square, parallelogram, etc.), and removing the portion of the image that is not within the boundary defining the collectible. The collectible may be, for example, trading cards (e.g., baseball cards, basketball cards, football cards, Pokémon® cards, etc.), coins, currency, works of art, stamps, antiques, comic books, toys, jewelry, etc. Images may be collected by an imaging device such as a camera or scanner.

[0060]

[0067] In step 704, a machine learning (ML) model (e.g., a set of grader models 105 in Figure 1) is applied to the preprocessed image to generate a set of defect confidence levels. Each defect confidence level from the set of defect confidence levels (1) relates to an eigenpart of the preprocessed image from an eigensubgroup of the preprocessed image, and (2) indicates the possibility that there is at least one defect within that eigenpart of the preprocessed image. In some implementations, this is performed automatically (e.g., without requiring human input) in response to 704 generating the preprocessed image in step 702. In some implementations, each defect confidence level is related to a numerical value (e.g., from 0 to 100, from 0% to 100%, from 1 to 10, etc.). In some implementations, each defect confidence level is related to a text label (e.g., like new, very good, good, near good, excellent, good, poor quality, etc.). In some implementations, each eigenpart from an eigensubgroup does not overlap with any other eigenpart from the eigensubgroup (e.g., one eigenpart in the upper half and another in the lower half). In some implementations, at least one eigenpart from an eigensubgroup (for example, one to all eigenparts from the eigensubgroup) overlaps with another eigenpart from the eigensubgroup (for example, the first eigenpart of the upper half, the second eigenpart of the lower half, and the third eigenpart of the central part which includes the subdivisions of the upper and lower halves).

[0061]

[0068] 706 displays the preprocessed image on a display. In some implementations, this is done automatically (e.g., without requiring human input) in response to 704 generating a set of confidence levels. In some implementations, a processor (e.g., processor 104) sends at least one electrical signal to a display (not shown in Figure 1) and the output is displayed on the display, and the display is operablely coupled to the processor by wired and / or wireless connection to display the preprocessed image on the display.

[0062]

[0069] 708 causes the display to show each eigenpart of a preprocessed image from the eigenpart associated with a defect confidence level from a set of defect confidence levels outside a predetermined range. In some implementations, this is done automatically (e.g., without requiring human input) in response to 706 displaying the preprocessed image. In some implementations, a defect confidence level being within a predetermined range indicates that the eigenpart associated with that defect confidence level is in a desired (or "sufficiently good") state (e.g., like new, excellent, good, etc.), and a defect confidence level being outside a predetermined range indicates that the eigenpart associated with that defect confidence level is not in a desired state (e.g., poor, low quality, etc.). In some implementations, the predetermined range can be adjusted for a specific use case (i.e., based on what is considered an acceptable state by the user, customer, organization, group, etc.) (e.g., by instructions input by the user and received by the processor).

[0063]

[0070] In some implementations, the ML model is a first ML model, and Method 700 further includes applying a second ML model to a preprocessed image to generate a first score indicating the surface condition of the collectible, applying a third ML model to the preprocessed image to generate a second score indicating the edge condition of the collectible, applying a fourth ML model to the preprocessed image to generate a third score indicating the corner condition of the collectible, and applying a fifth ML model to the preprocessed image to generate a fourth score indicating the centering condition of the collectible. Method 700 may further include assigning the collectible at least one label indicating the overall condition of the collectible based on the first, second, third, and fourth scores. In some implementations, the at least one label may indicate that the overall condition is one of the following: like new, very good, very good, good, near good, very good, good, good, good, good, good, good, good, or poor. In some implementations, a numerical value that is a function (e.g., sum, mean, weighted mean, etc.) of the first, second, third, and / or fourth scores corresponds to at least one label (e.g., within a numerical range associated with such label), and thus the numerical value can be calculated and used to determine at least one label. Method 700 may further include displaying each defect confidence level from the set of defect confidence levels as superimposed on the eigenparts of the preprocessed image associated with its defect confidence level. For example, if the preprocessed image contains N eigenparts (e.g., tiles), then N defect confidence levels can be displayed, each confidence level associated with a different eigenpart (superimposed on such eigenpart).

[0064]

[0071] In some implementations, Method 700 may further include applying a computer vision model to a preprocessed image to identify at least one of the card type, player information, character information, and / or other information related to the collectible, the card type, player information, character information, and / or other information being used by at least one of a first ML model for generating a set of defect confidence levels, a second ML model for generating a first score, a third ML model for generating a second score, a fourth ML model for generating a third score, or a fifth ML model for generating a fourth score. In some implementations, applying the first ML model takes place before applying the second to fifth ML models, and at least two of the applications of the second, third, fourth, or fifth ML models are performed in parallel. In some implementations, the first to fifth ML models can be applied sequentially, in parallel, or in any combination thereof.

[0065]

[0072] In some implementations, the preprocessing in 702 further includes resizing the image to a predetermined size. In some implementations, the preprocessing in 702 further includes resizing the image so that the image has a resolution within a predetermined range of resolutions.

[0066]

[0073] In some implementations, method 700 further includes determining at least one of the card type, player information, character information, and / or other information associated with the collectible with respect to a preprocessed image. The ML model can further be applied to at least one of the card type, player information, character information, and / or other information associated with the collectible to generate a set of failure confidence levels. In other words, the set of failure confidence levels can be generated by the ML model based at least partially on at least one of the card type, player information, character information, and / or other information associated with the collectible.

[0067]

[0074] Figure 8 is a flowchart of a method 800 for training a model using a training set containing a set of composite images, according to one embodiment. In some implementations, method 800 may be executed by a processor (e.g., processor 104 in Figure 1). For example, instructions for causing processor 104 to execute method 800 may be stored in memory 102 in Figure 1.

[0068]

[0075] In 802, a set of images of a collectible is augmented to generate a set of composite images of the collectible (e.g., trading cards only, coins only, currency only, a combination of cards, coins, and / or currency). In some implementations, the augmentation in 802 may include at least one of the following: rotating the first image from the set of images, vertically shifting the first image, horizontally shifting the first image, scaling the first image, adjusting the brightness of the first image, adjusting the contrast of the first image, vertically flipping the first image, or horizontally flipping the first image. In 804, the set of images of the collectible and the set of composite images of the collectible are combined to yield a training set. In 806, a set of machine learning models (e.g., a set of grader models 105 in Figure 1) are trained on the training set. Each machine learning model from the set of machine learning models is configured to generate grades for image attributes from a set of image attributes. A set of image attributes includes at least one of edge, corner, center, or surface. After training, a set of machine learning models is run to generate a set of grades for images of collectibles not included in the training set. In some implementations, a set of grades can be used to determine if a collectible not included in the training set is defective, and a signal can be sent to prompt at least one corrective action (e.g., flagging the image, flagging the collectible, notifying the user). In some implementations, at least one image from a set of images is captured using at least one first camera configuration, and images of collectibles not included in the training set are captured using at least one second camera configuration different from the first.

[0069]

[0076] It should be understood that the disclosed embodiments do not represent all of the innovations described in the claims. Therefore, certain aspects of this disclosure are not discussed herein. Alternative embodiments may not be presented for certain parts of the innovation, or further undescribed alternative embodiments that may be available for certain parts should not be considered exclusionary. Therefore, it should be understood that other embodiments may be utilized, and functional, logical, operational, organizational, structural, and / or topological modifications may be made without departing from the scope of this disclosure. For this reason, all examples and / or embodiments throughout this disclosure should be considered non-limiting.

[0070]

[0077] Some embodiments described herein relate to methods. It should be understood that such methods may be implemented by a computer (e.g., instructions stored in memory and executed on a processor). While the methods described above show that certain events occur in a certain order, the order of these events is modifiable. In addition, certain events can be executed sequentially as described above, as well as repeatedly and simultaneously within parallel processes where possible. Furthermore, certain embodiments may omit one or more of the described events.

[0071]

[0078] Some embodiments described herein relate to computer memory products having a non-temporary computer-readable medium (which may also be called a non-temporary processor-readable medium) having instructions or computer code thereon for performing various operations implemented by a computer. The computer-readable medium (or processor-readable medium) is non-temporary in the sense that it does not contain the temporary propagating signal itself (e.g., propagating electromagnetic waves that carry information on a transmission medium such as space or a cable). The medium and the computer code (which may also be called the code) may be designed and constructed for a particular purpose. Examples of non-temporary computer-readable mediums include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tapes; optical storage media such as compact disks / digital video disks (CDs / DVDs), compact disk read-only memory (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier signal processing modules; and hardware devices specifically configured to store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), read-only memory (ROMs), and random access memory (RAM) devices. Other embodiments described herein relate to computer program products that may include, for example, the instructions and / or computer code discussed herein.

[0072]

[0079] To address various issues and advance the field of the art, the entire application (including the cover page, title, headings, background, summary, brief description of the drawings, detailed description, claims, abstract, drawings, appendices, etc.) illustrates various embodiments that can put the embodiments into practice. The advantages and features of the application are merely representative examples of the embodiments and are not exhaustive and / or exclusive. They are provided to aid in and teach the understanding of the principles described in the claims.

[0073]

[0080] Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions such as those created by a compiler, code used to create web services, and files containing high-level instructions executed by a computer using an interpreter. For example, embodiments can be implemented using Python, Java, JavaScript, C++, and / or other programming languages, packages, and software development tools.

[0074]

[0081] The drawings are for illustrative purposes only and are not intended to limit the scope of the content described herein. The drawings are not necessarily drawn to scale, and in some cases, various aspects of the content disclosed herein may be exaggerated or enlarged in order to aid in understanding various features. In the drawings, similar reference letters generally refer to similar features (e.g., functionally similar and / or structurally similar elements).

[0075]

[0082] The actions performed as part of the disclosed method can be ordered in any appropriate manner. Therefore, embodiments can be constructed in which processes or steps are performed in an order different from that shown, and such embodiments may include the simultaneous execution of some steps or processes, even if they are shown as sequential actions in exemplary embodiments. In other words, it should be understood that such features are not necessarily limited to a specific execution order, and any number of threads, processes, services, servers, etc., can execute sequentially, asynchronously, concurrently, in parallel, simultaneously, synchronously, etc., in a manner more consistent with the disclosure. Therefore, some of these features may be contradictory in that they cannot coexist simultaneously within a single embodiment. Similarly, some features are applicable to certain embodiments of the innovation but not to others.

[0076]

[0083] When used herein and in its embodiments, the phrase “and / or” should be understood to mean “either or both” of the elements thus combined, that is, elements that exist sometimes associatively and sometimes separately. Similarly, any multiple elements listed using “and / or” should be interpreted as “one or more” of the elements thus combined. Elements other than those specifically identified by the “and / or” clause may exist, whether related to those specifically identified elements or not. Thus, as a non-restrictive example, when used in combination with non-restrictive language such as “includes,” a reference to “A and / or B” may refer to A only in one embodiment (optionally including elements other than B), to B only in another embodiment (optionally including elements other than A), and to both A and B in yet another embodiment (optionally including other elements), and so on.

[0077]

[0084] When used herein and in embodiments, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when dividing items in a list, “or” or “and / or” should be interpreted as inclusive, meaning that it includes some of the elements or at least one or more of the items in the list, and optionally, additional items not listed. Only terms that clearly indicate the opposite, such as “one of” or “exactly one of” or, when used in embodiments, “consisting of,” refer to including some of the elements or exactly one of the items in the list. In general, when used herein, the term “or” should be interpreted only as indicating an exclusive alternative (i.e., “one or the other, but not both”) when preceded by an exclusive term such as “either,” “one of,” “one of,” or “exactly one of.” When used in embodiments, “consisting essentially of” should have the usual meaning as used in the field of patent law.

Claims

1. A processor receiving a set of images of a plurality of collectibles, wherein each image in the set of images is assigned to one or more of the following: at least one defect type label, a first grade classification label relating to the surface condition of the collectibles from the plurality of collectibles, a second grade classification label relating to the edge condition of the collectibles, and a third grade classification label relating to the corner condition of the collectibles. In the aforementioned processor, for each image in the set of images, a boundary defining the collected items within that image is detected, a perspective warp transformation is performed on the image from the set of images where the boundary relating to that image does not have a predetermined shape, and the portion of the image that is not within the boundary defining the collected items is removed, thereby generating a set of preprocessed images based on the set of images. The processor trains a set of models based on each preprocessed image from the set of preprocessed images, wherein the set of models is (1) A first model trained using (a) each preprocessed image from the set of preprocessed images, and (b) the first grade classification label associated with the preprocessed image, (2) A second model trained using (a) each preprocessed image from the set of preprocessed images, and (b) the second grade classification label associated with the preprocessed image, and (3) Training a third model, which is trained using (a) each preprocessed image from the set of preprocessed images, and (b) the third grade classification label associated with the preprocessed image. In the aforementioned processor, the set of models is applied to a new image of a new collectible that is not included in the plurality of collectibles to indicate a defect and the type of defect associated with the defect, and The processor displays an output indicating that the new collected item contains the defect and the type of defect associated with the defect. Methods that include...

2. The method according to claim 1, wherein a first image from the set of images is taken under first lighting conditions, and a second image from the set of images is taken under second lighting conditions different from the first lighting conditions.

3. The method according to claim 1, wherein a first image from the set of images is taken at a first angle with respect to a first collectible from the plurality of collectibles, and a second image from the set of images is taken at a second angle with respect to the first collectible or one of the second collectibles from the plurality of collectibles that is different from the first collectible, and the second angle is different from the first angle.

4. The method according to claim 1, wherein a first image from the set of images is taken with a first background, and a second image from the set of images is taken with a second background different from the first background.

5. The method according to claim 1, wherein generating the set of preprocessed images further includes resizing each image from the set of images, which have a size other than the predetermined size, so that the images have the predetermined size.

6. The method according to claim 1, further comprising generating the set of preprocessed images by resizing each image from the set of images having a resolution not within a predetermined resolution range so that the images have a resolution within the predetermined resolution range.

7. The method according to claim 1, wherein the set of models includes at least one dropout layer to reduce over-adaptation.

8. The processor improves the hyperparameters associated with the set of models by using at least one of a random search algorithm, a hyperband algorithm, or a Bayesian optimization algorithm. The method according to claim 1, further comprising:

9. The method according to claim 1, wherein each image from the set of images is further associated with a fourth grade classification label indicating the centering status of the collected items, and the set of models further includes (1) a fourth model trained using (a) each preprocessed image from the set of preprocessed images and (b) the fourth grade classification label associated with the preprocessed image, and (2) a fifth model trained using (a) each preprocessed image from the set of preprocessed images and (b) the at least one defect type label associated with the preprocessed image.

Images