SYSTEM, METHOD, APPARATUS, AND NON-TRANSIENT COMPUTER-READABLE STORAGE MEDIUM FOR PROCESSING IMAGES OF A MOBILE DEVICE USING MACHINE LEARNING TO DETERMINE THE INTEGRITY STATE OF THE MOBILE DEVICE - Patent application

Abstract

Systems, apparatus, methods, and computer program products are provided for determining the integrity status of a mobile device. Images of a mobile device captured by the mobile device using a reflective surface are processed using various trained models, such as neural networks, to verify authenticity, detect damage, and detect occlusions. A mask may be generated to enable identification of concave occlusions or blocked corners of an object, such as a mobile device, in an image. Images of the front and / or back of the mobile device may be processed to determine the integrity status of the mobile device, such as verified, not verified, or indeterminate. A user may be prompted to remove a cover, remove an occlusion, and / or move the mobile device closer to a reflective surface. A real-time response regarding the integrity status of the mobile device may be provided. The trained models may be trained to increase the accuracy of the mobile device integrity status.

Description

[Technical field]

[0001] Embodiments of the present invention relate generally to computer technology, and more particularly to systems, methods, apparatus, and computer program products for utilizing machine learning to train and utilize mathematical model(s), such as predictive model(s), neural network(s), and / or the like, to determine an integrity status of a mobile device based on electronic processing of images. [Background technology]

[0002] Computer vision allows computers to see and understand images. In some cases, computer vision can be used to detect and analyze the content of an image, such as recognizing objects within an image. However, existing techniques are inadequate to meet the speed and accuracy requirements of many industries, and there is a need for improvements in computer vision techniques and technologies to enable high performance image processing. Moreover, human analysis does not lend itself to the speed and accuracy required for computer vision tasks. Through hard work, ingenuity, and innovation, many of these identified problems have been solved by developing solutions included in embodiments of the present invention, many examples of which are described in detail herein. Summary of the Invention [Problem to be solved by the invention]

[0003] Systems, methods, apparatus and computer program products are therefore provided for training and utilizing models using machine learning to determine an integrity state of a mobile device based on electronic processing of images. [Means for solving the problem]

[0004] In some use cases, the system needs to review images of an object to verify the object's integrity (e.g., to determine information about the object, to verify the object's operability or functionality, to verify the object's identity, or the like). Computer vision and image processing need to occur quickly and with high accuracy, which is lacking in many conventional image processing techniques. A further challenge is when the system cannot select the imaging device that captures the image and has no direct control over the image capture process, and thus the computer vision needs to be robust enough to absorb and / or detect issues with the capture process. In an example work environment, the system may attempt to verify the object's identity and integrity using only an image of the object (e.g., a mobile device) or using the image in combination with one or more data objects transmitted from the object or from another device. One example of such an environment may be when a user registers for a service, protection plan, or the like, which requires a remote system to verify the object (e.g., a mobile device) without the device being physically present. Some processes for purchasing coverage aftermarket require the consumer to visit a retailer, insurance company, or mobile device service provider to have the device inspected and verify the integrity of the device before an insurer issues a policy for coverage and registers the device. Other processes for purchasing and / or selling coverage allow the consumer to utilize a self-service web or mobile application to take a photo of their mobile device and submit the image for manual review prior to registration. However, such processes require review time and may delay confirmation of coverage for the consumer. Such processes may further expose providers to fraudulent activities, such as when a consumer submits a photo of a different, undamaged mobile device to attempt to obtain coverage for a previously damaged device.

[0005] Additional implementations provide a time-sensitive barcode, quick response (QR) code, or other computer-generated code that is displayed by the device and captured in a photograph using a mirror, thereby linking the photo submission to the device that displayed the code. However, such implementations may be susceptible to fraud, such as by allowing a user to recreate the code on another intact device and capture a photograph of that intact device. Still further, code implementations may only provide verification of the front (e.g., display side) of the device without positively verifying the condition or status of the back of the device and / or the device's bezel. Another disadvantage of such implementations is that if the code is displayed on the device display, it may obscure cracks or other damage present on the screen.

[0006] Example embodiments of the present disclosure provide improved determination of the integrity status of a mobile device. Example embodiments may prompt a user to capture an image of their device in a mirror or other reflective surface using the device's own sensors or camera. The mobile device's identification information may be processed along with the image to verify that the image was indeed taken from the subject device from which the image was captured and that there is no pre-existing damage that would render the device ineligible for compensation.

[0007] Example embodiments may utilize machine learning algorithms and associated mathematical model(s), such as, but not limited to, neural network(s), such as convolutional neural networks and / or the like, predictive model(s) and / or other types of "model(s)," as may be referred to herein, that may be trained to analyze and identify relevant information from images by using training images that are manually reviewed and labeled and / or characterized by a user. It will be understood that any reference to a "model" herein may include any type of model that may be used in a machine learning algorithm trained on training images to make predictions regarding certain features of other images. Example embodiments may utilize the trained model(s) to utilize information detected in subsequently received images to predict features in the images, such as, but not limited to, the integrity state of the mobile device.

[0008] Different models, each of which may be trained with a different training set, may be used to make different types of predictions. Utilization of a trained model(s) (e.g., neural networks) may enable an example embodiment to determine the integrity status of a mobile device in real-time or near real-time from the time an image is submitted (without additional human review, according to some embodiments) and / or forward inconclusive or high-risk predictions in real-time or near real-time for further review before finalizing the integrity status of the mobile device and / or enrolling the mobile device in a protection plan. In some embodiments, the output of one or more models may be input to subsequent models for more sophisticated analysis of the images and determination of the integrity status of the mobile device.

[0009] In some example environments, consumers may purchase insurance plans, warranties, extended warranties, and / or other device protection plans to protect their mobile devices and smartphones from damage, theft, loss, and / or the like. In some cases, consumers may purchase such plans at a storefront, so the condition of the device is known to be new and the device is eligible for coverage. However, in some cases, consumers may wish to purchase protection after the device is owned, either directly from an insurance company or through their mobile device service provider. The provider must be able to quickly verify the integrity of the device without physical access to the device and without the ability to directly manipulate the device. In such cases, manual review may not be able to meet the accuracy and speed required to verify the integrity of the device in a reasonable amount of time, and there may be a need for the systems, methods, and apparatus described herein. Similarly, consumers who purchase used or refurbished devices may wish to purchase insurance in the aftermarket, where the condition of the device is not known to the insurance company. Insurance companies may verify the condition of the device at the time the protection is purchased to minimize losses and prevent fraudulent purchases of protection for devices with existing damage.

[0010] It will be understood that references made herein to guarantees, extended warranties, insurance, policies, insurance policies, coverage, device protection plans, protection plans, and / or the like are not intended to limit the scope of the present disclosure, and example embodiments may relate to enrolling a mobile device into any such aforesaid or similar plans to protect the mobile device against loss, or to other environments using the computer vision and image processing systems, methods, and apparatus described herein. Similarly, any reference to verifying device integrity may relate to the eligibility of a mobile device for enrollment in any of the aforesaid plans or environments. Still further, determining the integrity status of a mobile device may be used for other purposes.

[0011] One example condition implemented in accordance with example embodiments described herein includes determining whether an occlusion is present in an image of a mobile device. It will be appreciated that the occlusion detection process disclosed herein may be used for other purposes, such as determining occlusion of any type of object in an image.

[0012] A method is provided that includes receiving a device integrity verification request associated with a mobile device and receiving a mobile device identification data object that includes information describing the mobile device. The method further includes displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device, and receiving at least one image captured by the one or more image sensors of the mobile device. The method may further include processing the at least one image with at least one trained model to determine an integrity state of the mobile device. In an embodiment, the at least one trained model may include a neural network.

[0013] According to an example embodiment, processing at least one image to determine an integrity status of the mobile device includes determining whether the at least one image includes a mobile device associated with a mobile device identification data object. Determining whether the at least one image includes a mobile device includes identifying a suspect mobile device in the at least one image, generating a prediction of an identification of the at least one suspect mobile device, and comparing the mobile device identification data object to the prediction of an identification of the at least one suspect mobile device to determine whether the suspect mobile device is the mobile device. Processing at least one image to determine an integrity status of the mobile device may further include determining an integrity status of the mobile device as verified if the suspect mobile device is determined to be the mobile device. Processing at least one image to determine an integrity status of the mobile device may further include sending a device integrity verification request and the at least one image to an internal user device for internal review if the integrity status of the mobile device is determined to be indeterminate.

[0014] According to some embodiments, processing the at least one image to determine an integrity status of the mobile device may include utilizing the at least one trained model to determine whether there is damage to the mobile device, and in response to determining that there is damage to the mobile device, determining that the integrity status of the mobile device is not verified.

[0015] In some embodiments, processing the at least one image to determine an integrity status of the mobile device includes determining an angle of the mobile device relative to a reflective surface when the at least one image was captured, and determining based on the angle that the at least one image includes a different mobile device than a mobile device associated with the mobile device identification data object. Processing the at least one image to determine an integrity status of the mobile device may further include, in response to determining that the at least one image captures a different mobile device based on the angle, causing a message to be displayed on the mobile device instructing a user to recapture the mobile device, and determining that the integrity status of the mobile device has not been verified.

[0016] According to some embodiments, processing the at least one image to determine an integrity state of the mobile device may include determining a location within the at least one image of the mobile device, the location defined as a bounding box, and if the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, causing a message to be displayed on the mobile device instructing the mobile device to move closer to a reflective surface. If the bounding box has a second predetermined relationship with the threshold ratio of the at least one image, processing the at least one image to determine an integrity state of the mobile device may further include cropping the at least one image according to the bounding box.

[0017] According to some embodiments, processing the at least one image to determine the integrity state of the mobile device includes using at least one trained model to determine that an object is occluding the mobile device in the at least one image and displaying a prompt on the mobile device to capture an image without the occlusion. Determining whether an occlusion of the mobile device is in the at least one image may include determining whether a concave occlusion is in the at least one image and determining whether a blocked corner is in the at least one image. Determining whether there is a concave occlusion in the at least one image may include using the at least one trained model to generate a mobile device mask that includes a reduced number of colors compared to the at least one image, extracting a polygonal subregion P of the mobile device mask, determining a convex hull of P and calculating a difference between P and the convex hull, removing or reducing thin discrepancies of at least one edge of P and the convex hull, identifying a maximum area of ​​the remaining region of P, and comparing the maximum area to a threshold to determine whether the at least one image includes a concave occlusion.

[0018] According to some embodiments, determining whether there are occluded corners in the at least one image may include generating a mobile device mask using the at least one trained model, the mobile device mask including a reduced number of colors compared to the at least one image, extracting a polygonal subregion P of the mobile device mask, determining a convex hull of P, identifying four major edges of the convex hull, determining intersections of adjacent major edges to identify corners, determining a respective distance from each corner to P, and comparing each distance to a distance threshold to determine whether any corners are occluded in the at least one image.

[0019] According to an embodiment, processing the at least one image to determine an integrity status of the mobile device includes determining, using the at least one trained model, whether the at least one image includes a front side of the mobile device, a back side of the mobile device, or a cover.

[0020] In response to receiving the at least one image, an example embodiment may provide a response for display on the mobile device in real time or near real time, the response provided depending on the determined integrity state of the mobile device.

[0021] Example embodiments may also include displaying, on the mobile device, a test pattern configured to provide improved accuracy in predicting characteristics of at least one image captured when the mobile device is displaying the test pattern as compared to accuracy in predicting characteristics of at least one image captured when the mobile device is displaying another display pattern.

[0022] Some example embodiments may identify a subset of conditions to be satisfied to determine the integrity state of the mobile device as verified, set the image state of the particular image as verified if all conditions in the subset of conditions are satisfied in the particular image, and determine the integrity state of the mobile device as verified if the respective image states for all required images are verified. In some embodiments, at least one condition of the subset of conditions to be satisfied is executed on the mobile device.

[0023] According to an embodiment, receiving at least one image includes receiving at least two images captured by a mobile device, where a first image of the at least two images is of a front side of the device and a second image of the at least two images is of a back side of the device, and processing the at least one image to determine an integrity state of the mobile device includes processing both the first image and the second image with at least one trained model, and determining the determined integrity state of the mobile device as verified if processing of both images results in a respective image state being verified.

[0024] Some embodiments may train at least one trained model by inputting training images and respective labels that describe characteristics of each training image. A method for detecting concave occlusions in an image is also provided, the method including generating a mask containing a reduced number of colors compared to the image using at least one trained model, extracting a polygonal subregion P of the mask, determining a convex hull of P, and calculating the difference between P and the convex hull. The method further includes removing or reducing thin discrepancies of at least one edge of P and the convex hull, recalculating P as the maximum area of ​​the remaining regions, and determining the concavity as the difference between P and the convex hull.

[0025] A method is provided for detecting occluded corners of an object in an image, the method including generating a mask containing a reduced number of colors compared to the image using at least one trained model, extracting a polygonal subregion P of the mask, determining a convex hull of P, identifying a predetermined number of major edges of the convex hull, determining intersections of adjacent major edges to identify corners, determining a respective distance from each corner to P, and comparing each distance to a distance threshold to determine whether any corners are occluded in the image.

[0026] An apparatus is provided with at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, using the processor, cause the apparatus to at least receive a device integrity verification request associated with a mobile device, and receive a mobile device identification data object including information describing the mobile device. The at least one memory and the computer program code are further configured to, using the processor, cause the apparatus to display a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device, and receive at least one image captured by the one or more image sensors of the mobile device. The at least one memory and the computer program code may be further configured, using the processor, to process the at least one image using the at least one trained model to determine an integrity state of the mobile device.

[0027] An apparatus for detecting concave occlusions in an image is provided, the apparatus comprising at least one processor and at least one memory containing computer program code configured to cause the apparatus, using the processor, to at least generate a mask containing a reduced number of colors compared to the image using the at least one trained model, extract a polygonal subregion P of the mask, determine a convex hull of P, calculate a difference between P and the convex hull, remove or reduce thin discrepancies of at least one edge of P and the convex hull, recalculate P as the maximum area of ​​the remaining regions, and determine a concavity as the difference between P and the convex hull.

[0028] An apparatus for detecting occluded corners of an object in an image is also provided, the apparatus comprising at least one processor and at least one memory containing computer program code configured to cause the apparatus, using the processor, to at least generate a mask containing a reduced number of colors compared to the image, extract a polygonal subregion P of the mask, determine a convex hull of P, identify a predetermined number of major edges of the convex hull, determine intersections of adjacent major edges to identify corners, determine a respective distance from each corner to P, and compare each distance to a distance threshold to determine whether any corners are occluded in the image.

[0029] A computer program product is provided that includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving a device integrity verification request associated with a mobile device, receiving a mobile device identification data object including information describing the mobile device, displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device, receiving the at least one image captured by the one or more image sensors of the mobile device, and processing the at least one image using the at least one trained model to determine an integrity status of the mobile device.

[0030] A computer program product for detecting concave occlusions in an image is also provided, the computer program product including at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for generating a mask including a reduced number of colors compared to the image using at least one trained model, extracting a polygonal subregion P of the mask, determining a convex hull of P, calculating the difference between P and the convex hull, removing or reducing thin discrepancies of at least one edge of P and the convex hull, recalculating P as the maximum area of ​​the remaining regions, and determining a concavity as the difference between P and the convex hull.

[0031] A computer program product for detecting occluded corners of an object in an image is also provided, the computer program product including at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for generating a mask including a reduced number of colors compared to the image using at least one trained model, extracting a polygonal subregion P of the mask, determining a convex hull of P, identifying a predetermined number of major edges of the convex hull, determining intersections of adjacent major edges to identify corners, determining a respective distance from each corner to P, and comparing each distance to a distance threshold to determine whether any corners are occluded in the image.

[0032] An apparatus is provided that includes means for receiving a device integrity verification request associated with a mobile device and means for receiving a mobile device identification data object that includes information describing the mobile device, means for displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device, means for receiving at least one image captured by the one or more image sensors of the mobile device, and means for processing the at least one image using the at least one trained model to determine an integrity state of the mobile device.

[0033] An apparatus is provided having means for detecting concave occlusions in an image, the apparatus including means for causing the apparatus to generate a mask containing a reduced number of colors compared to the image using at least one trained model and extracting a polygonal subregion P of the mask, means for determining a convex hull of P and calculating the difference between P and the convex hull, means for removing or reducing thin discrepancies of at least one edge of P and the convex hull, means for recalculating P as the maximum area of ​​the remaining regions, and means for determining a concavity as the difference between P and the convex hull.

[0034] There is also provided an apparatus comprising means for detecting occluded corners of an object in an image, the apparatus comprising: The method includes means for generating a mask using at least one trained model that contains a reduced number of colors compared to the image, means for extracting a polygonal subregion P of the mask, means for determining a convex hull of P and identifying a predetermined number of major edges of the convex hull, means for determining intersections of adjacent major edges to identify corners, means for determining a respective distance from each corner to P, and means for comparing each distance to a distance threshold to determine whether any corners are occluded in the image.

[0035] According to an embodiment, a method is provided that includes receiving an indication of a target image and processing the target image with at least one trained model, such as a model (e.g., a neural network) that can be used in a machine learning algorithm to determine whether the target image includes a mobile device. The model is trained on a plurality of training images, each of which is labeled as either including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device.

[0036] According to an embodiment, a method is provided that includes receiving an indication of a target image and processing the target image with at least one trained model trained on a plurality of training images, each associated with a bounding box indicating a location of a mobile device within the image, to determine a location of the mobile device within the target image. The method may further include cropping the target image based on the determined location of the mobile device within the target image.

[0037] According to one embodiment, a method is provided that includes receiving an indication of a target image of a target mobile device, and processing the target image of the target mobile device with at least one trained model trained on a plurality of training images of mobile devices labeled as including a cover on a respective mobile device or excluding a cover on a respective mobile device to determine whether the target image includes a cover on the target mobile device.

[0038] According to an embodiment, a method is provided that includes receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device with at least one trained model trained on a plurality of training images of the mobile device, where each training image is labeled as including either a front side of the respective mobile device or a rear side of the respective mobile device, to determine whether the target image includes a front side or a rear side of the target mobile device.

[0039] According to an embodiment, a method is provided that includes receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device with at least one trained model trained on a plurality of training images of mobile devices, where each training image is labeled as being captured by a respective mobile device included in the image or by a different device than the respective mobile device included in the image, to determine whether a target mobile device included in the target image was captured by the target mobile device or by a different device.

[0040] According to one embodiment, a method is provided that includes receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device with at least one trained model trained on a plurality of training images of the mobile device, where each training image is labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0041] According to an embodiment, the device comprises at least one processor and at least one memory containing computer program code configured to cause the device, using the processor, to at least receive an indication of a target image and process the target image using at least one trained model trained on a plurality of training images, each labeled as either including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device.

[0042] Also provided is an apparatus comprising at least one processor and at least one memory containing computer program code configured to, using the processor, cause the apparatus to at least receive an indication of a target image; process the target image using at least one trained model trained on a plurality of training images, each associated with a bounding box indicating a position of a mobile device within the image, to determine a position of the mobile device within the target image; and crop the target image based on the determined position of the mobile device within the target image.

[0043] According to an embodiment, an apparatus is provided that includes at least one processor and at least one memory including computer program code configured to, using the processor, cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained on a plurality of training images of mobile devices labeled as including a cover on the respective mobile device or excluding a cover on the respective mobile device to determine whether the target image includes a cover on the target mobile device.

[0044] Also provided is an apparatus comprising at least one processor and at least one memory containing computer program code configured, using the processor, to cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained on a plurality of training images of the mobile device, where each training image is labeled as including either a front side of the respective mobile device or including a rear side of the respective mobile device, to determine whether the target image includes a front side or a rear side of the target mobile device.

[0045] According to an embodiment, an apparatus comprises at least one processor and at least one memory including computer program code configured to, using the processor, cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained on a plurality of training images of mobile devices, where each training image is labeled as being captured by a respective mobile device included in the image or by a device different from the respective mobile device included in the image, to determine whether a target mobile device included in the target image was captured by the target mobile device or by a different device.

[0046] Also provided is an apparatus comprising at least one processor and at least one memory containing computer program code configured, using the processor, to cause the apparatus to at least receive an indication of target images of target mobile devices, and process the target images of the mobile devices using at least one trained model trained on a plurality of training images of the mobile devices, each training image being labeled with a damage assessment, to calculate a damage assessment for the target mobile devices in the target images.

[0047] According to an example embodiment, a computer program product is provided that includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an indication of a target image and processing the target image using at least one trained model trained on a plurality of training images, each labeled as either including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device.

[0048] Also provided is a computer program product that includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an indication of a target image and processing the target image using at least one trained model trained on a plurality of training images, each associated with a bounding box indicating a position of a mobile device within the image, to determine a position of the mobile device within the target image. The computer-executable program code instructions include program code instructions for cropping the target image based on the determined position of the mobile device within the target image.

[0049] Also provided is a computer program product that includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained on a plurality of training images of mobile devices labeled as including a cover on a respective mobile device or excluding a cover on a respective mobile device to determine whether the target image includes a cover on the target mobile device.

[0050] According to an embodiment, a computer program product is provided that includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained on a plurality of training images of mobile devices, where each training image is labeled as including a front side of the respective mobile device or including a rear side of the respective mobile device, to determine whether the target image includes a front side or a rear side of the target mobile device.

[0051] A computer program product is provided that includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained on a plurality of training images of mobile devices, where each training image is labeled as being captured by a respective mobile device included in the image or by a device different from the respective mobile device included in the image, to determine whether a target mobile device included in the target image was captured by the target mobile device or by a different device.

[0052] Also provided is a computer program product that includes at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained on a plurality of training images of the mobile device, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0053] The models and algorithms described herein may be used alone for their own intended purpose or may be used in one or more larger processes, such as those described herein. For example, in some embodiments, both rear and front camera images may be taken of the back and front of the device, and the various trained models described herein may be run on each image, either separately or as part of a larger process, to ensure that the device is intact and undamaged. In some embodiments, one or more of the models and algorithms may be run as part of an orientation process for a protected product and / or service contract or other device protection program where verification of the integrity of the device is required.

[0054] The foregoing summary is provided solely for the purpose of summarizing some example embodiments of the present invention in order to provide a basic understanding of some aspects of the present invention. As such, it will be understood that the foregoing example embodiments are merely examples and should not be construed as narrowing the scope or spirit of the present disclosure in any way. It will be understood that the scope of the present disclosure encompasses many potential embodiments, some of which are further described below in addition to those summarized herein.

[0055] Having described embodiments of the present invention in general terms, reference is now made to the accompanying drawings, which are not necessarily drawn to scale: [Brief description of the drawings]

[0056] [Figure 1] 1 illustrates a system for determining an integrity status of a mobile device according to some example embodiments. [Diagram 2] 1 illustrates a block diagram of an apparatus according to some example embodiments. [Diagram 3] 1 is a flow diagram illustrating operations for determining an integrity state of a mobile device according to some example embodiments. [Figure 4A-1] 1 is a flow diagram illustrating operations for determining an integrity state of a mobile device according to some example embodiments. [Figure 4A-2] 1 is a flow diagram illustrating operations for determining an integrity state of a mobile device according to some example embodiments. [Figure 4B] 1 illustrates data flow between models and / or their circuits according to some example embodiments. [Figure 5A] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5B] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5C] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5D] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5E] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5F] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5G] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5H]1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5I] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5J] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5K] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5L] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5M] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5N] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5O] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5P] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5Q] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5R] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5S] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5T] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5U] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5V] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5W] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5X]1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 5Y] 1 illustrates example user interfaces provided in accordance with some example embodiments. [Figure 6] 4 is a flow diagram illustrating operations for detecting occlusions in an image in accordance with some example embodiments. [Figure 7A] 1 illustrates an example captured image of a mobile device according to some example embodiments. [Figure 7B] 7A and 8B illustrate mobile device masks that may be generated from the images of FIGS. 7A and 8A, respectively, according to some example embodiments. [Figure 8A] 1 illustrates an example captured image of a mobile device according to some example embodiments. [Figure 8B] 7A and 8B illustrate mobile device masks that may be generated from the images of FIGS. 7A and 8A, respectively, according to some example embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0057] Certain embodiments of the present invention will now be described more fully below with reference to the accompanying drawings, in which some, but not all embodiments of the invention are shown. Indeed, various embodiments of the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, but rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numbers refer to like elements throughout.

[0058] As used herein, the terms "data," "content," "information," and similar terms may be used interchangeably to refer to data that may be captured, transmitted, received, displayed, and / or stored, according to various example embodiments. Thus, use of any such terms should not be construed to limit the spirit and scope of the present disclosure. Additionally, where a computing device that receives data from another computing device is described herein, it will be understood that the data may be received directly from the other computing device or indirectly via one or more intermediate computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and / or the like. Similarly, where a computing device that transmits data to another computing device is described herein, it will be understood that the data may be transmitted directly to the other computing device or indirectly via one or more intermediate computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and / or the like.

[0059] System Overview FIG. 1 illustrates a system 100 for determining an integrity state of a mobile device based on processing an image of the device, according to an example embodiment. The system of FIG. 1 may further be utilized to detect occlusions in an image, such as an image of a mobile device, according to an example embodiment. It will be understood that the system of FIG. 1 and the illustrations in the other figures are each provided as an example of an embodiment(s) and should not be construed to narrow the scope or spirit of the present disclosure in any manner. In this regard, the scope of the present disclosure encompasses many potential embodiments in addition to those illustrated and described herein. Thus, while FIG. 1 illustrates one example configuration, numerous other configurations may be used to implement embodiments of the present invention.

[0060] The system 100 may include any number of mobile devices 104, or simply "devices" as referred to herein. The mobile devices 104 may be embodied as any mobile computing device, such as, by way of non-limiting example, a mobile phone, a smart phone, a mobile communication device, a tablet computing device, any combination thereof, or the like. Although described as a mobile device, in some embodiments, the mobile device 104 may instead be substituted with any stationary computing device, or other device, without departing from the scope of the present disclosure. The mobile device 104 may be used by a user to download, install, and access a self-service app, such as one provided by a provider, to obtain coverage for the mobile device 104. Additionally or alternatively, the mobile device 104 may utilize a browser installed thereon to access a self-service web application, such as one hosted and / or provided by a provider. Still further, the mobile device 104 may be used to capture images for processing according to example embodiments.

[0061] The device integrity verification device 108 may be associated with a provider, or any other entity, and may be any processor-driven device that facilitates processing of requests for device integrity verification, such as those generated from a request to enroll a device in a device protection program. For example, the device integrity verification device 108 may include one or more computers, servers, server clusters, one or more network nodes, or cloud computing infrastructure configured to facilitate device integrity verification, enrollment in a device protection plan, and / or other services related to the provider. In an embodiment, some or all of the device integrity verification device 108 may be implemented on the mobile device 104.

[0062] In an example embodiment, the device integrity verification device 108 hosts or provides a service that allows access by the mobile device 104 to request coverage, and further prompts the user of the mobile device 104 to capture an image via the camera of the mobile device 104 using a mirror, as described in further detail herein. The device integrity verification device 108 may process the image using one or more of the computer vision and image processing embodiments described herein to determine whether the device is eligible for coverage, as described in further detail herein. The device integrity verification device 108 may include or have access to one or more models trained to analyze the image and extract relevant information, as described in further detail herein, to determine the integrity status of the device. According to some embodiments, the collection of training images and the training of the models may be performed in the device integrity verification device 108. The device integrity verification device 108 may be further configured to maintain information regarding subscribed and issued device protection plans and / or facilitate communication between the mobile device 104 and / or the optional internal user device 110.

[0063] The occlusion detection device 109 may be any processor-driven device that facilitates processing of an image to determine whether an object in the image is occluded. For example, the occlusion detection device 109 may include one or more computers, servers, server clusters, one or more network nodes, or a cloud computing infrastructure configured to facilitate processing of the image and identification of occlusion. According to an embodiment, the device integrity verification device 108 may be integrated with the occlusion detection device 109 to determine whether a mobile device in an image is occluded by a finger, and / or the like.

[0064] The optional internal user device 110 may include any computing device or devices that may be used by the provider and / or other entities to facilitate device integrity verification. As an example, the internal user device 110 may be implemented in a support center or central facility remote from the mobile device that may be staffed with one or more customer service representatives who may utilize applications provided by the device integrity verification device 108 to receive device integrity verification server results, which may allow for further processing or analysis of the images prior to verification or may facilitate additional review. For example, if the device integrity verification device 108 indicates that further internal review of an image is required for verification, such as by an indeterminate mobile device integrity status, the internal user device 110 may be used by the support staff to review the images to confirm or deny the integrity of the mobile device 104, thereby respectively confirming or denying coverage of the mobile device 104 in the device protection plan. The internal user device 110 may further be utilized by the internal user to capture and / or label training images with which to train the model(s). It will be understood that the internal user device 110 may be considered optional. In some embodiments, the device integrity verifier 108 may facilitate faster processing by automatically verifying or rejecting the integrity of the mobile device.

[0065] According to some embodiments, various components of the system 100 may be configured to communicate over a network, such as via the network 106. For example, the mobile device 104 may be configured to access the network 106 via a cellular connection, a wireless local area network connection, an Ethernet connection, and / or the like. As such, the network 106 may include a wired network, a wireless network (e.g., a cellular network, a wireless local area network, a wireless wide area network, some combination thereof, or the like), or a combination thereof, and in some example embodiments, includes at least a portion of the Internet.

[0066] As mentioned above, certain components of the system 100 may be optional. For example, the device integrity verification unit 108 may be optional, and the device integrity verification may be performed on the mobile device 104, such as by a self-service app installed on the mobile device 104.

[0067] 2, apparatus 200 is a computing device(s) configured to implement mobile device 104, device integrity verification device 108, image detection masking server 109, and / or internal user device 110, according to example embodiments. Apparatus 200 may at least partially or fully embody any of mobile device 104, device integrity verification device 108, image detection masking server 109, and / or internal user device 110. Apparatus 200 may be implemented as a distributed system including any of mobile device 104, device integrity verification device 108, image detection masking server 109, and / or internal user device 110, and / or associated network(s).

[0068] It should be noted that the components, devices, and elements illustrated and described in connection with FIG. 2 may not be required, and thus some may be omitted in certain embodiments. For example, FIG. 2 illustrates a user interface 216, which may be optional within the device integrity verification apparatus 108, as described in further detail below. Additionally, some embodiments may include additional or different components, devices, or elements beyond those illustrated and described in connection with FIG.

[0069] Apparatus 200 may include processing circuitry 210, which may be configured to perform operations according to one or more example embodiments disclosed herein. In this regard, processing circuitry 210 may be configured to perform and / or control the operation of one or more functions of apparatus 200 according to various example embodiments. Processing circuitry 210 may be configured to perform data processing, application execution, and / or other processing and management services according to one or more example embodiments. In some embodiments, apparatus 200, or a portion(s) or component(s) thereof, such as processing circuitry 210, may be embodied as or include a circuit chip. The circuit chip may constitute a means for performing one or more operations to provide the functionality described herein.

[0070] In some example embodiments, the processing circuitry 210 may include a processor 212, and in some embodiments, such as that illustrated in FIG. 2, may further include a memory 214. The processing circuitry 210 may communicate with or otherwise control a user interface 216, and / or a communication interface 218. As such, the processing circuitry 210 and / or the apparatus 200, such as those included within any of the mobile device 104, the device integrity verification apparatus 108, the image detection masking server 109, and / or the internal user apparatus 110, may be embodied as a circuit chip (e.g., an integrated circuit chip) configured (e.g., in hardware, software, or a combination of hardware and software) to perform the operations described herein.

[0071] The processor 212 may be embodied in a number of different ways. For example, the processor 212 may be embodied as a variety of processing means, such as one or more of a microprocessor or other processing element, a co-processor, a controller, or a variety of other computing or processing devices, including integrated circuits, such as, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. Although shown as a single processor, it will be understood that the processor 212 may include multiple processors. The multiple processors may be in operative communication with each other and may be collectively configured to perform one or more functions of the apparatus 200 as described herein. The multiple processors may be embodied on a single computing device or may be distributed across multiple computing devices collectively configured to function as the mobile device 104, the device integrity verification device 108, the image detection shielding server 109, the internal user device 110, and / or the apparatus 200. In some example embodiments, the processor 212 may be configured to execute instructions stored in the memory 214 or otherwise accessible to the processor 212. Thus, whether configured by hardware or a combination of hardware and software, processor 212 may represent an entity (e.g., physically embodied in a circuit--processing circuit 210) capable of performing operations in accordance with an embodiment of the present invention and simultaneously configured accordingly. Thus, for example, if processor 212 is embodied as an ASIC, FPGA, or the like, processor 212 may be hardware that is specifically configured to perform the operations described herein. As another example, if processor 212 is embodied as an executor of software instructions, the instructions may specifically configure processor 212 to perform one or more operations described herein.

[0072] In some example embodiments, the memory 214 may include one or more persistent memory devices, such as, for example, volatile and / or non-volatile memory, which may be either fixed or removable. In this regard, the memory 214 may include a persistent computer-readable storage medium. Although the memory 214 is shown as a single memory, it will be understood that the memory 214 may include multiple memories. The multiple memories may be embodied on a single computing device or distributed across multiple computing devices. The memory 214 may be configured to store information, data, applications, computer program code, instructions, and / or the like to enable the device 200 to perform various functions, according to one or more example embodiments. For example, when the device 200 is implemented as a mobile device 104, a device integrity verification device 108, an image detection masking server 109, and / or an internal user device 110, the memory 214 may be configured to store computer program code to perform its corresponding functions, as described herein, according to example embodiments.

[0073] Still further, the memory 214 may be configured to store the model(s) and / or training images used to train the model(s) to predict certain relevant information in subsequently received images. The memory 214 may be further configured to buffer input data for processing by the processor 212. Additionally or alternatively, the memory 214 may be configured to store instructions for execution by the processor 212. In some embodiments, the memory 214 may include one or more databases that may store various files, content, or data sets. Among the contents of the memory 214, applications may be stored for execution by the processor 212 to perform functions associated with each respective application. In some cases, the memory 214 may communicate with one or more of the processor 212, the user interface 216, and / or the communication interface 218 to pass information between components of the device 200.

[0074] The optional user interface 216 may be in communication with the processing circuit 210 to receive user input thereat and / or provide audible, visual, mechanical, or other output to the user. As such, the user interface 216 may include, for example, a keyboard, a mouse, a display, a touch screen display, a microphone, a speaker, and / or other input / output mechanisms. For example, in embodiments in which the apparatus 200 is implemented as a mobile device 104, the user interface 216 may provide a means for displaying instructions for capturing an image, in some example embodiments. In embodiments in which the apparatus is implemented as an internal user apparatus 110, the user interface 216 may provide a means for an internal user or colleague to review the images to verify or reject the integrity of the mobile device 104. The user interface 216 may further be used to label training images for purposes of training the model(s). In some example embodiments, aspects of the user interface 216 may be limited or the user interface 216 may not be present.

[0075] The communication interface 218 may include one or more interface mechanisms for enabling communication with other devices and / or networks. In some cases, the communication interface 218 may be any means, such as a device or circuit embodied in either hardware or a combination of hardware and software, configured to receive data from and / or transmit data to a network and / or any other device or module in communication with the processing circuit 210. By way of example, the communication interface 218 may be configured to enable communication over a network, such as the network 106, between any of the mobile device 104, the device integrity verification device 108, the internal user device 110, and / or the device 200. Accordingly, the communication interface 218 may include supporting hardware and / or software for enabling, for example, wireless and / or wired communication via cable, digital subscriber line (DSL), universal serial bus (USB), Ethernet, or other methods.

[0076] The apparatus 200 may include one or more image capture sensors 220, such as when the apparatus 200 is embodied by a mobile device 104. The image capture sensor 220 may be any sensor, such as a camera or other image capture device, configured to capture images and / or record video from the mobile device 104, and may include a front image capture sensor (e.g., a camera) configured on the same side as the display screen of the device, and / or a rear image capture sensor (e.g., a camera) on the rear side of the device (e.g., the side of the device without the display screen). In some embodiments, the mobile device 104 may include both front and rear image capture sensors, and in some embodiments, the mobile device 104 may include only one of a front image capture sensor or a rear image capture sensor. In some embodiments, any number of image capture sensors 220 may be present on the apparatus 200 (e.g., the mobile device 104).

[0077] Determining the Integrity Status of a Mobile Device Having now generally described example embodiments of the system 100 and apparatus for implementing the example embodiments, Figures 3 and 4A are flow diagrams illustrating example operations of the apparatus 200 according to some example embodiments. The operations may be performed by the apparatus 200, such as the mobile device 104, the device integrity verification apparatus 108, the occlusion detection apparatus 109, and / or the internal user apparatus 110.

[0078] 3 illustrates operations for determining an integrity status of a mobile device, such as for enrollment of the mobile device 104 in a device protection plan, according to some example embodiments. As shown in operation 302, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or the like, for receiving a device integrity verification request associated with the mobile device. In this regard, a user may access an application (or "app") installed on the mobile device 104 or a website hosted by the device integrity verification device 108 to request enrollment in a device protection plan. In some embodiments, the device integrity enrollment request may be generated by the device integrity verification device 108 or the internal user device 110 during device and / or user orientation. In this regard, according to an embodiment, the device integrity verification request may include or be accompanied by details regarding the requested insurance policy and / or coverage (e.g., an order), and / or other account information associated with the user, the user's contact information, and / or the like. It will be appreciated that a device integrity verification request may be generated for purposes other than device orientation in a protection plan.

[0079] Example embodiments may prompt the user to provide certain personal information, mobile device service provider information, and / or user-provided device information about their device, such as via the user interface 216. According to some embodiments, the user may be instructed to use the mobile device 104 to continue the registration process using the device that the user desires to register. For example, FIGS. 5A, 5B, 5C, and 5D are examples of user interfaces that may provide introductory information to the user and may be used to collect at least some data from the user, such as the user's mobile device service provider and / or mobile device information (e.g., make, model, and / or the like). For example, as shown in FIG. 5A, an introductory message 500 is provided. As shown in FIG. 5B, a prompt 502 for selecting a mobile device service provider and selectable options 504 or eligible mobile device service providers are provided. As shown in FIG. 5C, once a mobile device service provider is selected by the user, a confirmation 510 of the selected mobile device service provider is provided, as well as links to content and additional information 512.

[0080] As shown at operation 304, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or the like, for receiving a mobile device identification data object including information describing a mobile device, such as the mobile device 104. As previously mentioned, the user may be prompted via the user interface 216 to provide information describing the device the user wishes to register for protection. In some embodiments, the user may provide such information via a separate interface and / or network, such as by a personal computer or other mobile or fixed computing device. Any such data describing the device and / or its hardware, such as device type (e.g., manufacturer, model identifier), International Mobile Equipment Identity (IMEI), and / or the like, may be stored within the mobile device identification data object.

[0081] According to some embodiments, the mobile device identification information may not need to be provided by the user, and the mobile device data object may store the IMEI and / or other mobile device identification information systematically obtained by the website and / or app as the user accesses the website and / or app using the mobile device 104. The mobile device identification data object may thus include other information used to identify or uniquely identify the device, such as the device type, device model identifier, serial number, and / or the like. Although FIG. 5N (described in further detail below) illustrates a user interface that allows user entry of the device IMEI, it will be understood that according to some example embodiments, the IMEI may be systematically obtained as described above. Systematically obtaining the mobile device identification information may thus limit or reduce fraudulent activity, such as by preventing a user from entering the IMEI of a stolen, lost, or damaged device.

[0082] The mobile device identification data object may be used to enroll the device in a device discovery plan so that, upon subsequent claims, the consumer can generate device reflection data that matches the data (e.g., IMEI) stored in the mobile device identification data object. For claims related to lost or stolen devices, the mobile device service provider may use the data (e.g., IMEI) stored in the mobile device identification data object to prevent future network access and / or use of the device.

[0083] In operation 306, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or the like, for displaying a prompt on the mobile device 104 to capture at least one image of the mobile device using one or more sensors of the mobile device and a reflective surface, such as a mirror. Figures 5D, 5E, 5F, 5G, 5H, 5I, 5J, 5K, 5L, 5M, 5U, 5V, and 5W are example user interfaces that guide a user to capture an image of their mobile device using the front camera, the rear camera, or both. As shown in Figure 5D, instructional information 514 may be provided to the user to provide an overview of certain steps associated with photographing a mobile device. As shown in Figure 5E, an image capture instruction 516 is provided, as well as selectable prompts 518 and 518 indicating to capture an image of the front and rear (back) of the device, respectively. In response to the selection of the selectable prompt 518 and / or 518, the processor 212 of the example embodiment activates the image capture sensor 220 associated with the selected prompt 518 or 520, respectively. For example, in the case where the prompt 518 is selected to capture an image of the front of the mobile device 104, the processor 212 of the mobile device 104 may activate the front image capture sensor 220. In the case where the prompt 520 is selected to capture an image of the rear of the mobile device 104, the processor 212 of the mobile device 104 may activate the rear image capture sensor 220. According to some embodiments, any image capture sensor 220 with image capture capabilities may be utilized to capture an image. As shown in FIG. 5F, the example embodiment may provide an image capture instruction 526, which may be specific to the device type of the mobile device 104. For example, the image capture instruction 526 of FIG. 5F may indicate an instruction to utilize the “volume up” hard key of the mobile device 104 to capture an image. It will be appreciated that various implementations may be contemplated, such as utilizing hard or soft keys (not shown in FIG. 5F) to capture an image.In an embodiment, implementation may vary depending on the device type of the mobile device 104 .

[0084] As shown in FIG 5G, some mobile devices 104 may provide a security alert 528 to prompt the user to allow a mobile application provided by example embodiments provided herein to access or activate an image capture sensor 220, such as a camera. If the user has previously authorized or confirmed the request to allow the mobile application access, the message may not be displayed. In any case, as shown in FIG 5H, if the mobile application is permitted by the user to access the provided image camera 220, the display may reflect a viewfinder 532 to show images that may be captured.

[0085] In addition to providing prompts to capture an image, example embodiments may transition the user interface 216 to display a test pattern to provide improved accuracy in downstream processing of the captured image, including the device display. For example, the displayed test pattern may include a solid white display screen, as shown in FIG. 5H, or other test patterns identified as enabling efficient identification of damage, such as cracks and water damage, and / or enabling efficient identification of the display portion of the mobile device relative to the bezel.

[0086] In response, the system may prompt the user to hold the mobile device 104 in front of a mirror or other reflective surface and use the mobile device 104 to capture an image, for example, using one or more sensors (e.g., image capture sensor 220) of the device. Once an image is captured as instructed by the user, the captured image may be displayed as a confirmed captured image 528 in FIG. 5I. In response, as shown in FIG. 5J, the captured image 532 may be displayed within the area of ​​a selectable prompt 518 as shown in FIG. 5E, which may be selectable to allow the user to recapture the image. In FIG. 5J, a selectable prompt 520 is displayed similar to the display in FIG. 5E to indicate that a back photo has not yet been captured.

[0087] When the selectable prompt 520 is selected, the processor 212 may activate the rear image capture sensor 220 of the mobile device 104 to display a viewfinder 538 of the captured image, as shown in FIG. 5K. The user may follow the prompt or provide input to capture the image, and the captured image 542 may be displayed as provided in FIG. 5L. In response, the display shown in FIG. 5M may be updated to reflect the captured images 532 and 542 within the area of ​​the selectable prompts 518 and 518, respectively. The selectable prompts 518 and 518 may be selected to modify, edit, or review the captured image.

[0088] Returning to the description of FIG. 3, at operation 308, in response to the aforementioned image capture operation, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the image capture sensor 220, the communication interface 218, and / or the like, for receiving at least one image captured by the mobile device. The image may thus be captured by the mobile device 104 and transmitted to the device integrity verification apparatus 108 (e.g., via an app installed on the mobile device and / or a website of the device integrity verification apparatus 108). Additionally or alternatively, the image may be received locally by the mobile device 104 and further processed on the mobile device 104, as described below.

[0089] An identifier generated by the app or website is associated with the image to indicate whether a particular image was submitted as a front-of-device or back-of-device image. The identifier may be received at the mobile device 104 and / or by the device integrity verification unit 108 in association with the received image.

[0090] According to example embodiments, as indicated by operation 310, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for pre-processing the image. According to example embodiments, the received image may be cropped as described in further detail herein. According to example embodiments, the received image may be converted or reduced to a predefined size, such as, for example, 300 pixels by 300 pixels. According to certain example embodiments, some operations described herein may be performed using a single-shot detection algorithm, meaning that the complete image (which may be cropped and resized) is processed as described herein. However, in some embodiments, the image may be divided into sections for individual processing according to any of the operations described herein, and reconstructed such that example embodiments utilize respective data and / or predictions associated with the separate sections.

[0091] As indicated by operation 314, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for processing at least one image to determine an integrity status of the mobile device. Example operations for determining an integrity status of a mobile device, according to an example embodiment, are described below with respect to FIG. 4A, which spans two pages.

[0092] Determining the integrity state of the mobile device may include processing the image through a series of conditions implemented by respective algorithms and / or models. A prediction or outcome regarding the condition may indicate the integrity state of the mobile device. For example, the integrity state of the mobile device may include "verified," indicating that the mobile device identity has been confirmed and the mobile device is in an acceptable state for enrollment in a protection plan. A mobile device integrity state of "not verified" may indicate that the device has not yet been verified and / or any one or more conditions required for verification have not been satisfied.

[0093] According to some embodiments, an optional mobile device integrity status of "indeterminate" may indicate that the conditions that example embodiments have determined to be necessary for validation are likely to be satisfied, but further review should be performed before final validation. Accordingly, in an embodiment, the determination of the mobile device integrity status may be based on predictions made by various models and / or algorithms, and a confidence returned by any of the models and / or algorithms indicating the confidence of a prediction. In some embodiments, as described herein, the validation conditions may include detecting the presence and location of the mobile device, detecting occlusions in a mobile device image, and other related evaluations. Some embodiments may further evaluate whether the device is insured and / or uninsured.

[0094] For simplicity, the operations of FIG. 4A are described with respect to the processing of a single image, but it will be understood that the processing of the front and back images may occur simultaneously or in tandem, such that both the front and back of the device are considered in verifying the integrity of the device. According to some embodiments, only one image may need to be processed to verify the integrity of the device. In any event, an "image state" may relate to a prediction regarding one image (e.g., the front image, or the back image). An image state of "verified" may be required for one or more images (e.g., the front and / or back) for example embodiments to determine the integrity state of the mobile device as "verified". Such a determination is described in further detail below with respect to operations 440, 442, 446, 448, and 450.

[0095] According to some embodiments, determining whether a particular condition is satisfied may be implemented in a model(s) trained to make predictions regarding an image and / or other algorithms configured to determine the quality of an image. FIG. 4B provides an example hierarchy of model(s) that may be used to implement the operations of FIG. 4A according to an example embodiment. FIG. 4B illustrates the flow of data from one trained model to another trained model according to an example embodiment. Each model, configured on memory 214 and used and / or trained by an example embodiment, such as with processor 212, may, among others: A mobile device presence model 486 trained to detect whether a mobile device is present in an image; A location detection and cropping model 488 trained to detect the location of a mobile device and optionally crop an image; A cover detection model 490 trained to detect covers on a mobile device present in an image; A mobile device front / back discrimination model492, trained to determine whether an image reflects the front or back of a device; A mobile device authenticity model 494 trained to determine whether an image includes the mobile device from which the image was captured; an occlusion detection model 496 trained to generate a mask that is used to determine whether an object in an image is occluded; and A damage detection model trained to detect damage to mobile devices in images498 may include.

[0096] FIG. 4B reflects a hierarchy of models through which an image is fed in accordance with example embodiments. If a particular model predicts that an image does not satisfy a particular condition, example embodiments may prevent further processing by additional models. However, if a particular model predicts that an image satisfies its respective condition(s), the image may continue to be processed by the additional models illustrated in the tiered architecture of FIG. 4B. In this manner, efficiency of the system may be improved, enhanced, and / or maximized compared to a system that performs processing of all conditions regardless of other outcomes. It will be appreciated that the order of models through which an image flows or is processed may be arranged in a different or altered order than that illustrated in FIG. 4B. In some embodiments, any one or more models may be executed separately for its (their) intended purpose without requiring each step illustrated in FIG. 4B. In this regard, any of the models described herein and their respective predictions may be utilized and / or utilized for other purposes in addition to or in lieu of determining an image state and / or a mobile device integrity state.

[0097] Similarly, the order of operations and / or conditions described with respect to FIG. 4A may be altered. For example, operations identified as less resource intensive than others may be processed before those identified as more resource intensive. Additionally or alternatively, if a particular condition known to generally result in low confidence or accuracy of other predictions is not verified, then that particular condition may be intentionally configured to be processed prior to another condition. For example, if an example embodiment does not verify that an image includes a mobile device (operation 400, described below), it may not accurately determine whether the image is the front or back of a device (operation 406, described below).

[0098] Continuing with the description of FIG. 4A, as shown in operation 400, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the mobile device presence model 486, and / or the like, for determining whether at least one image includes a mobile device. The determination may be made, for example, using the mobile device presence model 486 deployed on the mobile device 104 and / or the device integrity verification apparatus 108. Although the user interface 216 prompts the user to use a mirror to capture an image of the user's device, the user may submit an image that does not include a mobile device. For example, some users may try to cheat by taking a photo of a paper model that looks like a mobile device. Other users may intentionally or inadvertently capture an image that does not include a mobile device.

[0099] The processor 212 may process the target image using at least one trained model (e.g., a neural network) trained on a plurality of training images, each labeled as either including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device.

[0100] In any event, if the example embodiment predicts, such as with the mobile device presence model 486, that at least one image does not include a mobile device, as shown in operation 430, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or the like, for providing feedback to the user indicating capturing an image of their device and / or determining the image status as “not verified.” The feedback may include causing one or more instructions to be transmitted to and / or displayed on the mobile device.

[0101] In this regard, the user may be given the opportunity to recapture the image for reprocessing and verification. A message such as that displayed in the user interface of FIG. 5R may be provided to the user. Although operation 430 illustrates optionally providing user feedback, it will be appreciated that, according to some embodiments, the user may be provided with more specific instructions (e.g., issues with the captured image) related to a particular condition that was processed but did not result in verification of device integrity as a result of certain or all of the conditional operations 400, 403, 405, 406, 410, 416, 420, 426 and / or 442. If the user provides new image(s), processing may return to operation 400 to process the newly captured image.

[0102] It will be appreciated that in an example embodiment, operation 400 may be performed in a single shot for each image, or the image may be subdivided into sections whereby each separate section may be processed as described herein.

[0103] If the example embodiment determines that at least one image includes a mobile device, further processing may continue to operation 403. At least some of the remaining operations of Figure 4A are described with reference to a mobile device in the image or a captured mobile device. It will be understood that such references refer to a processor-driven prediction that a mobile device is likely present in the image, and thus that the captured mobile device is a suspect mobile device.

[0104] At operation 403, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the location and cropping model 488, and / or the like, for determining the location of the mobile device within the image. In this regard, example embodiments may predict a bounding box, or sub-portion of the image within which the mobile device is located, such as with the location and cropping model 488 and / or its respective model. If the bounding box has a predetermined relationship (e.g., less than or equal to) compared to a threshold minimum percentage of the image (e.g., 25%), example embodiments may determine that the mobile device 104 was too far from a mirror or other reflective surface when the image was captured (e.g., too far away to provide additional processing and predictions regarding the mobile device with a threshold confidence). As such, the apparatus 200 may determine that the image state is "not verified," such as by operation 430, and may optionally provide feedback to the user, such as to indicate to hold the mobile device 104 closer to the mirror when recapturing the image.

[0105] If the bounding box is determined to have a different predetermined relationship (e.g., greater than or equal to) compared to a threshold minimum ratio of the image, example embodiments may determine that the mobile device 104 was close enough to the mirror when the image was captured (e.g., close enough to provide additional processing and prediction regarding the mobile device with a threshold confidence) and may thus continue processing.

[0106] At operation 404, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, the location detection and cropping model 488, and / or the like, to crop the image such that the area outside the bounding box is removed. The cropped image may then be reduced and resized to a predetermined size, such as 300 pixels by 300 pixels. To avoid overcomplicating the description, the cropped image may be referred to as the “image” or the “captured image” and processed as described further below, even though in some cases the image may differ from the originally captured image that may be cropped according to example embodiments.

[0107] As indicated by operation 405, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the cover detection model 490, and / or the like, to determine whether the captured mobile device in the image is free of or includes a cover(s), such as a cover that prevents an accurate assessment of the state of the mobile device 104. The determination may be made using the cover detection model 490 trained to detect a cover on the captured mobile device in the image. Example embodiments may process the image using a model (e.g., the cover detection model 490) trained to predict or detect whether the user captured the image with a cover on the mobile device. In response, at operation 430, example embodiments may provide feedback to the user to indicate to remove the cover and recapture the image, etc. Example embodiments may further determine the image state as "not verified." In this regard, the user may be given an opportunity to recapture the image for reprocessing.

[0108] If the example embodiment determines that the at least one image does not have a cover thereon, further processing may continue at operation 406. As indicated by operation 406, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for determining whether the at least one image includes a display surface of the mobile device. A "display surface" does not necessarily mean a user-displayed surface, but may mean a systematically-displayed display surface associated with a captured image, such as may be generated by an app due to a user being prompted separately to capture the front and back of the device.

[0109] The determination may be made, for example, using a mobile device front / back identification model 492 deployed on the mobile device 104 and / or device integrity verifier 108. Example embodiments may run the image through the model (e.g., mobile device front / back identification model 492) to verify that the image captures the indicated side (e.g., front or back). If it is determined that the user has captured the wrong side of the device, then at operation 430 example embodiments may provide feedback to the user, such as instructing them to capture the indicated side of the device (e.g., front or back). Example embodiments may further determine the image status as "not verified." In this regard, the user may be given an opportunity to recapture the image for reprocessing.

[0110] If the image is determined to reflect the indicated face of the device (e.g., front or back), processing may continue at operation 410. As indicated by operation 410, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, the mobile device authenticity model 494, and / or the like, to determine whether at least one image includes a mobile device associated with the mobile device identification data object. In other words, the example embodiment determines whether at least one image includes a mobile device from which the mobile device was captured. In some cases, a user may attempt to cheat by utilizing their mobile device and a mirror to capture an image of a different, intact phone. The example embodiment may utilize the mobile device authenticity model 494 to estimate the angle of the mobile device relative to the reflective surface using the image to predict whether the device present in the image is indeed the mobile device from which the image was captured or whether the captured device in the image is another device.

[0111] As another example, example embodiments may generate a prediction of an identity of a suspect mobile device in an image based on the image, such as with mobile device authenticity model 494. For example, mobile device authenticity model 494 may predict the make and / or model of the mobile device, and example embodiments may compare the predicted mobile device identity to an identity indicated by a mobile device identification data object (e.g., IMEI) to determine whether the image reflects characteristics of an expected device based on the mobile device identification data object.

[0112] If the mobile device in the image is determined to be different from the one from which the image was captured, feedback may optionally be provided to the user to capture an image of their mobile device using the same mobile device 104 from which the device integrity verification request was sent (e.g., the mobile device for which a protection plan is desired) in operation 430. Example embodiments may further determine the image status as "not verified."

[0113] If the example embodiment determines that the mobile device in the image is indeed the mobile device 104 from which the image was captured, processing may continue at operation 416. As indicated by operation 416, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, and / or the like, for determining whether the quality of at least one image is sufficient for further processing. According to an embodiment, the blurriness of the image may be determined by implementation of a Laplacian variance metric.

[0114] Due to various external factors of the environment and / or positioning of the mobile device 104 relative to the mirror, and / or the like, some images may be too blurry to be further processed to detect occlusion or damage (described in more detail below). Additionally or alternatively, images may be too blurry to make other predictions, including those described above, so it may be advantageous for example embodiments to evaluate image quality and blurriness prior to performing the operations described herein. In some examples, image quality may be sufficient to accomplish one task but not another, so various image quality validations may be performed through the process illustrated by FIG. 4A.

[0115] In any event, at operation 430, feedback may be provided to the user to recapture the image, and may include further guidance as to how to position the mobile device 104 relative to the mirror to capture an image having sufficient quality for further processing. FIG. 5S provides an example interface prompting the user to retake the photo because the photo is too blurry. Further instructions may be provided on how to adjust the angle or orientation of the mobile device 104 relative to the mirror, by moving the mobile device 104 closer to or further away from the mirror. Example embodiments may further determine the image status as "not verified," and the user may be given an opportunity to recapture the image for reprocessing.

[0116] If the example embodiment determines that the image quality is sufficient for processing, processing may continue at operation 420. As indicated by operation 420, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the occlusion detection server 109, the processor 212, the memory 214, the occlusion detection model 496, and / or the like, for determining whether at least one image is free of occlusion or includes an object occluding the mobile device 104 in at least one image. To avoid overcomplicating the flow diagram, operation 420 indicates that the image is either free of occlusion or has occlusion. However, it will be understood that the degree or amount of occlusion is determined and taken into account in determining whether the image state is set to "not verified" or "verified" as described herein.

[0117] For example, a user may inadvertently or intentionally cover a portion of the mobile device 104, such as a crack or other damage on the display screen, or other portion of the mobile device 104. Example embodiments may use the occlusion detection model 496 to generate a mask that is utilized in detecting occlusions, such as occluded corners (e.g., a finger covering a corner of the mobile device) and concave occlusions (e.g., a finger protruding into a portion of a captured mobile device), as described in further detail below.

[0118] While minor occlusions covering a bezel or outer portion of the surface of the mobile device 104 may be acceptable, major occlusions that obscure significant portions of the display screen or other significant portions of the device may not be acceptable. If the example embodiment determines that the mobile device 104 is obscured by an object and therefore the integrity of the device cannot be verified, the process may proceed to operation 430 to prompt the user to retake the image without the occlusion (e.g., by holding fingers only around the edges of the device and not covering the front or back of the device) and determine the mobile image status as "not verified." Further details regarding occlusion detection are provided below with respect to FIG. 6 in the section entitled "Occlusion Detection."

[0119] If no obstruction is detected or any such obstruction is minor so that the obstruction does not prevent further processing and analysis of damage or other conditions, the process may proceed to operation 426. As shown in operation 426, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, the damage detection model 498, and / or the like, to determine whether at least one image indicates that the device is free of damage or includes damage. Additionally or alternatively, the damage detection model 498 may determine or predict the presence of a particular type of damage, such as cracks, water damage, dents, and / or any other damage that would prevent the mobile device from being insured or protected. In an embodiment, if the model determines that damage is likely, a separate model may predict the type of damage. In any event, the example embodiment determines whether there is pre-existing damage to the mobile device 104 such that coverage for a protection plan should be denied. To avoid overcomplicating the flow chart, operation 426 indicates that the image is either undamaged or damaged, but it will be understood that the degree or amount of damage is taken into account in determining whether the image status is set to "unverified," "verified," or "indeterminate," as described herein.

[0120] Further details are provided below regarding the damage detection model 498, which utilizes training images and model(s) to detect damage to a mobile device. If damage is detected, example embodiments may provide a response to the user indicating that damage is detected and / or that a device protection plan cannot be issued, at operation 430. Example embodiments may further determine that the image status is "not verified."

[0121] In instances where it is determined that there is no damage to the mobile device 104 and / or that the physical condition and / or operability parameters of the mobile device 104 are sufficient for insurability purposes, processing may proceed to operation 440. As shown in operation 440, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, and / or the like, for determining the image status as "verified." It will be understood that certain operations illustrated in FIG. 4A may not be present in certain embodiments. Accordingly, the apparatus 200 may be configured to require any number of verifications and / or conditions described with respect to FIG. 4A, such that a "verified" image status may be determined when all desired (e.g., desired by the provider, etc.) verifications or conditions are performed.

[0122] In some embodiments, one or more models may run in parallel, such as occlusion detection model 496 and cover detection model 490. In some embodiments, the output of image cropping model 488 may be fed to one or more of cover detection model 490, mobile device front / back discrimination model 492, mobile device authenticity model 494, occlusion detection model 496, and / or damage detection model 498, simultaneously or in any order.

[0123] According to an example embodiment, a "verified" image status may be required for multiple images, for example a front image and a back image. Thus, in examples where both the front and back (and / or any other images) should be verified to confirm the integrity status of the mobile device, although not shown in Figure 4A to avoid overcomplicating the flow diagram, operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440 of Figure 4A may be repeated separately for each image requested by the insurer. For example, images directed to capture the front of the device may be processed according to operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440, and images directed to capture the back of the device may be processed according to operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440.

[0124] As such, as indicated by operation 442, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, and / or the like, for determining whether all required images (e.g., required for purposes of determining the integrity status of the mobile device) have an image status of “verified.” The particular images (e.g., front and back) may be pre-configured or set by the device integrity verification unit 108 and may relate to the provider's requirements for enrolling the device in a protection plan.

[0125] For example, if front and back images of a device are required and both images have an image status of "verified", the integrity status of the device may also be set to "verified". However, if both front and back images of a device are required and only one or neither image has an image status of "verified", the integrity status of the mobile device should be left null or set to "not verified" until at least both images have an image status of "verified". For example, as shown in operation 446, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, and / or the like, to determine the integrity status of the device to be "not verified". FIG. 5T provides an example user interface illustrating that the front photo is approved, but a back photo (e.g., a back photo) may still need to be captured and processed. If one or both images are "not verified", example embodiments may prompt the user to capture or recapture the respective image.

[0126] It will be appreciated that, as shown in operation 448, example embodiments may be configured to determine the integrity status of the mobile device as “verified” based on a first threshold confidence level (which may be configurable). For example, determining the integrity status of the mobile device as “verified” may require a “verified” status for all required images, as well as a minimum overall or average confidence level for all conditions evaluated. The first threshold confidence level test may thus be configured as an option, and may be configured in a variety of ways. For example, although not illustrated in FIG. 4A, in an embodiment, a threshold confidence level for a particular prediction (e.g., condition) may be created in association with any of the predictions made in operations 400, 403, 405, 406, 410, 416, 420, and / or 426. For example, some models may be configured to provide not only a prediction, but also a confidence level reflecting the confidence of the accurate prediction. Thus, a threshold confidence level may be required for each condition to be satisfied before proceeding to the next condition. According to an example embodiment, the average confidence for all conditions may need to be 95% or higher to set the integrity status of the mobile device to "verified." As another example, all conditions may need to have a confidence of 98% or higher to set the integrity status of the mobile device to "verified."

[0127] In any event, if all required images have an image status of “verified,” as indicated by operation 442, and a first threshold confidence level is satisfied, as indicated by operation 448, the integrity status of the mobile device may be determined to be “verified,” as indicated by operation 450.

[0128] In this regard, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like, for determining the device's integrity status as "verified." If the first threshold confidence is not implemented, in an embodiment, operation 448 may be omitted or bypassed and the verification that all images have an image status of "verified" in operation 442 may proceed to operation 450 and the mobile device's integrity status may be determined to be "verified."

[0129] According to some embodiments, if the integrity status of the mobile device is set to "Verified," the mobile device 104 may be automatically enrolled in the protection plan and confirmation may be provided to the user via the user interface 216. For example, FIG. 5Y provides confirmation that the device is insured. According to some examples, the automatic enrollment and confirmation may be provided in real-time, or near real-time, during the session in which coverage is requested and images are captured by the user.

[0130] Additionally or alternatively, in response to determining the mobile device integrity status as “verified,” the device may not necessarily be automatically enrolled in a protection plan, but rather may be forwarded to an internal user device 110 for internal review, such as by the mobile device 104 and / or the device integrity verification device 108. In such examples, embodiments may provide messages, such as those of FIGS. 5O, 5P, 5Q, and / or 5X, to indicate that the images have been submitted for review. Accordingly, if a provider wishes to further review images internally (e.g., and does not provide automatic enrollment) prior to enrolling any mobile device in a protection plan, example embodiments may nevertheless advantageously remove images predicted to be neither acceptable nor verifiable, and optionally provide feedback to the user to encourage efficient device enrollment.

[0131] As another example, as shown at operations 468 and 470, even if the first threshold confidence is not satisfied at 448, if the second threshold confidence is satisfied (e.g., 90%), example embodiments may determine the integrity status of the mobile device as “indeterminate” and indicate that further review should be performed, such as with the internal user equipment 110. Accordingly, example embodiments may be configured to “auto-register” devices determined to be low risk according to the process of FIG. 4A, but reserve the opportunity for the provider to further internally review the images before registering any mobile devices with associated images determined to be high risk. Still further, if the confidence does not satisfy either the first or second threshold confidence, the integrity status of the mobile device may be determined to be “not verified” (446) and any associated requests for insurance and / or the like may be denied without further manual review. Example embodiments may thus advantageously remove images predicted to be neither acceptable nor verifiable, and may optionally provide feedback to the user to encourage efficient device registration.

[0132] In any event, it will be understood that example embodiments may be configured to systematically perform any amount or all of the required set of checks and / or verifications, and that in some embodiments the level of systematic checks and / or verifications may be balanced with internal (e.g., human) review, for example, as desired by the provider. Various configurations and thresholds of confidence for various stages of processing may be contemplated.

[0133] Regardless of the variant(s) implemented above, certain example embodiments may provide additional user interface displays, examples of which are described below. In an embodiment, the user interface display of Figure 5N may be considered optional and may allow for entry, confirmation, or modification of device identification information, such as device IMEI 550. As previously mentioned, the IMEI may be detected and / or populated without explicit input by a user, and thus the user interface display of Figure 5N is optional. However, in an embodiment, processor 212 may receive a device identifier via a user interface, such as that of Figure 5N.

[0134] It will be further appreciated that certain updates and / or status may be provided to the user before, during, or after an operation related to determining the integrity status of the mobile device. For example, as shown in FIG. 5O, a pending review status message 556 may be displayed. As shown in FIG. 5P, the processor 212 may initiate a notification permission message 560, such as may be generated by the mobile device 104 in response to the example embodiment mobile application enabling or attempting to enable notifications on the mobile device 104. In this regard, the user may allow or deny the example embodiment mobile application from sending or pushing notifications. If notifications are allowed for the example embodiment mobile application, notifications may be provided to the user during various points in the processes and operations described herein.

[0135] In an embodiment, the user may access the mobile application of the example embodiment to view the status related to the request. For example, a review status message 564 may provide a status that the photo has been submitted and is currently under review. In an embodiment, if notifications for the mobile application of the example embodiment are enabled on the mobile device 104, a notification such as notification 570 of FIG. 5R may be displayed. Notification 570 indicates to the user that an image of the back of the mobile device needs to be retaken. The user may select the notification and access the mobile application to retake the image.

[0136] In response, an embodiment may provide feedback to a user when accessing a mobile application of an example embodiment, such as feedback summary 574 and reason 576 of Figure 5S. For example, feedback summary 574 indicates that an image of the back needs to be retaken, and reason 576 indicates that the image needs to be retaken because the photo is too blurry.

[0137] In an embodiment, as shown in Figure 5T, a message such as image approved message 580 may be displayed, such as in the area of ​​selectable prompt 518 of Figure 5J. As shown in Figure 5T, selectable prompt 520 may not be populated with a message because a back image has not yet been captured. In response, the user may select to capture an image, and the user interface displays of Figures 5U and 5V may be updated to provide viewfinder 538 and captured image 542, respectively.

[0138] In response, as shown in Figure 5W, an image approved message 580 is displayed for one image, such as an image of the front of the device, while the captured image 542 is displayed for another image, such as an image of the back of the device. In this regard, because the respective image has not yet been approved, a selectable prompt 520 is displayed to allow for recapture or editing of the captured image 142. In an embodiment, selecting the selectable prompt 520 may display a review status message 564, such as that of Figure 5X.

[0139] It will be appreciated that any of the user interface displays provided herein may be updated with a status, such as an image status, and / or the integrity status of the mobile device may be updated and / or populated as described with respect to FIG. 4A. In an embodiment, a mobile device integrity status, such as "not verified" or "indeterminate," may be presented to the user as a "pending" status, and a status, such as "verified," may be presented to the user as "verified." In response, the mobile device integrity status may be set to "indeterminate," but the associated image may be queued for manual review, and / or the like.

[0140] Still further, according to an embodiment, if notifications are allowed on the mobile device 104, the mobile application of the example embodiment may initiate a notification 590 if the integrity status of the mobile device is determined to be “verified,” so that the user may be notified that the image is approved and may or has enrolled in a protection plan for their mobile device.

[0141] Occlusion Detection 6 is a flow diagram of operations for detecting occlusion, such as by occlusion detection apparatus 109, according to an example embodiment. The operations of FIG. 6 may be triggered by operation 420, or in other examples, may be performed as a separate process not necessarily related to images of the mobile device. The operations of FIG. 6 may utilize or provide an image segmentation approach for detecting occlusion.

[0142] In operation 600, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, the occlusion detection model 496, and / or the like, for generating a mask that includes a reduced number of colors compared to the number of colors in the original image. According to example embodiments as described herein, the mask may be described with reference to the mobile device in the image, but it will be understood that the mask generation may be performed on any object detected or present in the original image. With respect to the example of mask generation for a mobile device (such as for purposes of determining an integrity state of the mobile device), the original image may be considered an image captured by the mobile device, and the generated mask may be considered a mobile device mask.

[0143] A mask may be considered an additional image generated from processing of an original image (e.g., an image of a mobile device captured by a mobile device, which may have been previously cropped according to the location detection and cropping model 488). A mask may be an image with a reduced number of colors compared to the original image. For example, FIGS. 7A and 8A are example images of a mobile device captured by a user and including a wide range of colors, and FIGS. 7B and 8B are masks generated according to example embodiments and including binary values ​​(represented as white and black pixels in FIGS. 7B and 8B, respectively). However, it will be understood that other configurations of colors may be selected to generate a mask. An example process for generating a mask is described in further detail below with respect to configuring, training, and deploying the occlusion detection model 496. It will be understood that separate occlusion detection models 496 and / or models thereof may be used for the front side of the device and the back side of the device.

[0144] According to an example embodiment, the model may return an array of values ​​indicating whether a particular pixel should belong to the mask or not. Example embodiments may then determine whether the pixel should be made white (e.g., included in the mask) or black (e.g., not included in the mask) based on a threshold value.

[0145] As indicated by operation 602, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for extracting a polygonal subregion P of the mask. Accordingly, example embodiments may apply an algorithm, such as a marching squares algorithm, to extract the maximum polygonal subregion P of the mask. In some embodiments, the maximum polygonal subregion P may be considered as the mobile device screen when the image cropping model 488 and other related models and pre-processing steps generated the original image in which the mobile device was detected and for the occlusion detection model 496, the image is substantially cropped before generating said mask. Accordingly, small islands (including, for example, small outlier polygons, black pixels appearing within a mostly white portion, such as those that may be caused by camera imperfections, dust / dirt, and / or other minor environmental factors present on the device or mirror, and / or the like) may be removed.

[0146] As indicated by operation 604, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for determining the convex hull of P. The convex hull may be calculated according to commonly known computational geometry algorithms.

[0147] Using the polygonal subregion P together with the convex hull of P enables example embodiments to identify concave occlusions, such as concave occlusion 700 of Figures 7A and 7B. Sub-processes for identifying concave occlusions are provided by operations 608, 612, 616, and 620. Additionally or alternatively, example embodiments may use the polygonal subregion P together with the convex hull of P to identify occluded corners, such as occluded corner 800 of Figures 8A and 8B. Sub-processes for identifying occluded corners are provided by operations 630, 634, 638, and 642. According to example embodiments, both sub-processes or one of the sub-processes may be implemented and performed.

[0148] As indicated by operation 608, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for calculating the difference between P and the convex hull. At operation 612, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for reducing or removing thin discrepancies at the edges of P and the convex hull. Example embodiments may reduce or remove the discrepancies by performing pixel erosion and pixel dilation techniques, such as may be provided by Shapely and / or other libraries. At operation 616, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for recalculating P as the maximum area of ​​the remaining regions. In this regard, P may be identified as the maximum area of ​​the remaining connected regions of P. P can thus be thought of as the estimated area of ​​the visible screen (eg, the portion of the screen that is not occluded).

[0149] At operation 620, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, and / or the like, for determining the concavity as the difference between P and the convex hull. In some examples, any such concavities may be compared to a threshold for further filtering, so that very small concavities are not necessarily flagged as such, but large concavities that may be obtrusive to other downstream tasks (such as determining whether damage is present on the device) may be flagged so that the user is prompted to recapture the image. For example, if a particular detected concavity is greater than, equal to, or larger than a predetermined threshold size, the region may remain predicted to be a concavity. If a region initially identified as a concavity is less than, equal to, or smaller than a predetermined threshold size, the region may be disregarded as a concavity. In this regard, if the example embodiment predicts that a concavity is present (and / or the concavity is large enough to be distracting to downstream tasks, as indicated by the use of a threshold), operation 420 may determine that the image contains an occlusion, causing the example embodiment to determine the image state as "not verified," and optionally prompt the user to recapture the image.

[0150] At operation 630, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for determining a predetermined number of dominant edges of the convex hull. For an example mobile device mask, an example embodiment may determine four dominant edges and, according to some embodiments, may identify the four most dominant edges. In this regard, the number of dominant edges identified may be based on the type of object for which the mask is created.

[0151] A Hough transform feature extraction technique may be implemented to identify the dominant edge or a predetermined number of the most dominant edges, and thus predict where the edges of the mobile device should be visible in the image (e.g., assuming that the edges are not occluded, even if they are occluded). As indicated by operation 634, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, and / or the like, for identifying the intersections of adjacent edges (identified based on their respective angles) to identify protruding corner points (which may be occluded or unoccluded) of the mobile device in the image. At operation 638, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification unit 108, the processor 212, the memory 214, and / or the like, for determining the distance from each protruding corner to P. For example, determining the distance may include measuring the shortest distance from the estimated corner to the nearest edge or point of P.

[0152] At operation 642, the apparatus 200 may include means, such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, for comparing each distance to a threshold to determine whether any corners are occluded. For example, if the distance is greater than, or greater than or equal to a predetermined threshold distance, example embodiments may determine that the respective corner is occluded. In this regard, when utilized to determine the integrity status of a mobile device, operation 420 may determine that the image includes an occlusion, causing example embodiments to determine the image status as "not verified" and optionally prompt the user to recapture the image. If no calculated distance is greater than, or greater than or equal to the predetermined threshold distance, example embodiments may determine that no corners of the object are occluded in the original image.

[0153] 6, in an example embodiment where an image is processed for both concave occlusions and occluded corners but neither is detected, operation 420 may determine, in accordance with the operations described above, that there is no occlusion of the mobile device in the image, i.e., no occlusion that may affect subsequent processing of the image, and thus small occlusions that do not affect subsequent processing may be tolerated.

[0154] Building, training, and deploying the model Training of the model(s) (e.g., neural network(s)) utilized by example embodiments may occur prior to deployment of the model (e.g., prior to use by the mobile device 104 and / or device integrity verifier 108 in determining whether the device is eligible for coverage under a plan, and / or the occlusion detection unit 109 in determining whether an object in an image is occluded). According to some example embodiments, training may be performed continuously by receiving images and associated classifications and / or labels that have been reviewed by either example embodiments and / or human reviewers. Machine learning may be used to develop specific pattern recognition algorithms (i.e., algorithms that represent specific parameter recognition problems), which may be based on statistical inference, and train the model(s) based thereon.

[0155] Example embodiments receive and store multiple types of data, including data sets, such as with the communications interface 218, memory 214, and / or the like, and use the data in multiple ways, such as with the processor 212. The device integrity verification unit 108 may receive the data sets from a computing device. The data sets may be stored in the memory 214 and utilized for a variety of purposes. The data sets may thus be used in modeling, machine learning, and artificial intelligence (AI). Machine learning and related artificial intelligence may be performed by the device integrity verification unit 108 based on a variety of modeling techniques.

[0156] For example, a set of clusters may be developed by unsupervised learning, in which the number of clusters and their respective sizes are based on a calculation of the similarity of features of patterns in a training set of previously collected patterns. In another example, a classifier representing a particular categorization problem or challenge may be developed using supervised learning based on the use of a training set of patterns and their respective known categorizations. Each training pattern is input to the classifier, and the difference between the output categorization produced by the classifier and the known categorization is used to adjust the classifier coefficients to more accurately represent the problem. Classifiers developed using supervised learning are also known as trainable classifiers.

[0157] In some embodiments, the dataset analysis includes a source-specific classifier that takes as input a source-specific representation of a dataset received from a particular source and generates an output that categorizes the input as likely to contain relevant data references or unlikely to contain relevant data references (e.g., likely or unlikely to satisfy a required criterion). In some embodiments, the source-specific classifier is a trainable classifier that can be optimized as more instances of the dataset are received from a particular source for analysis.

[0158] Alternatively or additionally, the trained model may be trained to extract one or more features from the historical data using pattern recognition based on solutions for unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, association rule learning, Bayesian learning, probabilistic graphical models, among other computational intelligence algorithms that may use an iterative process to extract patterns from data. In some examples, the historical data may include data that has been generated using user input, cloud-based input, or the like (e.g., user verification).

[0159] The model(s) may be initialized at multiple nodes. In some embodiments, an existing deep learning framework may be used to initialize the model(s). The model(s) may be implemented as a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) network, and / or the like. According to certain example embodiments, any of the models described herein may utilize existing or pre-trained models and further train such models with training data specific to its respective task and / or condition described herein. For example, the device integrity verifier 108 may develop templates, such as using any known or modified machine learning templating technique. In this regard, a templating model for each respective task and / or condition described herein may be utilized by example embodiments to further train the respective model. In this regard, example embodiments may utilize templates with domain-specific data schemes and models. For example, a template design for identifying certain textures in an image may be utilized for cover detection and / or damage prediction. A template for identifying whether a particular object is present in an image may be utilized for mobile device presence detection.

[0160] According to certain embodiments, CNNs and related deep learning algorithms may be particularly useful in machine learning applications to image processing by creating multiple connected layers of perceptrons. Each layer may be connected to its neighboring layers, which provides an efficient basis for scaling the weights of a loss function to identify patterns in the data. In response, the machine learning algorithms of example embodiments may train related models, such as CNNs, to learn which features are important in images, efficiently resulting in accurate predictions about images.

[0161] Using the techniques described herein, the model may then be trained to determine one or more features of the image to generate one or more predictions associated with the methods and embodiments described herein. The training data may also be selected from a predefined period, such as days, weeks, or months prior to today.

[0162] In example embodiments, labeled datasets, such as those associated with a particular task or prediction described herein, may be fed into the device integrity verification unit 108 to train a model(s). The model(s) may then be trained to identify and classify subsequently received images received from the computing device corresponding to one or more of the labeled criteria.

[0163] In some embodiments, the AI ​​and models described herein use deep learning modules. Deep learning is a subset of machine learning that generates models based on training datasets provided to it. Deep learning networks can be used to pull in large amounts of inputs and have the algorithm learn which inputs are relevant. In some embodiments, the training models may use unsupervised learning techniques including clustering, anomaly detection, and Hebbian learning, as well as learning latent variable models such as expectation maximization algorithms, moment methods (mean, covariance), and blind signal separation techniques including principal component analysis, independent component analysis, non-negative matrix factorization, and singular value decomposition.

[0164] Accordingly, example embodiments may use the multiple training images and corresponding labels to train a model(s) or input the initialized models for further training, and use the processor 212 to learn features via supervised or unsupervised deep learning.

[0165] In this regard, training images, some of which include a captured mobile device and others that do not, are input to each model along with associated labels that indicate various characteristics depending on the particular model being trained. The training set may include hundreds or thousands of images that have been reviewed and labeled by a user or data scientist (e.g., using internal user equipment 110, etc.).

[0166] The model may convert the image into a matrix representation of the image and process the image along with its confirmed labels (e.g., "contains mobile device", "does not contain mobile device") to learn image features that correlate with the labels. In an example, an image may have multiple labels for each state, so that one image may be used to train multiple different models and / or neural networks. For example, one image may be labeled "contains mobile device", "contains cover", and "contains damage", so that one image is used by the processor 212 of an example embodiment to train three separate models, such as a mobile device presence model, a cover detection model, and a damage detection model, respectively. In an example, an image may be used to train one model. In this regard, training data may be collected and used in a variety of ways.

[0167] The processor 212 may train the model(s) on the training images and adjust their respective parameters to reconstruct a matrix representation of the image through a series of deep learning iterations to capture those features or give more importance to those features that are strong indicators of a particular label or classification. Various techniques may be utilized during training of the models, such as, but not limited to, fractal dimension, which is a statistical analysis that may be employed by a machine learning algorithm to detect which features, at what scale, are stronger indicators of certain predictions and / or conditions, such as those described herein. The scaling of the training images may be adjusted according to the fractal dimension technique, which may vary depending on the particular task or prediction to be made. For example, detection of damage, such as water damage and / or cracks, using a machine learning algorithm may require higher resolution images than may be required for detection of whether a cover is on the device. In this regard, a fractal dimension algorithm may be utilized to adjust the resolution of the images to balance the accuracy and efficiency of one or each model.

[0168] Additional details regarding the configuration, training, and deployment of each model(s) for their respective tasks, conditions, and / or methods associated with example embodiments are described below. It will be further appreciated that some models may employ other classification techniques instead of, or in addition to, neural networks, such as, but not limited to, support vector machines, decision trees, random forests, naive Bayes classifiers, and logistic regression.

[0169] It will be further appreciated that, although the processor 212 of an embodiment may advantageously use separate models to make separate predictions for the various conditions described herein, an embodiment may utilize one model (e.g., a neural network) to assess the integrity status of the mobile device as “verified”, “unverified”, and / or “uncertain”. In this regard, the model (e.g., a neural network) may be trained using images labeled “verified”, “unverified”, and / or “uncertain”, and the model may essentially evaluate which images include the mobile device, which are front or back, and which include or are free of damage to determine which images should be predicted as “verified”, “unverified”, and / or “uncertain”. However, the use of a single model may require more training data to provide accurate or meaningful results than utilizing separate models for at least some of the conditions described herein. At least one additional advantage of using separate models and optionally generating respective confidences includes enabling the provision of specific feedback to the user capturing the image, such as "move your device closer to the mirror," "remove the cover of your device," "place your finger on the edge of your device and retake the photo," and / or the like. Such feedback can result in improved or enhanced automated verification rates while reducing uncertainty or the need for manual review.

[0170] Mobile Device Presence Model The mobile device presence model 486 enables example embodiments to automatically predict (e.g., without human review) whether a newly received image contains a mobile device or does not contain a mobile device. The processor 212 of example embodiments, such as the apparatus 200, may utilize an existing model, such as Squeezenet's Torchvision implementation, and pre-train the model using weights established by a visual database, such as ImageNet. Example embodiments may further train the model for a mobile device presence detection task by inputting model training images and corresponding labels, such as "contains device" and "does not contain device". In this regard, the model (e.g., a neural network) may be trained with at least two sets of images, such as a first set of training images that contain a mobile device and a second set of training images that do not contain a mobile device.

[0171] Various deep learning methods may then be used to process the training images and corresponding labels through a model to train the model to generate predictions for subsequently received images, according to example embodiments. Once deployed, according to example embodiments, the trained mobile device presence model 486 may generate an indication of the likelihood of an image containing a mobile device. For example, the indication may be a number between 0 and 1, with a number closer to 1 indicating a higher likelihood that a mobile device is present. As such, the indication may reflect the confidence of the prediction.

[0172] Accordingly, the processor 212 of the example embodiment may process the target image using at least one model trained on a plurality of training images, each labeled as either including or excluding a mobile device, to determine whether the target image includes a mobile device.

[0173] In the context of determining the integrity state or image state of a mobile device, example embodiments may utilize the trained mobile device presence model 486 in performing operations 400. Example embodiments may further implement configurable or predetermined quantifiable requirements to indicate a degree of confidence that a prediction must be satisfied to be accepted (e.g., not require further internal review).

[0174] Position detection and trimming model The location detection and cropping model 488 enables example embodiments to predict where a particular object, such as a mobile device, is in the image and to determine whether the object was too far from the image capture sensor (220) or, in the case of an example of a reflective surface, too far from a mirror when the image was captured. The location detection and cropping model 488 also enables cropping of the image accordingly. Example embodiments may utilize existing frameworks to further train the pre-trained model. For example, according to example embodiments, Tensorflow's object detection framework may be utilized to train a network with a Mobilenet backend pre-trained on the COCO (Collaborative Computing) dataset.

[0175] Training images in which reviewers locate the contours of mobile devices present in the images, along with their respective labels of the located contours, may be input to the model for further training. Thus, the model may be trained to predict a bounding box defined as (xmin, xmax, ymin, ymax) for an image in which an object, such as a mobile device, is likely to be located. The model may be further trained to generate an indicator, such as a number between 0 and 1, indicating that the bounding box is likely to contain a mobile device. For example, a number closer to 1 may indicate that the bounding box is more likely to contain a mobile device than a number closer to 0. Thus, the output of the location detection and cropping model 488 may indicate a confidence that the bounding box accurately captures the location of the object, such as a mobile device, in the image.

[0176] Accordingly, as described with respect to operation 403, the position detection and cropping model 488, once deployed, may make predictions regarding the proximity of the mobile device to the mirror at the time the image was captured, allowing feedback to be optionally provided to the user.

[0177] According to an embodiment, a careful edge detection algorithm may also be used to estimate a bounding box of the mobile device in the image. For example, careful edge detection may utilize a Gaussian filter to smooth the image and determine the intensity gradient to predict the strongest edges in the image. The bounding box may then be predicted accordingly.

[0178] The processor 212 of the example embodiment may process the target image using at least one model trained on a plurality of training images, each associated with a bounding box indicating the location of the mobile device in the image, to determine the location of the mobile device in the target image. The example embodiment may further determine a confidence level for pinpointing the location of the mobile device in the image.

[0179] In this regard, if the confidence does not satisfy a threshold, the image may remain "unverified" and thus may be subject to further manual review. Additionally or alternatively, if a certain threshold confidence is satisfied, as described with respect to operation 404, the image may be cropped according to a predicted bounding box. It will be appreciated that in some example embodiments, satisfaction of a threshold confidence may be optional. In embodiments not utilizing a threshold confidence, any or all images for which a bounding box may be calculated may be cropped accordingly.

[0180] Cover Detection Model The cover detection model 490 enables example embodiments to predict whether a user has captured an image of a mobile device with a cover attached. An existing model may be utilized to further train, such as with images and respective labels indicating whether the images include a mobile device with a cover attached thereon (e.g., a mobile device with a case). Example embodiments may utilize existing image processing frameworks to further train the model with training images and labels. Thus, example embodiments may train the model using the processor 212 to give more importance to certain features, such as those related to texture, that are strong indicators of whether a mobile device in a captured image has a cover thereon. In this regard, texture may be determined to be a strong indicator, learned by the model, of whether a mobile device in an image includes a cover. The processor 212 of example embodiments may therefore process a target image of a mobile device using at least one model trained with a plurality of training images of mobile devices labeled as including a cover on the respective mobile device or excluding a cover on the respective mobile device, to determine whether the target image includes a cover on the target mobile device. In response, the deployed cover detection model 490 may enable the processor 212 of the device 200 to provide predictions related to the newly captured image and, optionally, provide feedback to the user to remove the cover and recapture the image, as provided in operation 405.

[0181] Mobile device front / back identification model The mobile device front / back discrimination model 492 enables example embodiments to predict whether a user is capturing the front or back of their device. Existing models may be utilized and further trained, such as with images and respective labels indicating whether the images provide a view of the front of the device or the back of the device. Example embodiments may utilize existing image processing frameworks to further train the model with training images and labels. Thus, example embodiments may train the model to identify important features that may be specific to one side of the device. For example, a deformation in pixels associated with a display screen surrounded by a bezel may indicate the front of the device.

[0182] In this regard, processor 212 of apparatus 200 may process a target image of a mobile device using at least one trained model, trained on a plurality of training images of a mobile device, where each training image is labeled as including either the front side of the respective mobile device or the rear side of the respective mobile device, to determine whether the target image includes the front side or the rear side of the target mobile device.

[0183] In response, mobile device front / back discrimination model 490 may provide a prediction of whether the user is correctly capturing the front and / or back of the mobile device in association with a newly captured image, and may optionally provide feedback to the user to capture a pointing surface of the device, as provided in operation 406. Additionally or alternatively, example embodiments may determine whether a particular image is an image of the front or back of the device based on data identifying whether the image was captured with a front or back camera (e.g., whether front or back image capture 220 was used to capture the image).

[0184] According to some example embodiments, the cover detection model 490 and the mobile device front / back identification model 492 may be implemented as a single model. For example, since it may be advantageous to reject or remove any images that contain a cover, training images input to the model may include the labels "cover," "front," or "back," and thus the "cover" label should be used for any training images that contain a cover. Thus, example embodiments may reject or set the image status to "not verified" in any scenario where an image is predicted to contain a cover.

[0185] Mobile Device Authenticity Model The mobile device authenticity model 494 enables example embodiments to predict whether an image contains the same mobile device as the image was captured. Existing models can be utilized and further trained, such as by the processor 212, with images and respective labels indicating whether the image contains the captured mobile device (e.g., "same device") or another device (e.g., "different device"). For example, an internal user or data scientist may use a mobile device to capture images of both the mobile device capturing the image and the other device and label the images accordingly. Example embodiments may thus train the model with the training images and labels such that example embodiments learn to detect and measure edges of the mobile device in the image and predict or estimate the angle at which the mobile device is held relative to the mirror.

[0186] As such, the processor 212 of an example embodiment may process a target image of a mobile device using at least one model trained on a plurality of training images of mobile devices, where each training image is labeled as being captured by the respective mobile device contained in the image or by a device different from the respective mobile device contained in the image, to determine whether a target mobile device contained in the target image was captured by the target mobile device or by a different device.

[0187] In some embodiments, this process of determining the authenticity of the mobile device may further utilize bounding boxes drawn by the user during labeling of the training images. In any case, based on the angle, a further prediction may be made indicating whether the mobile device in the image is the same device with which the image was captured.

[0188] Occlusion detection model As introduced above, a mask, such as a mask used to detect occlusion, may be generated by a trained model. In this regard, example embodiments may utilize existing models, such as the UNet architecture, to train a model from scratch with a manually created mask. In this regard, a data scientist or other internal user may review an image and manually trace or input a contour in the form of a mask (e.g., a contour reflecting the example masks of FIGS. 7B and 8B) that includes exposed areas of the object of interest (e.g., a mobile device) but does not include occluding objects (e.g., fingers) and / or objects visible in the background. In this regard, each pixel of a training image (which may be reduced and / or cropped to a predetermined size, such as 300 pixels by 300 pixels) may be labeled with an associated indication as to whether each pixel belongs to the mask or not. A model may then be trained on the images and the labels.

[0189] Accordingly, the deployed and trained model may take an input image (e.g., a scaled and cropped image) and process the image using the processor 212 to provide a prediction of the mask in the form of an array of a predetermined size (e.g., 300×300) of indicators indicating whether each pixel belongs to the mask or not. For example, the array may include numerical values ​​ranging from 0 to 1, with values ​​closer to 1 corresponding to pixels that are more likely to belong to the mobile device (and thus be considered to be included in the mask) compared to values ​​closer to 0. Accordingly, the mask predicted by the model may be utilized by example embodiments, such as those described with respect to FIG. 6, to determine whether an occlusion of the mobile device 104 is present in the image. According to an example embodiment, one model and training set may be used for masks of images of the front of the device, and a separate model and training set may be used for masks of images of the back of the device.

[0190] Using a model (e.g., a neural network) to determine the mask can provide advantages that may not be offered by other image processing techniques. Models are useful in contextual predictions that are often made by humans and that traditional computer algorithms cannot. For example, colors within a screen are not uniform and some of the same color may appear within pixels outside the device in an image. While models such as neural networks can make such distinctions, traditional color detection or other image processing algorithms may not accurately distinguish between pixels that are far from the image and pixels that are not far from the image that otherwise have the same or similar color.

[0191] Damage Detection Model The damage detection model 498 enables example embodiments to predict whether a captured mobile device in an image is damaged (e.g., damaged to such an extent that the integrity status of the mobile device should be set to "not verified," such as because the damaged device is not eligible for coverage in a device protection plan). The existing model may be utilized to further train by the processor 212, such as with images and respective labels indicating whether the image contains a damaged mobile device, such as, but not limited to, cracks, water damage, dents, scratches, and / or any other damage that would prevent the mobile device from being insured or protected. According to example embodiments, special user interface tools may be utilized by a reviewer or data scientist to zoom in on the images of the mobile device to determine whether cracks or other damage are present and label the training images accordingly. In an embodiment, a binary label, such as "damaged" or "not damaged," may be applied to the training images. As another example, a reviewer may score the level of damage, such that small or seemingly inconsequential cracks may receive a relatively low score compared to images showing more significant cracks that may affect functionality. Any variation or scoring of damage labeling may be contemplated. As yet another example, certain damage may be specifically labeled as a "crack," "water damage," or any other type of damage that may affect the insurability of the mobile device.

[0192] In an embodiment, a first label of a training image of a mobile device may indicate "damaged" and a second label may indicate a type of damage. In this regard, one model may be trained to predict whether damage is present or absent. A separate model and / or model(s) may predict a particular type of identified damage, such as a crack, water damage, or a dent, if damage is predicted to be present. For example, one model may be trained only to detect water damage based on training images of water damage, and the same logic may be applied to other types of damage and any other visually detectable condition of the device.

[0193] Example embodiments may utilize existing image processing frameworks and further train the model using training images and labels, so example embodiments may place more importance on certain features, such as those related to changes in texture and / or color, that are strong indicators of damage and / or specific types of damage.

[0194] In this regard, the apparatus 200 may include means, such as the processor 212, to process the target image with at least one model trained on a plurality of training images of the mobile device, where each training image is labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image. In this regard, the damage assessment may include "no damage," "mild damage," "extensive damage," and / or the like. In an embodiment, the damage assessment may include a quantifiable assessment, such as on a scale of 1 to 10, where 1 indicates no damage detected and 10 indicates extensive or severe damage. The deployed damage detection model(s) 498 may thus provide a prediction regarding whether the device is damaged, the extent of the damage, and / or the type of damage.

[0195] In a scenario where damage is detected, and optionally depending on the type and / or extent of the damage, the image status may be "not verified", resulting in a mobile device integrity status of "not verified" or "indeterminate." As another example, a quantitative damage score may be generated.

[0196] Additionally or alternatively, it will be appreciated that separate damage detection models may be constructed, trained and deployed for each of the front, back and / or bezel of a mobile device, The bezel may be recognized as part of the device such that damage to the bezel may also be detected.

[0197] conclusion As described herein, example embodiments of the present disclosure provide technical advantages over alternative implementations: example embodiments can be implemented to consume fewer processing resources that might otherwise be wasted submitting every image captured by a user to a server for storage, potential review, and further processing.

[0198] In this regard, certain operations, such as any of those of Figures 3, 4A, and / or 6, may be performed on the mobile device 104 while other operations may be performed on the device integrity unit 108 and / or occlusion detection unit 109. As such, example embodiments may provide resource efficiency by strategically balancing such operations. For example, some initial image processing operations may be performed on the mobile device 104 before the image is sent to the device integrity unit 108 and / or occlusion detection unit 109. As such, some images may be filtered out on the mobile device 104 before being processed by other, potentially more resource intensive processes.

[0199] For example, some embodiments may employ models, such as neural networks, configured to run on a mobile device, such as mobile device 104. For example, TensorFlow Lite and / or other frameworks designed to be deployed on mobile devices may be utilized in accordance with example embodiments.

[0200] So, for example, example embodiments may provide real-time on-device verification in certain scenarios, such as when a high confidence image state or mobile device integrity state is determined. If not, an image and / or device integrity verification request may be sent to the device integrity device 108 for review by an agent. On the server side, similar algorithms implemented on the device side may be used to facilitate image review and / or process input from the review as a means of further calibrating algorithms and / or training models.

[0201] 3, 4A, and 6 each illustrate a flow diagram of a system, method, and computer program product according to some example embodiments. It will be understood that each block of the flow diagram, and combinations of blocks in the flow diagram, may be implemented by various means, such as hardware and / or a computer program product including one or more computer readable media having computer readable program instructions stored thereon. For example, one or more of the procedures described herein may be embodied by computer program instructions of a computer program product. In this regard, a computer program product(s) embodying a procedure described herein may include one or more memory devices (e.g., memory 214) of a computing device storing instructions executable by a processor (e.g., by processor 212) in the computing device. In some example embodiments, computer program instructions of a computer program product(s) embodying the aforementioned procedures may be stored by memory devices of multiple computing devices. As will be appreciated, any such computer program product may be loaded onto a computer or other programmable apparatus (e.g., the mobile device support apparatus 102, the mobile device 104, and / or other apparatus) to create a machine, such that the computer program product including instructions for executing on the computer or other programmable apparatus creates means for implementing the functions specified in the flowchart block(s). Additionally, the computer program product may include one or more computer readable memories on which computer program instructions may be stored such that the one or more computer readable memories can direct the computer or other programmable apparatus to function in a particular manner, such that the computer program product may include a product that implements the functions specified in the flowchart block(s).Computer program instructions of one or more computer program products may also be loaded onto a computer or other programmable apparatus (e.g., the mobile device 104 and / or other apparatus) and a series of operations executed on the computer or other programmable apparatus to generate a computer-implemented process such that the instructions executing on the computer or other programmable apparatus implement the functions specified in the flowchart block(s).

[0202] Accordingly, the blocks of the flow charts support a combination of means for performing the specified functions and a combination of operations for performing the specified functions. It will also be understood that one or more blocks of the flow charts, and combinations of blocks within the flow charts, can be implemented by a dedicated hardware-based computer system that performs the specified functions, or a combination of dedicated hardware and computer instructions.

[0203] Many embodiments of the subject matter described may include all, parts, or combinations of parts of the systems, apparatus, methods, and / or computer program products described herein. The subject matter described herein includes, but is not limited to, the following specific embodiments: 1. A method comprising: receiving a device integrity verification request associated with the mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; Receiving at least one image captured by one or more image sensors of a mobile device; and Processing at least one image with the at least one trained model to determine an integrity state of the mobile device. Includes.

[0204] 2. The method of embodiment 1, wherein processing at least one image to determine an integrity status of the mobile device includes: utilizing at least one trained model to determine whether there is damage to the mobile device; and In response to determining that there is damage to the mobile device, determining that an integrity state of the mobile device is not verified. Includes.

[0205] 3. The method of embodiment 1, wherein processing at least one image to determine an integrity status of the mobile device includes: determining an angle of the mobile device relative to the reflective surface at the time the at least one image was captured; and determining, based on the angle, that at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object; Includes.

[0206] 4. The method of embodiment 1, wherein processing at least one image to determine an integrity status of the mobile device includes: Determining whether at least one image includes a mobile device associated with a mobile device identification data object. Includes.

[0207] 5. The method of embodiment 4, wherein determining whether at least one image includes a mobile device comprises: Identifying a suspect mobile device within at least one image; generating a prediction of an identification of the at least one suspect mobile device and comparing the mobile device identification data object to the prediction of the identification of the at least one suspect mobile device to determine whether the suspect mobile device is the mobile device; and If the suspect mobile device is determined to be the mobile device, determining that the integrity state of the mobile device is verified. Includes.

[0208] 6. The method of embodiment 1, wherein the integrity state of the mobile device is determined to be indeterminate, the method further comprising: Transmitting a device integrity verification request and at least one image to an internal user device for internal review. Further includes:

[0209] 7. The method of embodiment 3, comprising: In response to determining that at least one of the images captures a different mobile device based on the angle, (a) causing a message to be displayed on the mobile device instructing the user to recapture the mobile device; and (b) determining that the integrity status of the mobile device has not been verified; Further includes:

[0210] 8. The method of embodiment 1, wherein processing at least one image to determine an integrity status of the mobile device includes: determining a location in at least one image of the mobile device, the location being defined as a bounding box; and If the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, displaying a message on the mobile device instructing the user to move the mobile device closer to the reflective surface. Includes.

[0211] 9. The method of embodiment 8, comprising: cropping the at least one image according to the bounding box if the bounding box has a second predetermined relationship with the threshold proportion of the at least one image. Further includes:

[0212] 10. The method of embodiment 1, wherein processing at least one image to determine an integrity status of the mobile device includes: determining, using the at least one trained model, that an object is occluding the mobile device in the at least one image; and Displaying a prompt on a mobile device to capture the unoccluded image. Includes.

[0213] 11. The method of embodiment 10, wherein determining whether an occlusion of the mobile device occurs within at least one image includes: determining whether a concave occlusion is present in at least one image; and Determine whether the occluded corner is present in at least one image. Includes.

[0214] 12. The method of embodiment 10, wherein determining whether a concave occlusion is present in at least one image comprises: generating a mobile device mask using the at least one trained model, the mobile device mask including a reduced number of colors compared to the at least one image; Extracting a polygonal subregion P of a mobile device mask; Determining the convex hull of P; Computing the difference between P and the convex hull; removing or reducing thin discrepancies between P and at least one edge of the convex hull; Identifying the largest area of ​​the remaining region of P; and Comparing the maximum area to a threshold to determine whether at least one image includes a concave occlusion. Includes.

[0215] 13. The method of embodiment 10, wherein determining whether there is an occluded corner in at least one image comprises: generating a mobile device mask using the at least one trained model, the mobile device mask including a reduced number of colors compared to the at least one image; Extracting a polygonal subregion P of a mobile device mask; Determining the convex hull of P; Identifying the four principal edges of the convex hull, determining the intersection of adjacent major edges to identify corners; Determine the distance from each corner to P, and Comparing each distance to a distance threshold to determine whether any corners are occluded in at least one of the images. Includes.

[0216] 14. The method of embodiment 1, wherein processing at least one image to determine an integrity status of the mobile device includes: Using the at least one trained model, determining whether at least one image includes a front side of the mobile device, a back side of the mobile device, or a cover. Includes.

[0217] 15. The method of embodiment 1, comprising: The method further includes providing a response for display on the mobile device in real time or near real time in response to receiving the at least one image, the response provided depending on the determined integrity state of the mobile device.

[0218] 16. The method of embodiment 1, Displaying a test pattern on a mobile device configured to provide improved accuracy in predicting a characteristic of at least one image captured while the mobile device is displaying the test pattern as compared to accuracy in predicting a characteristic of at least one image captured while the mobile device is displaying another display pattern. Further includes.

[0219] 17. The method of embodiment 1, comprising: identifying a subset of conditions that must be satisfied to determine an integrity state of the mobile device as verified; setting an image status of the particular image as verified if all conditions in the subset of conditions are satisfied in the particular image; and Determining the integrity status of the mobile device as verified if each image status for all required images is verified. Further includes.

[0220] 18. The method of embodiment 17, wherein at least one condition of the subset of conditions to be satisfied is performed on a mobile device. 19. The method of embodiment 1, wherein receiving the at least one image includes receiving at least two images captured by a mobile device, a first image of the at least two images being of a front side of the device and a second image of the at least two images being of a back side of the device, and processing the at least one image to determine an integrity status of the mobile device includes: processing both the first image and the second image with the at least one trained model; and determining that the determined integrity state of the mobile device is verified if processing of both images results in a verified image state for each; Includes.

[0221] 20. The method of embodiment 1, Training at least one trained model by inputting training images and respective labels that describe the properties of each training image. Further includes.

[0222] 21. The method of embodiment 1, wherein at least one trained model is a neural network. 22. A method for detecting concave occlusions in an image, the method comprising: generating a mask using at least one trained model that includes a reduced number of colors compared to the image; Extracting a polygonal subregion P of the mask; Determining the convex hull of P; Computing the difference between P and the convex hull; removing or reducing thin discrepancies between P and at least one edge of the convex hull; Recalculating P as the maximum area of ​​the remaining regions, and Determine the concavity as the difference between P and the convex hull Includes.

[0223] 23. A method for detecting occluded corners of an object in an image, the method comprising: generating a mask using at least one trained model that includes a reduced number of colors compared to the image; Extracting a polygonal subregion P of the mask; Determining the convex hull of P; identifying a predetermined number of dominant edges of the convex hull; determining the intersection of adjacent major edges to identify corners; Determine the distance from each corner to P, and Comparing each distance to a distance threshold to determine whether any corners are occluded in the image. Includes.

[0224] 24. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least: receiving a device integrity verification request associated with the mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying a prompt on the mobile device to cause the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; receiving at least one image captured by one or more image sensors of the mobile device; Processing at least one image with the at least one trained model to determine an integrity state of the mobile device. The method is configured to cause the

[0225] 25. The apparatus of embodiment 24, wherein processing at least one image to determine an integrity status of the mobile device includes: utilizing at least one trained model to determine whether there is damage to the mobile device; and In response to determining that there is damage to the mobile device, determining that an integrity state of the mobile device is not verified. Includes.

[0226] 26. The apparatus of embodiment 24, wherein processing at least one image to determine an integrity status of the mobile device includes: determining an angle of the mobile device relative to the reflective surface at the time the at least one image was captured; and determining, based on the angle, that at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object; Includes.

[0227] 27. The apparatus of embodiment 24, wherein processing at least one image to determine an integrity status of the mobile device includes: Determining whether at least one image includes a mobile device associated with a mobile device identification data object. Includes.

[0228] 28. The apparatus of embodiment 27, wherein determining whether at least one image includes a mobile device comprises: Identifying a suspect mobile device within at least one image; generating a prediction of an identification of the at least one suspect mobile device and comparing the mobile device identification data object to the prediction of the identification of the at least one suspect mobile device to determine whether the suspect mobile device is the mobile device; and If the suspect mobile device is determined to be the mobile device, determining that the integrity status of the mobile device is verified. Includes.

[0229] 29. The apparatus of embodiment 24, wherein the integrity state of the mobile device is determined to be indeterminate, and the at least one memory and computer program code, using a processor, cause the apparatus to at least: Transmitting a device integrity verification request and at least one image to an internal user device for internal review. The device is further configured to cause the

[0230] 30. The apparatus of embodiment 26, wherein the at least one memory and the computer program code, using a processor, cause the apparatus to at least: In response to determining that at least one of the images captures a different mobile device based on the angle, (a) causing a message to be displayed on the mobile device instructing the user to recapture the mobile device; and (b) determining that the integrity state of the mobile device is not verified; The device is further configured to cause the

[0231] 31. The apparatus of embodiment 24, wherein processing at least one image to determine an integrity status of the mobile device includes: determining a location within at least one image of the mobile device, the location being defined as a bounding box; and If the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, displaying a message on the mobile device instructing the user to move the mobile device closer to the reflective surface. Includes.

[0232] 32. The device of embodiment 31, wherein the at least one memory and the computer program code, using a processor, cause the device to at least: and causing the at least one image to be cropped according to the bounding box if the bounding box has a second predetermined relationship with a threshold ratio of the at least one image. Further configured.

[0233] 33. The apparatus of embodiment 24, wherein processing at least one image to determine an integrity status of the mobile device includes: determining, using the at least one trained model, that an object is occluding the mobile device in the at least one image; and Displaying a prompt on a mobile device to capture the unoccluded image. Includes.

[0234] 34. The apparatus of embodiment 33, wherein determining whether an occlusion of the mobile device occurs within at least one image comprises: determining whether a concave occlusion is present in at least one image; and Determine whether the occluded corner is present in at least one image. Includes.

[0235] 35. The apparatus of embodiment 33, wherein determining whether a concave occlusion is present in at least one image comprises: generating a mobile device mask using the at least one trained model, the mobile device mask including a reduced number of colors compared to the at least one image; Extracting a polygonal subregion P of a mobile device mask; Determining the convex hull of P; Computing the difference between P and the convex hull; removing or reducing thin discrepancies between P and at least one edge of the convex hull; Identifying the largest area of ​​the remaining region of P; and Comparing the maximum area to a threshold to determine whether at least one image includes a concave occlusion. Includes.

[0236] 36. The apparatus of embodiment 33, wherein determining whether an occluded corner is present in at least one image comprises: generating a mobile device mask using the at least one trained model, the mobile device mask including a reduced number of colors compared to the at least one image; Extracting a polygonal subregion P of a mobile device mask; Determining the convex hull of P; Identifying the four principal edges of the convex hull, determining the intersection of adjacent major edges to identify corners; Determine the distance from each corner to P, and Comparing each distance to a distance threshold to determine whether any corners are occluded in at least one of the images. Includes.

[0237] 37. The apparatus of embodiment 24, wherein processing at least one image to determine an integrity status of the mobile device includes: Using the at least one trained model, determining whether at least one image includes a front side of the mobile device, a back side of the mobile device, or a cover. Includes.

[0238] 38. The apparatus of embodiment 24, wherein determining whether an occluded corner is present in at least one image comprises: and providing a response for display on the mobile device in real time or near real time in response to receiving the at least one image, the response provided depending on the determined integrity state of the mobile device.

[0239] 39. The apparatus of embodiment 24, wherein determining whether an occluded corner is present in at least one image comprises: Displaying a test pattern on a mobile device configured to provide improved accuracy in predicting a characteristic of at least one image captured while the mobile device is displaying the test pattern as compared to accuracy in predicting a characteristic of at least one image captured while the mobile device is displaying another display pattern. Includes.

[0240] 40. The apparatus of embodiment 24, wherein the at least one memory and the computer program code, using a processor, cause the apparatus to at least: identifying a subset of conditions that must be satisfied to determine an integrity state of the mobile device as verified; setting an image status of the particular image as verified if all conditions in the subset of conditions are satisfied in the particular image; and Determining the integrity status of the mobile device as verified if each image status for all required images is verified. The device is further configured to cause the

[0241] 41. The apparatus of embodiment 40, wherein at least one condition of the subset of conditions to be satisfied is executed on a mobile device. 42. The apparatus of embodiment 24, wherein receiving the at least one image includes receiving at least two images captured by a mobile device, a first image of the at least two images being of a front side of the device and a second image of the at least two images being of a back side of the device, and processing the at least one image to determine an integrity status of the mobile device includes: processing both the first image and the second image with the at least one trained model; and determining that the determined integrity state of the mobile device is verified if processing of both images results in a verified image state for each; Includes.

[0242] 43. The apparatus of embodiment 24, wherein the at least one memory and the computer program code, using a processor, cause the apparatus to at least: Train at least one trained model by inputting training images and respective labels that describe the properties of each training image. The method is further configured as follows. 44. The apparatus of embodiment 24, wherein at least one trained model is a neural network.

[0243] 45. An apparatus for detecting concave occlusions in an image, the apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured, using the processor, to cause the apparatus to at least: generating a mask that includes a reduced number of colors compared to the image using at least one trained model; Extracting a polygonal subregion P of the mask; Determining the convex hull of P; Computing the difference between P and the convex hull; removing or reducing thin discrepancies between P and at least one edge of the convex hull; Recalculating P as the maximum area of ​​the remaining regions, and Determine the concavity as the difference between P and the convex hull The method is configured to cause the

[0244] 46. ​​An apparatus for detecting occluded corners of an object in an image, the apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured, using the processor, to cause the apparatus to at least: generating a mask that includes a reduced number of colors compared to the image using at least one trained model; Extracting a polygonal subregion P of the mask; Determining the convex hull of P; identifying a predetermined number of dominant edges of the convex hull; determining the intersection of adjacent major edges to identify corners; Determine the distance from each corner to P, and Comparing each distance to a distance threshold to determine whether any corners are occluded in the image. The method is configured to cause the

[0245] 47. A computer program product, comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: receiving a device integrity verification request associated with the mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying a prompt on the mobile device to cause the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; receiving at least one image captured by one or more image sensors of the mobile device; and Processing at least one image using the at least one trained model to determine an integrity state of the mobile device. The method includes program code instructions for:

[0246] 48. The computer program product of embodiment 47, wherein processing at least one image to determine an integrity status of the mobile device includes: utilizing at least one trained model to determine whether there is damage to the mobile device; and In response to determining that there is damage to the mobile device, determining that an integrity state of the mobile device is not verified. Includes.

[0247] 49. The computer program product of embodiment 47, wherein processing at least one image to determine an integrity status of the mobile device includes: determining an angle of the mobile device relative to the reflective surface at the time the at least one image was captured; and determining, based on the angle, that at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object; Includes.

[0248] 50. The computer program product of embodiment 47, wherein processing at least one image to determine an integrity status of the mobile device includes: Determining whether at least one image includes a mobile device associated with a mobile device identification data object. Includes.

[0249] 51. In the computer program product of embodiment 50, determining whether at least one image includes a mobile device comprises: Identifying a suspect mobile device within at least one image; generating a prediction of an identification of the at least one suspect mobile device and comparing the mobile device identification data object to the prediction of the identification of the at least one suspect mobile device to determine whether the suspect mobile device is the mobile device; and If the suspect mobile device is determined to be the mobile device, determining that the integrity state of the mobile device is verified. Includes.

[0250] 52. The computer program product of embodiment 47, wherein the integrity state of the mobile device is determined to be indeterminate, and the computer-executable program code instructions include: Transmitting the device integrity verification request and at least one image to an internal user computer program product for internal review. The method further includes program code instructions for:

[0251] 53. The computer program product of embodiment 49, wherein the computer-executable program code instructions include: In response to determining that at least one of the images captures a different mobile device based on the angle, (a) causing a message to be displayed on the mobile device instructing the user to recapture the mobile device; and (b) determining that the integrity status of the mobile device has not been verified; The method further includes program code instructions for:

[0252] 54. The computer program product of embodiment 47, wherein processing at least one image to determine an integrity status of the mobile device includes: determining a location in at least one image of the mobile device, the location being defined as a bounding box; and If the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, displaying a message on the mobile device instructing the user to move the mobile device closer to the reflective surface. Includes.

[0253] 55. The computer program product of embodiment 54, wherein the computer-executable program code instructions include: If the bounding box has a second predetermined relationship with the threshold ratio of the at least one image, cropping the at least one image according to the bounding box. The method further includes program code instructions for:

[0254] 56. The computer program product of embodiment 47, wherein processing at least one image to determine an integrity status of the mobile device includes: determining, using the at least one trained model, that an object is occluding the mobile device in the at least one image; and Displaying a prompt on a mobile device to capture the unoccluded image. Includes.

[0255] 57. In the computer program product of embodiment 56, determining whether an occlusion of the mobile device occurs within at least one image comprises: determining whether a concave occlusion is present in at least one image; and Determine whether the occluded corner is present in at least one image. Includes.

[0256] 58. The computer program product of embodiment 56, wherein determining whether a concave occlusion is present in at least one image comprises: generating a mobile device mask using the at least one trained model, the mobile device mask including a reduced number of colors compared to the at least one image; Extracting a polygonal subregion P of a mobile device mask; Determining the convex hull of P; Computing the difference between P and the convex hull; removing or reducing thin discrepancies between P and at least one edge of the convex hull; Identifying the largest area of ​​the remaining region of P; and Comparing the maximum area to a threshold to determine whether at least one image includes a concave occlusion. Includes.

[0257] 59. The computer program product of embodiment 56, wherein determining whether an occluded corner is present in at least one image comprises: generating a mobile device mask using the at least one trained model, the mobile device mask including a reduced number of colors compared to the at least one image; Extracting a polygonal subregion P of a mobile device mask; Determining the convex hull of P; Identifying the four principal edges of the convex hull, determining the intersection of adjacent major edges to identify corners; Determine the distance from each corner to P, and Comparing each distance to a distance threshold to determine whether any corners are occluded in at least one of the images. Includes.

[0258] 60. The computer program product of embodiment 47, wherein processing at least one image to determine an integrity status of the mobile device includes: Using the at least one trained model, determining whether at least one image includes a front side of the mobile device, a back side of the mobile device, or a cover. Includes.

[0259] 61. The computer program product of embodiment 47, wherein determining whether an occluded corner is present in at least one image comprises: and providing a response for display on the mobile device in real time or near real time in response to receiving the at least one image, the response provided depending on the determined integrity state of the mobile device.

[0260] 62. The computer program product of embodiment 47, wherein determining whether there is an occluded corner in at least one image comprises: Displaying a test pattern on a mobile device configured to provide improved accuracy in predicting a characteristic of at least one image captured while the mobile device is displaying the test pattern as compared to accuracy in predicting a characteristic of at least one image captured while the mobile device is displaying another display pattern. Includes.

[0261] 63. The computer program product of embodiment 47, wherein the computer-executable program code instructions include: identifying a subset of conditions that must be satisfied to determine an integrity state of the mobile device as verified; setting an image status of the particular image as verified if all conditions in the subset of conditions are satisfied in the particular image; and Determining the integrity status of the mobile device as verified if each image status for all required images is verified. The method further includes program code instructions for:

[0262] 64. The computer program product of embodiment 63, wherein at least one condition of the subset of conditions to be satisfied is executed on a mobile device. 65. The computer program product of embodiment 47, wherein receiving at least one image includes receiving at least two images captured by a mobile device, a first image of the at least two images being of a front side of the device and a second image of the at least two images being of a back side of the device, and processing the at least one image to determine an integrity status of the mobile device includes: processing both the first image and the second image with the at least one trained model; and determining that the determined integrity state of the mobile device is verified if processing of both images results in a verified image state for each; Includes.

[0263] 66. The computer program product of embodiment 47, wherein the computer-executable program code instructions include: Train at least one trained model by inputting training images and respective labels that describe the properties of each training image. The method further includes program code instructions for:

[0264] 67. The computer program product of embodiment 47, wherein at least one trained model is a neural network. 68. A computer program product for detecting concave occlusions in an image, the computer program product including at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: generating a mask that includes a reduced number of colors compared to the image using at least one trained model; Extracting a polygonal subregion P of the mask; Determining the convex hull of P; Computing the difference between P and the convex hull; removing or reducing thin discrepancies between P and at least one edge of the convex hull; Recalculating P as the maximum area of ​​the remaining regions, and Determine the concavity as the difference between P and the convex hull The method includes program code instructions for:

[0265] 69. A computer program product for detecting occluded corners of an object in an image, the computer program product including at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: generating a mask that includes a reduced number of colors compared to the image using at least one trained model; Extracting a polygonal subregion P of the mask; Determining the convex hull of P; identifying a predetermined number of dominant edges of the convex hull; determining the intersection of adjacent major edges to identify corners; Determine the distance from each corner to P, and Comparing each distance to a distance threshold to determine whether any corners are occluded in the image. The method includes program code instructions for:

[0266] 70. Receiving an indication of a target image; and Processing the target image with at least one trained model, the trained model being trained with a plurality of training images, each of which is labeled as either including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device. The method includes:

[0267] 71. Receiving an indication of a target image; processing the target image with at least one trained model trained on a plurality of training images, each associated with a bounding box indicating a location of the mobile device within the image, to determine a location of the mobile device within the target image; and Cropping the target image based on the determined position of the mobile device within the target image The method includes:

[0268] 72. Receiving an indication of a target image of a target mobile device; and Processing the target image of the mobile device with at least one trained model trained with a plurality of training images of the mobile device labeled as including a cover on the respective mobile device or excluding a cover on the respective mobile device to determine whether the target image includes a cover on the target mobile device. The method includes:

[0269] 73. Receiving an indication of a target image of a target mobile device; and Processing a target image of a mobile device with at least one trained model trained on a plurality of training images of the mobile device, each training image labeled as including a front side of the respective mobile device or including a rear side of the respective mobile device, to determine whether the target image includes a front side or a rear side of the target mobile device. The method includes:

[0270] 74. Receiving an indication of a target image of a target mobile device; and Processing target images of the mobile devices with at least one trained model trained on a plurality of training images of the mobile devices, where each training image is labeled as being captured by a respective mobile device contained in the image or by a device different from the respective mobile device contained in the image, to determine whether a target mobile device contained in the target image was captured by the target mobile device or by a different device. The method includes:

[0271] 75. Receiving an indication of a target image of a target mobile device; and Processing target images of the mobile device with at least one trained model trained on a plurality of training images of the mobile device, each training image being labeled with a damage rating, to calculate a damage rating for the target mobile device in the target images. The method includes:

[0272] 76. An apparatus having at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform at least one receiving an indication of a target image; and Processing the target image with at least one trained model, the trained model being trained with a plurality of training images, each of which is labeled as either including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device. The method is configured to cause the

[0273] 77. An apparatus having at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform at least one receiving an indication of a target image; processing the target image with at least one trained model trained on a plurality of training images, each associated with a bounding box indicating a location of the mobile device within the image, to determine a location of the mobile device within the target image; and Cropping the target image based on the determined position of the mobile device within the target image The method is configured to cause the

[0274] 78. An apparatus having at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform at least one receiving an indication of a target image for a target mobile device; and Processing the target image of the mobile device with at least one trained model trained with a plurality of training images of the mobile device labeled as including a cover on the respective mobile device or excluding a cover on the respective mobile device to determine whether the target image includes a cover on the target mobile device. The method is configured to cause the

[0275] 79. An apparatus having at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform at least one receiving an indication of a target image for a target mobile device; and Processing a target image of a mobile device with at least one trained model trained on a plurality of training images of the mobile device, each training image labeled as including a front side of the respective mobile device or including a rear side of the respective mobile device, to determine whether the target image includes a front side or a rear side of the target mobile device. The method is configured to cause the

[0276] 80. An apparatus having at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform at least one receiving an indication of a target image for a target mobile device; and Processing target images of the mobile devices with at least one trained model trained on a plurality of training images of the mobile devices, where each training image is labeled as being captured by a respective mobile device contained in the image or by a device different from the respective mobile device contained in the image, to determine whether a target mobile device contained in the target image was captured by the target mobile device or by a different device. The method is configured to cause the

[0277] 81. An apparatus having at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to cause the apparatus to perform at least one receiving an indication of a target image for a target mobile device; and Processing target images of the mobile device with at least one trained model trained on a plurality of training images of the mobile device, each training image being labeled with a damage rating, to calculate a damage rating for the target mobile device in the target images. The method is configured to cause the

[0278] 82. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: receiving an indication of a target image; and Processing the target image with at least one trained model, the trained model being trained with a plurality of training images, each of which is labeled as either including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device. The method includes program code instructions for:

[0279] 83. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: receiving an indication of a target image; processing the target image with at least one trained model trained on a plurality of training images, each associated with a bounding box indicating a location of the mobile device within the image, to determine a location of the mobile device within the target image; and Cropping the target image based on the determined position of the mobile device within the target image The method includes program code instructions for:

[0280] 84. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: receiving an indication of a target image for a target mobile device; and Processing the target image of the mobile device with at least one trained model trained with a plurality of training images of the mobile device labeled as including a cover on the respective mobile device or excluding a cover on the respective mobile device to determine whether the target image includes a cover on the target mobile device. The method includes program code instructions for:

[0281] 85. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: receiving an indication of a target image for a target mobile device; and Processing a target image of a mobile device with at least one trained model trained on a plurality of training images of the mobile device, each training image labeled as including a front side of the respective mobile device or including a rear side of the respective mobile device, to determine whether the target image includes a front side or a rear side of the target mobile device. The method includes program code instructions for:

[0282] 86. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: receiving an indication of a target image for a target mobile device; and Processing target images of the mobile devices with at least one trained model trained on a plurality of training images of the mobile devices, where each training image is labeled as being captured by a respective mobile device contained in the image or by a device different from the respective mobile device contained in the image, to determine whether a target mobile device contained in the target image was captured by the target mobile device or by a different device. The method includes program code instructions for:

[0283] 87. A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: receiving an indication of a target image for a target mobile device; and Processing target images of the mobile device with at least one trained model trained on a plurality of training images of the mobile device, each training image being labeled with a damage rating, to calculate a damage rating for the target mobile device in the target images. The method includes program code instructions for:

[0284] Many modifications and other embodiments of the inventions described herein will occur to those skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing description and the accompanying drawings. It is therefore understood that the invention is not limited to the specific embodiments disclosed, and that other embodiments are intended to fall within the scope of the appended claims. Moreover, while the foregoing description and the associated drawings describe example embodiments in the context of certain example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, it is contemplated that combinations of elements and / or functions different from those explicitly described above may be set forth in some of the appended claims. Although certain terms are employed herein, they are used in a generic and descriptive sense only, and not for purposes of limitation.

Claims

1. receiving a device integrity verification request associated with the mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; receiving the at least one image captured by the one or more image sensors of the mobile device; and processing the at least one image using the at least one trained model to determine an integrity state of the mobile device; Processing the at least one image to determine an integrity status of the mobile device includes: determining a location of the mobile device within the at least one image, the location being defined as a bounding box; and if the bounding box has a first predetermined relationship to a threshold ratio of the at least one image, displaying a message on the mobile device instructing the mobile device to move closer to the reflective surface.

2. Processing the at least one image to determine an integrity status of the mobile device includes: utilizing the at least one trained model to determine whether there is damage to the mobile device; The method of claim 1 , further comprising: in response to determining that there is damage to the mobile device, determining that an integrity state of the mobile device is not verified.

3. Processing the at least one image to determine an integrity status of the mobile device includes: determining an angle of the mobile device relative to the reflective surface at the time the at least one image was captured; The method of claim 1 , further comprising predicting, based on the angle, that the at least one image includes a different mobile device than the mobile device associated with the mobile device identification data object.

4. in response to predicting that the at least one image captures a different mobile device based on the angle, (a) displaying a message on the mobile device instructing a user to recapture the mobile device; (b) determining that the integrity status of the mobile device is not verified; The method of claim 3 further comprising:

5. Processing the at least one image to determine an integrity status of the mobile device includes: The method of claim 1 , further comprising determining whether the at least one image includes the mobile device associated with the mobile device identification data object.

6. Determining whether the at least one image includes the mobile device includes: identifying a suspect mobile device within the at least one image; generating a prediction of an identification of the at least one suspect mobile device and comparing the mobile device identification data object with the prediction of the identification of the at least one suspect mobile device to determine whether the suspect mobile device is the mobile device; and determining an integrity state of the mobile device as verified if the suspect mobile device is determined to be the mobile device.

7. An integrity state of the mobile device is determined to be indeterminate, and the method further comprises: The method of claim 1 , further comprising transmitting the device integrity verification request and the at least one image to an internal user device for internal review.

8. The method of claim 1 , further comprising cropping the at least one image according to the bounding box if the bounding box has a second predetermined relationship to the threshold ratio of the at least one image.

9. Processing the at least one image to determine an integrity status of the mobile device includes: determining that an object is occluding the mobile device in the at least one image using the at least one trained model; and The method of claim 1 , further comprising: displaying a prompt on the mobile device to capture the unoccluded image.

10. Processing the at least one image to determine an integrity status of the mobile device includes:

10. The method of claim 1 , comprising using the at least one trained model to determine whether the at least one image includes a front side of the mobile device, a back side of the mobile device, or a cover.

11. 10. The method of claim 1, further comprising providing a response for display on the mobile device in real time or near real time in response to receiving the at least one image, the response provided depending on the determined integrity state of the mobile device.

12. 10. The method of claim 1, further comprising: displaying on the mobile device the test pattern configured to provide improved accuracy in predicting characteristics of the at least one image captured when the mobile device is displaying a test pattern as compared to accuracy in predicting the characteristics of the at least one image captured when the mobile device is displaying another display pattern.

13. identifying a subset of conditions that must be satisfied to determine an integrity state of the mobile device as verified; setting an image status of a particular image as verified if all of the conditions in the subset of conditions are satisfied in the particular image; The method of claim 1 , further comprising: determining an integrity status of the mobile device as verified if the respective image statuses for all required images are verified.

14. Receiving the at least one image includes receiving at least two images captured by the mobile device, a first image of the at least two images being of a front side of the mobile device and a second image of the at least two images being of a back side of the mobile device, and processing the at least one image to determine an integrity status of the mobile device includes: processing both the first image and the second image using the at least one trained model; and and determining the determined mobile device integrity state as verified if the processing of both images results in a respective image state being verified.

15. The method of claim 1 , further comprising training the at least one trained model by inputting a plurality of training images and respective labels describing characteristics for each of the plurality of training images.

16. 1. An apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least: receiving a device integrity verification request associated with the mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; receiving at least one image captured by the one or more image sensor mobile devices; and processing the at least one image using the at least one trained model to determine an integrity state of the mobile device; Processing the at least one image to determine an integrity status of the mobile device includes: determining a location of the mobile device within the at least one image, the location being defined as a bounding box; and if the bounding box has a first predetermined relationship to a threshold ratio of the at least one image, displaying a message on the mobile device instructing the mobile device to move closer to the reflective surface.

17. At least one persistent computer-readable storage medium having computer-executable program code instructions stored thereon, the computer-executable program code instructions comprising: receiving a device integrity verification request associated with the mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; receiving the at least one image captured by the one or more image sensor mobile devices; and processing the at least one image using the at least one trained model to determine an integrity state of the mobile device. Processing the at least one image to determine an integrity status of the mobile device includes: determining a location of the mobile device within the at least one image, the location being defined as a bounding box; and if the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, displaying a message on the mobile device instructing the mobile device to move closer to the reflective surface.

Images