System, method, apparatus, and sustainable computer readable storage medium for processing image of mobile device using machine learning to determine integrity state of mobile device

Abstract

To provide a system, an apparatus, a method, and a computer program product for determining the integrity state of a mobile device.SOLUTION: Images of a mobile device captured by the mobile device using reflective surfaces are processed using various trained models, such as neural networks, to verify authenticity, detect damage, and detect shielding. A mask may be generated to enable identification of concave shielding or occluded corners of an object, such as a mobile device, in an image. Images of the front and / or back of the mobile device may be processed to determine the integrity state of the mobile device, such as verified, unverified, or indeterminate. A user may be prompted to remove the cover, remove the shielding, and / or move the mobile device closer to the reflective surface.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments of the present invention generally relate to computer technology, and more particularly to systems, methods, apparatuses, and computer program products for training and utilizing machine learning to train and utilize mathematical models such as prediction models, neural networks, and / or the like, and for determining the complete state of a mobile device based on the electronic processing of images.

Background Art

[0002] Computer vision enables a computer to look at and understand images. In some cases, computer vision can be used to detect and analyze the content of an image, such as the recognition of objects within the image. However, existing technologies are insufficient to meet the speed and accuracy requirements of many industries, and there is a need for improvements in computer vision technology and techniques to enable high-performance image processing. Moreover, human analysis is not useful for the speed and accuracy required for computer vision tasks. Through the noted efforts, ingenuity, and innovation, many of these identified problems have been solved by developing the solutions included in embodiments of the present invention, and many examples thereof are described in detail herein.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Systems, methods, apparatuses, and computer program products are therefore provided for training and utilizing a model using machine learning to determine the complete state of a mobile device based on the electronic processing of images.

Means for Solving the Problems

[0004] In some use cases, the system needs to review an image of an object to verify the integrity of the object (e.g., to determine information about the object, to verify the operability or functionality of the object, to verify the identification of the object, or the like). Computer vision and image processing need to occur quickly and with high accuracy, which is lacking in many conventional image processing techniques. A further challenge is when the system cannot select the imaging device that captures the image and cannot directly control the image capture process, and thus computer vision needs to be robust enough to absorb and / or detect problems related to the capture process. In an example of a working environment, the system may attempt to verify the identification and integrity of an object using only an image of the object (e.g., a mobile device), or in combination with one or more data objects sent from that object or from another device. An example of such an environment can be when a user registers for a service, a protection plan, or the like, which requires a remote system to verify an object (e.g., a mobile device) without the device being physically present. According to some processes for purchasing compensation in the aftermarket, consumers need to visit a retailer, an insurance company, or a mobile device service provider to have the device inspected and verify the integrity of the device before the insurer issues a security for the compensation coverage and registers the device. Other processes for purchasing and / or selling compensation enable consumers to take a photo of their mobile device using a self-service web application or mobile application and submit the image for manual review before registration. However, such processes require review time and the confirmation of compensation for consumers may be delayed. Such processes may further expose the provider to fraud, such as when a consumer submits a photo of a different undamaged mobile device to obtain compensation for a previously damaged device.

[0005] Additional embodiments provide a time - limited barcode, quick response (QR) code, or other computer - generated code that is displayed by a device, captured in a photo using a mirror, thereby linking the photo submission to the device that displayed its code. However, such embodiments may be vulnerable to fraud, such as by enabling a user to recreate the code on another pristine device and capture a photo of that pristine device. Even further, code embodiments may provide verification only of the front of the device (e.g., the display side) without ensuring verification of the condition or state of the back of the device and / or the bezel of the device. Another drawback of such embodiments is that when the code is displayed on the device display, it may obscure cracks or other damage present on the screen.

[0006] Example embodiments of the present disclosure provide an improved determination of the pristine state of a mobile device. Example embodiments may prompt a user to capture an image of their device in a mirror or other reflective surface using the device's own sensors or camera. The identification information of the mobile device is processed with the image to confirm that the image was truly taken from the target device from which the image was captured and that there are no existing damages that would render the device unfit for compensation.

[0007] Example embodiments can utilize prediction models and / or other types of "models" such as, but not limited to, machine learning algorithms and related mathematical model(s), e.g., neural network(s) such as convolutional neural networks and / or the like, which can be trained to analyze and identify relevant information from images by using training images that are manually reviewed and labeled and / or characterized by a user. It is understood that any reference to a "model" herein can include any type of model that is used in a machine learning algorithm trained with training images to make predictions regarding certain features of other images. Example embodiments can utilize information detected in an image received later to predict features in the image, such as, but not limited to, the complete state of a mobile device, by using a trained model.

[0008] Different models, each of which can be trained with a different training set, can be used to make different types of predictions. The utilization of trained model(s) (e.g., neural network) can enable an example embodiment to determine the complete state of a mobile device in real time or substantially real time from when an image is submitted, (without additional human review in accordance with some embodiments) and / or transfer for further review in real time or substantially real time before finalizing the complete state of the mobile device and / or registering the mobile device with a protection plan for uncertain or high-risk predictions. In some embodiments, the output of one or more models can be used as input to a subsequent model for more refined analysis of the image and determination of the complete state of the mobile device.

[0009] In some environmental examples, consumers may purchase insurance plans, warranties, extended warranties, and / or other device protection plans to protect their mobile devices and smartphones from damage, theft, loss, and / or the like. In some cases, consumers may purchase such plans in-store, so the condition of the device is known to be new and the device is eligible for compensation. However, in some cases, consumers may wish to purchase protection directly from an insurance company or through their mobile device service provider after the device has been owned. The provider must be able to quickly verify the integrity of the device without physically accessing the device and without the ability to directly operate the device. In such cases, manual review cannot satisfy the accuracy and speed required to verify the integrity of the device in a reasonable time, and there may be a need for the systems, methods, and devices described herein. Similarly, consumers who purchase used or refurbished devices may wish to purchase insurance in the aftermarket if the condition of the device is unknown to the insurance company. The insurance company checks the condition of the device at the time the protection is purchased to minimize losses and prevent fraudulent purchases of protection for devices with existing damage.

[0010] References herein to warranties, extended warranties, insurance, insurance policies, insurance certificates, coverage, device protection plans, protection plans, and / or the like are not intended to limit the scope of the present disclosure, and embodiments may relate to the registration of a mobile device with any such foregoing plan or similar plan to protect against loss of the mobile device or to other environments in which the computer vision and image processing systems, methods, and devices described herein are used. Similarly, any reference to verifying the integrity of a device may relate to the eligibility of a mobile device for registration in any of the foregoing plans or environments. Still further, determination of the pristine condition of a mobile device may be used for other purposes.

[0011] One conditional example implemented in accordance with the example embodiments described herein includes determining whether there is occlusion in an image of a mobile device. It will be understood that the occlusion detection process disclosed herein can be used for other purposes, such as determining occlusion of any type of object within an image.

[0012] A method is provided that includes receiving a device integrity verification request associated with a mobile device and receiving a mobile device identification data object that includes information describing the mobile device. The method further includes displaying, on the mobile device, a prompt for causing at least one image of the mobile device to be captured using one or more image sensors of the mobile device and a reflective surface, and receiving at least one image captured by one or more image sensors of the mobile device. The method may further include processing the at least one image using at least one trained model to determine the integrity state of the mobile device. In some embodiments, the at least one trained model may include a neural network.

[0013] According to one example embodiment, determining the integrity state of a mobile device by processing at least one image includes determining whether the at least one image includes a mobile device associated with a mobile device identification data object. Determining whether the at least one image includes a mobile device includes identifying a suspected mobile device within the at least one image, generating a prediction of the identification of the at least one suspected mobile device, and comparing the mobile device identification data object with the prediction of the identification of the at least one suspected mobile device to determine whether the suspected mobile device is that mobile device. Determining the integrity state of a mobile device by processing at least one image may further include determining that the integrity state of the mobile device is verified if the suspected mobile device is determined to be that mobile device. Determining the integrity state of a mobile device by processing at least one image may further include transmitting a device integrity verification request and the at least one image to an internal user device for internal review if the integrity state of the mobile device is determined to be uncertain.

[0014] According to some embodiments, determining the integrity state of a mobile device by processing at least one image may include determining whether there is damage to the mobile device using at least one trained model, and in response to determining that there is damage to the mobile device, determining that the integrity state of the mobile device has not been verified.

[0015] In some embodiments, determining the full state of a mobile device by processing at least one image includes determining the angle of the mobile device relative to the reflective surface when the at least one image was captured, and based on that angle, determining that the at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object. Determining the full state of a mobile device by processing at least one image may further include, in response to a determination, based on the angle, that the at least one image captured a different mobile device, displaying on the mobile device a message instructing the user to recapture the mobile device, and determining that the full state of the mobile device has not been verified.

[0016] According to some embodiments, determining the full state of a mobile device by processing at least one image includes determining a position within at least one image of the mobile device, the position being defined as a bounding box, and if the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, may include displaying on the mobile device a message instructing the mobile device to move closer to the reflective surface. If the bounding box has a second predetermined relationship with a threshold ratio of the at least one image, determining the full state of a mobile device by processing at least one image may further include trimming the at least one image according to the bounding box.

[0017] According to some embodiments, determining the perfect state of a mobile device by processing at least one image includes using at least one trained model to determine that an object is obscuring the mobile device within at least one image, and displaying, on the mobile device, a prompt for capturing an image without such obscuration. Determining whether there is an obscuration of the mobile device within at least one image may include determining whether there is a concave obscuration within at least one image and determining whether there is an occluded corner within at least one image. Determining whether there is a concave obscuration within at least one image may include using at least one trained model to generate a mobile device mask that includes a smaller number of colors compared to the at least one image, extracting a polygonal partial region P of the mobile device mask, determining a convex hull of P, calculating a difference between P and the convex hull, removing or reducing a thin mismatch of at least one edge between P and the convex hull, identifying a maximum area of the remaining region of P, and comparing the maximum area with a threshold to determine whether the at least one image includes a concave obscuration.

[0018] According to some embodiments, determining whether there is an occluded corner within at least one image includes using at least one trained model to generate a mobile device mask that includes a smaller number of colors compared to the at least one image, extracting a polygonal partial region P of the mobile device mask, determining a convex hull of P, identifying four major edges of the convex hull, determining intersections of adjacent major edges to identify corners, determining respective distances from each corner to P, and comparing each distance with a distance threshold to determine whether any corner is occluded within the at least one image.

[0019] According to one embodiment, determining the perfect state of a mobile device by processing at least one image includes determining, using at least one trained model, whether the at least one image includes the front of the mobile device, the back of the mobile device, or a cover.

[0020] In response to receiving the at least one image, some example embodiments may provide a response for display on the mobile device in real time or near real time, and the response provided is determined by the determined perfect state of the mobile device.

[0021] Example embodiments may also include causing a test pattern configured to provide improved accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying a test pattern, compared to the accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying another display pattern, to be displayed on the mobile device.

[0022] Some example embodiments may identify a subset of conditions to be satisfied to determine that the perfect state of the mobile device has been verified. When all conditions within the subset of conditions are satisfied in a particular image, the image state of the particular image is set as verified. When the respective image states for all required images are verified, it is determined that the perfect state of the mobile device has been verified. In some embodiments, at least one of the conditions in the subset of conditions to be satisfied is executed on the mobile device.

[0023] According to one embodiment, receiving at least one image includes receiving at least two images captured by a mobile device, wherein a first image of the at least two images is of the front of the device and a second image of the at least two images is of the back of the device, and processing the at least one image to determine a fully functional state of the mobile device includes processing both the first image and the second image using at least one trained model, and when the respective image states are verified in the processing of both images, determining that the fully functional state of the mobile device being determined is verified.

[0024] Some embodiments may train at least one trained model by inputting training images and respective labels describing characteristics of each training image. A method for detecting a concave occlusion in an image is also provided. The method includes generating a mask containing a smaller number of colors compared to the image using at least one trained model, extracting a polygonal partial region P of the mask, determining a convex hull of P, and calculating a difference between P and the convex hull. The method further includes removing or reducing a thin mismatch of at least one edge between P and the convex hull, recalculating P as the largest area of the remaining region, and determining the indentation as the difference between P and the convex hull.

[0025] A method for detecting an occluded corner of an object in an image is provided. The method includes generating a mask containing a smaller number of colors compared to the image using at least one trained model, extracting a polygonal partial region P of the mask, determining a convex hull of P, identifying a predetermined number of major edges of the convex hull, determining intersections of adjacent major edges to identify corners, determining respective distances from each corner to P, and comparing each distance with a distance threshold to determine whether any corner is occluded in the image.

[0026] An apparatus is provided comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause the processor, at least in the apparatus, to receive device integrity verification requirements associated with a mobile device, and to receive a mobile device identification data object including information describing the mobile device. The at least one memory and the computer program code are further configured to cause the processor, in the apparatus, to display on the mobile device a prompt for capturing at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device, and to receive at least one image captured by one or more image sensors of the mobile device. The at least one memory and the computer program code may be further configured to cause the processor, using at least one trained model, to process at least one image to determine the integrity state of the mobile device.

[0027] An apparatus for detecting a concave occlusion in an image is provided, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to cause the processor, at least in the apparatus, to generate a mask including a smaller number of colors compared to the image using at least one trained model, to extract a polygonal partial region P of the mask, to determine a convex hull of P, to calculate a difference between P and the convex hull, to remove or reduce a thin mismatch of at least one edge between P and the convex hull, to recalculate P as the largest area of the remaining region, and to determine a depression as the difference between P and the convex hull.

[0028] An apparatus for detecting occluded corners of an object in an image is also provided. The apparatus comprises at least one processor and at least one memory including computer program code. The at least one memory and the computer program code are configured, using the processor, to cause the apparatus to at least generate a mask containing a smaller number of colors compared to the image, using at least one trained model; extract a polygonal partial region P of the mask; determine the convex hull of P; identify a predetermined number of major edges of the convex hull; determine intersections of adjacent major edges to identify corners; determine respective distances from each corner to P; and compare each distance with a distance threshold to determine whether any of the corners are occluded in the image.

[0029] A computer program product is provided. The computer program product includes at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein. The computer-executable program code instructions include program code instructions for receiving a device integrity verification request associated with a mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying, on the mobile device, a prompt for causing at least one image of the mobile device to be captured using one or more image sensors and a reflective surface of the mobile device; receiving at least one image captured by one or more image sensors of the mobile device; and processing the at least one image using at least one trained model to determine a full state of the mobile device.

[0030] A computer program product for detecting concave shielding within an image is also provided. This computer program product includes at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein. The computer-executable program code instructions are for using at least one trained model to generate a mask that includes a smaller number of colors compared to the image, extracting a polygonal partial region P of the mask, determining the convex hull of P, calculating the difference between P and the convex hull, removing or reducing thin mismatches of at least one edge between P and the convex hull, recalculating P as the largest area of the remaining region, and determining the concavity as the difference between P and the convex hull, and include program code instructions for doing so.

[0031] A computer program product for detecting occluded corners of an object within an image is also provided. This computer program product includes at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein. The computer-executable program code instructions are for using at least one trained model to generate a mask that includes a smaller number of colors compared to the image, extracting a polygonal partial region P of the mask, determining the convex hull of P, identifying a predetermined number of major edges of the convex hull, determining the intersections of adjacent major edges to identify corners, determining the respective distances from each corner to P, and comparing each distance with a distance threshold to determine whether any corner is occluded within the image, and include program code instructions for doing so.

[0032] An apparatus is provided, the apparatus including means for receiving a device integrity verification request associated with a mobile device, and means for receiving a mobile device identification data object including information describing the mobile device, means for displaying on the mobile device a prompt for causing at least one image of the mobile device to be captured using one or more image sensors and a reflective surface of the mobile device, means for receiving at least one image captured by one or more image sensors of the mobile device, and means for using at least one trained model to process the at least one image to determine the integrity state of the mobile device.

[0033] An apparatus is provided having means for detecting a concave occlusion in an image, the apparatus including means for using at least one trained model to cause the apparatus to generate a mask including a smaller number of colors compared to the image and extract a polygonal partial region P of the mask, means for determining a convex hull of P and calculating a difference between P and the convex hull, means for removing or reducing a thin mismatch of at least one edge between P and the convex hull, means for recalculating P as the maximum area of the remaining region, and means for determining the indentation as the difference between P and the convex hull.

[0034] An apparatus is also provided comprising means for detecting an occluded corner of an object in an image, the apparatus including means for using at least one trained model to generate a mask including a smaller number of colors compared to the image, means for extracting a polygonal partial region P of the mask, means for determining a convex hull of P and identifying a predetermined number of major edges of the convex hull, means for determining intersections of adjacent major edges to identify corners, means for determining respective distances from each corner to P, and means for comparing each distance with a distance threshold to determine whether any corner is occluded in the image.

[0035] According to one embodiment, a method is provided that includes receiving an indication of a target image and processing the target image using at least one trained model, such as a model (e.g., a neural network) that can be used in a machine learning algorithm. The model is trained with a plurality of training images each labeled as either including or excluding a mobile device, to determine whether the target image includes a mobile device.

[0036] According to one embodiment, a method is provided that includes receiving an indication of a target image and processing the target image using at least one trained model trained with a plurality of training images each associated with a bounding box indicating the position of a mobile device within the image, to determine the position of the mobile device within the target image. The method may further include trimming the target image based on the determined position of the mobile device within the target image.

[0037] According to one embodiment, a method is provided that includes receiving an indication of a target image of a target mobile device and processing the target image of the target mobile device using at least one trained model trained with a plurality of training images of mobile devices each labeled as either including or excluding a cover on each respective mobile device, to determine whether the target image includes a cover on the target mobile device.

[0038] According to one embodiment, a method is provided that includes receiving an indication of a target image of a target mobile device and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices each labeled as either including the front face or the back face of each respective mobile device, to determine whether the target image includes the front face or the back face of the target mobile device.

[0039] According to one embodiment, there is provided a method including receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as being captured by the respective mobile device in which it is included in the image or being captured by a device different from the respective mobile device included in the image, to determine whether the target mobile device included in the target image was captured by the target mobile device or by a different device.

[0040] According to one embodiment, there is provided a method including receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0041] According to one embodiment, an apparatus comprises at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least receive an indication of a target image and process the target image using at least one trained model trained with a plurality of training images each labeled as either including or excluding a mobile device, to determine whether the target image includes a mobile device.

[0042] There is also provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least receive an instruction of a target image, process the target image using at least one trained model trained with a plurality of training images each associated with a bounding box indicating the position of a mobile device in the image, to determine the position of the mobile device in the target image, and trim the target image based on the determined position of the mobile device in the target image.

[0043] According to an embodiment, there is provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least receive an instruction of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices labeled as including or excluding a cover on each mobile device, to determine whether the target image includes a cover on the target mobile device.

[0044] There is also provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least receive an instruction of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices labeled as including the front or the back of each mobile device, to determine whether the target image includes the front or the back of the target mobile device.

[0045] According to one embodiment, the apparatus comprises at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, using the processor, to cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as being captured by the respective mobile device in which it is included in the image or captured by a device different from the respective mobile device included in the image, to determine whether the target mobile device included in the target image was captured by the target mobile device or by a different device.

[0046] There is also provided an apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, using the processor, to cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0047] According to an example embodiment, there is provided a computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an instruction of a target image and processing the target image using at least one trained model trained with a plurality of training images each labeled as either including a mobile device or excluding a mobile device to determine whether the target image includes a mobile device.

[0048] There is also provided a computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an instruction of a target image and processing the target image using at least one trained model trained with a plurality of training images each associated with a bounding box indicating a position of a mobile device in the image to determine a position of the mobile device in the target image. The computer-executable program code instructions include program code instructions for trimming the target image based on the determined position of the mobile device in the target image.

[0049] There is also provided a computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an instruction of a target image of a target mobile device and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices each labeled as either including a cover on the respective mobile device or excluding a cover on the respective mobile device to determine whether the target image includes a cover on the target mobile device.

[0050] According to an embodiment, there is provided a computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an instruction of a target image of a target mobile device and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as including the front face of each mobile device or the back face of each mobile device, to determine whether the target image includes the front face or the back face of the target mobile device.

[0051] According to an embodiment, there is provided a computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions including program code instructions for receiving an instruction of a target image of a target mobile device and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as being captured by each mobile device included in the image or captured by a device different from each mobile device included in the image, to determine whether the target mobile device included in the target image was captured by the target mobile device or by a different device.

[0052] A computer program product is also provided that includes at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions being for receiving an indication of a target image of a target mobile device and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of the mobile device, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device within the target image.

[0053] The models and algorithms described herein can be used alone or in one or more larger processes such as those described herein for their intended purposes. For example, in some embodiments, both rear and front camera images can capture the back and front of the device, and the various trained models described herein can be run on each image either separately or as part of a larger process to ensure that the device is intact and not damaged. In some embodiments, one or more of the models and algorithms can be run as part of an orientation process for a protection product and / or service contract or other device protection program where verification of the device's integrity is required.

[0054] The foregoing summary is provided for the purpose of summarizing some embodiments of the present invention in order to provide a basic understanding of some aspects of the present invention. Accordingly, it will be understood that the foregoing embodiments are merely examples and should not be construed as narrowing the scope or spirit of the present disclosure in any way. The scope of the present disclosure encompasses many potential embodiments, some of which will be further described below in addition to those summarized herein.

[0055] Although embodiments of the present invention have been described in general terms, reference is now made to the accompanying drawings, which are not necessarily drawn to scale:

Brief Description of the Drawings

[0056]

Figure 1

Figure 2

Figure 3

Figure 4A-1

Figure 4A-2

Figure 4B

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 5G

Figure 5H

Figure 5I

Figure 5J

Figure 5K

Figure 5L

Figure 5M

Figure 5N

Figure 5O

Figure 5P

Figure 5Q

Figure 5R

Figure 5S

Figure 5T

Figure 5U

Figure 5V

Figure 5W

Figure 5X

Figure 5Y

Figure 6

Figure 7A

Figure 7B

Figure 8A

Figure 8B

DETAILED DESCRIPTION OF THE INVENTION

[0057] Some embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the invention are shown. In fact, the various embodiments of the present invention may be embodied in many different forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like reference numerals refer to like elements throughout.

[0058] As used herein, the terms "data," "content," "information," and similar terms may be used interchangeably to refer to data that can be captured, transmitted, received, displayed, and / or stored according to various example embodiments. Accordingly, the use of any such terms should not be construed as limiting the spirit and scope of the present disclosure. Further, when a computing device that receives data from another computing device is described herein, it will be understood that the data can be received directly from the other computing device or indirectly via one or more intermediate computing devices such as, for example, one or more servers, relays, routers, network access points, base stations, and / or the like. Similarly, when a computing device that transmits data to another computing device is described herein, it will be understood that the data can be transmitted directly to the other computing device or indirectly via one or more intermediate computing devices such as, for example, one or more servers, relays, routers, network access points, base stations, and / or the like.

[0059] System Overview FIG. 1 shows a system 100 for determining the full state of a mobile device based on the processing of an image of the device, according to an example embodiment. The system of FIG. 1 can be further utilized to detect occlusions in an image, such as an image of a mobile device, according to an example embodiment. The examples in FIG. 1 and in other drawings are each provided as an example of an embodiment(s), and it will be understood that they should not be construed as narrowing the scope or spirit of the present disclosure in any way. In this regard, the scope of the present disclosure includes many potential embodiments in addition to those illustrated and described herein. Thus, while FIG. 1 shows one example configuration, numerous other configurations can be used to implement embodiments of the present invention.

[0060] System 100 may include any number of mobile devices 104, or simply "devices" as referred to herein. Mobile device 104 may be embodied as any mobile computing device, such as, by way of non-limiting example, a cellular phone, smartphone, mobile communication device, tablet computing device, any combination thereof, or the like. Although described as a mobile device, in some embodiments, mobile device 104 may instead be replaced by any fixed computing device or other device without departing from the scope of the present disclosure. Mobile device 104 may be used by a user to download, install, and access a self-service app, such as one provided by a provider, to obtain a compensation scope for mobile device 104. Additionally or alternatively, mobile device 104 may utilize a browser installed thereon to access a self-service web application, such as one hosted and / or provided by a provider. Still further, mobile device 104 may be used to capture an image for processing according to an embodiment example.

[0061] Device integrity verification device 108 may be associated with a provider or any other entity and may be any processor-driven device that facilitates the processing of requests for device integrity verification, such as those generated from a request to register a device with a device protection program. For example, device integrity verification device 108 may include one or more computers, servers, server clusters, one or more network nodes, or a cloud computing infrastructure configured to facilitate device integrity verification, registration with a device protection plan, and / or other services related to a provider. In some embodiments, some or all of device integrity verification device 108 may be implemented on mobile device 104.

[0062] In one example embodiment, the device integrity verification apparatus 108 hosts or provides a service that enables access by the mobile device 104 to request a compensation range, and further prompts the user of the mobile device 104 to capture an image through the camera of the mobile device 104 using a mirror, as described in more detail herein. The device integrity verification apparatus 108 processes the image using one or more of the computer vision and image processing embodiments described herein to determine whether the device is eligible for the compensation range, as described in more detail herein. The device integrity verification apparatus 108 may include or access one or more models trained to analyze the image to extract relevant information to determine the integrity status of the device. According to some embodiments, the collection of training images and the training of the model may be performed by the device integrity verification apparatus 108. The device integrity verification apparatus 108 may further be configured to maintain information regarding the applied and issued device protection plans and / or facilitate communication between the mobile device 104 and / or an optional internal user device 110.

[0063] The occlusion detection device 109 can be any processor-driven device that facilitates the processing of an image to determine whether an object in the image is occluded. For example, the occlusion detection device 109 may include one or more computers, servers, server clusters, one or more network nodes, or a cloud computing infrastructure configured to facilitate the processing of images and the identification of occlusions. According to one embodiment, the device integrity verification apparatus 108 is integrated with the occlusion detection device 109 to determine whether the mobile device in the image is occluded by a finger and / or the like.

[0064] The optional internal user device 110 can include any computing device or plurality of computing devices that can be used to facilitate device integrity verification by a provider and / or other entity. As an example, the internal user device 110 can be implemented at a support center or central facility far from a mobile device to which one or more customer service representatives can be assigned who can utilize an application provided by the device integrity verification device 108 to receive the results of the device integrity verification server, which can enable further processing or analysis of an image prior to verification or facilitate additional reviews. For example, if the device integrity verification device 108 indicates that further internal review of an image is necessary for verification, such as due to an uncertain mobile device integrity state, the internal user device 110 is used by support staff to review the image to confirm or reject the integrity of the mobile device 104, whereby the coverage of the mobile device 104 in the device protection plan can be confirmed or rejected respectively. The internal user device 110 can further be utilized by an internal user to capture and / or label training images with which to train a model(s). It will be understood that the internal user device 110 can be considered optional. In some embodiments, the device integrity verification device 108 can facilitate faster processing by automatically verifying or rejecting the integrity of the mobile device.

[0065] According to some embodiments, various components of system 100 can be configured to communicate over a network, such as via network 106. For example, mobile device 104 can be configured to access network 106 via a mobile communication connection, a wireless local area network connection, an Ethernet® connection, and / or the like. Thus, network 106 can include a wired network, a wireless network (e.g., a mobile communication network, a wireless local area network, a wireless wide area network, some combination thereof, or the like), or a combination thereof, and in some example embodiments, includes at least a portion of the Internet.

[0066] As described above, certain components of system 100 can be optional. For example, device integrity verification device 108 can be optional, and device integrity verification can be performed on mobile device 104, such as by a self-service application installed on mobile device 104.

[0067] Referring now to FIG. 2, device 200 is a computing device(s) configured to implement mobile device 104, device integrity verification device 108, image detection masking server 109, and / or internal user device 110, according to an example embodiment. Device 200 can at least partially or fully embody any one of mobile device 104, device integrity verification device 108, image detection masking server 109, and / or internal user device 110. Device 200 can be implemented as a distributed system that includes any one of mobile device 104, device integrity verification device 108, image detection masking server 109, and / or internal user device 110, and / or associated network(s).

[0068] The components, devices, and elements illustrated and described in connection with FIG. 2 may not be essential, and thus, it should be noted that some may be omitted in some embodiments. For example, FIG. 2 shows a user interface 216, which may be optional within the device integrity verification apparatus 108, as will be described in more detail below. Additionally, some embodiments may include additional or different components, devices, or elements beyond those illustrated and described in connection with FIG. 2.

[0069] Device 200 may include a processing circuit 210, which may be configured to perform operations according to one or more example embodiments disclosed herein. In this regard, processing circuit 210 may be configured to perform and / or control the operation of one or more functions of device 200 according to various example embodiments. Processing circuit 210 may be configured to perform data processing, application execution, and / or other processing and management services according to one or more example embodiments. In some embodiments, device 200, or portions or components thereof, such as processing circuit 210, may be embodied as or may include a circuit chip. The circuit chip may constitute means for performing one or more operations to provide the functions described herein.

[0070] In some example embodiments, processing circuit 210 may include a processor 212 and, in some embodiments, may further include a memory 214, such as that illustrated in FIG. 2. Processing circuit 210 may communicate with or otherwise control user interface 216 and / or communication interface 218. Thus, processing circuit 210 and / or device 200, such as those included within any of mobile device 104, device integrity verification apparatus 108, image detection masking server 109, and / or internal user device 110, may be embodied as a circuit chip (e.g., an integrated circuit chip) configured (e.g., in hardware, software, or a combination of hardware and software) to perform the operations described herein.

[0071] Processor 212 can be embodied in several different ways. For example, processor 212 can be embodied as a microprocessor or other processing element, a coprocessor, a controller, or various other computing or processing means including integrated circuits such as, for example, an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), or the like. Although shown as a single processor, it will be understood that processor 212 can include multiple processors. The multiple processors can communicate operably with each other and can be configured as a whole to perform one or more functions of device 200 as described herein. The multiple processors can be embodied on a single computing device or can be distributed across multiple computing devices configured to function as mobile device 104, device integrity verification device 108, image detection masking server 109, internal user device 110, and / or device 200 as a whole. In some example embodiments, processor 212 can be configured to execute instructions stored in memory 214 or otherwise accessible to processor 212. Thus, whether configured by hardware or by a combination of hardware and software, processor 212 can represent an entity (e.g., in the form of a circuit - processing circuit 210, physically embodied) configured to and operable to perform operations in accordance with embodiments of the present invention. Thus, for example, if processor 212 is embodied as an ASIC, an FPGA, or the like, processor 212 can be hardware specifically configured to perform the operations described herein. As another example, if processor 212 is embodied as an executor of software instructions, the instructions can specifically configure processor 212 to perform one or more of the operations described herein.

[0072] In some example embodiments, the memory 214 may include one or more persistent memory devices, such as volatile and / or non-volatile memory, which may be either fixed or removable, for example. In this regard, the memory 214 may include a persistent computer-readable storage medium. Although the memory 214 is shown as a single memory, it will be understood that the memory 214 may include multiple memories. The multiple memories may be embodied on a single computing device or may be distributed across multiple computing devices. The memory 214 may be configured to store information, data, applications, computer program code, instructions, and / or the like in order to enable the device 200 to perform various functions, according to one or more example embodiments. For example, if the device 200 is implemented as the mobile device 104, the device integrity verification device 108, the image detection masking server 109, and / or the internal user device 110, the memory 214 may be configured to store computer program code to perform its corresponding functions, as described herein according to the example embodiments.

[0073] Furthermore, the memory 214 may be configured to store models (s), and / or training images used to train the models (s) to predict certain relevant information within the subsequently received images. The memory 214 may be further configured to buffer input data for processing by the processor 212. Additionally or alternatively, the memory 214 may be configured to store instructions for execution by the processor 212. In some embodiments, the memory 214 may include one or more databases that may store various files, content, or data sets. Among the contents of the memory 214, applications may be stored for execution by the processor 212 to perform functions associated with each respective application. In some cases, the memory 214 may communicate with one or more of the processor 212, the user interface 216, and / or the communication interface 218 to pass information between components of the device 200.

[0074] The optional user interface 216 communicates with the processing circuit 210 to receive user input at the user interface 216 and / or provide audible, visual, mechanical, or other output to the user. To that end, the user interface 216 can include, for example, a keyboard, a mouse, a display, a touch screen display, a microphone, a speaker, and / or other input / output mechanisms. For example, in an embodiment where the device 200 is implemented as the mobile device 104, the user interface 216 can, in some example embodiments, provide means for displaying instructions for capturing an image. In an embodiment where the device is implemented as the internal user device 110, the user interface 216 can provide means for an internal user or a colleague to review an image to verify or reject the integrity of the mobile device 104. The user interface 216 can further be used to label training images for the purpose of training the model(s). In some example embodiments, aspects of the user interface 216 can be limited or the user interface 216 may not exist.

[0075] The communication interface 218 may include one or more interface mechanisms to enable communication with other devices and / or networks. In some cases, the communication interface 218 may be any means such as hardware, or a combination of hardware and software, embodied as a device or circuit configured to receive data from any other device or module communicating with the network and / or the processing circuit 210, and / or to transmit data to any other device or module communicating with the network and / or the processing circuit 210. By way of example, the communication interface 218 may be configured to enable communication over a network, such as network 106, between any of the mobile device 104, the device integrity verification device 108, the internal user device 110, and / or the device 200. Accordingly, the communication interface 218 may include, for example, support hardware and / or software to enable wired communication via wireless and / or cable, digital subscriber line (DSL), universal serial bus (USB), Ethernet, or other means.

[0076] When the device 200 is embodied by the mobile device 104, for example, the device 200 may include one or more image capture sensors 220. The image capture sensors 220 may be any sensor such as a camera or other image capture device configured to capture images from the mobile device 104 and / or record video, and may include a front image capture sensor (e.g., a camera) configured on the same plane as the display screen of the device, and / or a rear image capture sensor (e.g., a camera) on the back of the device (e.g., the side of the device without a display screen). In some embodiments, the mobile device 104 may include both front and rear image capture sensors, and in some embodiments, the mobile device 104 may include only one of the front image capture sensor or the rear image capture sensor. In some embodiments, any number of image capture sensors 220 may be present on the device 200 (e.g., the mobile device 104).

[0077] Determination of the Complete State of a Mobile Device So far, embodiments of system 100 and the apparatus for implementing the embodiments have been generally described. FIGS. 3 and 4A are flowcharts showing operation examples of apparatus 200 according to some embodiments. The operations can be performed by apparatus 200, such as mobile device 104, device integrity verification apparatus 108, shielding detection apparatus 109, and / or internal user apparatus 110.

[0078] FIG. 3 illustrates operations for determining the complete state of a mobile device, such as for registration within the device protection plan of mobile device 104, according to some embodiments. As shown in operation 302, apparatus 200 may include means such as mobile device 104, device integrity verification apparatus 108, processor 212, memory 214, user interface 216, communication interface 218, and / or the like, to receive a device integrity verification request associated with the mobile device. In this regard, the user may access an application (or "app") installed on mobile device 104 or a website hosted by device integrity verification apparatus 108 to request registration in the device protection plan. In some embodiments, the device integrity registration request may be generated by device integrity verification apparatus 108 or internal user apparatus 110 during device and / or user orientation. In this regard, according to one embodiment, the device integrity verification request may include or be accompanied by details regarding the requested insurance certificate and / or scope of compensation (e.g., order), and / or other account information associated with the user, the user's contact information, and / or the like. It will be understood that the device integrity verification request may be generated for purposes other than device orientation in the protection plan.

[0079] Embodiment examples may prompt the user to provide certain personal information, mobile device service provider information, and / or user-provided device information regarding their device, such as via the user interface 216. According to some embodiments, the user may be instructed to use the mobile device 104 to continue the registration process using the device they wish to register. For example, FIGS. 5A, 5B, 5C, and 5D may provide introductory information to the user and may be used to collect at least some data from the user, such as the user's mobile device service provider and / or mobile device information (e.g., manufacturer, model, and / or the like). For example, as shown in FIG. 5A, an introductory message 500 is provided. As shown in FIG. 5B, a prompt 502 for selecting a mobile device service provider and selectable options 504 or eligible mobile device service providers are provided. As shown in FIG. 5C, once a mobile device service provider is selected by the user, a confirmation 510 of the selected mobile device service provider, as well as a link to the details and additional information 512, are provided.

[0080] As shown in operation 304, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or the like, to receive a mobile device identification data object that includes information describing a mobile device, such as the mobile device 104. As described above, the user may be prompted to provide, via the user interface 216, information describing the device that the user wishes to register for protection. In some embodiments, the user may provide such information via a separate interface and / or network, such as by a personal computer or other mobile or stationary computing device. Any such data describing the device and / or its hardware, such as the device type (e.g., manufacturer, model identifier), the International Mobile Equipment Identity (IMEI), and / or the like, may be stored within the mobile device identification data object.

[0081] According to some embodiments, the mobile device identification information may not need to be provided by the user, and the mobile device data object may store the IMEI and / or other mobile device identification information systematically obtained by the website and / or application when the user accesses the website and / or application using the mobile device 104. The mobile device identification data object may thus include information such as the type of device, device model identifier, serial number, and / or the like, or other information used to identify or uniquely identify the device. FIG. 5N (described in more detail below) shows a user interface that enables user input of the device IMEI, but it will be understood that according to some example embodiments, the IMEI may be obtained systematically as described above. Systematically obtaining the mobile device identification information can thus limit or reduce fraud, for example, by preventing the user from entering the IMEI of a stolen, lost, or damaged device.

[0082] The mobile device identification data object may be used to register the device with the device detection plan, such that subsequently, when a claim is made, the consumer can generate device reflection data that matches the data (e.g., IMEI) stored within the mobile device identification data object. For claims regarding a lost or stolen device, the mobile device service provider may use the data (e.g., IMEI) stored within the mobile device identification data object to prevent future network access and / or use of the device.

[0083] In operation 306, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or the like, to cause the mobile device to display a prompt on the mobile device 104 for capturing at least one image of the mobile device using one or more sensors of the mobile device and a reflective surface such as a mirror. FIGS. 5D, 5E, 5F, 5G, 5H, 5I, 5J, 5K, 5L, 5M, 5U, 5V, 5W are examples of user interfaces for guiding a user to capture an image of their mobile device using the front camera, the rear camera, or both. As shown in FIG. 5D, instruction information 514 may be provided to the user to provide an overview of certain steps related to taking a picture of the mobile device. As shown in FIG. 5E, an image capture instruction 516, as well as selectable prompts 518 and 518 indicating to capture images of the front and back (rear) of the device respectively, are provided. In response to the selection of the selectable prompt 518 and / or 518, the processor 212 of the exemplary embodiment activates the image capture sensor 220 associated with the selected prompt 518 or 520 respectively. For example, in the case where prompt 518 is selected to capture an image of the front of the mobile device 104, the processor 212 of the mobile device 104 may activate the front image capture sensor 220. In the case where prompt 520 is selected to take a picture of the back of the mobile device 104, the processor 212 of the mobile device 104 may activate the rear image capture sensor 220. According to some embodiments, any image capture sensor 220 with an image capture function may be utilized to capture an image. As shown in FIG. 5F, the exemplary embodiment may provide an image capture instruction 526, which may be specific to the device type of the mobile device 104. For example, the image capture instruction 526 in FIG. 5F may indicate an instruction to utilize the "volume up" hard key of the mobile device 104 to capture an image. It will be understood that various embodiments may be contemplated, such as utilizing a hard key or a soft key (not shown in FIG. 5F) to capture an image.In one embodiment, the implementation may vary depending on the device type of the mobile device 104.

[0084] As shown in FIG. 5G, a mobile device 104 may provide a security alert 528 to prompt the user to allow or enable a mobile application provided by an example embodiment herein to access or activate an image capture sensor 220, such as a camera. If the user has previously permitted or confirmed a request to enable mobile application access, that message may not be displayed. In any case, as shown in FIG. 5H, if the user permits the mobile application to access the provided image camera 220, the display may reflect a viewfinder 532 to show the image that can be captured.

[0085] In addition to providing a prompt to capture an image, an example embodiment may transition the user interface 216 to display a test pattern to provide improved accuracy in the downstream processing of the captured image, including the device display. For example, the test pattern displayed may be a solid white display screen as shown in FIG. 5H, or other test patterns identified as enabling efficient identification of damage such as cracks and water damage and / or enabling efficient identification of the display portion of the mobile device relative to the bezel.

[0086] Accordingly, the system may instruct the user to hold the mobile device 104 in front of a mirror or other reflective surface and use the mobile device 104, for example, one or more centers of the device (e.g., the image capture sensor 220), to capture an image. Once the image is captured as directed by the user, the captured image may be displayed as the confirmed captured image 528 in FIG. 5I. Accordingly, as shown in FIG. 5J, the captured image 532 may be displayed within the area of the selectable prompt 518 as shown in FIG. 5E, which may be selectable to allow the user to recapture the image. In FIG. 5J, the selectable prompt 520 is displayed in the same manner as the display in FIG. 5E, indicating that the rear photo has not yet been captured.

[0087] When the selectable prompt 520 is selected, the processor 212 may activate the rear image capture sensor 220 of the mobile device 104 and display a viewfinder 538 for the image to be captured, as shown in FIG. 5K. The user may follow the prompt or provide an input to capture the image, and the captured image 542 may be displayed as provided in FIG. 5L. Accordingly, the display shown in FIG. 5M may be updated to reflect the captured images 532 and 542 within the areas of the selectable prompts 518 and 518, respectively. The selectable prompts 518 and 518 may be selected to change, edit, or view the captured images.

[0088] Returning to the description of FIG. 3, at operation 308, in response to the aforementioned image capture operation, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the image capture sensor 220, the communication interface 218, and / or the like to receive at least one image captured by the mobile device. The image may thus be captured by the mobile device 104 and transmitted to the device integrity verification apparatus 108 (e.g., via an application installed on the mobile device and / or the website of the device integrity verification apparatus 108). Additionally or alternatively, the image may be received locally by the mobile device 104 and further processed on the mobile device 104 as described below.

[0089] An identifier generated by the application or website is associated with the image to indicate whether a particular image was submitted as an image of the front or back of the device. The identifier may be received at the mobile device 104 and / or by the device integrity verification apparatus 108 in relation to the received image.

[0090] According to an example embodiment, as indicated by operation 310, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like, to preprocess an image. According to an example embodiment, as will be described in more detail herein, the received image may be trimmed. According to an example embodiment, the received image may be converted or reduced to a predetermined size, such as, for example, 300 pixels × 300 pixels. According to certain example embodiments, some of the operations described herein may be performed using a single-shot detection algorithm, meaning that the complete image (which may be trimmed and resized) is processed as described herein. However, in some embodiments, the image may be split into sections for individual processing according to any of the operations described herein and reconstructed such that the example embodiments utilize their respective data and / or predictions associated with the separate sections.

[0091] As indicated by operation 314, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like, to process at least one image to determine the integrity state of the mobile device. An example of an operation for determining the integrity state of the mobile device will be described below with respect to FIG. 4A, which spans two pages, according to an example embodiment.

[0092] Determining the integrity state of the mobile device may include processing the image through a series of conditions implemented by respective algorithms and / or models. Predictions or results regarding the conditions may indicate the integrity state of the mobile device. For example, the integrity state of the mobile device may include "verified", indicating that the mobile device identification has been confirmed and the mobile device is in an acceptable state for registration to a protection plan. An integrity state of the mobile device of "not verified" may indicate that the device has not yet been verified and / or that any one or more of the conditions required for verification are not satisfied.

[0093] According to some embodiments, the complete state of a mobile device with an "uncertain" selection may indicate that the conditions determined by the example embodiments to be necessary for verification are likely to be met, but that further review should be performed prior to final verification. Accordingly, in one embodiment, the determination of the complete state of the mobile device may be based on predictions made by various models and / or algorithms, and the confidence returned by either a model and / or algorithm indicating the confidence of a prediction. In some embodiments, as described herein, the verification conditions may include detecting the presence and location of the mobile device, detecting occlusions within the mobile device image, as well as other related evaluations. Some embodiments may further evaluate whether the device is or is not insured.

[0094] For simplicity, the operations of FIG. 4A are described with respect to the processing of a single image, but it will be understood that the processing of the front and back images may occur simultaneously or sequentially such that both the front and back of the device are considered in the verification of the device's integrity. According to some embodiments, only one image may need to be processed to verify the integrity of the device. In any case, the "image state" may thus be related to the prediction regarding one image (e.g., the front image or the back image). The "verified" image state may be required for one or more images (e.g., the front and / or the back) for the example embodiments to determine that the complete state of the mobile device is "verified". Such determination is described in further detail below with respect to operations 440, 442, 446, 448, and 450.

[0095] According to some embodiments, the determination of whether certain conditions are satisfied can be implemented with a model(s) trained to make predictions regarding an image and / or other algorithms configured to determine the quality of the image. FIG. 4B provides an example of a hierarchy of model(s) that can be used to implement the operations of FIG. 4A, according to an example embodiment. FIG. 4B shows the flow of data from one trained model to another trained model, according to an example embodiment. Each model, configured on memory 214 and used and / or trained by an example embodiment, such as using processor 212, can include, among others: · a mobile device presence model 486 trained to detect whether a mobile device is present within an image; · a position detection and trimming model 488 trained to detect the position of the mobile device and optionally trim the image; · a cover detection model 490 trained to detect a cover on the mobile device present within the image; · a mobile device front / back identification model 492 trained to determine whether the image reflects the front or back of the device; · a mobile device authenticity model 494 trained to determine whether the image includes the mobile device from which the image was captured; · a occlusion detection model 496 trained to generate a mask used to determine whether an object within the image is occluded; and · a damage detection model 498 trained to detect damage to the mobile device within the image.

[0096] Figure 4B reflects the hierarchy of the models through which the images are supplied according to an embodiment example. If a particular model predicts that an image does not meet a particular condition, the embodiment example may prevent further processing by an additional model. However, if a particular model predicts that an image meets its respective condition(s), the image may have its processing continued by additional models illustrated in the step-by-step architecture of Figure 4B. In this way, the efficiency of the system can be improved, enhanced, and / or maximized as compared to a system that executes the processing of all conditions regardless of other results. It will be understood that the order of the models through which the images flow or are processed may be different from or configured in a modified order from that illustrated in Figure 4B. In some embodiments, any one or more of the models may be executed separately for their intended purpose(s) without requiring each step shown in Figure 4B. In this regard, any of the models described herein, and their respective predictions, may be utilized and / or exploited for other purposes in addition to or instead of determining the state of the image and / or the integrity state of the mobile device.

[0097] Similarly, the order and / or conditions of the operations described with respect to Figure 4A may be changed. For example, an operation identified as consuming fewer resources than others may be processed before one identified as consuming more resources. Additionally or alternatively, if a particular condition that is generally known to result in a low confidence or low accuracy of other predictions is not verified, that particular condition may be intentionally configured to be processed prior to another condition. For example, if an embodiment example does not verify that an image contains a mobile device (operation 400, described below), it may not accurately determine whether the image is of the front or back of the device (operation 406, described below).

[0098] Continuing with the description of FIG. 4A, as shown in operation 400, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the mobile device presence model 486, and / or the like, to determine whether at least one image includes a mobile device. The determination may be made, for example, using the mobile device presence model 486 deployed on the mobile device 104 and / or the device integrity verification apparatus 108. The user interface 216 prompts the user to capture an image of the user's device using a mirror, but the user may submit an image that does not include a mobile device. For example, some users may attempt fraud by taking a photo of a paper model that looks like a mobile device. Other users may accidentally or intentionally capture an image that does not include a mobile device.

[0099] The processor 212 may process the target image using at least one trained model (e.g., a neural network) trained with a plurality of training images each labeled as either including or excluding a mobile device to determine whether the target image includes a mobile device.

[0100] In any case, if an embodiment predicts that at least one image does not include a mobile device, as shown in operation 430, e.g., using the mobile device presence model 486, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or the like, to provide feedback to the user indicating to capture an image of their device and / or to determine the image status as "not verified". The feedback may include sending one or more instructions to the mobile device and / or displaying them on the mobile device.

[0101] In this regard, the user may be given the opportunity to recapture an image for reprocessing and verification. A message such as that displayed in the user interface of FIG. 5R may be provided to the user. Operation 430 is shown as optionally providing user feedback, but according to some embodiments, as a result of certain or all of conditional operations 400, 403, 405, 406, 410, 416, 420, 426 and / or 442, more specific instructions related to particular conditions (e.g., problems with the captured image) that have been processed but have not led to verification of device integrity may be provided to the user. If the user provides a new image(s), the process may return to operation 400 to process the newly captured image.

[0102] In one example embodiment, it will be understood that operation 400 may be performed as a single shot per image, or the image may be subdivided into sections, whereby each individual section may be processed as described herein.

[0103] If an embodiment determines that at least one image includes a mobile device, further processing may follow operation 403. At least a portion of the remaining operations of FIG. 4A are described with reference to the mobile device within the image, or the captured mobile device. Such reference is understood to refer to a processor-driven prediction that the mobile device is likely to be present within the image, and thus the captured mobile device is a suspected mobile device.

[0104] In operation 403, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the position detection and trimming model 488, and / or the like, to determine the position of the mobile device within the image. In this regard, an exemplary embodiment may predict a bounding box, or a lower portion of the image, in which the mobile device is located, such as by using the position detection and trimming model 488 and / or its respective models. If the bounding box has a predetermined relationship (e.g., less than, or below) compared to a threshold minimum ratio of the image (e.g., 25%), the exemplary embodiment may determine that the mobile device 104 was too far from the mirror or other reflective surface when the image was captured (e.g., too far to provide additional processing and prediction regarding the mobile device with a threshold confidence). Thus, the apparatus 200 may determine that the image state is "not verified" by operation 430, and optionally provide feedback to the user to indicate, for example, holding the mobile device 104 closer to the mirror when recapturing the image.

[0105] If it is determined that the bounding box has a different predetermined relationship (e.g., greater than, or above) compared to the threshold minimum ratio of the image, the exemplary embodiment may determine that the mobile device 104 was close enough to the mirror when the image was captured (e.g., close enough to provide additional processing and prediction regarding the mobile device with a threshold confidence), and thus the process may continue.

[0106] In operation 404, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the position detection and trimming model 488, and / or the like, to trim the image such that the area outside the bounding box is removed. The trimmed image may then be resized and reduced to a predetermined size, such as 300 pixels by 300 pixels. The trimmed image, even if in some cases the image is different from the originally captured image that may be trimmed according to an embodiment example, may be referred to as the "image" or "captured image" for the sake of simplicity and to avoid complicating the explanation more than necessary, and may be processed as further described below.

[0107] As indicated by operation 405, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the cover detection model 490, and / or the like, to determine whether there is no cover (s), or a cover, such as a cover that impedes accurately assessing the state of the mobile device 104, within the captured mobile device in the image. The determination may be made using a cover detection model 490 trained to detect covers on the captured mobile device in the image. Embodiment examples may process the image using a model (e.g., the cover detection model 490) trained to predict or detect whether the user has captured an image with a cover on the mobile device. Accordingly, in operation 430, the embodiment example may provide feedback to the user indicating, for example, to remove the cover and recapture the image. The embodiment example may further determine that the image state is "not verified". In this regard, the user may be given an opportunity to recapture the image for reprocessing.

[0108] If the embodiment determines that at least one of the images does not have a cover thereon, further processing may continue at operation 406. As indicated by operation 406, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like to determine whether at least one of the images includes the pointing surface of the mobile device. The "pointing surface" may not necessarily mean the surface pointed to by the user, but may mean the pointing surface systematically indicated in relation to the captured image that may be generated by the application due to the user being separately prompted to capture the front and back of the device.

[0109] The determination may be made, for example, using the mobile device front / back identification model 492 deployed on the mobile device 104 and / or the device integrity verification apparatus 108. The embodiment may run the image through a model (e.g., the mobile device front / back identification model 492) to confirm that the image captures the indicated surface (e.g., the front or back). If the user is determined to be capturing the wrong surface of the device, at operation 430, the embodiment may provide feedback to the user and instruct the user to capture the pointing surface (e.g., the front or back) of the device. The embodiment may further determine the image status as "not verified". In this regard, the user may be given the opportunity to recapture the image for reprocessing.

[0110] If the image is determined to reflect the surface (e.g., front or back) of the indicated device, the process may continue at operation 410. As indicated by operation 410, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the mobile device authenticity model 494, and / or the like, to determine whether at least one image includes a mobile device associated with a mobile device identification data object. In other words, embodiments determine whether at least one image includes the mobile device from which the mobile device was captured. In some cases, a user may attempt fraud by using their mobile device and a mirror to capture an image of a different pristine phone. Embodiments may utilize the mobile device authenticity model 494 to estimate, using the image, the angle relative to the reflective surface of the mobile device, to predict whether the device present in the image is truly the mobile device from which the image was captured, or whether the captured device in the image is a different device.

[0111] As another example, embodiments may generate, based on an image, a prediction of the identification of a suspected mobile device in the image, such as by using the mobile device authenticity model 494. For example, the mobile device authenticity model 494 may predict the manufacturer and / or model of the mobile device, and embodiments may compare the predicted identification of the mobile device with the identification (e.g., IMEI) indicated by the mobile device identification data object, to determine whether the image reflects the characteristics of the device expected based on the mobile device identification data object.

[0112] If it is determined that the mobile device in the image is different from the one from which the image was captured, at operation 430, feedback may optionally be provided to the user to capture an image of their mobile device using the same mobile device 104 (e.g., the mobile device for which a protection plan is desired) for which the device integrity verification request was sent. An example embodiment may further determine the image status as "not verified".

[0113] If an example embodiment determines that the mobile device in the image is indeed the mobile device 104 from which the image was captured, the process may continue at operation 416. As indicated by operation 416, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like, to determine whether the quality of at least one image is sufficient for further processing. According to an embodiment, the blurriness of the image may be determined by the implementation of a Laplacian variance metric.

[0114] Due to various external factors of the environment, and / or the positioning of the mobile device 104 relative to the mirror, and / or the like, some images may be too blurry and may not be further processable to detect occlusion or damage (described in more detail below). Additionally, or alternatively, the images may be too blurry and may not be able to perform other predictions, including those described above, and thus it may be advantageous for an example embodiment to evaluate the image quality and blurriness before performing the operations described herein. In some examples, the image quality may be sufficient for one task but not for another, and thus various image quality verifications may be performed through the process illustrated by FIG. 4A.

[0115] In any case, at operation 430, feedback for recapturing the image can be provided to the user, and may include further guidance on how to position the mobile device 104 relative to the mirror in order to capture an image having sufficient quality for further processing. FIG. 5S provides an example interface that prompts the user to retake the photo because the photo is too blurry. Further instructions may be provided on how to move the mobile device 104 closer to or farther from the mirror to adjust the angle or orientation of the mobile device 104 relative to the mirror. Example embodiments may further determine the image state as "not verified", and the user may be given the opportunity to recapture the image for reprocessing.

[0116] If an example embodiment determines that the image quality is sufficient for processing, the processing may continue at operation 420. As indicated by operation 420, the apparatus 200 may include means such as the mobile device 104, the device integrity verification device 108, the occlusion detection server 109, the processor 212, the memory 214, the occlusion detection model 496, and / or the like, to determine whether there is no occlusion in at least one image or whether the at least one image includes an object that occludes the mobile device 104. To avoid making the flowchart overly complex, operation 420 indicates that there is either no occlusion in the image or there is occlusion. However, as described herein, it will be understood that the degree or amount of occlusion is determined and considered in the determination of whether the image state is set to "not verified" or "verified".

[0117] For example, a user may inadvertently or intentionally cover a portion of the mobile device 104, such as a crack or other damage on the display screen, or other parts of the mobile device 104. Example embodiments may use a occlusion detection model 496 to generate a mask and utilize it in the detection of occlusions such as a blocked corner (e.g., a finger covering a corner of the mobile device) and a concave occlusion (e.g., a finger protruding into a portion of the captured mobile device), as will be described in more detail below.

[0118] Small occlusions that cover the bezel or outer portion of the surface of the mobile device 104 may be tolerated, but large occlusions that obscure important portions of the display screen or other important portions of the device may not be tolerated. If an example embodiment determines that the mobile device 104 is obscured by an object and thus the integrity of the device cannot be verified, the process proceeds to operation 430 to prompt the user to retake the image without the occlusion (e.g., by holding only the edges of the device with a finger and not covering the front or back of the device), and the mobile image status may be determined to be "not verified". Further details regarding the detection of occlusions are provided below in the section titled "Occlusion Detection" with respect to FIG. 6.

[0119] If no occlusion is detected or any such occlusion is small enough so that it does not prevent further processing and analysis of damage or other conditions, the process may proceed to operation 426. As shown in operation 426, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the damage detection model 498, and / or the like to determine whether at least one image indicates that there is no damage to the device or includes damage. Additionally or alternatively, the damage detection model 498 may determine or predict the presence of a particular type of damage such as cracks, water damage, dents, and / or any other damage that prevents the mobile device from being insured or protected. In some embodiments, if the model determines that there is likely damage, a separate model may predict the type of damage. In any case, the embodiments determine whether there is pre-existing damage to the mobile device 104 such that the coverage of the protection plan should be denied. To avoid making the flowchart overly complex, operation 426 indicates whether there is no damage or there is damage in the image. However, as described herein, it will be understood that the degree or amount of damage is considered in determining whether the image state is set to "not verified", "verified", or "uncertain".

[0120] Further details regarding the damage detection model 498 that utilizes training images and model(s) to detect damage to the mobile device are provided below. If damage is detected, in operation 430, the embodiments may provide a response to the user indicating that damage has been detected and / or that a device protection plan cannot be issued. The embodiments may further determine that the image state is "not verified".

[0121] In cases where it is determined that there is no damage to the mobile device 104 and / or the physical state and / or operational parameters of the mobile device 104 are sufficient for insurance purposes, the process may proceed to operation 440. As shown in operation 440, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like, to determine that the image state is "verified". It will be understood that certain operations illustrated in FIG. 4A may not exist in some embodiments. Accordingly, the apparatus 200 may be configured to require any number of verifications and / or conditions described with respect to FIG. 4A, such that a "verified" image state may be determined when all desired (e.g., as desired by a provider) verifications or conditions are performed.

[0122] In some embodiments, one or more models, such as the occlusion detection model 496 and the cover detection model 490, may be executed in parallel. In some embodiments, the output of the image trimming model 488 may be supplied to one or more of the cover detection model 490, the mobile device front / back identification model 492, the mobile device authenticity model 494, the occlusion detection model 496, and / or the damage detection model 498, simultaneously or in any order.

[0123] According to an example embodiment, a "verified" image state may be required for a plurality of images, e.g., a front image and a back image. Therefore, in an example where both the front and back (and / or any other images) should be verified to check the full state of the mobile device, although not shown in FIG. 4A to avoid making the flowchart overly complex, operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440 of FIG. 4A can be repeated separately for each image required by the insurer. For example, an image instructed to capture the front of the device can be processed according to operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440, and an image instructed to capture the back of the device can be processed according to operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440.

[0124] Therefore, as indicated by operation 442, the apparatus 200 can include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like, to determine whether all the required images (e.g., required for the purpose of determining the full state of the mobile device) have an "verified" image state. Specific images (e.g., the front and back) can be pre-configured or set by the device integrity verification apparatus 108 and can be related to the requirements of the provider for registering the device with a protection plan.

[0125] For example, if images of the front and back of a device are required and both images have an "approved" image state, the complete state of the device can also be set to "approved". However, if images of both the front and back of the device are required and only one or neither of the images has an "approved" image state, the complete state of the mobile device should remain null or be set to "not verified" until at least both images have an "approved" image state. For example, as shown in operation 446, apparatus 200 may include means such as mobile device 104, device integrity verification apparatus 108, processor 212, memory 214, and / or the like to determine that the complete state of the device is "not verified". FIG. 5T provides an example of a user interface showing that the front photo is approved but the back photo (e.g., rear photo) may still need to be captured and processed. If one or both of the images are "not verified", embodiments may prompt the user to capture or recapture each image.

[0126] As shown in operation 448, it will be understood that embodiments may be configured to determine the complete state of a mobile device as "verified" based on a first threshold confidence level (which may be configurable), or the first threshold confidence level. For example, determining the complete state of a mobile device as "verified" may require a "verified" state for all necessary images, and may also require a minimum overall or average confidence level for all conditions being evaluated. The first threshold confidence level test may thus be optionally configured and may be configured in various ways. For example, although not illustrated in FIG. 4A, in one embodiment, the threshold confidence level for a particular prediction (e.g., condition) may be created in relation to any of the predictions made in operations 400, 403, 405, 406, 410, 416, 420, and / or 426. For example, some models may be configured to provide not only predictions but also confidence levels that reflect the confidence in the accurate predictions. As such, the threshold confidence level may be required for each condition to be satisfied before proceeding to the next condition. According to one example embodiment, the average confidence level for all conditions may need to be 95% or more in order to set the complete state of the mobile device as "verified". As another example, all conditions may need to have a confidence level of 98% or more in order to set the complete state of the mobile device as "verified".

[0127] In any case, as shown by operation 442, if all necessary images have a "verified" image state and, as shown by operation 448, the first threshold confidence level is satisfied, the complete state of the mobile device may be determined as "verified" as shown by operation 450.

[0128] In this regard, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like, to determine that the complete state of the device is "verified". If the first threshold reliability is not implemented, in some embodiments, operation 448 may be omitted or bypassed, and the verification that all images in operation 442 have an image state of "verified" proceeds to operation 450, and the complete state of the mobile device may be determined to be "verified".

[0129] According to some embodiments, when the complete state of the mobile device is set to "verified", the mobile device 104 may be automatically registered with a protection plan, and confirmation may be provided to the user via the user interface 216. For example, FIG. 5Y provides confirmation that the device is insured. According to some examples, automatic registration and confirmation may be provided in real-time or substantially real-time during a session in which a compensation scope is requested and images are captured by the user.

[0130] Additionally or alternatively, in response to determining that the mobile device is in a "verified" complete state, the device may not necessarily be automatically registered with a protection plan. Instead, it may be transferred to an internal user device 110 for internal review, such as by the mobile device 104 and / or the device integrity verification apparatus 108. In such an example, the embodiment may provide a message, such as those of FIGS. 5O, 5P, 5Q, and / or 5X, to indicate that the image is being submitted for review. Accordingly, if the provider wishes to further internally review the image (e.g., and not provide automatic registration) before registering any mobile device with a protection plan, the embodiment example may nevertheless advantageously remove images predicted to be unacceptable or unverifiable and optionally provide feedback to the user to encourage efficient device registration.

[0131] As another example, as shown in operations 468 and 470, even if the first threshold confidence level is not satisfied at 448, if the second threshold confidence level is satisfied (e.g., 90%), the embodiment may indicate that the full state of the mobile device should be determined as "uncertain" and further review should be performed, such as using the internal user device 110. Accordingly, the embodiment may be configured to "automatically register" devices determined to be of low risk according to the process of FIG. 4A, but the provider may reserve the opportunity to further internally review the image before registering any mobile device with a related image determined to be of high risk. Even further, if the confidence level does not satisfy either the first or the second threshold confidence level, the full state of the mobile device may be determined as "not verified" (446), and related requests for insurance and / or the like may be rejected without further manual review. The embodiment may thus advantageously remove images predicted to be neither acceptable nor verifiable and optionally provide feedback to the user to encourage efficient device registration.

[0132] In any case, the embodiment may be configured to systematically perform any amount, or all, of the set of confirmations and / or verifications, and in some embodiments, it will be understood that the level of systematic confirmation and / or verification may be balanced with internal (e.g., human) review, for example, as required by the provider. Various configurations and thresholds of confidence levels for various stages of the process may be contemplated.

[0133] Regardless of the aforementioned implemented variant(s), an embodiment may provide additional user interface displays, examples of which are described below. In some embodiments, the user interface display of FIG. 5N may be considered optional and may allow for the input, verification, or change of device identification information such as the device IMEI 550. As described above, the IMEI may be detected and / or populated without explicit input by the user, so the user interface display of FIG. 5N is optional. However, in some embodiments, the processor 212 may receive a device identifier via a user interface such as that of FIG. 5N.

[0134] It will be further appreciated that certain updates and / or situations may be provided to the user before, during, or after an operation related to determining the full state of the mobile device. For example, as shown in FIG. 5O, a pending review status message 556 may be displayed. As shown in FIG. 5P, the processor 212 may initiate a notification permission message 560, such as may be generated by the mobile device 104, in response to a mobile application of an example embodiment that enables or attempts to enable notifications on the mobile device 104. In this regard, the user may permit or deny the example embodiment mobile application to send or push notifications. If notifications are permitted for the example embodiment mobile application, the notifications may be provided to the user during various points of the processes and operations described herein.

[0135] In some embodiments, the user may access the example embodiment mobile application to view the situation related to the request. For example, a review status message 564 may provide the situation that a photo has been submitted and is currently under review. In some embodiments, if notifications for the example embodiment mobile application are enabled on the mobile device 104, a notification such as notification 570 of FIG. 5R may be displayed. Notification 570 indicates to the user that it is necessary to retake an image of the back of the mobile device. The user may select the notification and access the mobile application to retake the image.

[0136] Accordingly, in certain embodiments, when accessing the mobile application of the embodiment example, feedback such as the feedback summary 574 and reason 576 of FIG. 5S may be provided to the user. For example, the feedback summary 574 indicates that it is necessary to retake the image of the back, and the reason 576 indicates that it is necessary to retake the image because the photo is too blurry.

[0137] In certain embodiments, as shown in FIG. 5T, a message such as the image approved message 580 may be displayed within the area of the selectable prompt 518 of FIG. 5J. As shown in FIG. 5T, the selectable prompt 520 may not have been input with a message because the back image has not yet been captured. Accordingly, the user may select to capture an image, and the user interface displays of FIGS. 5U and 5V may be updated to provide the viewfinder 538 and the captured image 542, respectively.

[0138] Accordingly, as shown in FIG. 5W, the image approved message 580 is displayed for one image, such as an image of the front of the device, while the captured image 542 is displayed for another image, such as an image of the back of the device. In this regard, since each image has not yet been approved, a selectable prompt 520 is displayed to enable recapture or editing of the captured image 142. In certain embodiments, when the selectable prompt 520 is selected, a review status message 564, such as that of FIG. 5X, may be displayed.

[0139] Any of the user interface displays provided herein is updated along with the state, such as the image state, and / or the complete state of the mobile device is updated and / or launched as described with respect to FIG. 4A. It will be understood that in certain embodiments, the complete state of the mobile device, such as "not verified" or "uncertain", may be shown to the user as a "pending" state, and a state such as "verified" may be shown to the user as "verified". Accordingly, the complete state of the mobile device is set to "uncertain", but the associated image may be placed in a queue for manual review and / or the like.

[0140] Furthermore, according to certain embodiments, when notifications are permitted on mobile device 104, the exemplary mobile application may initiate a notification 590 when the complete state of the mobile device is determined to be "verified". Thus, the user may be notified that the image has been approved and / or that they can or are registered for a protection plan for their mobile device.

[0141] Occlusion Detection FIG. 6 is a flowchart of an operation for detecting occlusion, such as by an occlusion detection device 109, according to an exemplary embodiment. The operations of FIG. 6 may be triggered by operation 420 and, in other examples, may be executed as a separate process not necessarily related to an image of the mobile device. The operations of FIG. 6 may utilize or provide an image segmentation approach for detecting occlusion.

[0142] In operation 600, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, the occlusion detection model 496, and / or the like to generate a mask that contains a smaller number of colors compared to the number of colors in the original image. According to an example embodiment as described herein, although the mask may be described in relation to the mobile device within the image, it will be understood that mask generation may be performed on any object detected or present within the original image. For an example of mask generation for a mobile device (such as for the purpose of determining the integrity state of the mobile device), the original image may be considered an image captured by the mobile device, and the generated mask may be considered a mobile device mask.

[0143] The mask may be considered an additional image generated from the processing of the original image (e.g., an image of the mobile device captured by the mobile device, which may have been previously trimmed according to the position detection and trimming model 488). The mask may be an image with a reduced number of colors compared to the original image. For example, FIGS. 7A and 8A are examples of images of a mobile device captured by a user and containing a wide range of colors, and FIGS. 7B and 8B are masks generated according to an example embodiment and containing binary values (represented as white and black pixels in FIGS. 7B and 8B), respectively. However, it will be understood that other configurations of colors may be selected to generate the mask. An example process for generating the mask will be described in more detail below with respect to the configuration, training, and deployment of the occlusion detection model 496. It will be understood that separate occlusion detection models 496 and / or their models may be used for the front and the back of the device.

[0144] According to an example embodiment, the model may return an array of values indicating whether a particular pixel should belong to the mask. The example embodiment may then determine, based on a threshold, whether the pixel should be made white (e.g., included within the mask) or black (e.g., not included within the mask).

[0145] As shown by operation 602, apparatus 200 may include means such as mobile device 104, device integrity verification apparatus 108, processor 212, memory 214, and / or the like to extract the polygonal partial region P of the mask. Accordingly, an embodiment example may apply an algorithm such as a marching square algorithm to extract the maximum polygonal partial region of the mask. In some embodiments, the maximum polygonal partial region P may be considered as the mobile device screen when the original image is generated for the image trimming model 488 and other related models and the preprocessing steps in which the mobile device is detected and the image is substantially trimmed before generating the aforementioned mask for the occlusion detection model 496. Accordingly, small islands (including, for example, small outlier polygons, black pixels that appear within a mostly white portion such as may be caused by camera defects, dust / stains, and / or other minor environmental factors present on the device or mirror, and / or the like) may be removed.

[0146] As shown by operation 604, apparatus 200 may include means such as mobile device 104, device integrity verification apparatus 108, processor 212, memory 214, and / or the like to determine the convex hull of P. The convex hull may be calculated according to generally known computational geometry algorithms.

[0147] Using the polygonal partial region P together with the convex hull of P enables an embodiment example to identify concave occlusions such as the concave occlusion 700 of FIGS. 7A and 7B. The sub-process for identifying concave occlusions is provided by operations 608, 612, 616, and 620. Additionally or alternatively, an embodiment example may use the polygonal partial region P together with the convex hull of P to identify occluded corners such as the occluded corner 800 of FIGS. 8A and 8B. The sub-process for identifying occluded corners is provided by operations 630, 634, 638, and 642. According to an embodiment example, both sub-processes or one of the sub-processes may be implemented and executed.

[0148] As shown by operation 608, apparatus 200 may include means such as mobile device 104, device integrity verification device 108, processor 212, memory 214, and / or the like, to calculate the difference between P and the convex hull. At operation 612, apparatus 200 may include means such as mobile device 104, device integrity verification device 108, processor 212, memory 214, and / or the like, to reduce or remove thin discrepancies at the edges of P and the convex hull. Example embodiments may reduce or remove the discrepancies by performing pixel erosion and pixel dilation techniques, such as may be provided by Shapely and / or other libraries. At operation 616, apparatus 200 may include means such as mobile device 104, device integrity verification device 108, processor 212, memory 214, and / or the like, to recalculate P as the largest area of the remaining region. In this regard, P may be identified as the largest area of the remaining connected regions of P. P may thus be considered as the estimated area of the visible screen (e.g., the unobscured portion of the screen).

[0149] In operation 620, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like to determine the indentation as the difference between P and the convex hull. In some examples, any such indentation may be compared to a threshold for further filtering, so that very small indentations are not necessarily flagged as such, but large indentations that may be obtrusive for other downstream tasks (such as determining whether there is damage on the device) may be flagged so that the user is prompted to recapture the image. For example, if a particular detected indentation is greater than, or equal to, or greater than a predetermined threshold size, that area may remain predicted to be an indentation. If the area initially identified as an indentation is less than, or equal to, or less than a predetermined threshold size, that area may be ignored as an indentation. In this regard, if an embodiment predicts that an indentation exists (and / or the indentation is large enough to be obtrusive for downstream tasks as indicated by using a threshold), operation 420 may determine that the image includes occlusion and cause the embodiment to determine the image state as "not verified" and optionally prompt the user to recapture the image.

[0150] In operation 630, the apparatus 200 may include means such as the mobile device 104, the device integrity verification apparatus 108, the processor 212, the memory 214, and / or the like to determine a predetermined number of major edges of the convex hull. In an example of a mobile device mask, an embodiment may determine four major edges and, according to some embodiments, may identify the four most major edges. In this regard, the number of major edges identified may be based on the type of object for which the mask is created.

[0151] The Huff transform feature extraction technique is implemented to identify the major edges or a predetermined number of the most major edges, and thus can predict where the edges of the mobile device should appear in the image (for example, even if the edges are occluded, assume they are not occluded). As shown by operation 634, the apparatus 200 may include means such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, to identify the intersection points of adjacent edges (identified based on their respective angles) to identify the protruding corner points of the mobile device in the image (which may or may not be occluded). At operation 638, the apparatus 200 may include means such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, to determine the distance from each protruding corner to P. For example, determining the distance may include measuring the shortest distance from the estimated corner to the closest edge or point of P.

[0152] At operation 642, the apparatus 200 may include means such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or the like, to compare each distance with a threshold to determine whether any of the corners are blocked. For example, if the distance is greater than, or greater than or equal to, a predetermined threshold distance, the embodiments may determine that the respective corner is blocked. In this regard, when used to determine the integrity state of the mobile device, operation 420 may determine that the image includes occlusion, cause the embodiments to determine that the image state is "not verified", and optionally prompt the user to recapture the image. If there is no calculated distance greater than, or greater than or equal to, the predetermined threshold distance, the embodiments may determine that none of the corners of the object are blocked in the original image.

[0153] Although not shown in FIG. 6, in one example embodiment where the image is processed for both concave occlusion and occluded corners and neither is detected, in accordance with the operations described above, operation 420 may determine that there is no occlusion of the mobile device within the image, i.e., no occlusion that can affect subsequent processing of the image. Thus, small occlusions that do not affect subsequent processing may be tolerated.

[0154] Model Configuration, Training, and Deployment Training of the model(s) (e.g., neural network(s)) utilized by an example embodiment may occur prior to deployment of the model (e.g., prior to use by mobile device 104 and / or device integrity verification device 108 in determining whether a device is eligible for compensation under a plan, and / or prior to use by occlusion detection device 109 in determining whether an object in an image is occluded). According to some example embodiments, training may be continuously performed by receiving images that have been verified by either the example embodiment and / or a human reviewer, along with associated classifications and / or labels. Machine learning may be used to develop a particular pattern recognition algorithm (i.e., an algorithm representing a particular parameter recognition problem) based on statistical inference and to train the model(s) based thereon.

[0155] Example embodiments receive and store multiple types of data, including a dataset, using a communication interface 218, memory 214, and / or the like, and use that data in multiple ways, such as using a processor 212. Device integrity verification device 108 may receive a dataset from a computing device. The dataset may be stored in memory 214 and utilized for various purposes. The dataset may thus be used in modeling, machine learning, and artificial intelligence (AI). Machine learning and related artificial intelligence may be performed by device integrity verification device 108 based on various modeling techniques.

[0156] For example, a set of clusters can be developed by unsupervised learning, where the number and size of each cluster are based on the calculation of the similarity of the features of the patterns within a previously collected training set of patterns. In another example, a classifier representing a particular categorization problem or task can be developed using supervised learning based on a training set of patterns and their respective known categorizations. Each training pattern is input into the classifier, and the difference between the output categorization generated by the classifier and the known categorization is used to adjust the classifier coefficients to more accurately represent the problem. A classifier developed using supervised learning is also known as a trainable classifier.

[0157] In some embodiments, the dataset analysis takes as input a source-specific representation of a dataset received from a particular source, and generates an output that categorizes the input as likely or unlikely to include relevant data references (e.g., likely or unlikely to meet the required criteria), including a source-specific classifier. In some embodiments, the source-specific classifier is a trainable classifier that can be optimized for more instances of the dataset since the analysis is received from a particular source.

[0158] Alternatively or additionally, the trained model can be trained to extract one or more features from historical data using pattern recognition, among other computational intelligence algorithms that can use an interactive process to extract patterns from data, based on, inter alia, unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, correlation rule learning, Bayesian learning, solutions for probabilistic graphical models. In some examples, the historical data can include data generated using user input, cloud-based input, or the like (e.g., user confirmation).

[0159] The model(s) can be initialized with multiple nodes. In some embodiments, an existing deep learning framework can be used to initialize the model(s). The model(s) can be implemented as a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) network, and / or the like. According to an example embodiment, any of the models described herein can utilize an existing or pre-trained model, and such a model can be further trained with training data specific to each respective task and / or condition described herein. For example, the device integrity verification apparatus 108 can develop a template, such as by using any known or modified machine learning templating technique. In this regard, a templated model for each respective task and / or condition described herein can be utilized by an embodiment to further train each model. In this regard, an embodiment can utilize a template having a domain-specific data scheme and model. For example, a template design for identifying a certain texture in an image can be utilized for cover detection and / or damage prediction. A certain template for identifying whether a particular object exists in an image can be utilized for mobile device presence detection.

[0160] According to an embodiment, CNNs and related deep learning algorithms can be particularly useful in the application of machine learning to image processing by generating multiple combined layers of perceptrons. Each layer can be coupled to its adjacent layer, which provides an efficient basis for measuring the weights of a loss function to identify patterns in the data. Accordingly, the machine learning algorithm of an embodiment can train a related model, such as a CNN, to learn which features are important in an image, resulting in accurate predictions regarding the image efficiently.

[0161] Using the techniques described herein, the model can then be trained to determine one or more features of an image and generate one or more predictions associated with the methods and embodiments described herein. The training data can also be selected from a predetermined period, such as several days, weeks, or months prior to today.

[0162] In an example embodiment, a labeled dataset, such as one associated with a particular task or prediction described herein, can be fed into the device integrity verification apparatus 108 to train the model(s). The model(s) can then be trained to identify and classify subsequent received images received from the computing device corresponding to one or more of the labeled references.

[0163] In some embodiments, the AI and models described herein use deep learning modules. Deep learning is a subset of machine learning that generates a model based on a training dataset provided thereto. A deep learning network can be used to draw in large-scale inputs and have the algorithm learn which inputs are relevant. In some embodiments, the training model can use unsupervised learning techniques including clustering, anomaly detection, Hebbian learning, as well as learning latent variable models such as the expectation maximization algorithm, moment methods (mean, covariance), and blind signal separation techniques including principal component analysis, independent component analysis, non-negative matrix factorization, and singular value decomposition.

[0164] Accordingly, an example embodiment can input a plurality of training images and corresponding labels into an initialized model for training or further training the model(s) using them, and learn features via supervised or unsupervised deep learning using the processor 212.

[0165] Regarding this, the training images, some of which include the photographed mobile device and some of which do not, are input into each model together with associated labels indicating various characteristics according to the specific model being trained. The training set may include hundreds or thousands of images that have been reviewed and labeled by a user or data scientist (e.g., using the internal user device 110).

[0166] The model can convert the image into a tabular representation of the image and process the image together with its identified labels (e.g., "including a mobile device", "not including a mobile device") to learn the features of the image that are correlated with those labels. In one example, an image can have multiple labels for each state, so that one image can be used to train multiple different models and / or neural networks. For example, one image can be labeled as "including a mobile device", "including a cover", and "including damage", so that one image is used by the processor 212 of the embodiment example to train three separate models such as a mobile device presence model, a cover detection model, and a damage detection model, respectively. In one example, an image can be used to train one model. Regarding this, training data can be collected and used in various ways.

[0167] Processor 212 trains the model(s) with training images and adjusts their respective parameters to reconstruct the tabular representation of the images through a series of deep learning iterations to capture their features or to place more emphasis on those features that are strong indicators of a particular label or classification. Various techniques, such as but not limited to fractal dimension, can be utilized during the training of the model, and the fractal dimension is a statistical analysis that can be employed by a machine learning algorithm to detect which features are stronger indicators of a certain prediction and / or condition, such as those described herein, at which scale. The scaling of the training images can be adjusted according to the fractal dimension technique, which can vary depending on the particular task or prediction being made. For example, the detection of damage, such as flooding and / or cracks, using a machine learning algorithm may require a higher resolution image than that required for the detection of whether a cover is present on the device. In this regard, the fractal dimension algorithm can be utilized to adjust the resolution of the image to achieve a balance between the accuracy and efficiency of one model or each model.

[0168] Additional details regarding the configuration, training, and deployment of each model(s) for their respective tasks, conditions, and / or methods associated with the exemplary embodiments are described below. It will be further understood that some models may employ other classification techniques, such as but not limited to support vector machines, decision trees, random forests, naive Bayes classifiers, and logistic regression, instead of or in addition to neural networks.

[0169] The processor 212 of one embodiment may advantageously use separate models to make separate predictions regarding the various conditions described herein. However, it will be further understood that one embodiment may utilize one model (e.g., a neural network) to evaluate the overall state of a mobile device as "verified", "not verified", and / or "uncertain". In this regard, the model (e.g., a neural network) may be trained using images labeled as "verified", "not verified", and / or "uncertain", and the model may essentially evaluate which images contain a mobile device, which are front or back, which contain damage or not, and determine which images should be predicted as "verified", "not verified", and / or "uncertain". However, the use of a single model may require more training data to produce accurate or meaningful results compared to using separate models for at least some of the conditions described herein. At least one additional advantage of using separate models and optionally generating respective confidence levels is to enable the provision of specific feedback to the user capturing the image, such as "Please bring the device closer to the mirror", "Please remove the cover of your device", "Please place your finger on the edge of the device and retake the photo", and / or the like. Such feedback may result in an improved or enhanced automatic verification rate while reducing the uncertainty or the need for manual review.

[0170] Mobile device presence model The Mobile Device Presence Model 486 enables embodiments to automatically predict (e.g., without human review) whether a newly received image contains a mobile device or not. A processor 212 of an embodiment, such as the apparatus 200, may pre-train the model using an existing model such as the Torchvision implementation of Squeezenet and using weights established by a visual database such as ImageNet. Embodiments may further train a model for the mobile device presence detection task by inputting model training images and corresponding labels such as "containing a device" and "not containing a device". In this regard, the model (e.g., a neural network) may be trained with at least two sets of images, such as a first set of training images containing a mobile device and a second set of training images not containing a mobile device.

[0171] Various deep learning methods may then be used, according to embodiments, to process the training images and corresponding labels through the model to train the model to generate predictions regarding later received images. According to embodiments, once deployed, the trained Mobile Device Presence Model 486 may generate an indication of the likelihood of an image containing a mobile device. For example, the indication may be a number between 0 and 1, and a number close to 1 indicates a high likelihood of the presence of a mobile device. Thus, the indication may reflect the confidence of the prediction.

[0172] Accordingly, a processor 212 of an embodiment may process a target image using at least one model trained with a plurality of training images each labeled as either including or excluding a mobile device to determine whether the target image contains a mobile device.

[0173] In the context of determining the complete state or image state of a mobile device, an embodiment may utilize a mobile device presence model 486 trained in the execution of operation 400. The embodiment may further implement configurable or predetermined quantifiable requirements to indicate a confidence level that needs to be satisfied for a prediction to be approved (e.g., no further internal review is required).

[0174] Position Detection and Trimming Model The position detection and trimming model 488 enables an embodiment to predict where in an image a particular object, such as a mobile device, is located and further determines whether, when the image was captured, the object was not too far from the image capture sensor (220) or, in the case of an example of a reflective surface, not too far from a mirror. The position detection and trimming model 488 further enables trimming of the image accordingly. An embodiment may utilize an existing framework to further train a pre-trained model. For example, according to an embodiment, the object detection framework of TensorFlow (registered trademark) may be utilized to train a network using a Mobilenet backend pre-trained with respect to the COCO (Common Computing) dataset.

[0175] Training images in which a reviewer has located the outline of a mobile device present in an image can be input into the model for further training, along with the respective labels of those located outlines. Thus, the model can be trained to predict a bounding box defined as (xmin, xmax, ymin, ymax) for an image in which an object, such as a mobile device, is likely to be located. The model is further trained to generate an indicator, such as a number between 0 and 1, to indicate that the bounding box is likely to contain the mobile device. For example, a number closer to 1 may indicate that the bounding box is more likely to contain the mobile device than a number closer to 0. Thus, the output of the position detection and trimming model 488 can indicate the confidence level of a bounding box that accurately captures the position of an object, such as a mobile device, in the image.

[0176] Accordingly, as described with respect to operation 403, once deployed, the position detection and trimming model 488 may predict the proximity of the mobile device to the mirror when the image is captured, enabling feedback to optionally be provided to the user.

[0177] According to an embodiment, a careful edge detection algorithm may also be used to estimate the bounding box of the mobile device within the image. For example, careful edge detection may utilize a Gaussian filter to smooth the image, determine the intensity gradient, and predict the strongest edges within the image. The bounding box may then be predicted accordingly.

[0178] The processor 212 of the exemplary embodiment may process the target image using at least one model trained with a plurality of training images each associated with a bounding box indicating the position of the mobile device within the image to determine the position of the mobile device within the target image. The exemplary embodiment may further determine a confidence level indicating the accuracy of the position of the mobile device within the image.

[0179] In this regard, if the confidence level does not meet the threshold, the image may remain "unverified" and thus may be subject to further manual review. Additionally or alternatively, as described with respect to operation 404, if a certain threshold confidence level is met, the image may be trimmed according to the predicted bounding box. It will be understood that in some exemplary embodiments, meeting the threshold confidence level may be optional. In embodiments that do not utilize a threshold confidence level, any or all images for which a bounding box is calculated may be trimmed accordingly.

[0180] Cover Detection Model The cover detection model 490 enables embodiments to predict whether a user is capturing an image of a mobile device with a cover attached. Existing models can be further trained, such as using the image and respective labels indicating whether the image includes a mobile device (e.g., a mobile device with a case) having a cover attached thereto. Embodiments can utilize an existing image processing framework and further train the model with training images and labels. Thus, embodiments can train the model using the processor 212 to place more emphasis on certain features, such as those related to texture, which is a strong indicator of whether the mobile device in the captured image has a cover thereon. In this regard, the texture can be determined to be a strong indicator of whether the mobile device in the image, learned by the model, includes a cover. The processor 212 of the embodiments can thus process the target image of the mobile device using at least one model trained with a plurality of training images of mobile devices labeled as including a cover on each mobile device or excluding the cover on each mobile device, to determine whether the target image includes a cover on the target mobile device. Accordingly, the deployed cover detection model 490 can enable the processor 212 of the apparatus 200 to provide a prediction related to the newly captured image and optionally provide feedback to the user to remove the cover and recapture the image as provided in operation 405.

[0181] Mobile Device Front / Back Identification Model The mobile device front / back identification model 492 enables embodiments to predict whether a user is capturing the front or back of their device. Existing models can be further trained using images and their respective labels indicating whether the image provides a view of the front or back of the device. Embodiments can utilize an existing image processing framework and further train the model using training images and labels. Thus, embodiments can train the model to identify important features that may be unique to one side of the device. For example, deformations in the pixels associated with the display screen surrounded by the bezel may indicate the front of the device.

[0182] In this regard, the processor 212 of the device 200 processes a target image of the mobile device using at least one trained model trained with a plurality of training images of the mobile device, each labeled as including the front of the respective mobile device or including the back of the respective mobile device, to determine whether the target image includes the front or the back of the target mobile device.

[0183] Accordingly, the mobile device front / back identification model 490 provides a prediction as to whether the user is accurately capturing the front and / or back of the mobile device in relation to a newly captured image, and optionally provides feedback to the user to capture the indicated side of the device, as provided in operation 406. Additionally or alternatively, embodiments can determine whether a particular image is an image of the front or back of the device based on data identifying which of the front or back cameras captured the image (e.g., which of front or back image capture 220 was used to capture the image).

[0184] According to some example embodiments, the cover detection model 490 and the mobile device front / back identification model 492 can be implemented as a single model. For example, since it may be convenient to reject or remove any image that includes a cover, the training images input into the model can include labels of "cover", "front", or "back", and thus the "cover" label should be used for any training image that includes a cover. Therefore, the example embodiments can reject or set the image state to "not verified" in any scenario where the image is predicted to include a cover.

[0185] Mobile device authenticity model The mobile device authenticity model 494 enables example embodiments to predict whether an image includes the same mobile device on which the image was captured. An existing model can be utilized and further trained by the processor 212, for example, using the images and their respective labels indicating whether the image includes the mobile device on which the image was captured (e.g., "same device") or another device (e.g., "different device"). For example, an internal user or data scientist can use a mobile device to capture images of both the mobile device on which the image is being captured and other devices, and label the images accordingly. The example embodiments can thus train the model using the training images and labels such that the example embodiments detect the edges of the mobile device in the image, measure the edges, and learn to predict or estimate the angle at which the mobile device is held relative to the mirror.

[0186] Therefore, the processor 212 in the embodiment example can process the target image of the mobile device using at least one model trained with a plurality of training images of the mobile device, where each training image is labeled as being captured by each mobile device included in the image or by a device different from each mobile device included in the image, and determine whether the target mobile device included in the target image was captured by the target mobile device or by a different device.

[0187] In one embodiment, this process of determining the authenticity of the mobile device can further utilize the bounding boxes drawn by the user during the labeling of the training images. In any case, further predictions can be made based on the angle to indicate whether the mobile device in the image is the same device as the one by which the image was captured.

[0188] Occlusion detection model As introduced above, a mask such as the one used to detect occlusion can be generated by a trained model. In this regard, the embodiment example can utilize an existing model such as the UNet architecture to train the model from scratch using a manually created mask. In this regard, a data scientist or other internal user can review the image and manually trace the outline that includes the exposed area of the object of interest (e.g., the mobile device) but does not include the occluding object (e.g., a finger) and / or the objects visible in the background, or input it in the form of a mask (e.g., the outline reflecting the mask examples in FIGS. 7B and 8B). In this regard, each pixel of the training image (which can be resized and / or trimmed to a predetermined size such as 300 pixels × 300 pixels) can be labeled to have a relevant indicator regarding whether each pixel belongs to the mask. The model can then be trained with respect to the image and the label.

[0189] Accordingly, the deployed and trained model may take the input image (e.g., the resized and trimmed image), process the image using the processor 212, and provide a prediction of the mask in the form of an array of a predetermined size of indicators (e.g., 300×300) indicating whether each pixel belongs to the mask. For example, the array may include numerical values ranging from 0 to 1, and values closer to 1 correspond to pixels that are more likely to belong to the mobile device (and thus are considered to be included within the mask) compared to values closer to 0. Accordingly, the mask predicted by the model may be utilized by an embodiment as described with respect to FIG. 6 to determine whether the shielding of the mobile device 104 is present within the image. According to one embodiment, one model and training set may be used for the mask of the image of the front of the device, and a separate model and training set may be used for the mask of the image of the back of the device.

[0190] Using a model (e.g., a neural network) to determine the mask may provide advantages that may not be provided by other image processing techniques. The model is often useful in contextual prediction that is done by humans and not possible with conventional computer algorithms. For example, the color within the screen is not uniform, and some of the same color may appear within pixels outside the device in the image. A model such as a neural network can make such a distinction, while conventional color detection or other image processing algorithms may not accurately distinguish between pixels away from the image and pixels not away from the image that otherwise have the same or similar color.

[0191] Damage detection model The damage detection model 498 enables an embodiment to predict whether a captured mobile device in an image is damaged (e.g., the complete state of the mobile device should be set as "not verified" to the extent that the damaged device is not eligible for compensation under the device protection plan, etc.). An existing model can be used, and the processor 212 can be further trained using, for example, the image and each label indicating whether the image contains a damaged mobile device, including but not limited to cracks, water damage, dents, scratches, and / or any other damage that prevents the mobile device from being insured or protected. According to an embodiment, a special user interface tool can be used by a reviewer or data scientist to zoom in on an image of a mobile device to determine whether there are cracks or other damage and label the training image accordingly. In one embodiment, binary labels such as "damaged" or "not damaged" can be applied to the training image. As another example, a reviewer can score the level of damage, so that small or seemingly minor cracks can be given a relatively low score compared to an image showing a more significant crack that can affect functionality. Any variation or scoring of damage labeling can be contemplated. As yet another example, a certain damage can be specifically labeled as "crack", "water damage", or any other type of damage that can affect the insurability of the mobile device.

[0192] In one embodiment, the first label of a training image of a mobile device can indicate "damaged", and the second label can indicate the type of damage. In this regard, one model can be trained to predict whether damage exists or not. A separate model and / or models can predict the specific type of identified damage, such as a crack, water damage, or dent, if damage is predicted to exist. For example, one model can be trained to detect only water damage based on training images of water damage, and the same logic can be applied to other types of damage to the device and any other visibly detectable conditions.

[0193] The exemplary embodiments can utilize an existing image processing framework and further train the model using training images and labels. Therefore, the exemplary embodiments may place more emphasis on certain features, such as those related to texture and / or color changes, which are strong indicators of damage and / or the severity of a particular type of damage.

[0194] In this regard, the apparatus 200 includes means, such as a processor 212, to process a target image using at least one model trained with a plurality of training images of a mobile device, where each training image is labeled with a damage assessment, to calculate a damage assessment of the target mobile device within the target image. In this regard, the damage assessment may include "no damage", "minor damage", "extensive damage", and / or the like. In one embodiment, the damage assessment may include a quantifiable assessment, such as on a scale of 1 to 10, where 1 indicates no damage detected and 10 indicates extensive or severe damage. The deployed damage detection model(s) 498 can thus provide a prediction regarding whether the device is damaged, the extent of the damage, and / or the type of damage.

[0195] In scenarios where damage is detected and optionally depends on the type and / or extent of the damage, the image status is "not verified", resulting in a "not verified" or "uncertain" full device status of the mobile device. As another example, a quantitative damage score can be generated.

[0196] It will be understood that additional or alternative separate damage detection models can be configured, trained, and deployed for each of the front, back, and / or bezel of the mobile device. The bezel can be recognized as part of the device so that damage to the bezel can also be detected.

[0197] Conclusion As described herein, embodiments of the present disclosure provide a technical advantage over alternative embodiments. Embodiments may be implemented to consume fewer processing resources, which otherwise could be wasted submitting all images captured by a user to a server for storage, potential review, and further processing.

[0198] In this regard, certain operations, such as any of the operations of FIGS. 3, 4A, and / or 6, may be executed on the mobile device 104, while other operations may be executed on the device integrity apparatus 108 and / or the occlusion detection apparatus 109. Thus, embodiments may provide resource efficiency by strategically balancing such operations. For example, some initial image processing operations may be executed on the mobile device 104 before the image is sent to the device integrity apparatus 108 and / or the occlusion detection apparatus 109. Thus, an image may be removed on the mobile device 104 before being processed by other, potentially more resource-intensive processes.

[0199] For example, some embodiments may employ a model, such as a neural network configured to operate on a mobile device, such as the mobile device 104. For example, TensorFlow Lite, and / or other frameworks designed to be deployed on a mobile device may be utilized according to embodiments.

[0200] Thus, for example, embodiments may provide real-time verification on the device in certain scenarios, such as when a high-confidence image state or mobile device integrity state is determined. Otherwise, image and / or device integrity verification requests may be sent to the device integrity apparatus 108 for review by an agent. On the server side, an algorithm similar to that implemented on the device side may be used to facilitate image review and / or process inputs from the review as a means for further calibration of the algorithm and / or training of the model.

[0201] FIG. 3, FIG. 4A, and FIG. 6 each illustrate a flowchart of a system, method, and computer program product according to some embodiments. It will be understood that each block of the flowchart, and combinations of blocks in the flowchart, can be implemented by various means, such as hardware and / or a computer program product including one or more computer-readable media having computer-readable program instructions stored thereon. For example, one or more of the procedures described herein can be embodied by computer program instructions of a computer program product. In this regard, a computer program product (s) embodying the procedures described herein can include one or more memory devices (e.g., memory 214) of a computing device that stores instructions executable by a processor in the computing device (e.g., by processor 212). In some embodiments, the computer program instructions of a computer program product (s) embodying the foregoing procedures can be stored by the memory devices of a plurality of computing devices. As will be understood, any such computer program product can be loaded onto a computer or other programmable apparatus (e.g., mobile device support device 102, mobile device 104, and / or other devices) to create a machine, and thus a computer program product including instructions for execution on the computer or other programmable apparatus creates means for implementing the functions specified in the flowchart block (s). Further, the computer program product can include one or more computer-readable memories on which computer program instructions can be stored such that the one or more computer-readable memories can direct a computer or other programmable apparatus to function in a particular manner, and thus the computer program product can include a product for implementing the functions specified in the flowchart block (s).Computer program instructions of one or more computer program products are also loaded onto a computer or other programmable apparatus (e.g., mobile device 104 and / or other apparatus), and a series of operations are executed on the computer or other programmable apparatus to generate a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus can implement the functions specified within the flowchart block(s).

[0202] Accordingly, the blocks of the flowchart support combinations of means for performing the specified functions and combinations of operations for performing the specified functions. It will also be understood that one or more blocks of the flowchart, and combinations of blocks within the flowchart, can be implemented by a dedicated hardware-based computer system for performing the specified functions, or by combinations of dedicated hardware and computer instructions.

[0203] Many embodiments of the subject matter described herein may include all, a portion, or a combination of portions of the systems, apparatus, methods, and / or computer program products described herein. The subject matter described herein includes, but is not limited to, the following specific embodiments: 1. A method comprising: receiving a device integrity verification requirement associated with a mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying, on the mobile device, a prompt to capture at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; receiving at least one image captured by one or more image sensors of the mobile device; and processing the at least one image using at least one trained model to determine a full state of the mobile device.

[0204] 2. The method of Embodiment 1, wherein processing at least one image to determine the perfect state of a mobile device comprises: determining whether there is damage to the mobile device using at least one trained model, and in response to determining that there is damage to the mobile device, determining that the perfect state of the mobile device has not been verified.

[0205] 3. The method of Embodiment 1, wherein processing at least one image to determine the perfect state of a mobile device comprises: determining the angle of the mobile device with respect to the reflecting surface when at least one image is captured, and based on that angle, determining that at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object.

[0206] 4. The method of Embodiment 1, wherein processing at least one image to determine the perfect state of a mobile device comprises: determining whether at least one image includes the mobile device associated with the mobile device identification data object.

[0207] 5. The method of Embodiment 4, wherein determining whether at least one image includes the mobile device comprises: identifying a suspected mobile device in at least one image, generating a prediction of the identification of at least one suspected mobile device and comparing the mobile device identification data object with the prediction of the identification of at least one suspected mobile device to determine whether the suspected mobile device is that mobile device, and if it is determined that the suspected mobile device is that mobile device, determining that the perfect state of the mobile device has been verified.

[0208] 6. The method of Embodiment 1, wherein the complete state of the mobile device is determined to be uncertain, and the method further includes sending a device integrity verification request and at least one image to an internal user device for internal review.

[0209] 7. The method of Embodiment 3, wherein in response to a determination that, based on an angle, at least one image captures different mobile devices, (a) displaying a message on the mobile device instructing the user to recapture the mobile device, and (b) further determining that the complete state of the mobile device has not been verified.

[0210] 8. The method of Embodiment 1, wherein determining the complete state of the mobile device by processing at least one image includes determining the position of the mobile device within at least one image, the position being defined as a bounding box, and when the bounding box has a first predetermined relationship with a threshold ratio of at least one image, displaying a message on the mobile device instructing the user to bring the mobile device closer to a reflective surface.

[0211] 9. The method of Embodiment 8, wherein when the bounding box has a second predetermined relationship with a threshold ratio of at least one image, further trimming at least one image according to the bounding box.

[0212] 10. The method of Embodiment 1, wherein determining the complete state of the mobile device by processing at least one image includes using at least one trained model to determine that an object is obscuring the mobile device within at least one image, and Displaying a prompt for capturing an unobscured image on a mobile device.

[0213] 11. The method of embodiment 10, wherein determining whether a mobile device occlusion is within at least one image comprises determining whether a concave occlusion is within at least one image, and determining whether a blocked corner is within at least one image.

[0214] 12. The method of embodiment 10, wherein determining whether a concave occlusion is within at least one image comprises using at least one trained model to generate a mobile device mask that contains a smaller number of colors compared to at least one image, extracting a polygonal partial region P of the mobile device mask, determining the convex hull of P, calculating the difference between P and the convex hull, removing or reducing a thin mismatch of at least one edge between P and the convex hull, identifying the maximum area of the remaining region of P, and comparing the maximum area with a threshold to determine whether at least one image contains a concave occlusion.

[0215] 13. The method of embodiment 10, wherein determining whether there is a blocked corner within at least one image comprises using at least one trained model to generate a mobile device mask that contains a smaller number of colors compared to at least one image, extracting a polygonal partial region P of the mobile device mask, determining the convex hull of P, identifying four main edges of the convex hull, determining the intersection of adjacent main edges to identify a corner, determining the respective distances from each corner to P, and Including comparing each distance with a distance threshold to determine whether any corner is blocked in at least one image.

[0216] 14. The method of Embodiment 1, wherein processing at least one image to determine the perfect state of the mobile device includes using at least one trained model to determine whether at least one image includes the front of the mobile device, the back of the mobile device, or a cover.

[0217] 15. The method of Embodiment 1 further includes providing, in real-time or substantially real-time, a response for display on the mobile device in response to receiving at least one image, and the provided response is determined by the determined perfect state of the mobile device.

[0218] 16. The method of Embodiment 1 further includes displaying on the mobile device a test pattern configured to provide improved accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying a test pattern, as compared to the accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying another display pattern.

[0219] 17. The method of Embodiment 1 identifying a subset of conditions to be satisfied to determine that the perfect state of the mobile device has been verified, setting the image state of a specific image as verified when all conditions within the subset of conditions are satisfied in the specific image, and further includes determining that the perfect state of the mobile device has been verified when the respective image states for all required images have been verified.

[0220] 18. The method of embodiment 17, wherein at least one of the subset of conditions to be satisfied is executed on a mobile device. 19. The method of embodiment 1, wherein receiving at least one image includes receiving at least two images captured by a mobile device, a first image of the at least two images being of the front of the device and a second image of the at least two images being of the back of the device, and processing at least one image to determine the complete state of the mobile device includes processing both the first image and the second image using at least one trained model, and determining that the complete state of the mobile device is verified when the respective image states are verified in the processing of both images.

[0221] 20. The method of embodiment 1, wherein further includes training at least one trained model by inputting training images and respective labels describing the characteristics of each training image.

[0222] 21. The method of embodiment 1, wherein the at least one trained model is a neural network. 22. A method for detecting a concave occlusion in an image, the method comprising generating a mask using at least one trained model that contains a smaller number of colors compared to the image, extracting a polygonal partial region P of the mask, determining the convex hull of P, calculating the difference between P and the convex hull, removing or reducing a thin mismatch of at least one edge between P and the convex hull, recalculating P as the largest area of the remaining region, and determining the concavity as the difference between P and the convex hull.

[0223] 23. A method for detecting occluded corners of an object in an image, the method comprising: generating, using at least one trained model, a mask that contains a smaller number of colors compared to the image; extracting a polygonal sub-region P of the mask; determining the convex hull of P; identifying a predetermined number of major edges of the convex hull; identifying corners by determining the intersections of adjacent major edges; determining the respective distances from each corner to P, and comparing each distance with a distance threshold to determine whether any of the corners are occluded within the image.

[0224] 24. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured, using the processor, to cause the apparatus to at least: receive device integrity verification requirements associated with a mobile device; receive a mobile device identification data object including information describing the mobile device; display, on the mobile device, a prompt for capturing at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; receive at least one image captured by one or more image sensors of the mobile device; process the at least one image using at least one trained model to determine the integrity state of the mobile device.

[0225] 25. The apparatus of embodiment 24, wherein processing the at least one image to determine the integrity state of the mobile device comprises: determining, using at least one trained model, whether there is damage to the mobile device, and In response to a determination that there is damage to the mobile device, determining that the full state of the mobile device has not been verified.

[0226] 26. The apparatus of embodiment 24, wherein processing at least one image to determine the full state of the mobile device comprises: determining the angle of the mobile device with respect to the reflecting surface when at least one image was captured, and based on that angle, determining that at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object.

[0227] 27. The apparatus of embodiment 24, wherein processing at least one image to determine the full state of the mobile device comprises: determining whether at least one image includes the mobile device associated with the mobile device identification data object.

[0228] 28. The apparatus of embodiment 27, wherein determining whether at least one image includes the mobile device comprises: identifying a suspected mobile device within at least one image, generating a prediction of the identification of at least one suspected mobile device and comparing the mobile device identification data object with the prediction of the identification of at least one suspected mobile device to determine whether the suspected mobile device is that mobile device, and if the suspected mobile device is determined to be that mobile device, determining that the full state of the mobile device has been verified.

[0229] 29. The apparatus of embodiment 24, wherein the full state of the mobile device is determined to be uncertain, and at least one memory and computer program code cause the apparatus, using a processor, to at least: It is further configured to cause the device integrity verification request and at least one image to be sent to an internal user device for internal review.

[0230] 30. The apparatus of embodiment 26, wherein the at least one memory and the computer program code, using a processor, cause the apparatus to at least Based on the angle, in response to a determination that at least one image is capturing different mobile devices, (a) cause a message to be displayed on the mobile device instructing the user to recapture the mobile device, and (b) is further configured to cause a determination that the integrity state of the mobile device has not been verified.

[0231] 31. The apparatus of embodiment 24, wherein processing at least one image to determine the integrity state of the mobile device is determining a position within at least one image of the mobile device, the position being defined as a bounding box, and when the bounding box has a first predetermined relationship with a threshold ratio of at least one image, causing a message to be displayed on the mobile device instructing the mobile device to move closer to the reflecting surface.

[0232] 32. The apparatus of embodiment 31, wherein the at least one memory and the computer program code, using a processor, cause the apparatus to at least when the bounding box has a second predetermined relationship with a threshold ratio of at least one image, further configure to trim at least one image according to the bounding box.

[0233] 33. The apparatus of embodiment 24, wherein processing at least one image to determine the integrity state of the mobile device Using at least one trained model, determining that an object is obscuring a mobile device in at least one image, and displaying a prompt to cause the mobile device to capture an image without that obscuration.

[0234] 34. The apparatus of embodiment 33, wherein determining whether an obscuration of the mobile device is in at least one image comprises determining whether a concave obscuration is in at least one image, and determining whether a blocked corner is in at least one image.

[0235] 35. The apparatus of embodiment 33, wherein determining whether a concave obscuration is in at least one image comprises using at least one trained model to generate a mobile device mask that contains a smaller number of colors compared to at least one image, extracting a polygonal partial region P of the mobile device mask, determining a convex hull of P, calculating a difference between P and the convex hull, removing or reducing a thin discrepancy of at least one edge between P and the convex hull, identifying a maximum area of the remaining region of P, and comparing the maximum area with a threshold to determine whether at least one image contains a concave obscuration.

[0236] 36. The apparatus of embodiment 33, wherein determining whether there is a blocked corner in at least one image comprises using at least one trained model to generate a mobile device mask that contains a smaller number of colors compared to at least one image, extracting a polygonal partial region P of the mobile device mask, determining a convex hull of P, identifying four main edges of the convex hull, Identifying corners by determining intersections of adjacent major edges Determining respective distances from each corner to P, and Comparing each distance with a distance threshold to determine whether any corner is occluded in at least one image

[0237] 37. The apparatus of embodiment 24, wherein processing at least one image to determine the perfect state of the mobile device comprises Using at least one trained model to determine whether at least one image includes the front of the mobile device, the back of the mobile device, or a cover

[0238] 38. The apparatus of embodiment 24, wherein determining whether there is an occluded corner in at least one image comprises In response to receiving at least one image, providing a response for display on the mobile device in real-time or substantially real-time, the provided response being determined by the determined perfect state of the mobile device

[0239] 39. The apparatus of embodiment 24, wherein determining whether there is an occluded corner in at least one image comprises Displaying on the mobile device a test pattern configured to provide improved accuracy compared to the accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying another display pattern, in predicting the characteristics of at least one image captured when the mobile device is displaying a test pattern

[0240] 40. The apparatus of embodiment 24, wherein at least one memory and computer program code, using a processor, cause the apparatus to at least Identify a subset of conditions to be satisfied to determine that the perfect state of the mobile device has been verified If all conditions within a subset of conditions are satisfied within a particular image, set the image state of the particular image as verified, and When the respective image states for all required images are verified, it is further configured to cause a determination that the complete state of the mobile device is verified.

[0241] 41. The apparatus of Embodiment 40, wherein at least one condition of the subset of conditions to be satisfied is executed on a mobile device. 42. The apparatus of Embodiment 24, wherein receiving at least one image includes receiving at least two images captured by the mobile device, a first image of the at least two images being of the front of the device, a second image of the at least two images being of the back of the device, and determining the complete state of the mobile device by processing at least one image includes processing both the first image and the second image using at least one trained model, and when the respective image states are verified in the processing of both images, determining that the complete state of the mobile device determined is verified.

[0242] 43. The apparatus of Embodiment 24, wherein the at least one memory and computer program code are further configured, using a processor, to cause the apparatus to at least train at least one trained model by inputting training images and respective labels describing characteristics of the respective training images. 44. The apparatus of Embodiment 24, wherein the at least one trained model is a neural network.

[0243] 45. An apparatus for detecting a concave-shaped occlusion within an image, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and computer program code being configured, using the processor, to cause the apparatus to at least Using at least one trained model to generate a mask that contains a smaller number of colors compared to the image, Extracting a polygonal partial region P of the mask, Determining the convex hull of P, Calculating the difference between P and the convex hull, Removing or reducing a thin mismatch of at least one edge between P and the convex hull, Recalculating P as the maximum area of the remaining region, and Configured to cause determination of concavities as the difference between P and the convex hull.

[0244] 46. An apparatus for detecting blocked corners of an object in an image, the apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code causing the processor to cause the apparatus to at least, Using at least one trained model to generate a mask that contains a smaller number of colors compared to the image, Extracting a polygonal partial region P of the mask, Determining the convex hull of P, Identifying a predetermined number of major edges of the convex hull, Identifying corners by determining intersections of adjacent major edges, Determining respective distances from each corner to P, and Configured to cause comparison of each distance with a distance threshold to determine whether any corner is blocked within the image.

[0245] 47. A computer program product comprising at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions, Receiving a device integrity verification requirement associated with a mobile device, Receiving a mobile device identification data object including information describing the mobile device, Displaying, on a mobile device, a prompt for causing at least one image of the mobile device to be captured using one or more image sensors and a reflective surface of the mobile device; Receiving at least one image captured by one or more image sensors of the mobile device; and Including program code instructions for using at least one trained model to process at least one image to determine a complete state of the mobile device.

[0246] 48. The computer program product of embodiment 47, wherein processing at least one image to determine a complete state of the mobile device comprises: Using at least one trained model to determine whether there is damage to the mobile device; and In response to a determination that there is damage to the mobile device, determining that the complete state of the mobile device has not been verified.

[0247] 49. The computer program product of embodiment 47, wherein processing at least one image to determine a complete state of the mobile device comprises: Determining an angle of the mobile device with respect to the reflective surface when at least one image is captured; and Based on the angle, determining that at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object.

[0248] 50. The computer program product of embodiment 47, wherein processing at least one image to determine a complete state of the mobile device comprises: Determining whether at least one image includes the mobile device associated with the mobile device identification data object.

[0249] 51. A computer program product according to Embodiment 50, determining whether at least one image includes a mobile device, identifying a suspected mobile device within at least one image, generating a prediction of the identification of at least one suspected mobile device, and comparing the mobile device identification data object with the prediction of the identification of at least one suspected mobile device to determine whether the suspected mobile device is that mobile device, and if the suspected mobile device is determined to be that mobile device, determining that the complete state of the mobile device has been verified.

[0250] 52. A computer program product according to Embodiment 47, wherein the complete state of the mobile device is determined to be uncertain, and the computer-executable program code instructions further include program code instructions for sending a device integrity verification request and at least one image to an internal user computer program product for internal review.

[0251] 53. A computer program product according to Embodiment 49, wherein the computer-executable program code instructions in response to a determination, based on an angle, that at least one image captures different mobile devices, (a) causing a message to be displayed instructing the user to recapture the mobile device on the mobile device, and (b) further including program code instructions for determining that the complete state of the mobile device has not been verified.

[0252] 54. A computer program product according to Embodiment 47, determining the complete state of a mobile device by processing at least one image, is determining the position of the mobile device within at least one image, the position being defined as a bounding box, and When the bounding box has a first predetermined relationship with the threshold ratio of at least one image, it includes causing a message to be displayed on the mobile device instructing the mobile device to move closer to the reflective surface.

[0253] 55. A computer program product according to Embodiment 54, wherein the computer-executable program code instructions further include program code instructions for trimming at least one image according to the bounding box when the bounding box has a second predetermined relationship with the threshold ratio of at least one image.

[0254] 56. A computer program product according to Embodiment 47, wherein determining the perfect state of the mobile device by processing at least one image includes determining, using at least one trained model, that an object is obscuring the mobile device within at least one image, and causing a prompt to be displayed on the mobile device to capture an unobscured image thereof.

[0255] 57. A computer program product according to Embodiment 56, wherein determining whether there is an occlusion of the mobile device within at least one image includes determining whether there is a concave occlusion within at least one image, and determining whether there is an occluded corner within at least one image.

[0256] 58. A computer program product according to Embodiment 56, wherein determining whether there is a concave occlusion within at least one image includes generating a mobile device mask containing a smaller number of colors compared to at least one image using at least one trained model, extracting a polygonal partial region P of the mobile device mask, determining the convex hull of P, Calculating the difference between P and the convex hull, Removing or reducing the thin mismatches between P and at least one edge of the convex hull, Identifying the maximum area of the remaining region of P, and Comparing the maximum area with a threshold value to determine whether at least one image contains a concave occlusion.

[0257] 59. For the computer program product of embodiment 56, determining whether there is an occluded corner in at least one image Using at least one trained model to generate a mobile device mask that contains a smaller number of colors compared to at least one image, Extracting the polygonal partial region P of the mobile device mask, Determining the convex hull of P, Identifying the four main edges of the convex hull, Determining the intersection points of adjacent main edges to identify corners, Determining the respective distances from each corner to P, and Comparing each distance with a distance threshold value to determine whether any corner is occluded in at least one image.

[0258] 60. For the computer program product of embodiment 47, processing at least one image to determine the complete state of a mobile device Using at least one trained model to determine whether at least one image includes the front of a mobile device, the back of a mobile device, or a cover.

[0259] 61. For the computer program product of embodiment 47, determining whether there is an occluded corner in at least one image In response to receiving at least one image, providing a response for display on a mobile device in real time or substantially in real time, where the provided response is determined by the determined complete state of the mobile device.

[0260] 62. The computer program product of Embodiment 47, wherein determining whether there is a blocked corner in at least one image is when predicting the characteristics of at least one image captured when the mobile device is displaying a test pattern, compared to the accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying another display pattern, including causing a test pattern configured to provide improved accuracy to be displayed on the mobile device.

[0261] 63. The computer program product of Embodiment 47, wherein the computer-executable program code instructions identifying a subset of conditions to be satisfied to determine that the full state of the mobile device has been verified, when all conditions within the subset of conditions are satisfied in a specific image, setting the image state of the specific image as verified, and further including program code instructions for determining that the full state of the mobile device has been verified when the respective image states for all necessary images have been verified.

[0262] 64. The computer program product of Embodiment 63, wherein at least one of the conditions in the subset of conditions to be satisfied is executed on the mobile device. 65. The computer program product of Embodiment 47, wherein receiving at least one image includes receiving at least two images captured by the mobile device, the first of the at least two images being of the front of the device and the second of the at least two images being of the back of the device, and determining the full state of the mobile device by processing at least one image is processing both the first image and the second image using at least one trained model, and When the respective image states are verified in the processing of both images, it includes determining that the perfect state of the mobile device to be determined has been verified.

[0263] 66. A computer program product according to Embodiment 47, wherein the computer-executable program code instructions further include program code instructions for training at least one trained model by inputting training images and respective labels describing the characteristics of the respective training images.

[0264] 67. A computer program product according to Embodiment 47, wherein at least one trained model is a neural network. 68. A computer program product for detecting concave occlusions in an image, the computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising using at least one trained model to generate a mask containing a smaller number of colors compared to the image, extracting a polygonal partial region P of the mask, determining the convex hull of P, calculating the difference between P and the convex hull, removing or reducing thin mismatches of at least one edge between P and the convex hull, recalculating P as the maximum area of the remaining region, and program code instructions for determining the concavity as the difference between P and the convex hull.

[0265] 69. A computer program product for detecting blocked corners of an object in an image, the computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising Using at least one trained model to generate a mask that contains a smaller number of colors compared to the image, Extracting a polygonal partial region P of the mask, Determining the convex hull of P, Identifying a predetermined number of major edges of the convex hull, Determining the intersections of adjacent major edges to identify corners, Determining the respective distances from each corner to P, and Comparing each distance with a distance threshold and including program code instructions for determining whether any corner is blocked within the image.

[0266] 70. Receiving an indication of a target image, and Processing the target image using at least one trained model trained with a plurality of training images each labeled as either including or excluding a mobile device, to determine whether the target image includes a mobile device.

[0267] 71. Receiving an indication of a target image, Processing the target image using at least one trained model trained with a plurality of training images each associated with a bounding box indicating the position of a mobile device within the image, to determine the position of the mobile device within the target image, and Trimming the target image based on the determined position of the mobile device within the target image.

[0268] 72. Receiving an indication of a target image of a target mobile device, and Processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices each labeled as either including or excluding a cover on the respective mobile device, to determine whether the target image includes a cover on the target mobile device.

[0269] 73. Receiving an instruction for a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as including the front face of each mobile device or including the back face of each mobile device, to determine whether the target image includes the front face or the back face of the target mobile device. A method including this.

[0270] 74. Receiving an instruction for a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as being captured by each mobile device included in the image or being captured by a device different from each mobile device included in the image, to determine whether the target mobile device included in the target image was captured by the target mobile device or by a different device. A method including this.

[0271] 75. Receiving an instruction for a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image. A method including this.

[0272] 76. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code causing the processor to cause the apparatus to at least receive an instruction for a target image, and The target image is processed using at least one trained model trained with a plurality of training images each labeled as either including or excluding a mobile device, so as to determine whether the target image includes a mobile device.

[0273] 77. An apparatus comprising at least one processor and at least one memory including computer program code, wherein the at least one memory and the computer program code, when used by the processor, cause the apparatus to at least receive an indication of a target image, process the target image using at least one trained model trained with a plurality of training images each associated with a bounding box indicating the position of a mobile device in the image to determine the position of the mobile device in the target image, and trim the target image based on the determined position of the mobile device in the target image.

[0274] 78. An apparatus comprising at least one processor and at least one memory including computer program code, wherein the at least one memory and the computer program code, when used by the processor, cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices each labeled as either including or excluding a cover on the respective mobile device to determine whether the target image includes a cover on the target mobile device.

[0275] An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as including the front of the respective mobile device or including the back of the respective mobile device, to determine whether the target image includes the front or the back of the target mobile device.

[0276] An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least receive an indication of a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained with a plurality of training images of mobile devices, each training image being labeled as being captured by the respective mobile device included in the image or being captured by a device different from the respective mobile device included in the image, to determine whether the target mobile device included in the target image was captured by the target mobile device or by a different device.

[0277] An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code being configured to, using the processor, cause the apparatus to at least receive an indication of a target image of a target mobile device, and The target image of the mobile device is processed using at least one trained model trained with a plurality of training images of the mobile device, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device within the target image, and is configured to cause this to be done.

[0278] 82. A computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: Receiving an instruction of a target image, and Processing the target image using at least one trained model trained with a plurality of training images each labeled as either including or excluding a mobile device to determine whether the target image includes a mobile device, and including program code instructions for doing so.

[0279] 83. A computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: Receiving an instruction of a target image, Processing the target image using at least one trained model trained with a plurality of training images each associated with a bounding box indicating the position of a mobile device within the image to determine the position of the mobile device within the target image, and Including program code instructions for trimming the target image based on the determined position of the mobile device within the target image.

[0280] 84. A computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: Receiving an instruction of a target image of a target mobile device, and Processing the target image of the mobile device using at least one trained model trained with a plurality of training images of the mobile device labeled as including or excluding the cover on each mobile device to determine whether the target image includes the cover on the target mobile device, and including program code instructions for doing so.

[0281] 85. A computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: Receiving an indication of a target image of a target mobile device, and Processing the target image of the mobile device using at least one trained model trained with a plurality of training images of the mobile device labeled as including the front or the back of each mobile device to determine whether the target image includes the front or the back of the target mobile device, and including program code instructions for doing so.

[0282] 86. A computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising: Receiving an indication of a target image of a target mobile device, and The program code instructions include processing a target image of a mobile device using at least one trained model trained with a plurality of training images of the mobile device, where each training image is labeled as being captured by each mobile device included in the image or by a device different from each mobile device included in the image, to determine whether the target mobile device included in the target image was captured by the target mobile device or by a different device.

[0283] A computer program product including at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein, the computer-executable program code instructions comprising receiving an indication of a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained with a plurality of training images of the mobile device, where each training image is labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0284] Many modifications and other embodiments of the invention described herein will come to mind to those skilled in the art related to these inventions who obtain the benefits of the technology presented in the foregoing description and the accompanying drawings. Accordingly, it is understood that the invention is not limited to the specific embodiments disclosed, and that other embodiments are intended to be included within the scope of the appended claims. Moreover, the foregoing description and the related drawings illustrate embodiments by way of examples of combinations of elements and / or functions, but it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, combinations of elements and / or functions different from those explicitly described above are also contemplated as may be described in some of the appended claims. Specific terms are employed herein, but they are used only in a general and descriptive sense and not for purposes of limitation.

Claims

1. Receiving a device integrity verification request associated with a mobile device; Receiving a mobile device identification data object including information describing the mobile device; Displaying, on the mobile device, a prompt for capturing at least one image of the mobile device using one or more image sensors and a reflective surface of the mobile device; Receiving the at least one image captured by the one or more image sensors of the mobile device; Processing the at least one image using at least one trained model to determine a complete state of the mobile device, including: Processing the at least one image to determine the complete state of the mobile device includes: Determining a position of the mobile device within the at least one image of the mobile device, wherein the position is defined as a bounding box; and If the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, displaying, on the mobile device, a message instructing the mobile device to move closer to the reflective surface.

2. Processing the at least one image to determine the complete state of the mobile device includes: Using the at least one trained model to determine whether there is damage to the mobile device; and In response to determining that there is damage to the mobile device, further determining that the complete state of the mobile device has not been verified. The method according to claim 1.

3. Processing the at least one image to determine the complete state of the mobile device includes: Determining an angle of the mobile device with respect to the reflective surface when the at least one image is captured; and Based on the angle, further predicting that the at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object. The method according to claim 1.

4. In response to the prediction that the at least one image captures a different mobile device based on the angle, (a) causing a message to be displayed on the mobile device instructing the user to recapture the mobile device; (b) determining that the full state of the mobile device has not been verified; The method according to claim 3, further comprising.

5. Processing the at least one image to determine the full state of the mobile device includes: The method according to claim 1, further comprising determining whether the at least one image includes the mobile device associated with the mobile device identification data object.

6. Determining whether the at least one image includes the mobile device includes: identifying a suspected mobile device within the at least one image; generating a prediction of the identification of the at least one suspected mobile device and comparing the mobile device identification data object with the prediction of the identification of the at least one suspected mobile device to determine whether the suspected mobile device is the mobile device; The method according to claim 5, further comprising determining that the full state of the mobile device has been verified when it is determined that the suspected mobile device is the mobile device.

7. The full state of the mobile device is determined to be uncertain, and the method further includes: The method according to claim 1, further comprising transmitting the device integrity verification request and the at least one image to an internal user device for internal review.

8. The method according to claim 1, further comprising trimming the at least one image according to the bounding box when the bounding box has a second predetermined relationship with the threshold ratio of the at least one image.

9. Processing the at least one image to determine the full state of the mobile device includes: using the at least one trained model to determine that an object is obscuring the mobile device within the at least one image; The method according to claim 1, further comprising displaying a prompt on the mobile device to capture an unobscured image.

10. Processing the at least one image to determine the full state of the mobile device includes: The method of claim 1, comprising determining, using the at least one trained model, whether the at least one image includes the front of the mobile device, the back of the mobile device, or a cover.

11. The method of claim 1, further comprising providing, in real time or substantially real time in response to receipt of the at least one image, a response for display on the mobile device, the response provided being determined by the determined complete state of the mobile device.

12. The method of claim 1, further comprising causing the test pattern to be displayed on the mobile device, the test pattern being configured to provide improved accuracy in predicting characteristics of the at least one image captured when the mobile device is displaying the test pattern as compared to the accuracy in predicting the characteristics of the at least one image captured when the mobile device is displaying another display pattern.

13. identifying a subset of conditions to be satisfied in order to determine that the complete state of the mobile device has been verified; setting the image state of the particular image to verified if all of the conditions within the subset of conditions are satisfied within the particular image; and The method of claim 1, further comprising determining that the complete state of the mobile device has been verified if the respective image states for all of the required images have been verified.

14. Receiving the at least one image includes receiving at least two images captured by the mobile device, a first image of the at least two images being of the front of the mobile device and a second image of the at least two images being of the back of the mobile device, and processing the at least one image to determine the complete state of the mobile device includes processing both the first image and the second image using the at least one trained model; and The method of claim 1, comprising determining that the determined complete state of the mobile device has been verified if the respective image states of both images are verified in the processing of both images.

15. The method according to claim 1, further comprising training the at least one trained model by inputting a plurality of training images and respective labels describing characteristics for each of the plurality of training images.

16. An apparatus comprising at least one processor and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured, using the processor, to cause the apparatus to at least receive a device integrity verification request associated with a mobile device; receive a mobile device identification data object including information describing the mobile device; display, on the mobile device, a prompt for capturing at least one image of the mobile device using one or more image sensors and a reflecting surface of the mobile device; receive at least one image captured by the one or more image sensors of the mobile device; process the at least one image using at least one trained model to determine a complete state of the mobile device; Processing the at least one image to determine a complete state of the mobile device comprises determining a position of the mobile device within the at least one image of the mobile device, the position being defined as a bounding box; displaying, on the mobile device, a message instructing the mobile device to move closer to the reflecting surface when the bounding box has a first predetermined relationship with a threshold ratio of the at least one image.

17. At least one persistent computer-readable storage medium storing computer-executable program code instructions, the computer-executable program code instructions receiving a device integrity verification request associated with a mobile device; receiving a mobile device identification data object including information describing the mobile device; displaying, on the mobile device, a prompt for capturing at least one image of the mobile device using one or more image sensors and a reflecting surface of the mobile device; Receiving the at least one image captured by the one or more image sensor mobile devices; Including program code instructions for using at least one trained model to process the at least one image to determine a complete state of the mobile device; Processing the at least one image to determine the complete state of the mobile device comprises: Determining a position of the mobile device within the at least one image of the mobile device, the position being defined as a bounding box; At least one persistent computer-readable storage medium including, when the bounding box has a first predetermined relationship with a threshold ratio of the at least one image, causing a message to be displayed on the mobile device instructing the mobile device to move closer to the reflecting surface.

Images