A system, method, apparatus, and persistent computer-readable storage medium for processing images of mobile devices using machine learning to determine the integrity status of mobile devices.

Abstract

To provide a system, an apparatus, a method, and a computer program product for determining the integrity state of a mobile device.SOLUTION: Images of a mobile device captured by the mobile device using reflective surfaces are processed using various trained models, such as neural networks, to verify authenticity, detect damage, and detect shielding. A mask may be generated to enable identification of concave shielding or occluded corners of an object, such as a mobile device, in an image. Images of the front and / or back of the mobile device may be processed to determine the integrity state of the mobile device, such as verified, unverified, or indeterminate. A user may be prompted to remove the cover, remove the shielding, and / or move the mobile device closer to the reflective surface.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] Embodiments of the present invention generally relate to computer technology, and more particularly to systems, methods, apparatuses, and computer program products for training and utilizing mathematical models (such as prediction models, neural networks, and / or the like) using machine learning to determine the full state of a mobile device based on electronic processing of images.

Background Art

[0002] Computer vision enables a computer to see and understand images. In some cases, computer vision can be used to detect and analyze the content of an image, such as recognizing objects within the image. However, existing technologies are insufficient to meet the speed and accuracy requirements of many industries, and there is a need for improvements in computer vision technology and techniques to enable high-performance image processing. Moreover, human analysis is not useful for the speed and accuracy required for computer vision tasks. Through the noted efforts, ingenuity, and innovation, many of these identified problems have been solved by developing the solutions included in embodiments of the present invention, and many examples thereof are described in detail herein.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Systems, methods, apparatuses, and computer program products are therefore provided for training and utilizing models using machine learning to determine the full state of a mobile device based on electronic processing of images.

Means for Solving the Problems

[0004] In some use cases, a system needs to review images of an object to verify its integrity (e.g., to determine information about the object, to verify its operability or functionality, to verify its identification, or similar). Computer vision and image processing must occur quickly and with high accuracy, which is lacking in many conventional image processing techniques. A further challenge arises when the system cannot select the imaging device that captures the image and cannot directly control the image acquisition process; therefore, computer vision needs to be robust enough to absorb and / or detect problems related to the acquisition process. In example work environments, a system may attempt to verify the identification and integrity of an object (e.g., a mobile device) using only an image of the object, or using the image in combination with one or more data objects transmitted from that object or from another device. An example of such an environment might be when a user registers for a service, protection plan, or similar, which requires a remote system to verify an object (e.g., a mobile device) without the physical presence of the device. According to some processes for purchasing aftermarket coverage, consumers must visit a retailer, insurance company, or mobile device service provider to have their device inspected and its integrity verified before the insurer issues a policy for coverage and registers the device. Other processes for purchasing and / or selling coverage allow consumers to take photos of their mobile device using a self-service web or mobile application and submit the images for manual review before registration. However, such processes require review time and may delay the confirmation of coverage to the consumer. Such processes may also expose providers to further fraudulent activity, such as consumers submitting photos of different, undamaged mobile devices in an attempt to obtain coverage for a previously damaged device.

[0005] An additional embodiment provides a time-constrained barcode, quick-response (QR) code, or other computer-generated code that is displayed by a device and captured in a photograph using a mirror, thereby linking the photographic submission to the device that displayed the code. However, such embodiments may be susceptible to fraud, such as by allowing a user to recreate the code on another intact device and capture a photograph of that intact device. Furthermore, the code embodiment may only provide verification of the front of the device (e.g., the display side) without reliably verifying the condition or state of the back and / or bezel of the device. Another drawback of such embodiments is that if the code is displayed on the device display, it may obscure cracks or other damage present on the screen.

[0006] Examples of embodiments of this disclosure provide an improved determination of the integrity status of a mobile device. An example embodiment may prompt a user to capture an image of their device in a mirror or other reflective surface using the device's own sensors or camera. Identification information of the mobile device may be processed along with the image to verify that the image was indeed taken from the device from which the image was captured and that the device is free from any existing damage that would disqualify it from compensation.

[0007] Examples of embodiments may utilize machine learning algorithms and associated mathematical models, including, but not limited to, neural networks, such as convolutional neural networks and / or similar, predictive models and / or other types of “models,” as may be referred to herein, which can be trained to analyze and identify relevant information from images by using training images that are manually reviewed, labeled, and / or characterized by a user. Any reference to “model” herein will be understood to include any type of model that can be used in a machine learning algorithm trained on training images to predict certain features of other images. Examples of embodiments may utilize the trained models to utilize information detected in subsequently received images to predict features in images, such as, but not limited to, the integrity state of a mobile device.

[0008] Different models, each trained on a different training set, can be used to make different types of predictions. The use of trained models (e.g., neural networks) may enable one embodiment to determine the integrity status of a mobile device in real time or near real time from the moment an image is submitted, and / or to transfer uncertain or high-risk predictions in real time or near real time for further review before finalizing the integrity status of the mobile device and / or enrolling the mobile device in a protection plan, without additional human review (according to some embodiments). In some embodiments, the output of one or more models may be input to subsequent models for more sophisticated analysis of the image and determination of the integrity status of the mobile device.

[0009] In some scenarios, consumers may purchase insurance plans, warranties, extended warranties, and / or other device protection plans to protect their mobile devices and smartphones from damage, theft, loss, and / or similar incidents. In some cases, consumers may purchase such plans over the counter, thus knowing that the device is in new condition and is eligible for coverage. However, in some cases, consumers may wish to purchase protection after the device has been acquired, either directly from an insurance company or through their mobile device service provider. The provider must be able to quickly verify the device's integrity without physical access to the device or the ability to directly operate it. In such cases, manual inspection may not satisfy the accuracy and speed required to verify the device's integrity in a reasonable time, and there may be a need for the systems, methods, and apparatus described herein. Similarly, consumers purchasing used or refurbished devices may wish to purchase insurance in the aftermarket when the insurance company is unaware of the device's condition. Insurance companies verify the device's condition at the time the protection is purchased to minimize losses and prevent fraudulent purchases of protection for devices with pre-existing damage.

[0010] References made herein to warranties, extended warranties, insurance, insurance policies, insurance certificates, coverage, device protection plans, protection plans, and / or similar are not intended to limit the scope of this disclosure, and examples of embodiments may relate to the registration of a mobile device to any such aforementioned plan or similar plan for protection against loss, or to other environments using the computer vision and image processing systems, methods, and apparatus described herein. Similarly, any reference to the verification of the integrity of the apparatus may relate to the eligibility of the mobile device for registration in either of the aforementioned plans or environments. Furthermore, the determination of the integrity status of the mobile device may be used for other purposes.

[0011] One example of a condition implemented according to the embodiments described herein includes determining whether occlusion is present in an image from a mobile device. It will be understood that the occlusion detection process disclosed herein may be used for other purposes, such as determining the occlusion of any type of object in an image.

[0012] A method is provided which includes receiving a device integrity verification request associated with a mobile device and receiving a mobile device identification data object containing information describing the mobile device. The method further includes prompting the mobile device to capture at least one image of the mobile device using one or more image sensors and reflective surfaces of the mobile device, and receiving at least one image captured by one or more image sensors of the mobile device. The method may further include processing at least one image using at least one trained model to determine the integrity state of the mobile device. In one embodiment, the at least one trained model may include a neural network.

[0013] According to one embodiment, processing at least one image to determine the integrity status of a mobile device includes determining whether at least one image contains a mobile device associated with a mobile device identification data object. Determining whether at least one image contains a mobile device includes identifying a suspicious mobile device within at least one image, generating a prediction of the identification of at least one suspicious mobile device, and comparing the mobile device identification data object with the prediction of the identification of at least one suspicious mobile device to determine whether the suspicious mobile device is that mobile device. Processing at least one image to determine the integrity status of a mobile device may further include determining that the integrity status of the mobile device has been verified if the suspicious mobile device is determined to be that mobile device. Determining the integrity status of a mobile device may further include sending a device integrity verification request and at least one image to an internal user device for internal review if the integrity status of the mobile device is determined to be uncertain.

[0014] According to some embodiments, processing at least one image to determine the integrity state of a mobile device may include using at least one trained model to determine whether there is damage to the mobile device, and in response to the determination that there is damage to the mobile device, determining that the integrity state of the mobile device has not been verified.

[0015] In some embodiments, processing at least one image to determine the integrity status of a mobile device includes determining the angle of the mobile device relative to a reflective surface when the at least one image was captured, and, based on that angle, determining that the at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object. Processing at least one image to determine the integrity status of a mobile device may further include, in response to the determination based on the angle that the at least one image captures a different mobile device, displaying a message on the mobile device instructing the user to recapture the mobile device, and determining that the integrity status of the mobile device has not been verified.

[0016] According to some embodiments, processing at least one image to determine the integrity state of a mobile device may include determining the position of the mobile device within at least one image, where the position is defined as a bounding box, and if the bounding box has a first predetermined relationship with the threshold ratio of at least one image, displaying a message on the mobile device instructing it to move closer to a reflective surface. If the bounding box has a second predetermined relationship with the threshold ratio of at least one image, processing at least one image to determine the integrity state of a mobile device may further include cropping the at least one image according to the bounding box.

[0017] According to some embodiments, processing at least one image to determine the integrity state of a mobile device includes using at least one trained model to determine that an object occludes the mobile device in at least one image and displaying a prompt on the mobile device to capture an unoccluded image. Determining whether occlusion of a mobile device is in at least one image may include determining whether concave occlusion is in at least one image and whether blocked corners are in at least one image. Determining whether concave occlusion is in at least one image may include using at least one trained model to generate a mobile device mask containing fewer colors than at least one image, extracting a polygonal subregion P of the mobile device mask, determining the convex hull of P, calculating the difference between P and the convex hull, removing or reducing thin discrepancies at at least one edge of P and the convex hull, identifying the maximum area of ​​the remaining region of P, and comparing the maximum area to a threshold to determine whether at least one image contains concave occlusion.

[0018] According to some embodiments, determining whether there is a closed corner in at least one image may include: generating a mobile device mask containing fewer colors than at least one image using at least one trained model; extracting a polygonal subregion P of the mobile device mask; determining the convex hull of P; identifying four major edges of the convex hull; identifying corners by determining the intersections of adjacent major edges; determining the respective distances from each corner to P; and comparing each distance to a distance threshold to determine whether any of the corners are closed in at least one image.

[0019] According to one embodiment, processing at least one image to determine the integrity state of a mobile device includes using at least one trained model to determine whether at least one image includes the front, back, or cover of the mobile device.

[0020] In response to the reception of at least one image, one embodiment may provide a response for display on a mobile device in real time or near real time, the response provided being determined by the integrity status of the determined mobile device.

[0021] An example embodiment may also include displaying a test pattern on a mobile device that is configured to provide improved accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying a test pattern, compared to the accuracy in predicting the characteristics of at least one image captured when the mobile device is displaying a different display pattern.

[0022] In some embodiments, a subset of conditions that must be satisfied to determine if the integrity state of a mobile device is verified can be identified. If all conditions within the subset of conditions are satisfied within a particular image, the image state of that image is set to verified. If the respective image states for all required images are verified, the integrity state of the mobile device is determined to be verified. In some embodiments, at least one condition from the subset of conditions to be satisfied is performed on the mobile device.

[0023] According to one embodiment, receiving at least one image includes receiving at least two images captured by a mobile device, the first of the at least two images being of the front of the device, the second of the at least two images being of the back of the device, and processing at least one image to determine the integrity state of the mobile device includes processing both the first and second images using at least one trained model, and determining the integrity state of the mobile device to be determined is verified if the processing of both images verifies the respective image states.

[0024] In some embodiments, at least one trained model can be trained by inputting training images and respective labels describing the characteristics of each training image. A method for detecting concave occlusions in an image is also provided, which includes generating a mask containing fewer colors than the image using at least one trained model, extracting a polygonal subregion P of the mask, determining the convex hull of P, and calculating the difference between P and the convex hull. The method further includes removing or reducing thin discrepancies at at least one edge of P and the convex hull, recalculating P as the maximum area of ​​the remaining region, and determining the concave as the difference between P and the convex hull.

[0025] A method is provided for detecting closed corners of an object in an image, the method comprising: generating a mask containing fewer colors than the image using at least one trained model; extracting a polygonal subregion P of the mask; determining the convex hull of P; identifying a predetermined number of major edges of the convex hull; identifying corners by determining the intersections of adjacent major edges; determining the respective distances from each corner to P; and comparing each distance to a distance threshold to determine whether any corner is closed in the image.

[0026] An apparatus is provided that includes at least one processor and at least one memory including computer program code. The at least one memory and the computer program code are configured to cause the processor, in the apparatus, to at least receive device integrity verification requirements associated with a mobile device, and receive a mobile device identification data object including information describing the mobile device. The at least one memory and the computer program code are further configured to cause the processor, in the apparatus, to display, on the mobile device, a prompt for capturing at least one image of the mobile device using one or more image sensors and a reflecting surface of the mobile device, and receive at least one image captured by one or more image sensors of the mobile device. The at least one memory and the computer program code may be further configured to cause the processor to process the at least one image using at least one trained model to determine the integrity state of the mobile device.

[0027] An apparatus for detecting concave shielding in an image is provided. The apparatus includes at least one processor and at least one memory including computer program code. The at least one memory and the computer program code are configured to cause the processor, in the apparatus, to at least use at least one trained model to generate a mask that includes a smaller number of colors compared to the image, extract a polygonal partial region P of the mask, determine the convex hull of P, calculate the difference between P and the convex hull, remove or reduce a thin mismatch of at least one edge between P and the convex hull, recalculate P as the maximum area of the remaining region, and determine the depression as the difference between P and the convex hull.

[0028] An apparatus for detecting occluded corners of an object in an image is also provided. The apparatus includes at least one processor and at least one memory including computer program code. The at least one memory and the computer program code are configured, using the processor, to cause the apparatus to at least generate a mask that includes a smaller number of colors compared to the image, extract a polygonal partial region P of the mask, determine the convex hull of P, identify a predetermined number of major edges of the convex hull, determine intersections of adjacent major edges to identify corners, determine respective distances from each corner to P, and compare each distance with a distance threshold to determine whether any of the corners are occluded in the image.

[0029] A computer program product is provided, which includes at least one persistent computer-readable storage medium having computer-executable program code instructions stored therein. The computer-executable program code instructions include program code instructions for receiving a device integrity verification request associated with a mobile device, receiving a mobile device identification data object including information describing the mobile device, displaying, on the mobile device, a prompt for causing at least one image of the mobile device to be captured using one or more image sensors and a reflective surface of the mobile device, receiving at least one image captured by one or more image sensors of the mobile device, and processing the at least one image using at least one trained model to determine the integrity state of the mobile device.

[0030] A computer program product for detecting concave occlusions in an image is also provided, which includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, and which include program code instructions for generating a mask containing fewer colors than the image using at least one trained model, extracting a polygonal subregion P of the mask, determining the convex hull of P, calculating the difference between P and the convex hull, removing or reducing a thin mismatch of at least one edge between P and the convex hull, recalculating P as the maximum area of ​​the remaining region, and determining the concave as the difference between P and the convex hull.

[0031] A computer program product for detecting blocked corners of an object in an image is also provided, which comprises at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, the computer executable program code instructions comprising program code instructions for generating a mask containing fewer colors than the image, using at least one trained model, extracting a polygonal subregion P of the mask, determining the convex hull of P, identifying a predetermined number of major edges of the convex hull, determining the intersection of adjacent major edges to identify corners, determining the respective distance from each corner to P, and comparing each distance to a distance threshold to determine whether any corner is blocked in the image.

[0032] An apparatus is provided, which includes means for receiving a device integrity verification request associated with a mobile device, and means for receiving a mobile device identification data object containing information describing the mobile device, means for displaying a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and reflective surfaces of the mobile device, means for receiving at least one image captured by one or more image sensors of the mobile device, and means for processing at least one image to determine the integrity state of the mobile device using at least one trained model.

[0033] An apparatus is provided having means for detecting concave occlusion in an image, the apparatus including means for causing the apparatus to generate a mask containing fewer colors than the image using at least one trained model, and for extracting a polygonal subregion P of the mask; means for determining the convex hull of P and calculating the difference between P and the convex hull; means for removing or reducing a thin mismatch of at least one edge between P and the convex hull; means for recalculating P as the maximum area of ​​the remaining region; and means for determining the concave as the difference between P and the convex hull.

[0034] An apparatus is also provided that has means for detecting blocked corners of an object in an image, the apparatus including means for generating a mask containing fewer colors than the image using at least one trained model, means for extracting a polygonal subregion P of the mask, means for determining the convex hull of P and identifying a predetermined number of major edges of the convex hull, means for determining the intersection of adjacent major edges and identifying corners, means for determining the respective distance from each corner to P, and means for comparing each distance to a distance threshold to determine whether any corner is blocked in the image.

[0035] According to one embodiment, a method is provided which includes receiving an indication of a target image and processing the target image using at least one trained model, such as a neural network, which can be used in a machine learning algorithm. The model is trained on a plurality of training images, each labeled either as containing a mobile device or excluding a mobile device, to determine whether the target image contains a mobile device.

[0036] According to one embodiment, a method is provided which includes receiving instructions for a target image and processing the target image using at least one trained model trained on a plurality of training images, each associated with a bounding box indicating the location of a mobile device in the image, to determine the location of the mobile device in the target image. The method may further include cropping the target image based on the determined location of the mobile device in the target image.

[0037] According to one embodiment, a method is provided which includes receiving an instruction for a target image of a target mobile device, and processing the target image of the target mobile device using at least one trained model trained on multiple training images of mobile devices labeled as either including or excluding the cover on each mobile device, to determine whether the target image includes the cover on the target mobile device.

[0038] According to one embodiment, a method is provided which includes receiving instructions for a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained on multiple training images of the mobile device, each training image being labeled as either including the front of the mobile device or including the back of the mobile device, to determine whether the target image includes the front or back of the target mobile device.

[0039] According to one embodiment, a method is provided which includes receiving instructions for a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained on multiple training images of mobile devices, each training image being labeled as either captured by the respective mobile device contained in the image, or captured by a different device from the respective mobile device contained in the image, to determine whether the target mobile device contained in the target image was captured by the target mobile device or by a different device.

[0040] According to one embodiment, a method is provided which includes receiving instructions for a target image of a target mobile device, and processing the target image of the mobile device using at least one trained model trained on multiple training images of the mobile device, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0041] According to one embodiment, the device comprises at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to cause the device to use the processor to at least receive instructions for a target image and process the target image using at least one trained model trained on a plurality of training images, each labeled either as containing a mobile device or excluding a mobile device, to determine whether the target image contains a mobile device.

[0042] A device is also provided, comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to cause the device to use the processor to at least: receive instructions for a target image; process the target image using at least one trained model trained on a plurality of training images, each associated with a bounding box indicating the position of a mobile device in the image, to determine the position of the mobile device in the target image; and crop the target image based on the determined position of the mobile device in the target image.

[0043] According to one embodiment, a device is provided comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to cause the device to use the processor to at least receive instructions for a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained on a plurality of training images of mobile devices labeled as either including or excluding the cover on each mobile device, to determine whether the target image includes the cover on the target mobile device.

[0044] A device is also provided, comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and computer program code are configured to use the processor to cause the device to at least receive instructions for a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained on multiple training images of the mobile device, each training image being labeled as either including the front of the mobile device or including the back of the mobile device, to determine whether the target image includes the front or back of the target mobile device.

[0045] According to one embodiment, the device comprises at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are configured to use the processor to cause the device to at least receive instructions for a target image of a target mobile device, and to process the target image of the mobile device using at least one trained model trained on multiple training images of mobile devices, each of which is labeled as either captured by the respective mobile device contained in the image or captured by a different device, in order to determine whether the target mobile device contained in the target image was captured by the target mobile device or by a different device.

[0046] A device is also provided, comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and computer program code are configured to use the processor to cause the device to at least receive instructions for a target image of a target mobile device, and process the target image of the mobile device using at least one trained model trained on multiple training images of the mobile device, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0047] According to an embodiment, a computer program product is provided which includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, the computer executable program code instructions include program code instructions for receiving instructions for a target image and processing the target image using at least one trained model trained on a plurality of training images, each labeled either as including a mobile device or excluding a mobile device, to determine whether the target image includes a mobile device.

[0048] A computer program product is also provided which includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, the computer executable program code instructions include program code instructions for receiving instructions for a target image and processing the target image using at least one trained model trained on a plurality of training images, each associated with a bounding box indicating the location of a mobile device in the image, to determine the location of the mobile device in the target image. The computer executable program code instructions also include program code instructions for cropping the target image based on the determined location of the mobile device in the target image.

[0049] A computer program product is also provided which includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, the computer executable program code instructions include instructions for receiving a target image of a target mobile device, and processing that target image of the mobile device using at least one trained model trained on multiple training images of mobile devices labeled to include or exclude the cover on each mobile device, to determine whether the target image includes the cover on the target mobile device.

[0050] According to one embodiment, a computer program product is provided which includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, the computer executable program code instructions include a program code instruction for receiving an instruction for a target image of a target mobile device, and processing that target image of the mobile device using at least one trained model trained on a plurality of training images of the mobile device, each training image being labeled as either including the front of the mobile device or including the back of the mobile device, to determine whether the target image includes the front or the back of the target mobile device.

[0051] A computer program product is provided which includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, the computer executable program code instructions include instructions for receiving instructions for a target image of a target mobile device, and processing that target image of the mobile device using at least one trained model trained on multiple training images of mobile devices, each training image being labeled as being captured by the respective mobile device in which it is contained in the image, or by a different device, to determine whether the target mobile device contained in the target image is captured by that target mobile device or by a different device.

[0052] A computer program product is also provided which includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, the computer executable program code instructions include program code instructions for receiving instructions for a target image of a target mobile device, and processing that target image of the mobile device using at least one trained model trained on multiple training images of the mobile device, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device in the target image.

[0053] The models and algorithms described herein may be used independently for their intended purposes, or they may be used in one or more larger processes, such as those described herein. For example, in some embodiments, both rear and front camera images may capture the back and front of the device, and the various trained models described herein may be run on each image, either separately or as part of a larger process, to ensure that the device is intact and undamaged. In some embodiments, one or more models and algorithms may be run as part of an orientation process to a protective product and / or service contract or other device protection program that requires verification of the device's integrity.

[0054] The above summary is provided solely for the purpose of summarizing some embodiments of the invention in order to provide a basic understanding of some aspects of the invention. It will be understood that the above embodiments are merely examples and should not be construed as narrowing the scope or spirit of the disclosure in any way. It will be understood that the scope of the disclosure encompasses many potential embodiments, some of which, in addition to those summarized herein, will be further described below.

[0055] Embodiments of the present invention have been described in general terms, with references made to the accompanying drawings, which are not necessarily drawn to actual size: [Brief explanation of the drawing]

[0056] [Figure 1] A system for determining the integrity status of a mobile device is shown according to several embodiments. [Figure 2] A block diagram of the apparatus according to several embodiments is shown. [Figure 3] This flowchart illustrates the operations for determining the integrity status of a mobile device according to several embodiments. [Figure 4A-1] This flowchart illustrates the operations for determining the integrity status of a mobile device according to several embodiments. [Figure 4A-2] This flowchart illustrates the operations for determining the integrity status of a mobile device according to several embodiments. [Figure 4B] The data flow between the model and / or its circuitry is shown according to several embodiments. [Figure 5A] Examples of user interfaces provided according to several embodiments are shown. [Figure 5B] Examples of user interfaces provided according to several embodiments are shown. [Figure 5C] Examples of user interfaces provided according to several embodiments are shown. [Figure 5D] Examples of user interfaces provided according to several embodiments are shown. [Figure 5E] Examples of user interfaces provided according to several embodiments are shown. [Figure 5F] Examples of user interfaces provided according to several embodiments are shown. [Figure 5G] Examples of user interfaces provided according to several embodiments are shown. [Figure 5H]Examples of user interfaces provided according to several embodiments are shown. [Figure 5I] Examples of user interfaces provided according to several embodiments are shown. [Figure 5J] Examples of user interfaces provided according to several embodiments are shown. [Figure 5K] Examples of user interfaces provided according to several embodiments are shown. [Figure 5L] Examples of user interfaces provided according to several embodiments are shown. [Figure 5M] Examples of user interfaces provided according to several embodiments are shown. [Figure 5N] Examples of user interfaces provided according to several embodiments are shown. [Figure 5O] Examples of user interfaces provided according to several embodiments are shown. [Figure 5P] Examples of user interfaces provided according to several embodiments are shown. [Figure 5Q] Examples of user interfaces provided according to several embodiments are shown. [Figure 5R] Examples of user interfaces provided according to several embodiments are shown. [Figure 5S] Examples of user interfaces provided according to several embodiments are shown. [Figure 5T] Examples of user interfaces provided according to several embodiments are shown. [Figure 5U] Examples of user interfaces provided according to several embodiments are shown. [Figure 5V] Examples of user interfaces provided according to several embodiments are shown. [Figure 5W] Examples of user interfaces provided according to several embodiments are shown. [Figure 5X]Examples of user interfaces provided according to several embodiments are shown. [Figure 5Y] Examples of user interfaces provided according to several embodiments are shown. [Figure 6] This flowchart illustrates an operation for detecting occlusion in an image according to several embodiments. [Figure 7A] Examples of captured images from a mobile device are illustrated according to several embodiments. [Figure 7B] The following are examples of mobile device masks that can be generated from the images in Figures 7A and 8A, respectively, according to several embodiments. [Figure 8A] Examples of captured images from a mobile device are illustrated according to several embodiments. [Figure 8B] The following are examples of mobile device masks that can be generated from the images in Figures 7A and 8A, respectively, according to several embodiments. [Modes for carrying out the invention]

[0057] Some embodiments of the present invention are described more fully below with reference to the accompanying drawings, which show some, but not all, embodiments of the present invention. In practice, various embodiments of the present invention can be embodied in many different ways and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided to satisfy the legal requirements to which this disclosure is applicable. Similar reference numbers refer to similar elements throughout.

[0058] As used herein, the terms “data,” “content,” “information,” and similar terms may be used interchangeably to refer to data that can be captured, transmitted, received, displayed, and / or stored, according to various embodiments. Therefore, no use of such terms should be construed as limiting the spirit and scope of this disclosure. Furthermore, where a computing device that receives data from another computing device is described herein, it will be understood that the data may be received directly from the other computing device or indirectly through one or more intermediate computing devices, such as one or more servers, relay devices, routers, network access points, base stations, and / or similar. Similarly, where a computing device that transmits data to another computing device is described herein, it will be understood that the data may be transmitted directly to the other computing device or indirectly through one or more intermediate computing devices, such as one or more servers, relay devices, routers, network access points, base stations, and / or similar.

[0059] System Overview Figure 1 shows a system 100 for determining the integrity state of a mobile device based on the processing of the device's image, according to an example embodiment. The system in Figure 1 may be further used, according to an example embodiment, to detect occlusion in an image, such as an image of a mobile device. The system in Figure 1 and the examples in other drawings are provided as examples of each embodiment (maybe multiple embodiments) and should not be construed as narrowing the scope or spirit of the disclosure in any way. In this regard, the scope of the disclosure includes many potential embodiments in addition to those illustrated and described herein. Thus, although Figure 1 shows one example configuration, numerous other configurations may also be used to implement embodiments of the invention.

[0060] System 100 may include any number of mobile devices 104, or simply “devices” as referred to herein. Mobile devices 104 may be embodied, in non-limiting examples, as any mobile computing device, such as a mobile phone, smartphone, mobile communication device, tablet computing device, any combination thereof, or similar. While described as mobile devices, in some embodiments, mobile devices 104 may instead be replaced by any fixed computing device or other device without departing from the scope of this disclosure. Mobile devices 104 may be used by a user to download, install, and access self-service applications, such as those provided by a provider, in order to obtain compensation for mobile devices 104. Additionally or alternatively, mobile devices 104 may use a browser installed on them to access self-service web applications, such as those hosted and / or provided by a provider. Furthermore, mobile devices 104 may be used to capture images for processing according to exemplary embodiments.

[0061] The device integrity verifier 108 may be associated with a provider or any other entity and may be any processor-driven device that facilitates the processing of requests for device integrity verification, such as those generated from requests to register a device with a device protection program. For example, the device integrity verifier 108 may include one or more computers, servers, server clusters, one or more network nodes, or cloud computing infrastructure configured to facilitate device integrity verification, registration with a device protection plan, and / or other services associated with a provider. In one embodiment, part or all of the device integrity verifier 108 may be implemented on a mobile device 104.

[0062] In one embodiment, the device integrity verifier 108 hosts or provides a service that enables access by the mobile device 104 to request compensation coverage, further encouraging the user of the mobile device 104 to capture an image via the mobile device 104's camera using a mirror, as described in more detail herein. The device integrity verifier 108 may process the image using one or more computer vision and image processing embodiments described herein to determine whether the device is eligible for compensation coverage, as described in more detail herein. The device integrity verifier 108 may include or access one or more models trained to analyze images and extract relevant information, as described in more detail herein, to determine the device's integrity status. According to some embodiments, the collection of training images and the training of models may be performed in the device integrity verifier 108. The device integrity verifier 108 may be further configured to maintain information regarding applied-for and issued device protection plans and / or to facilitate communication between the mobile device 104 and / or an optional internal user device 110.

[0063] The occlusion detection device 109 may be any processor-driven device that facilitates image processing to determine whether an object in the image is occluded. For example, the occlusion detection device 109 may include one or more computers, servers, server clusters, one or more network nodes, or cloud computing infrastructure configured to facilitate image processing and occlusion identification. According to one embodiment, a device integrity verifier 108 may be integrated with the occlusion detection device 109 to determine whether a mobile device in the image is occluded by a finger and / or similar object.

[0064] An optional internal user device 110 may include any or more computing devices that can be used by a provider and / or other entity to facilitate device integrity verification. For example, the internal user device 110 may be implemented in a support center or central facility far from the mobile device where one or more customer service personnel may be stationed who can receive the results of the device integrity verification server using an application provided by the device integrity verification device 108, and which may enable further processing or analysis of images before verification or facilitate additional review. For example, if the device integrity verification device 108 indicates that further internal review of an image is required for verification, such as due to an uncertain mobile device integrity status, the internal user device 110 may be used by support staff to review the image and confirm or reject the integrity of the mobile device 104, thereby confirming or rejecting the coverage of the mobile device 104 in the device protection plan, respectively. The internal user device 110 may also be used by an internal user to capture and / or label training images for training a model(s) with which it may be used. It will be understood that the internal user device 110 may be considered optional. In some embodiments, the device integrity verification device 108 may facilitate faster processing by automatically verifying or rejecting the integrity of the mobile device.

[0065] According to some embodiments, various components of system 100 may be configured to communicate through a network, such as via network 106. For example, a mobile device 104 may be configured to access network 106 via a mobile communication connection, a wireless local area network connection, an Ethernet® connection, and / or similar. Thus, network 106 may include a wired network, a wireless network (e.g., a mobile communication network, a wireless local area network, a wireless wide area network, any combination thereof, or similar), or a combination thereof, and in some embodiments, it may include at least a portion of the Internet.

[0066] As mentioned above, some components of system 100 may be optional. For example, the device integrity verification device 108 may be optional, and device integrity verification may be performed on the mobile device 104, for example, by a self-service application installed on the mobile device 104.

[0067] Referring here to Figure 2, the device 200 is a computing device(s) configured to implement a mobile device 104, a device integrity verifier 108, an image detection occlusion server 109, and / or an internal user device 110, according to an example embodiment. The device 200 may at least partially or completely embody any of the mobile device 104, the device integrity verifier 108, the image detection occlusion server 109, and / or the internal user device 110. The device 200 may be implemented as a distributed system including any of the mobile device 104, the device integrity verifier 108, the image detection occlusion server 109, and / or the internal user device 110, and / or an associated network(s).

[0068] It should be noted that the components, devices, and elements illustrated and described in relation to Figure 2 may not be essential, and therefore some may be omitted in certain embodiments. For example, Figure 2 shows a user interface 216, which may be optional within the device integrity verification apparatus 108, as will be described in more detail below. In addition, some embodiments may include additional or different components, devices, or elements beyond those illustrated and described in relation to Figure 2.

[0069] The device 200 may include a processing circuit 210, which may be configured to perform operations according to one or more embodiments disclosed herein. In this regard, the processing circuit 210 may be configured to perform and / or control the operation of one or more functions of the device 200 according to various embodiments. The processing circuit 210 may be configured to perform data processing, application execution, and / or other processing and management services according to one or more embodiments. In some embodiments, the device 200, or any part or component thereof, such as the processing circuit 210, may be embodied as a circuit chip, or may include a circuit chip. The circuit chip may constitute means for performing one or more operations to provide the functions described herein.

[0070] In some embodiments, the processing circuit 210 may include a processor 212, and in some embodiments, such as those illustrated in Figure 2, it may further include a memory 214. The processing circuit 210 may communicate with or be controlled by a user interface 216 and / or a communication interface 218. Thus, the processing circuit 210 and / or devices 200, such as those contained within a mobile device 104, a device integrity verification device 108, an image detection occlusion server 109, and / or an internal user device 110, may be embodied as a circuit chip (e.g., an integrated circuit chip) configured (e.g., in hardware, software, or a combination of hardware and software) to perform the operations described herein.

[0071] The processor 212 can be embodied in several different ways. For example, the processor 212 can be embodied as a microprocessor or other processing element, a coprocessor, a controller, or one or more other computing or processing devices including integrated circuits, such as an ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or similar. Although shown as a single processor, it will be understood that the processor 212 may comprise multiple processors. Multiple processors may communicate with each other in an operable manner and may be configured collectively to perform one or more functions of the device 200 as described herein. Multiple processors may be embodied on a single computing device or distributed across multiple computing devices configured collectively to function as the mobile device 104, the device integrity verification device 108, the image detection occlusion server 109, the internal user device 110, and / or the device 200. In some embodiments, the processor 212 may be stored in memory 214 or otherwise configured to execute instructions accessible to the processor 212. Therefore, whether composed of hardware or a combination of hardware and software, the processor 212 may, in accordance with embodiments of the present invention, simultaneously be configured accordingly to represent an entity capable of performing operations (e.g., physically embodied in the form of a circuit - processing circuit 210). Thus, for example, if the processor 212 is embodied as an ASIC, FPGA, or similar, the processor 212 may be hardware specifically configured to perform the operations described herein. As another example, if the processor 212 is embodied as a software instruction executor, the instructions may specifically configure the processor 212 to perform one or more operations described herein.

[0072] In some embodiments, memory 214 may include one or more persistent memory devices, such as volatile and / or nonvolatile memory, which may be fixed or removable. In this regard, memory 214 may include a persistent computer-readable storage medium. Although memory 214 is shown as a single memory, it will be understood that memory 214 may include multiple memories. Multiple memories may be embodied on a single computing device or distributed across multiple computing devices. Memory 214 may be configured to store information, data, applications, computer program code, instructions and / or similar items to enable the device 200 to perform various functions, according to one or more embodiments. For example, if the device 200 is implemented as a mobile device 104, a device integrity verification device 108, an image detection occlusion server 109, and / or an internal user device 110, memory 214 may be configured to store computer program code to perform its corresponding function, as described herein according to embodiments.

[0073] Furthermore, memory 214 may be configured to store model(s) and / or training images used to train those models(s) and to predict some relevant information in subsequently received images. Memory 214 may be further configured to buffer input data for processing by processor 212. Additionally or alternatively, memory 214 may be configured to store instructions for execution by processor 212. In some embodiments, memory 214 may include one or more databases that can store various files, contents, or datasets. Among the contents of memory 214, applications may be stored for execution by processor 212 to perform functions associated with each respective application. In some cases, memory 214 may communicate with processor 212, user interface 216, and / or communication interface 218 to pass information between components of device 200.

[0074] An optional user interface 216 may communicate with the processing circuit 210 to receive user input at the user interface 216 and / or provide the user with audible, visual, mechanical, or other output. Therefore, the user interface 216 may include, for example, a keyboard, mouse, display, touchscreen display, microphone, speaker, and / or other input / output mechanisms. For example, in an embodiment where the device 200 is implemented as a mobile device 104, the user interface 216 may, in some embodiments, provide means for displaying instructions for capturing an image. In an embodiment where the device is implemented as an internal user device 110, the user interface 216 may provide means for an internal user or colleague to review an image to verify or reject the integrity of the mobile device 104. The user interface 216 may further be used to label training images for the purpose of training a model(s). In some embodiments, the aspects of the user interface 216 may be limited, or the user interface 216 may not exist.

[0075] The communication interface 218 may include one or more interface mechanisms for enabling communication with other devices and / or networks. In some cases, the communication interface 218 may be any means, such as a device or circuit embodied in either hardware or a combination of hardware and software, configured to receive data from any other device or module communicating with the network and / or processing circuit 210 and / or transmit data to any other device or module communicating with the network and / or processing circuit 210. For example, the communication interface 218 may be configured to enable communication over a network, such as network 106, between any of the mobile device 104, device integrity verifier 108, internal user device 110, and / or device 200. Accordingly, the communication interface 218 may include supporting hardware and / or software to enable, for example, wireless and / or wired communication over cable, digital subscriber line (DSL), universal serial bus (USB), Ethernet, or other methods.

[0076] When the device 200 is embodied by a mobile device 104, the device 200 may include one or more image capture sensors 220. The image capture sensors 220 can be any sensor, such as a camera or other image capture device, configured to capture images from the mobile device 104 and / or record video, and may include a front image capture sensor (e.g., a camera) configured on the same surface as the device's display screen, and / or a rear image capture sensor (e.g., a camera) on the back of the device (e.g., the side of the device without a display screen). In some embodiments, the mobile device 104 may include both front and rear image capture sensors, and in some embodiments, the mobile device 104 may include only one of either the front or rear image capture sensor. In some embodiments, any number of image capture sensors 220 may be present on the device 200 (e.g., the mobile device 104).

[0077] Determining the integrity status of a mobile device We have broadly described the embodiments of System 100 and the devices for implementing these embodiments, but Figures 3 and 4A are flowcharts showing examples of operation of the device 200 according to several embodiments. The operation may be performed by the device 200, such as the mobile device 104, the device integrity verification device 108, the shielding detection device 109, and / or the internal user device 110.

[0078] Figure 3 illustrates operations for determining the integrity status of a mobile device, such as for registration of the mobile device 104 within a device protection plan, according to several embodiments. As shown in operation 302, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or similar, to receive device integrity verification requests associated with the mobile device. In this regard, a user may request registration to the device protection plan by accessing an application (or “app”) installed on the mobile device 104 or a website hosted by the device integrity verifier 108. In some embodiments, the device integrity registration request may be generated by the device integrity verifier 108 or the internal user apparatus 110 during device and / or user orientation. In this regard, according to one embodiment, a device integrity verification request may include, or may be accompanied by, the requested insurance policy and / or details relating to coverage (e.g., an order), as well as other account information related to the user, the user's contact information, and / or similar. It will be understood that a device integrity verification request may be generated for purposes other than device orientation in a protection plan.

[0079] Examples of embodiments may prompt the user to provide certain personal information, mobile device service provider information, and / or user-provided device information about their device, such as via a user interface 216. According to some embodiments, the user may be instructed to use a mobile device 104 to continue the registration process using the device they wish to register. For example, Figures 5A, 5B, 5C, and 5D are examples of user interfaces that may provide introductory information to the user and may be used to collect at least some data from the user, such as the user's mobile device service provider and / or mobile device information (e.g., manufacturer, model, and / or similar). For example, as shown in Figure 5A, an introductory message 500 is provided. As shown in Figure 5B, a prompt 502 for selecting a mobile device service provider and selectable options 504 or eligible mobile device service providers are provided. As shown in Figure 5C, once a mobile device service provider is selected by the user, a confirmation 510 of the selected mobile device service provider, as well as a link to content and additional information 512, are provided.

[0080] As shown in operation 304, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, the user interface 216, the communication interface 218, and / or similar, to receive a mobile device identification data object containing information describing a mobile device, such as the mobile device 104. As previously stated, the user may be prompted via the user interface 216 to provide information describing a device that the user wishes to register for protection. In some embodiments, the user may provide such information via a separate interface and / or network, such as by a personal computer or other mobile or fixed computing device. Any such data describing the device and / or its hardware, such as the device type (e.g., manufacturer, model identifier), the International Mobile Equipment Identity Number (IMEI), and / or similar, may be stored within the mobile device identification data object.

[0081] According to some embodiments, mobile device identification information may not need to be provided by the user, and the mobile device data object may store the IMEI and / or other mobile device identification information systematically obtained by the website and / or app when the user accesses the website and / or app using the mobile device 104. The mobile device identification data object may therefore include other information used to identify or uniquely identify the device, such as the device type, device model identifier, serial number, and / or similar. Figure 5N (described in more detail below) shows a user interface that allows user input of the device IMEI, but according to some embodiments, it will be understood that the IMEI may be systematically obtained as described above. Systematically obtaining mobile device identification information may therefore limit or reduce fraudulent activity, such as by preventing a user from entering the IMEI of a stolen, lost, or damaged device.

[0082] A mobile device identification data object may be used to enroll a device in a device discovery plan, thereby allowing the consumer to generate device reflection data that matches the data stored within the mobile device identification data object (e.g., IMEI) when making a claim. For claims regarding lost or stolen devices, the mobile device service provider may use the data stored within the mobile device identification data object (e.g., IMEI) to prevent future network access and / or use of the device.

[0083] In operation 306, the apparatus 200 may include means such as the mobile device 104, device integrity verifier 108, processor 212, memory 214, user interface 216, communication interface 218, and / or similar, to display prompts on the mobile device 104 to capture at least one image of the mobile device using one or more sensors of the mobile device and a reflective surface such as a mirror. Figures 5D, 5E, 5F, 5G, 5H, 5I, 5J, 5K, 5L, 5M, 5U, 5V, and 5W are examples of user interfaces that guide a user to capture an image of their mobile device using a front camera, a rear camera, or both. As shown in Figure 5D, instruction information 514 may be provided to the user to provide an overview of certain steps related to capturing an image of the mobile device. As shown in Figure 5E, an image capture instruction 516 is provided, as well as selectable prompts 518 and 518 indicating to capture an image of the front and rear (back) of the device, respectively. In response to the selection of selectable prompts 518 and / or 518, the processor 212 of the embodiment activates the image capture sensor 220 associated with the selected prompt 518 or 520, respectively. For example, in a case where prompt 518 is selected to capture an image of the front of the mobile device 104, the processor 212 of the mobile device 104 may activate the front image capture sensor 220. In a case where prompt 520 is selected to capture the back of the mobile device 104, the processor 212 of the mobile device 104 may activate the rear image capture sensor 220. According to some embodiments, any image capture sensor 220 with image capture capabilities may be used to capture an image. As shown in Figure 5F, the embodiment may provide an image capture instruction 526, which may be specific to the device type of the mobile device 104. For example, the image capture instruction 526 in Figure 5F may indicate an instruction to use the “volume up” hard key of the mobile device 104 to capture an image. It will be understood that various embodiments may be considered, such as using hard keys or soft keys (not shown in Figure 5F) to capture images.In one embodiment, the embodiment may differ depending on the device type of the mobile device 104.

[0084] As shown in Figure 5G, a mobile device 104 may provide a security alert 528 prompting the user to allow a mobile application provided by the embodiments provided herein to access or activate an image capture sensor 220, such as a camera. If the user has previously permitted or confirmed a request to allow access to the mobile application, the message may not be displayed. In any case, as shown in Figure 5H, if the user is permitted to allow the mobile application to access the provided image camera 220, the display may reflect a viewfinder 532 to show the image that can be captured.

[0085] In addition to providing prompts for capturing an image, the example embodiment may transition the user interface 216 to display a test pattern to provide improved accuracy in the downstream processing of the captured image, including the device display. For example, the displayed test pattern may include a solid white display screen, as shown in Figure 5H, or other test patterns identified as enabling efficient identification of damage such as cracks and water damage, and / or efficient identification of the display portion of the mobile device relative to the bezel.

[0086] Accordingly, the system may instruct the user to hold the mobile device 104 in front of a mirror or other reflective surface and to use the mobile device 104 to capture an image, for example, using one or more centers of the device (e.g., image capture sensor 220). Once an image is captured as instructed by the user, the captured image may be displayed as a confirmed captured image 528 in Figure 5I. Accordingly, as shown in Figure 5J, the captured image 532 may be displayed within the area of ​​a selectable prompt 518 as shown in Figure 5E, which may be selectable to allow the user to recapture the image. In Figure 5J, the selectable prompt 520 is displayed similarly to the display in Figure 5E, indicating that a rear view photograph has not yet been captured.

[0087] When a selectable prompt 520 is selected, the processor 212 may activate the rear image capture sensor 220 of the mobile device 104 to display the image viewfinder 538 to be captured, as shown in Figure 5K. The user may follow prompts or provide input to capture an image, and the captured image 542 may be displayed as shown in Figure 5L. Accordingly, the display shown in Figure 5M may be updated to reflect the captured images 532 and 542 in the areas of the selectable prompts 518 and 518, respectively. The selectable prompts 518 and 518 may be selected to modify, edit, or review the captured image.

[0088] Returning to the description of Figure 3, in operation 308, in response to the aforementioned image capture operation, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, the image capture sensor 220, the communication interface 218, and / or similar, to receive at least one image captured by the mobile device. The image may then be captured by the mobile device 104 and transmitted to the device integrity verifier 108 (for example, via an app installed on the mobile device and / or the website of the device integrity verifier 108). Additionally or alternatively, the image may be received locally by the mobile device 104 and further processed on the mobile device 104 as described below.

[0089] An identifier generated by the app or website is associated with the image to indicate whether a particular image was submitted as an image of the front or back of the device. The identifier may be received by the mobile device 104 and / or by the device integrity verification device 108 in connection with the received image.

[0090] According to an example embodiment, as shown by operation 310, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar for preprocessing the image. According to an example embodiment, the received image may be cropped, as will be described in more detail herein. According to an example embodiment, the received image may be converted to or reduced to a predetermined size, such as 300 pixels × 300 pixels. According to one example embodiment, some operations described herein may be performed using a single-shot detection algorithm, meaning that the complete image (which may be cropped and resized) is processed as described herein. However, in some embodiments, the image may be divided into sections for individual processing according to any of the operations described herein, and then reconstructed so that the example embodiment can utilize the respective data and / or predictions associated with the separate sections.

[0091] As shown by operation 314, the apparatus 200 may include means such as the mobile device 104, the device integrity verification device 108, the processor 212, the memory 214, and / or similar, for processing at least one image to determine the integrity status of the mobile device. An example of an operation for determining the integrity status of the mobile device is described below with reference to Figure 4A, spanning two pages, according to an example embodiment.

[0092] Determining the integrity status of a mobile device may involve processing an image through a set of conditions implemented by the respective algorithm and / or model. Predictions or results regarding the conditions may indicate the integrity status of the mobile device. For example, a mobile device integrity status may include "Verified," indicating that the mobile device identification has been confirmed and the mobile device is in an acceptable state for enrollment in a protection plan. A "Not Verified" mobile device integrity status may indicate that the device has not yet been verified and / or that one or more of the conditions required for verification have not been met.

[0093] According to some embodiments, an "uncertain" optional integrity state of a mobile device may indicate that the conditions that the embodiment determines are necessary for verification are likely to be satisfied, but that further review should be performed before final verification. Accordingly, in some embodiments, the determination of the integrity state of a mobile device may be based on predictions made by various models and / or algorithms, as well as confidence levels returned by any of the models and / or algorithms indicating the confidence level of certain predictions. In some embodiments, as described herein, verification conditions may include detecting the presence and location of a mobile device, detecting occlusions in the mobile device image, and other relevant assessments. Some embodiments may further evaluate whether the device is insurable and / or not.

[0094] For simplicity, the operation in Figure 4A is described in relation to the processing of a single image, but it will be understood that the processing of the front and rear images may occur simultaneously or sequentially, so that both the front and rear of the device are considered in verifying the device's integrity. According to some embodiments, only one image may need to be processed to verify the device's integrity. In any case, the “image state” may therefore relate to a prediction concerning one image (e.g., a front image or a rear image). A “verified” image state may be required for one or more images (e.g., front and / or rear) for an embodiment to determine that the integrity state of the mobile device is “verified.” Such determinations are described in more detail below with respect to operations 440, 442, 446, 448, and 450.

[0095] According to some embodiments, the determination of whether or not certain conditions are met can be implemented by a model(s) trained to make predictions about an image and / or other algorithms configured to determine the quality of the image. Figure 4B provides an example of a model(s) hierarchy that can be used to implement the operation in Figure 4A, according to an example embodiment. Figure 4B shows the flow of data from one trained model to another, according to an example embodiment. Each model, configured on memory 214 and used and / or trained by an embodiment such as using a processor 212, may include, among others: a mobile device presence model 486 trained to detect whether a mobile device is present in an image; a position detection and cropping model 488 trained to detect the position of a mobile device and optionally crop an image; a cover detection model 490 trained to detect covers on a mobile device present in an image; a mobile device front / back identification model 492 trained to determine whether an image reflects the front or back of a device; a mobile device authenticity model 494 trained to determine whether an image contains a mobile device that has been captured in the image; an occlusion detection model 496 trained to generate a mask used to determine whether an object in an image is occluded; and a damage detection model 498 trained to detect damage to a mobile device in an image.

[0096] Figure 4B reflects a model hierarchy through which an image is fed according to an embodiment. If a particular model predicts that an image does not satisfy a particular condition, the embodiment may prevent further processing by an additional model. However, if a particular model predicts that an image satisfies each of its conditions, the image may continue to be processed by additional models illustrated in the stepwise architecture of Figure 4B. In this way, the efficiency of the system may be improved, enhanced, and / or maximized compared to a system that processes all conditions regardless of other outcomes. It will be understood that the order of models through which an image flows or is processed may be different from or modified from the order illustrated in Figure 4B. In some embodiments, any one or more models may be executed separately for their intended purposes without requiring each step shown in Figure 4B. In this regard, any of the models described herein, and their respective predictions, may be utilized and / or used for other purposes in addition to, or instead of, determining the image state and / or the integrity state of the mobile device.

[0097] Similarly, the order and / or conditions of the operations described with respect to Figure 4A may be changed. For example, an operation identified as consuming fewer resources than others may be processed before one identified as consuming more resources. Additionally or alternatively, if certain conditions that are generally known to result in lower confidence or accuracy of other predictions are not verified, those conditions may be intentionally configured to be processed before other conditions. For example, if an embodiment does not verify that an image includes a mobile device (operation 400, described below), it may not accurately determine whether the image is the front or back of the device (operation 406, described below).

[0098] Continuing the explanation of Figure 4A, as shown in operation 400, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, the mobile device presence model 486, and / or similar, to determine whether at least one image contains a mobile device. The determination may be made, for example, using the mobile device presence model 486 deployed on the mobile device 104 and / or the device integrity verifier 108. The user interface 216 prompts the user to capture an image of their device using a mirror, but the user may submit an image that does not contain a mobile device. For example, some users may attempt to cheat by taking a photograph of a paper model made to look like a mobile device. Other users may intentionally or unintentionally capture an image that does not contain a mobile device.

[0099] The processor 212 may process the target image using at least one trained model (e.g., a neural network) trained on multiple training images, each labeled either as containing a mobile device or excluding a mobile device, in order to determine whether the target image contains a mobile device.

[0100] In any case, if an embodiment predicts that at least one image does not include a mobile device, as shown in operation 430, such as using mobile device presence model 486, the device 200 may include means such as the mobile device 104, device integrity verifier 108, processor 212, memory 214, user interface 216, communication interface 218, and / or similar to provide the user with feedback indicating that it captures an image of its device and / or to determine that the image state is “unverified.” The feedback may include causing one or more instructions to be sent to and / or displayed on the mobile device.

[0101] In this regard, the user may be given the opportunity to recapture the image for reprocessing and verification. A message such as that shown in the user interface of Figure 5R may be provided to the user. Operation 430 indicates that user feedback is provided at the user's discretion, but according to some embodiments, it will be understood that, as a result of certain or all of the conditional operations 400, 403, 405, 406, 410, 416, 420, 426 and / or 442, the user may be provided with more specific instructions relating to certain conditions (e.g., problems with the captured image) that have been processed but have not reached device integrity verification. If the user provides a new image(s), the process may return to operation 400 to process the newly captured image.

[0102] In one embodiment, it will be understood that operation 400 may be performed in a single shot for each image, or the image may be subdivided into sections, so that each separate section is processed as described herein.

[0103] If the embodiment determines that at least one image contains a mobile device, further processing may follow operation 403. At least part of the remaining operations in Figure 4A are described with reference to a mobile device in an image, or a captured mobile device. Such reference will be understood to mean a processor-driven prediction that a mobile device is likely present in the image, and therefore the captured mobile device is a suspected mobile device.

[0104] In operation 403, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, the position detection and cropping model 488 and / or similar, to determine the position of the mobile device in the image. In this regard, an example embodiment may predict the bounding box or lower portion of the image in which the mobile device is located, for example, using the position detection and cropping model 488 and / or each of its models. If the bounding box has a predetermined relationship (e.g., less than or equal to) with respect to the minimum threshold ratio of the image (e.g., 25%), the example embodiment may determine that the mobile device 104 is too far from the mirror or other reflective surface when the image is captured (e.g., too far to provide additional processing and predictions regarding the mobile device with threshold confidence). Thus, the apparatus 200 may determine, for example, in operation 430, that the image state is "unverified" and optionally provide feedback to the user, such as indicating that the mobile device 104 should be held closer to the mirror when recapturing the image.

[0105] If the bounding box is determined to have a different predetermined relationship (e.g., greater than or equal to, or greater than) compared to the minimum threshold ratio of the image, the embodiment may determine that the mobile device 104 was close enough to the mirror when the image was captured (e.g., close enough to provide additional processing and predictions with respect to the mobile device with threshold confidence), and therefore processing can continue.

[0106] In operation 404, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, a position detection and cropping model 488, and / or similar, to crop the image so that the area outside the bounding box is removed. The cropped image may then be reduced and resized to a predetermined size, such as 300 pixels × 300 pixels. The cropped image may be processed as described further below, even if in some cases the image is different from the originally captured image that may be cropped according to the embodiment, to avoid unnecessarily complicating the explanation, and the cropped image will be referred to as the “image” or “captured image”.

[0107] As shown by operation 405, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, the cover detection model 490 and / or similar, to determine whether the captured mobile device in the image is without a cover(s) or has a cover(s), such as a cover(s) that would prevent an accurate assessment of the state of the mobile device 104. The determination may be made using the cover detection model 490, which has been trained to detect covers on the captured mobile device in the image. An example embodiment may process the image using a model (e.g., the cover detection model 490) that has been trained to predict or detect whether the user has placed a cover on the mobile device when capturing the image. Accordingly, in operation 430, the example embodiment may provide feedback to the user, such as suggesting that the cover be removed and the image recaptured. The example embodiment may further determine that the image state is “unverified”. In this regard, the user may be given the opportunity to recapture the image for reprocessing.

[0108] If the embodiment determines that at least one image does not have a cover over it, further processing may continue in operation 406. As shown by operation 406, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, to determine whether at least one image includes a indicating surface of a mobile device. "Indicating surface" does not necessarily mean a surface pointed to by a user, but may mean an indicating surface that is systematically pointed to in relation to a captured image, which may be generated by the app due to a user being separately prompted to capture the front and back of the device.

[0109] The determination may be made, for example, using a mobile device front / back identification model 492 deployed on the mobile device 104 and / or device integrity verification device 108. In the embodiment, the image may be run through the model (e.g., mobile device front / back identification model 492) to confirm that the image captures the designated face (e.g., front or back). If the user determines that the wrong face of the device has been captured, in operation 430, the embodiment may provide feedback to the user, such as instructing them to capture the designated face of the device (e.g., front or back). The embodiment may further determine that the image state is "unverified". In this regard, the user may be given the opportunity to recapture the image for reprocessing.

[0110] If it is determined that the image reflects a face of the indicated device (e.g., front or back), processing may continue in operation 410. As shown by operation 410, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, a mobile device authenticity model 494, and / or similar, to determine whether at least one image contains a mobile device associated with a mobile device identification data object. In other words, an example embodiment determines whether at least one image contains a mobile device from which a mobile device was captured. In some cases, a user might attempt to commit fraud by using their mobile device and a mirror to capture images of different intact phones. An example embodiment may use the mobile device authenticity model 494 to estimate the angle of the mobile device relative to the reflective surface using the image to predict whether the device present in the image is indeed the mobile device from which the image was captured, or whether the captured device in the image is a different device.

[0111] As another example, an embodiment may generate predictions of the identification of a suspicious mobile device in an image based on the image, for example, using a mobile device authenticity model 494. For example, the mobile device authenticity model 494 may predict the manufacturer and / or model of the mobile device, and the embodiment may compare the predicted mobile device identification with the identification indicated by a mobile device identification data object (e.g., IMEI) to determine whether the image reflects the characteristics of the device expected based on the mobile device identification data object.

[0112] If it is determined that the mobile device in the image is different from the one from which the image was captured, operation 430 may optionally provide the user with feedback to capture an image of their own mobile device using the same mobile device 104 (e.g., the mobile device for which a protection plan is desired) from which the device integrity verification request was sent. An example embodiment may further determine that the image state is "unverified".

[0113] If the embodiment determines that the mobile device in the image is indeed the mobile device 104 from which the image was captured, processing may continue in operation 416. As shown by operation 416, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, and / or similar, to determine whether the quality of at least one image is sufficient for further processing. According to one embodiment, image blurriness may be determined by an implementation of the Laplacian variance metric.

[0114] Due to various external environmental factors and / or the positioning of the mobile device 104 relative to the mirror and / or similar factors, some images may be too blurry to be further processed for detecting occlusion or damage (as described in more detail below). Additionally, or alternatively, images may be too blurry to make other predictions, including those described above, and therefore it may be advantageous for the embodiments to evaluate image quality and blurriness before performing the operations described herein. In some examples, image quality may be sufficient to perform one task but not another, and therefore various image quality verifications may be performed through the processes illustrated by Figure 4A.

[0115] In any case, operation 430 may provide the user with feedback to recapture the image, and may include further guidance on how to position the mobile device 104 relative to the mirror to capture an image that is of sufficient quality for further processing. Figure 5S provides an example interface that prompts the user to retake a photo because the photo is too blurry. Further instructions may be provided on how to adjust the angle or orientation of the mobile device 104 relative to the mirror by moving the mobile device 104 closer to or further away from the mirror. The example embodiment may further determine that the image state is "unverified," and the user may be given the opportunity to recapture the image for reprocessing.

[0116] If the embodiment determines that the image quality is sufficient for processing, processing may continue in operation 420. As shown by operation 420, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the occlusion detection server 109, the processor 212, the memory 214, the occlusion detection model 496, and / or similar, to determine whether at least one image is unoccluded or contains an object that occludes the mobile device 104. To avoid unnecessarily complicating the flowchart, operation 420 indicates that the image is either unoccluded or occluded. However, it will be understood that the degree or amount of occlusion is determined and considered in determining whether the image state is set to “unverified” or “verified,” as described herein.

[0117] For example, a user may inadvertently or intentionally cover a part of the mobile device 104, such as a crack or other damage on the display screen, or another part of the mobile device 104. An example embodiment may use the occlusion detection model 496 to generate a mask, which is used in detecting occlusions such as blocked corners (e.g., fingers covering the corners of the mobile device) and concave occlusions (e.g., fingers protruding into a part of the captured mobile device), as will be described in more detail below.

[0118] Small occlusions covering the bezel or outer portion of the surface of the mobile device 104 may be acceptable, but large occlusions obscuring a significant portion of the display screen or other significant parts of the device may not be acceptable. If an embodiment determines that the mobile device 104 is obscured by an object and therefore the integrity of the device cannot be verified, the process may proceed to operation 430, prompting the user to retake the image without occlusion (for example, by holding only the edges of the device with their fingers, without covering the front or back of the device), and determine that the mobile image state is "unverified". Further details regarding occlusion detection are provided below with respect to Figure 6 in a section titled "Occlusion Detection".

[0119] If no obstruction is detected, or any such obstruction is small and does not interfere with further processing and analysis of damage or other conditions, the process may proceed to operation 426. As shown in operation 426, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, the damage detection model 498, and / or similar, to determine whether at least one image indicates that the device is undamaged or contains damage. Additionally or alternatively, the damage detection model 498 may determine or predict the presence of certain types of damage, such as cracks, water damage, dents, and / or any other damage that would prevent the mobile device from being insured or protected. In some embodiments, if a model determines that damage is likely, a separate model may predict the type of damage. In any case, the embodiment determines whether there is any pre-existing damage to the mobile device 104 such that coverage of the protection plan should be denied. To avoid unnecessarily complicating the flowchart, operation 426 indicates that the image is either undamaged or damaged. However, it will be understood that the degree or amount of damage is taken into consideration in determining whether the image state is set to “unverified,” “verified,” or “uncertain,” as described herein.

[0120] Further details regarding the damage detection model 498, which utilizes training images and / or models to detect damage to mobile devices, are provided below. If damage is detected, in operation 430, the embodiment may provide the user with a response indicating that damage has been detected and / or that a device protection plan cannot be issued. The embodiment may further determine that the image state is “unverified”.

[0121] In cases where it is determined that there is no damage to the mobile device 104 and / or that the physical condition and / or operability parameters of the mobile device 104 are sufficient for the purpose of insuranceability, the process may proceed to operation 440. As shown in operation 440, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, and / or similar, to determine that the image state is “verified.” It will be understood that certain operations illustrated in Figure 4A may not be present in certain embodiments. Accordingly, the apparatus 200 may be configured to require any number of verifications and / or conditions described with respect to Figure 4A, so that an image state can be determined to be “verified” when all desired (e.g., by a provider, etc.) verifications or conditions have been performed.

[0122] In some embodiments, one or more models, such as the occlusion detection model 496 and the cover detection model 490, may be executed in parallel. In some embodiments, the output of the image cropping model 488 may be supplied simultaneously or in any order to one or more of the cover detection model 490, the mobile device front / back identification model 492, the mobile device authenticity model 494, the occlusion detection model 496, and / or the damage detection model 498.

[0123] According to one embodiment, the "verified" image state may be required for multiple images, such as a front image and a back image. Therefore, in an example where both the front and back (and / or any other image) should be verified to confirm the integrity status of the mobile device, operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440 in Figure 4A may be repeated separately for each image requested by the insurer. For example, an image instructed to capture the front of the device may be processed according to operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440, and an image instructed to capture the back of the device may be processed according to operations 400, 403, 404, 405, 406, 410, 416, 420, 426, 430, and / or 440.

[0124] Therefore, as shown by operation 442, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, and / or similar, to determine whether all necessary images (e.g., those required for the purpose of determining the integrity status of the mobile device) have a “verified” image state. Certain images (e.g., front and back) may be pre-configured or set by the device integrity verifier 108 and may be related to the provider’s requirements for registering the device with a protection plan.

[0125] For example, if front and back images of a device are required and both images have a “verified” image state, the device integrity state may also be set to “verified.” However, if both front and back images of a device are required and only one or neither image has a “verified” image state, the mobile device integrity state should remain blank or be set to “unverified” until at least both images have a “verified” image state. For example, as shown in operation 446, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, and / or similar to determine that the device integrity state is “unverified.” Figure 5T provides an example user interface showing that a front photograph may be approved, but a back photograph (e.g., a rear photograph) may still need to be captured and processed. If one or both images are “unverified,” the example embodiment may prompt the user to capture or recapture each image.

[0126] As shown in operation 448, it will be understood that the embodiment example may be configured to determine the integrity state of a mobile device as “verified” based on a first threshold confidence level (which may be configurable), or based on a first threshold confidence level. For example, determining the integrity state of a mobile device as “verified” may require not only a “verified” state for all required images, but also a minimum overall or average confidence level for all conditions being evaluated. The first threshold confidence level test may therefore be of any choice and may be configured in various ways. For example, although not illustrated in Figure 4A, in one embodiment, the threshold confidence level for a particular prediction (e.g., a condition) may be calculated in relation to any of the predictions made in operations 400, 403, 405, 406, 410, 416, 420, and / or 426. For example, some models may be configured to provide not only predictions, but also confidence levels that reflect the reliability of the accurate predictions. Thus, a threshold confidence level may be required for each condition that must be satisfied before proceeding to the next condition. In one embodiment, the average confidence level for all conditions may need to be 95% or higher to set the mobile device's integrity status to "verified." In another example, all conditions may need to have a confidence level of 98% or higher to set the mobile device's integrity status to "verified."

[0127] In any case, if all required images have a “verified” image status, as shown by operation 442, and the first threshold confidence level is satisfied, as shown by operation 448, the integrity status of the mobile device can be determined to be “verified,” as shown by operation 450.

[0128] In this regard, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, and / or similar, to determine that the device integrity state is "verified". If the first threshold confidence level is not implemented, in one embodiment, operation 448 may be omitted or bypassed, and the verification in operation 442 that all images have a "verified" image state may proceed to operation 450, and the integrity state of the mobile device may be determined to be "verified".

[0129] According to some embodiments, if the integrity status of a mobile device is set to "verified," the mobile device 104 may be automatically enrolled in a protection plan, and confirmation may be provided to the user via the user interface 216. For example, Figure 5Y provides confirmation that the device is insured. According to some examples, automatic enrollment and confirmation may be provided in real time or near real time during a session in which coverage is requested and images are captured by the user.

[0130] As an addition or alternative, in response to a determination of "verified" mobile device integrity status, the device may not necessarily be automatically enrolled in the protection plan, but rather may be forwarded to an internal user device 110 for internal review, such as by the mobile device 104 and / or the device integrity verification device 108. In such an example, the embodiment may provide a message indicating that the image has been submitted for review, such as those in Figures 5O, 5P, 5Q, and / or 5X. Accordingly, if the provider wishes to further internally review the image before enrolling any mobile device in the protection plan (for example, and not providing automatic enrollment), the embodiment may nevertheless conveniently remove images that are predicted to be neither acceptable nor verifiable, and optionally provide feedback to the user to facilitate efficient device enrollment.

[0131] As another example, as shown in operations 468 and 470, even if the first threshold confidence level is not satisfied at 448, if the second threshold confidence level is satisfied (e.g., 90%), the embodiment may determine the integrity status of the mobile device to be "uncertain" and indicate that further review should be performed, such as using the internal user device 110. Accordingly, the embodiment may be configured to "automatically register" devices determined to be low risk, following the process in Figure 4A, but the provider may reserve the opportunity to further internally review images before registering any mobile device with associated images determined to be high risk. Furthermore, if the confidence level does not satisfy either the first or second threshold confidence level, the integrity status of the mobile device may be determined to be "unverified" (446), and associated requests for insurance and / or similar may be rejected without further manual review. The embodiment may therefore conveniently remove images predicted to be neither acceptable nor verifiable and optionally provide feedback to the user to facilitate efficient device registration.

[0132] In any case, the examples of embodiments may be configured to systematically perform any amount or all of the set of verifications and / or validations required, and in some embodiments, the level of systematic verification and / or validation may be balanced with internal (e.g., human) review, for example, as required by the provider. Various configurations and thresholds of confidence for different stages of processing may be contemplated.

[0133] Regardless of the implemented variations described above, one embodiment may provide additional user interface displays, such examples are described below. In one embodiment, the user interface display in Figure 5N may be considered optional and may allow input, confirmation, or modification of device identification information such as the device IMEI 550. As previously stated, the IMEI may be detected and / or data may be entered without explicit input by the user, hence the user interface display in Figure 5N is optional. However, in one embodiment, the processor 212 may receive the device identifier via a user interface, such as that in Figure 5N.

[0134] It will be further understood that certain updates and / or statuses may be provided to the user before, during, or after an operation related to determining the integrity status of the mobile device. For example, an pending review status message 556 may be displayed, as shown in Figure 5O. As shown in Figure 5P, the processor 212 may invoke a notification permission message 560, such as one that may be generated by the mobile device 104, in response to the mobile application of the embodiment that enables or attempts to enable notifications on the mobile device 104. In this regard, the user may allow or deny the mobile application of the embodiment to send or push notifications. If notifications are permitted for the mobile application of the embodiment, notifications may be provided to the user at various points in time during the processes and operations described herein.

[0135] In one embodiment, the user can access the example mobile application to view the status related to the request. For example, a review status message 564 may provide the status that a photograph has been submitted and is currently under review. In one embodiment, if notifications to the example mobile application are enabled on the mobile device 104, a notification such as notification 570 in Figure 5R may be displayed. Notification 570 indicates to the user that the image of the back of the mobile device needs to be retaken. The user can select the notification and access the mobile application to retake the image.

[0136] Accordingly, one embodiment may provide feedback to the user when accessing the mobile application of the embodiment example, such as the feedback summary 574 and reason 576 in Figure 5S. For example, the feedback summary 574 indicates that the back image needs to be retaken, and the reason 576 indicates that the image needs to be retaken because the photo is too blurry.

[0137] In one embodiment, as shown in Figure 5T, a message such as an image approved message 580 may be displayed within the area of ​​the selectable prompt 518 in Figure 5J, for example. As shown in Figure 5T, the selectable prompt 520 may not be populated with a message because the back image has not yet been captured. Accordingly, the user may choose to capture an image, and the user interface displays in Figures 5U and 5V may be updated to provide the finder 538 and the captured image 542, respectively.

[0138] Accordingly, as shown in Figure 5W, an image approved message 580 is displayed for one image, such as an image of the front of the device, while a captured image 542 is displayed for another image, such as an image of the back of the device. In this regard, since each image has not yet been approved, a selectable prompt 520 is displayed to allow the captured image 142 to be recaptured or edited. In one embodiment, selecting the selectable prompt 520 may display a review status message 564, such as that in Figure 5X.

[0139] It will be understood that any user interface display provided herein is updated in accordance with the status, such as the image status, and / or the integrity status of the mobile device is updated and / or entered as described with respect to Figure 4A. In one embodiment, a mobile device integrity status such as "unverified" or "uncertain" may be shown to the user as "pending," and a status such as "verified" may be shown to the user as "verified." Accordingly, the integrity status of the mobile device is set to "uncertain," but the associated image may be put into a queue for manual review and / or similar.

[0140] Furthermore, according to one embodiment, if notifications are permitted on the mobile device 104, the mobile application of the embodiment may initiate a notification 590 if the integrity status of the mobile device is determined to be "verified". Thus, the user may be notified that the image is approved, that they can or have already enrolled in a protection plan for their mobile device.

[0141] Shielding detection Figure 6 is a flowchart of operations for detecting occlusion, such as by an occlusion detection device 109, according to an example embodiment. The operations in Figure 6 may be triggered by operation 420 and, in other examples, may be performed as a separate process not necessarily related to the mobile device image. The operations in Figure 6 may utilize or provide an image segmentation approach for detecting occlusion.

[0142] In operation 600, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, an occlusion detection model 496, and / or similar, to generate a mask containing fewer colors compared to the number of colors in the original image. As described herein in the examples of embodiments, the mask may be described in relation to a mobile device in the image, but it will be understood that mask generation may be performed on any object detected or present in the original image. With respect to the example of mask generation for a mobile device (for example, for the purpose of determining the integrity status of the mobile device), the original image may be considered an image captured by the mobile device, and the generated mask may be considered a mobile device mask.

[0143] A mask can be thought of as an additional image generated from processing the original image (for example, an image of a mobile device captured by another mobile device, which may have been previously cropped according to position detection and cropping model 488). A mask may be an image with a reduced number of colors compared to the original image. For example, Figures 7A and 8A are examples of images of a mobile device captured by a user, containing a wide range of colors, while Figures 7B and 8B are masks generated according to an example embodiment, containing binary values ​​(represented as black and white pixels in Figures 7B and 8B). However, it will be understood that other configurations of colors may be selected to generate a mask. An example of the process for generating a mask will be described in more detail below with respect to the configuration, training, and deployment of occlusion detection model 496. It will be understood that separate occlusion detection models 496 and / or models thereof may be used for the front and back of the device.

[0144] According to one embodiment, the model may return an array of values ​​indicating whether a particular pixel should belong to a mask. The embodiment may then determine, based on a threshold, whether the pixel should be white (e.g., included in the mask) or black (e.g., not included in the mask).

[0145] As shown by operation 602, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, for extracting polygonal subregions P of the mask. Accordingly, in the embodiment, an algorithm such as a marching square algorithm may be applied to extract the largest polygonal subregion of the mask. In some embodiments, the largest polygonal subregion P may be the mobile device screen when the image cropping model 488 and other related models and preprocessing steps have generated the original image for an occlusion detection model 496 in which the mobile device is detected and the image is substantially cropped before generating the aforementioned mask. Accordingly, small islands (including, for example, small outlier polygons, black pixels appearing in mostly white areas, such as those that may result from camera defects, dust / dirt, and / or other minor environmental factors present on the device or mirror, and / or similar) may be removed.

[0146] As shown by operation 604, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, for determining the convex hull of P. The convex hull can be computed according to a commonly known computational geometry algorithm.

[0147] Using a polygonal subregion P together with its convex hull enables the embodiment to identify concave occlusions, such as the concave occlusion 700 in Figures 7A and 7B. Subprocesses for identifying concave occlusions are provided by operations 608, 612, 616, and 620. Additionally or alternatively, the embodiment may use a polygonal subregion P together with its convex hull to identify blocked angles, such as the blocked angle 800 in Figures 8A and 8B. Subprocesses for identifying blocked angles are provided by operations 630, 634, 638, and 642. According to the embodiment, both subprocesses or one of the subprocesses may be implemented and executed.

[0148] As shown by operation 608, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, to calculate the difference between P and the convex hull. In operation 612, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, to reduce or eliminate fine discrepancies at the edges of P and the convex hull. An example embodiment may reduce or eliminate discrepancies by performing pixel erosion and pixel expansion techniques, as may be provided by Shapely and / or other libraries. In operation 616, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, to recalculate P as the maximum area of ​​the remaining region. In this regard, P may be identified as the maximum area of ​​the remaining connected region of P. P can therefore be considered as the estimated region of the visible screen (e.g., the unobstructed portion of the screen).

[0149] In operation 620, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, to determine a depression as the difference between P and the convex hull. In some examples, any such depression may be compared to a threshold for further filtering, so that very small depressions are not necessarily flagged as such, but larger depressions that may be distracting for other downstream tasks (such as determining whether damage is present on the device) may be flagged so that the user is prompted to recapture the image. For example, if a particular detected depression is larger than, equal to, or greater than a predetermined threshold size, the area may remain predicted to be a depression. If an area initially identified as a depression is smaller than, equal to, or less than a predetermined threshold size, the area may be ignored as a depression. In this regard, if the embodiment predicts the presence of a depression (and / or that the depression is large enough to be distracting to downstream tasks, as indicated by the use of a threshold), operation 420 may determine that the image contains occlusion and cause the embodiment to determine the image state as "unverified," and optionally prompt the user to recapture the image.

[0150] In operation 630, the apparatus 200 may include means such as a mobile device 104, a device integrity verifier 108, a processor 212, a memory 214, and / or similar, for determining a predetermined number of principal edges of the convex hull. In the mobile device mask example, an example embodiment may determine four principal edges, and according to some embodiments, four most principal edges may be identified. In this regard, the number of principal edges identified may depend on the type of object for which the mask is created.

[0151] Hough transform feature extraction techniques are implemented to identify major edges or a predetermined number of most major edges, and thus predict where the edges of the mobile device should appear in the image (e.g., assuming an edge is unoccluded, even if it is occluded). As shown by operation 634, the apparatus 200 may include means such as the mobile device 104, device integrity verifier 108, processor 212, memory 214, and / or similar to identify the protruding corner points of the mobile device in the image (which may be occluded or unoccluded) by identifying the intersections of adjacent edges (identified based on their respective angles). In operation 638, the apparatus 200 may include means such as the mobile device 104, device integrity verifier 108, processor 212, memory 214, and / or similar to determine the distance from each protruding corner to P. For example, determining the distance may include measuring the shortest distance from the estimated corner to the nearest edge or point of P.

[0152] In operation 642, the apparatus 200 may include means such as the mobile device 104, the device integrity verifier 108, the processor 212, the memory 214, and / or similar, to compare each distance to a threshold to determine whether any corner is blocked. For example, if a distance is greater than, or equal to, a predetermined threshold distance, the embodiment may determine that each corner is blocked. In this regard, when used to determine the integrity status of a mobile device, operation 420 may determine that the image contains occlusion and cause the embodiment to determine the image state as "unverified," prompting the user to optionally recapture the image. If there is no calculated distance greater than, or equal to, a predetermined threshold distance, the embodiment may determine that no corner of the object is blocked in the original image.

[0153] Although not shown in Figure 6, in one embodiment where the image is processed for both concave occlusion and blocked corners, but neither is detected, operation 420 may determine, following the operations described above, that there is no occlusion on the mobile device in the image, i.e., no occlusion that could affect subsequent processing of the image. Therefore, small occlusions that do not affect subsequent processing may be acceptable.

[0154] Model configuration, training, and deployment Training of the model(s) used in the embodiment(s) (e.g., neural networks(s)) may occur before the model is deployed (e.g., before use by the mobile device 104 and / or device integrity verifier 108 in determining whether the device is eligible for compensation in the plan, and / or before use by the occlusion detection device 109 in determining whether an object in an image is occluded). According to some embodiments, training may be performed continuously by receiving images and associated classifications and / or labels that have been reviewed by either the embodiment or / or a human reviewer. Machine learning may be used to develop a specific pattern recognition algorithm (i.e., an algorithm representing a specific parameter recognition problem) that can be derived from statistical inference, and to train a model(s) based on it.

[0155] Examples of embodiments include receiving and storing multiple types of data, including datasets, using a communication interface 218, memory 214, and / or similar, and using the data in multiple ways, such as using a processor 212. The device integrity verifier 108 may receive datasets from a computing device. The datasets are stored in memory 214 and can be used for various purposes. The datasets can therefore be used for modeling, machine learning, and artificial intelligence (AI). Machine learning and related artificial intelligence can be performed by the device integrity verifier 108 based on various modeling techniques.

[0156] For example, a set of clusters may be developed by unsupervised learning, where the number of clusters and their respective sizes are based on the calculation of the similarity of the pattern features within a previously collected training set of patterns. In another example, a classifier representing a particular categorization problem or task may be developed using supervised learning based on a training set of patterns and their respective known categories. Each training pattern is input to the classifier, and the difference between the output categories produced by the classifier and the known categories is used to adjust the classifier coefficients to more accurately represent the problem. Classifiers developed using supervised learning are also known as trainable classifiers.

[0157] In some embodiments, the dataset analysis includes a source-specific classifier that takes a source-specific representation of a dataset received from a particular source as input and generates outputs that categorize the input as likely to contain relevant data references or unlikely to contain relevant data references (e.g., likely or unlikely to satisfy a required criterion). In some embodiments, the source-specific classifier is a trainable classifier that can be optimized as more instances of the dataset, since the analysis is received from a particular source.

[0158] Alternatively or additionally, trained models may be trained to extract one or more features from historical data using pattern recognition, based on unsupervised learning, supervised learning, semi-supervised learning, reinforcement learning, correlation rule learning, Bayesian learning, and solutions for probabilistic graphical models, among other computational intelligence algorithms that can use an interactive process to extract patterns from data. In some examples, historical data may include data generated using user input, cloud-based input, or similar (e.g., user confirmation).

[0159] A model(s) can be initialized with multiple nodes. In some embodiments, an existing deep learning framework can be used to initialize the model(s). The model(s) can be implemented as a convolutional neural network (CNN), a recurrent neural network (RNN), a long-shortening memory (LSTM) network, and / or similar. According to one embodiment, any of the models described herein can utilize an existing or pre-trained model, and such model can be further trained with training data specific to its respective task and / or condition as described herein. For example, the device integrity verification device 108 can develop templates, such as using any known or modified machine learning templating technique. In this regard, a templated model for each of the tasks and / or conditions described herein can be utilized by an embodiment to further train its respective model. In this regard, an embodiment can utilize a template with domain-specific data schemes and models. For example, a template design for identifying a certain texture in an image can be used for cover detection and / or damage prediction. A template for identifying whether a particular object is present in an image can be utilized for mobile device presence detection.

[0160] According to one embodiment, CNNs and related deep learning algorithms can be particularly useful in the application of machine learning to image processing by generating multiple connected layers of perceptrons. Each layer may be connected to its neighboring layers, which provides an efficient basis for measuring the weights of the loss function and identifying patterns in the data. Accordingly, the machine learning algorithms of the embodiment can efficiently produce accurate predictions about images by training related models, such as CNNs, to learn which features are important in the image.

[0161] Using the techniques described herein, a model may then be trained to determine one or more features of an image to generate one or more predictions associated with the methods and embodiments described herein. The training data may also be selected from a predetermined period, such as several days, weeks, or months prior to today.

[0162] In an example embodiment, labeled datasets, such as those associated with a specific task or prediction as described herein, may be fed into the device integrity verifier 108 to train a model(s). The model(s) may then be trained to identify and classify subsequently received images received from the computing device in correspondence with one or more labeled criteria.

[0163] In some embodiments, the AI ​​and models described herein utilize deep learning modules. Deep learning is a subset of machine learning that generates models based on training datasets provided to it. Deep learning networks can be used to draw in large amounts of input and train algorithms to learn which inputs are relevant. In some embodiments, the trained models may use unsupervised learning techniques including clustering, anomaly detection, and Hebb learning, as well as learned latent variable models such as expectation maximization algorithms, methods of moments (mean, covariance), and blind signal separation techniques including principal component analysis, independent component analysis, non-negative matrix factorization, and singular value decomposition.

[0164] Accordingly, the embodiment may input multiple training images and corresponding labels into an initialized model(s) to be used to train or further train a model(s), and use the processor 212 to learn features via supervised or unsupervised deep learning.

[0165] In this regard, the training images, some of which include the mobile device from which they were captured, and others which do not, are input into each model along with associated labels that exhibit various characteristics depending on the specific model being trained. The training set may include hundreds or thousands of images that have been reviewed and labeled by a user or data scientist (for example, using an internal user device 110).

[0166] The model can transform an image into a matrix representation of the image and process the image together with its verified labels (e.g., "contains a mobile device", "does not contain a mobile device") to learn image features that correlate with those labels. In one example, an image may have multiple labels for each state, and therefore one image can be used to train multiple different models and / or neural networks. For example, one image may be labeled "contains a mobile device", "contains a cover", and "contains damage", and therefore one image can be used by the processor 212 of the embodiment to train three distinct models, such as a mobile device presence model, a cover detection model, and a damage detection model, respectively. In one example, an image can be used to train a single model. In this regard, training data can be collected and used in various ways.

[0167] The processor 212 trains the model(s) on training images, adjusting the parameters of each model through a series of deep learning iterations to reconstruct the matrix representation of the images, thereby capturing their features or giving more weight to those features that are strong indicators of a particular label or classification. Various techniques, including but not limited to fractal dimension, may be used during model training. Fractal dimension is a statistical analysis that can be employed by machine learning algorithms to detect which features are stronger indicators of a certain prediction and / or condition, such as those described herein, and at what scale. The scaling of the training images may be adjusted according to fractal dimension techniques, which may vary depending on the specific task or prediction being made. For example, the detection of damage such as flooding and / or cracks using a machine learning algorithm may require images with a higher resolution than what may be required for detecting whether cover is on a device. In this regard, fractal dimension algorithms can be used to adjust the image resolution to balance the accuracy and efficiency of one or each model.

[0168] Further details regarding the configuration, training, and deployment of each model(s) for each of the tasks, conditions, and / or methods associated with the examples of embodiments are described below. It will be further understood that some models may employ other classification techniques, such as, but not limited to, support vector machines, decision trees, random forests, naive Bayesian classifiers, and logistic regression, instead of or in addition to neural networks.

[0169] While a processor 212 in one embodiment may conveniently use separate models to make separate predictions for the various conditions described herein, it will be further understood that in one embodiment, the integrity state of a mobile device can be assessed as “verified,” “unverified,” and / or “uncertain” by utilizing a single model (e.g., a neural network). In this regard, the model (e.g., a neural network) may be trained with images labeled “verified,” “unverified,” and / or “uncertain,” and the model can determine which images should be predicted as “verified,” “unverified,” and / or “uncertain” by essentially assessing which images contain a mobile device, which are front or back, and which contain or do not contain damage. However, the use of a single model may require more training data to produce accurate or meaningful results compared to utilizing separate models for at least some of the conditions described herein. At least one additional benefit of using separate models and generating confidence levels for each at will is that it allows the user capturing the image to provide specific feedback, such as "Move your device closer to the mirror," "Remove the cover from your device," "Place your finger on the edge of the device and retake the photo," and / or similar. Such feedback can result in an improved or increased automated verification rate, while reducing the need for uncertainty or manual review.

[0170] Mobile device existence model The mobile device presence model 486 enables an embodiment to automatically predict (e.g., without human review) whether a newly received image contains or does not contain a mobile device. The embodiment's processor 212, such as device 200, may utilize an existing model, such as an implementation of Torchvision by SqueezeNet, and pre-train the model using weights established by a visual database such as ImageNet. The embodiment may further train the model for the mobile device presence detection task by inputting model training images as well as corresponding labels such as “contains device” and “does not contain device.” In this regard, the model (e.g., a neural network) may be trained on at least two sets of images, such as a first set of training images containing mobile devices and a second set of training images not containing mobile devices.

[0171] Various deep learning methods may then be used to process training images and corresponding labels through the model, according to the embodiment, to train the model to generate predictions for subsequently received images. According to the embodiment, once deployed, the trained mobile device presence model 486 may generate an index of the likelihood of an image containing a mobile device. For example, the index may be a number between 0 and 1, with a number close to 1 indicating a high probability of a mobile device being present. Thus, the index may reflect the confidence of the prediction.

[0172] Accordingly, the processor 212 of the embodiment may process the target image using at least one model trained on a plurality of training images, each labeled either as containing a mobile device or excluding a mobile device, in order to determine whether the target image contains a mobile device.

[0173] In the context of determining the integrity state or image state of a mobile device, an example embodiment may utilize a mobile device presence model 486 trained in the execution of operation 400. The example embodiment may further implement configurable or predetermined quantifiable requirements to indicate a level of confidence that must be satisfied for the prediction to be acceptable (e.g., without requiring further internal review).

[0174] Position detection and trimming model The position detection and cropping model 488 enables the embodiment to predict where a specific object, such as a mobile device, is located in an image, and further determine whether the object was too far from the image capture sensor (220) or, in the case of a reflective surface, too far from the mirror when the image was captured. The position detection and cropping model 488 further enables cropping of the image accordingly. The embodiment may utilize existing frameworks to further train a pre-trained model. For example, according to the embodiment, the Tensorflow® object detection framework may be used to train a network using a Mobilenet backend pre-trained on the COCO (Cooperative Computing) dataset.

[0175] Training images in which reviewers have identified the contours of mobile devices present in the image, along with the labels of each identified contour, can be input into the model for further training. Thus, the model can be trained to predict bounding boxes defined as (xmin, xmax, ymin, ymax) for images where objects, such as mobile devices, are likely to be located. The model can be further trained to generate indices, such as numbers between 0 and 1, to indicate how likely the bounding box is to contain a mobile device. For example, a number closer to 1 may indicate a higher likelihood of the bounding box containing a mobile device compared to a number closer to 0. Therefore, the output of the position detection and cropping model 488 can indicate the confidence that the bounding box accurately captures the location of an object, such as a mobile device, within the image.

[0176] Accordingly, as described with respect to operation 403, once deployed, the position detection and cropping model 488 can make predictions about the proximity of the mobile device to the mirror when the image is captured, allowing feedback to be optionally provided to the user.

[0177] According to one embodiment, a careful edge detection algorithm can also be used to estimate the bounding box of a mobile device in an image. For example, careful edge detection may utilize a Gaussian filter to smooth the image, determine the intensity gradient, and predict the strongest edges in the image. The bounding box can then be predicted accordingly.

[0178] The processor 212 of the embodiment may process a target image using at least one model trained on multiple training images, each associated with a bounding box indicating the location of a mobile device in the image, in order to determine the location of the mobile device in the target image. The embodiment may further determine the confidence level of accurately indicating the location of the mobile device in the image.

[0179] In this regard, if the confidence level does not satisfy the threshold, the image may remain "unverified" and therefore subject to further manual review. Additionally or alternatively, as described with respect to operation 404, if a certain threshold confidence level is satisfied, the image may be cropped according to the predicted bounding box. In some embodiments, it will be understood that satisfying the threshold confidence level may be optional. In embodiments that do not utilize threshold confidence levels, any or all images for which a bounding box can be calculated may be cropped accordingly.

[0180] Cover detection model The cover detection model 490 enables the embodiment to predict whether a user has captured an image of a mobile device with a cover attached. Existing models may be used and further trained, for example, with images and respective labels indicating whether the image includes a mobile device with a cover attached on it (e.g., a mobile device with a case). The embodiment may utilize an existing image processing framework and further train the model with training images and labels. Thus, the embodiment may train the model using processor 212 to place more importance on certain features, such as those related to texture, which are strong indicators of whether a mobile device in a captured image has a cover on it. In this regard, texture may be determined by the model to be a strong indicator of whether a mobile device in an image includes a cover. The embodiment's processor 212 may therefore process a target image of a mobile device using at least one model trained on multiple training images of mobile devices labeled to include or exclude a cover on each mobile device, to determine whether the target image includes a cover on the target mobile device. Accordingly, the deployed cover detection model 490 may allow the processor 212 of the device 200 to provide predictions related to the newly captured image and optionally provide feedback to the user to remove the cover and recapture the image, as provided in operation 405.

[0181] Mobile device front / back identification model The mobile device front / back identification model 492 enables embodiments to predict whether a user is capturing the front or back of their device. Existing models may be utilized and further trained, for example, with images and labels indicating whether the image provides a view of the front or back of the device. Embodiments may utilize existing image processing frameworks and further train the model with training images and labels. Thus, embodiments may train the model to identify important features that may be unique to one side of the device. For example, deformation in pixels associated with a bezel-enclosed display screen may indicate the front of the device.

[0182] In this regard, the processor 212 of the device 200 may process a target image of a mobile device using at least one trained model trained on multiple training images of a mobile device, each of which is labeled as either including the front of the mobile device or including the back of the mobile device, to determine whether the target image includes the front or back of the target mobile device.

[0183] Accordingly, the mobile device front / back identification model 490 may provide a prediction of whether the user is accurately capturing the front and / or back of the mobile device in relation to the newly captured image, and optionally provide feedback to the user to capture the indicating surface of the device, as provided in operation 406. Additional or alternative, an example embodiment may determine whether a particular image is an image of the front or back of the device based on data that identifies whether the image was captured by the front or rear camera (e.g., whether the front or rear image capture 220 was used to capture the image).

[0184] According to some embodiments, the cover detection model 490 and the mobile device front / back identification model 492 can be implemented as a single model. For example, since it may be advantageous to reject or remove any image containing a cover, the training images input to the model may include labels such as “cover”, “front”, or “back”, and therefore the “cover” label should be used for any training image containing a cover. Thus, the embodiments may reject or set the image state to “unverified” in any scenario in which an image is expected to contain a cover.

[0185] Mobile device authenticity model The mobile device authenticity model 494 enables an embodiment to predict whether an image contains the same mobile device from which the image was captured. Existing models may be utilized and further trained by, for example, the processor 212, with images and respective labels indicating whether the image contains the same mobile device from which the image was captured (e.g., “the same device”) or a different device (e.g., “a different device”). For example, an internal user or data scientist may use a mobile device to capture images of both the mobile device capturing the image and other devices, and label the images accordingly. The embodiment may then train the model with training images and labels so that the embodiment learns to detect and measure the edges of mobile devices in an image and predict or estimate the angle at which the mobile device is held relative to a mirror.

[0186] Therefore, the processor 212 of the embodiment can process a target image of a mobile device using at least one model trained on multiple training images of mobile devices, each of which is labeled as either captured by the respective mobile device contained in the image or captured by a different device, to determine whether the target mobile device contained in the target image was captured by that mobile device or by a different device.

[0187] In one embodiment, this process for determining the authenticity of a mobile device may further utilize bounding boxes drawn by the user during the labeling of training images. In any case, based on the angle, further predictions can be made to indicate whether the mobile device in the image is the same device that captured the image.

[0188] Obstruction detection model As described above, masks, such as those used to detect occlusion, can be generated by a trained model. In this regard, an example embodiment shows that an existing model, such as the UNet architecture, can be used to train the model from scratch using manually created masks. In this regard, a data scientist or other internal user can review an image and manually trace the outline, or input it in the form of a mask (e.g., the outlines reflecting the mask examples in Figures 7B and 8B), which includes the exposed area of ​​the object in question (e.g., a mobile device) but does not include occluding objects (e.g., fingers) and / or objects visible in the background. In this regard, each pixel of the training image (which may be reduced to a predetermined size, such as 300 pixels × 300 pixels, and / or cropped) can be labeled so that each pixel has an relevant index indicating whether or not it belongs to the mask. The model can then be trained on the image and labels.

[0189] Accordingly, a deployed and trained model may take an input image (e.g., a reduced and cropped image), process the image using the processor 212, and provide a mask prediction in the form of a predetermined-size array (e.g., 300 x 300) of indices indicating whether each pixel belongs to the mask or not. For example, the array may contain numerical values ​​ranging from 0 to 1, where values ​​closer to 1 correspond to pixels that are more likely to belong to the mobile device (and therefore considered to be included in the mask) compared to values ​​closer to 0. Accordingly, the mask predicted by the model can be used in an embodiment, as described with respect to Figure 6, to determine whether occlusion of the mobile device 104 exists in the image. According to one embodiment, one model and training set may be used for the mask of the image of the front of the device, and a separate model and training set may be used for the mask of the image of the back of the device.

[0190] Using models (e.g., neural networks) to determine masks can offer advantages that other image processing techniques may not provide. Models are often useful in contextual predictions that are typically done by humans and that conventional computer algorithms cannot. For example, colors in a screen are not uniform, and some of the same color may appear in pixels outside the device in the image. Models such as neural networks can make such distinctions, whereas conventional color detection or other image processing algorithms may not accurately distinguish between pixels far from the image and pixels that are not far from the image but otherwise have the same or similar color.

[0191] Damage detection model The damage detection model 498 enables an embodiment to predict whether a captured mobile device in an image is damaged (for example, damage to such an extent that the integrity status of the mobile device should be set to "unverified," such as when the damaged device is not eligible for coverage under a device protection plan). Existing models may be utilized and further trained by the processor 212 with images and respective labels indicating whether the image contains a damaged mobile device, such as, but not limited to, cracks, water damage, dents, scratches, and / or any other damage that prevents the mobile device from being insured or protected. In an embodiment, a special user interface tool may be used by a reviewer or data scientist to zoom in on images of mobile devices to determine whether cracks or other damage are present and to label the training images accordingly. In one embodiment, a binary label such as "damaged" or "not damaged" may be applied to the training images. As another example, reviewers may score the level of damage, so a small or seemingly insignificant crack may receive a relatively lower score compared to images showing more significant cracks that could affect functionality. Any modifications or scores for damage labeling may be attempted. As yet another example, some damage may be specifically labeled as “crack,” “water damage,” or any other type of damage that could affect the insurability of the mobile device.

[0192] In one embodiment, a first label on a training image of a mobile device may indicate "damaged," and a second label may indicate the type of damage. In this regard, one model may be trained to predict whether or not damage is present. Another model and / or model(s) may, if damage is predicted to be present, predict a specific type of identified damage, such as a crack, water damage, or a dent. For example, one model may be trained solely to detect water damage based on a training image of water damage, and the same logic may be applied to other types of damage to the device and any other visiblely detectable condition.

[0193] The example embodiments may utilize existing image processing frameworks and further train the model using training images and labels. Therefore, the example embodiments may place greater emphasis on certain features, such as those related to texture and / or color changes, which are strong indicators of damage and / or specific types of damage.

[0194] In this regard, the apparatus 200 may process the target image using at least one model trained on multiple training images of a mobile device, each training image being labeled with a damage assessment, and including means such as a processor 212, to calculate a damage assessment of the target mobile device in the target image. In this regard, the damage assessment may include “no damage,” “minor damage,” “extensive damage,” and / or similar. In one embodiment, the damage assessment may include a quantifiable assessment such as on a scale of 1 to 10, where 1 indicates no damage detected and 10 indicates extensive or significant damage. The deployed damage detection model(s) 498 may therefore provide predictions about whether the device is damaged, the extent of the damage, and / or the type of damage.

[0195] In scenarios where damage is detected and, optionally, depends on the type and / or extent of the damage, the image state may be "unverified," resulting in a mobile device integrity state of "unverified" or "uncertain." As another example, a quantitative damage score may be generated.

[0196] It will be understood that, as an addition or alternative, separate damage detection models may be configured, trained, and deployed for each of the front, back, and / or bezel of the mobile device. The bezel may be recognized as part of the device so that damage to the bezel can also be detected.

[0197] conclusion As described herein, the embodiments of this disclosure offer technical advantages over alternative embodiments. The embodiments may be implemented to consume fewer processing resources, which could otherwise be wasted by submitting all images captured by the user to a server for storage, potential review, and further processing.

[0198] In this regard, some operations, such as those shown in Figure 3, Figure 4A, and / or Figure 6, may be performed on the mobile device 104, while other operations may be performed on the device integrity device 108 and / or the occlusion detection device 109. Thus, the embodiment may provide resource efficiency by strategically balancing such operations. For example, some initial image processing operations may be performed on the mobile device 104 before the image is sent to the device integrity device 108 and / or the occlusion detection device 109. Thus, some images may be removed on the mobile device 104 before being processed by other, potentially more resource-intensive processes.

[0199] For example, some embodiments may employ models such as neural networks configured to run on mobile devices, such as mobile device 104. For example, TensorFlow Lite and / or other frameworks designed to be deployed on mobile devices may be used according to the embodiments.

[0200] Therefore, for example, the embodiment may provide real-time verification on the device in certain scenarios, such as when a highly reliable image state or mobile device integrity state is determined. Otherwise, an image and / or device integrity verification request may be sent to the device integrity device 108 for review by an agent. On the server side, an algorithm similar to that implemented on the device side may be used to process input from the review as a means of facilitating image review and / or further calibration of the algorithm and / or training the model.

[0201] Figures 3, 4A, and 6 illustrate flowcharts of systems, methods, and computer program products according to several embodiments, respectively. It will be understood that each block in the flowchart, and combinations of blocks within the flowchart, can be implemented by various means, such as hardware and / or computer program products, which include one or more computer-readable media on which computer-readable program instructions are stored. For example, one or more of the procedures described herein can be embodied by computer program instructions of a computer program product. In this regard, a computer program product(s) that embodied the procedures described herein may include one or more memory devices (e.g., memory 214) of a computing device that store instructions executable by a processor in the computing device (e.g., by processor 212). In some embodiments, the computer program instructions of a computer program product(s) that embodied the aforementioned procedures may be stored in memory devices of multiple computing devices. As can be understood, any such computer program product is loaded onto a computer or other programmable device (e.g., mobile device support device 102, mobile device 104 and / or other device) to create a machine, and a computer program product containing instructions to be executed on a computer or other programmable device creates means for implementing the functions specified in the flowchart block(s). Furthermore, a computer program product may include one or more computer-readable memories on which computer program instructions can be stored, so that one or more computer-readable memories can be instructed to function in a particular way on a computer or other programmable device, and thus a computer program product may include products that implement the functions specified in the flowchart block(s).Computer program instructions for one or more computer program products are also loaded onto a computer or other programmable device (e.g., mobile device 104 and / or other device) to perform a series of operations on the computer or other programmable device to generate a computer implementation process, and so the instructions executed on the computer or other programmable device may implement the functions specified in the flowchart block(s).

[0202] Accordingly, the blocks in the flowchart support combinations of means for performing a specified function, and combinations of operations for performing a specified function. It will also be understood that one or more blocks in the flowchart, and combinations of blocks within the flowchart, can be implemented by a dedicated hardware-based computer system or a combination of dedicated hardware and computer instructions for performing a specified function.

[0203] Many embodiments of the subject matter described herein may include all, some, or combinations of parts thereof of the systems, apparatus, methods, and / or computer program products described herein. The subject matter described herein includes, but is not limited to, the following specific embodiments: 1. A method, Receiving a device integrity verification request associated with a mobile device, Receiving a mobile device identification data object containing information describing a mobile device, To display a prompt on a mobile device to capture at least one image of the mobile device using one or more image sensors and reflective surfaces of the mobile device. Receiving at least one image captured by one or more image sensors of a mobile device, and, This includes processing at least one image using at least one trained model to determine the integrity status of a mobile device.

[0204] 2. The method of Embodiment 1, wherein processing at least one image to determine the integrity status of a mobile device, To determine whether there is damage to the mobile device using at least one trained model, and This includes determining, in response to a determination that there is damage to the mobile device, that the integrity status of the mobile device has not been verified.

[0205] 3. The method of Embodiment 1, wherein processing at least one image to determine the integrity status of a mobile device, To determine the angle of the mobile device relative to the reflective surface when at least one image is captured, and This includes determining, based on the angle, that at least one image contains a mobile device different from the mobile device associated with the mobile device identification data object.

[0206] 4. The method of Embodiment 1, wherein processing at least one image to determine the integrity status of a mobile device, This includes determining whether at least one image contains a mobile device associated with a mobile device identification data object.

[0207] 5. The method of Embodiment 4, wherein determining whether at least one image includes a mobile device, Identifying a suspicious mobile device in at least one image, To generate predictions for the identification of at least one suspicious mobile device, and to compare a mobile device identification data object with the predictions for the identification of at least one suspicious mobile device to determine whether the suspicious mobile device is that mobile device, and This includes determining that the integrity status of a mobile device has been verified if the suspected mobile device is determined to be that mobile device.

[0208] 6. The method of Embodiment 1, in which the integrity state of the mobile device is determined to be uncertain, and the method This further includes sending a device integrity verification request and at least one image to an internal user device for internal review.

[0209] 7. The method of Embodiment 3, Based on the angle, in response to the determination that at least one image captures a different mobile device, (a) Display a message on the mobile device instructing the user to recapture the mobile device, and (b) Further including determining that the integrity status of the mobile device has not been verified.

[0210] 8. The method of Embodiment 1, wherein processing at least one image to determine the integrity status of a mobile device, Determining the position of a mobile device within at least one image, wherein the position is defined as a bounding box, and The method includes displaying a message on the mobile device instructing it to move closer to the reflective surface if the bounding box has a first predetermined relationship with the threshold ratio of at least one image.

[0211] 9. The method of Embodiment 8, If the bounding box has a second predetermined relationship with the threshold ratio of at least one image, the method further includes cropping at least one image according to the bounding box.

[0212] 10. The method of Embodiment 1, wherein processing at least one image to determine the integrity status of a mobile device, Using at least one trained model, determine that an object is occluding a mobile device in at least one image, and This includes displaying a prompt to capture the unobstructed image on a mobile device.

[0213] 11. The method of Embodiment 10, wherein determining whether the occlusion of a mobile device is in at least one image is: To determine whether a concave occlusion is present in at least one image, and This includes determining whether the blocked corner is present in at least one image.

[0214] 12. The method of Embodiment 10, wherein determining whether a concave occlusion is present in at least one image is: To generate a mobile device mask containing fewer colors than at least one image, using at least one trained model. Extracting a polygonal subregion P of a mobile device mask, Determine the convex hull of P. Calculate the difference between P and the convex hull. To eliminate or reduce the fine mismatch between P and at least one edge of the convex hull. Identifying the maximum area of ​​the remaining region P, and This includes comparing the maximum area to a threshold to determine whether at least one image contains concave occlusion.

[0215] 13. The method of Embodiment 10, wherein determining whether there is a blocked corner in at least one image is: To generate a mobile device mask containing fewer colors than at least one image, using at least one trained model. Extracting a polygonal subregion P of a mobile device mask, Determine the convex hull of P. Identifying the four main edges of the convex hull, Identifying corners by determining the intersections of adjacent major edges. Determining the distance from each angle to P, and This involves comparing each distance to a distance threshold to determine whether any of the angles are blocked in at least one image.

[0216] 14. The method of Embodiment 1, wherein processing at least one image to determine the integrity status of a mobile device, This includes using at least one trained model to determine whether at least one image includes the front, back, or cover of a mobile device.

[0217] 15. The method of Embodiment 1, The system further includes, in response to the reception of at least one image, providing a response for display on a mobile device in real time or near real time, the response provided being determined by the integrity status of the determined mobile device.

[0218] 16. The method of Embodiment 1, The further includes displaying a test pattern on a mobile device that is configured to provide improved accuracy in predicting the properties of at least one image captured when the mobile device is displaying a test pattern, compared to the accuracy in predicting the properties of at least one image captured when the mobile device is displaying a different display pattern.

[0219] 17. The method of Embodiment 1, Identifying a subset of conditions that must be satisfied to determine if the integrity status of a mobile device has been verified. If all conditions within a subset of the conditions are satisfied within a particular image, the image state of that image is set to verified, and If the image status of each required image is verified, this further includes determining that the integrity status of the mobile device has been verified.

[0220] 18. The method of Embodiment 17, wherein at least one condition from a subset of the conditions to be satisfied is performed on a mobile device. 19. The method of Embodiment 1, wherein receiving at least one image includes receiving at least two images captured by a mobile device, the first of the at least two images being of the front of the device, the second of the at least two images being of the back of the device, and processing at least one image to determine the integrity status of the mobile device. To process both the first and second images using at least one trained model, and If the processing of both images verifies the state of each image, this includes determining that the integrity state of the mobile device being considered has been verified.

[0221] 20. The method of Embodiment 1, This further includes training at least one trained model by inputting training images and respective labels describing the characteristics of each training image.

[0222] 21. The method of Embodiment 1, wherein at least one trained model is a neural network. 22. A method for detecting concave occlusion in an image, wherein this method is Using at least one trained model, generate a mask containing fewer colors than the image. Extracting the polygonal subregion P of the mask, Determine the convex hull of P. Calculate the difference between P and the convex hull. To eliminate or reduce the fine mismatch between P and at least one edge of the convex hull. Recalculate P as the maximum area of ​​the remaining region, and This includes determining the depression as the difference between P and the convex hull.

[0223] 23. A method for detecting the blocked corners of an object in an image, wherein this method is Using at least one trained model, generate a mask containing fewer colors than the image. Extracting the polygonal subregion P of the mask, Determine the convex hull of P. Identifying a predetermined number of major edges of the convex hull, Identifying corners by determining the intersections of adjacent major edges. Determining the distance from each angle to P, and This includes comparing each distance to a distance threshold to determine whether any of the angles are blocked within the image.

[0224] 24. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and computer program code are used by the processor to provide at least one to the device. Receiving a device integrity verification request associated with a mobile device, Receiving a mobile device identification data object containing information describing a mobile device, Display a prompt on the mobile device to capture at least one image of the mobile device using one or more image sensors and reflective surfaces of the mobile device. Receiving at least one image captured by one or more image sensors of a mobile device, It is configured to use at least one trained model to process at least one image and determine the integrity status of a mobile device.

[0225] 25. The apparatus of Embodiment 24, which processes at least one image to determine the integrity status of a mobile device, To determine whether there is damage to the mobile device using at least one trained model, and This includes determining, in response to a determination that there is damage to the mobile device, that the integrity status of the mobile device has not been verified.

[0226] 26. The apparatus of Embodiment 24, which processes at least one image to determine the integrity status of a mobile device, To determine the angle of the mobile device relative to the reflective surface when at least one image is captured, and This includes determining, based on the angle, that at least one image contains a mobile device different from the mobile device associated with the mobile device identification data object.

[0227] 27. The apparatus of Embodiment 24, which processes at least one image to determine the integrity status of a mobile device, This includes determining whether at least one image contains a mobile device associated with a mobile device identification data object.

[0228] 28. In the apparatus of Embodiment 27, determining whether at least one image includes a mobile device is: Identifying a suspicious mobile device in at least one image, To generate predictions for the identification of at least one suspicious mobile device, and to compare a mobile device identification data object with the predictions for the identification of at least one suspicious mobile device to determine whether the suspicious mobile device is that mobile device, and This includes determining that the integrity status of a mobile device has been verified if the suspected mobile device is determined to be that mobile device.

[0229] 29. The apparatus of Embodiment 24, wherein the integrity state of the mobile device is determined to be uncertain, and at least one memory and computer program code are provided to the apparatus using a processor, It is further configured to send a device integrity verification request and at least one image to an internal user device for internal review.

[0230] 30. The apparatus of Embodiment 26, wherein at least one memory and computer program code are provided to the apparatus using a processor, Based on the angle, in response to the determination that at least one image captures a different mobile device, (a) Display a message on the mobile device instructing the user to recapture the mobile device, and (b) Further configured to cause it to determine that the integrity status of the mobile device has not been verified.

[0231] 31. The apparatus of Embodiment 24, which processes at least one image to determine the integrity status of a mobile device, Determining a location within at least one image of a mobile device, wherein the location is defined as a bounding box, and The method includes displaying a message on the mobile device instructing it to move closer to the reflective surface if the bounding box has a first predetermined relationship with the threshold ratio of at least one image.

[0232] 32. The apparatus of Embodiment 31, wherein at least one memory and computer program code are provided to the apparatus using a processor, If the bounding box has a second predetermined relationship with the threshold ratio of at least one image, the system is further configured to crop at least one image according to the bounding box.

[0233] 33. The apparatus of Embodiment 24, which processes at least one image to determine the integrity status of a mobile device, Using at least one trained model, determining that an object is obscuring a mobile device in at least one image, and displaying a prompt to cause the mobile device to capture an image without that obscuration.

[0234] 34. The apparatus of embodiment 33, wherein determining whether an obscuration of the mobile device is in at least one image comprises determining whether a concave obscuration is in at least one image, and determining whether a blocked corner is in at least one image.

[0235] 35. The apparatus of embodiment 33, wherein determining whether a concave obscuration is in at least one image comprises using at least one trained model to generate a mobile device mask that includes a smaller number of colors compared to at least one image, extracting a polygonal partial region P of the mobile device mask, determining the convex hull of P, calculating the difference between P and the convex hull, removing or reducing a thin mismatch of at least one edge between P and the convex hull, identifying the maximum area of the remaining region of P, and comparing the maximum area with a threshold to determine whether at least one image includes a concave obscuration.

[0236] 36. The apparatus of embodiment 33, wherein determining whether there is a blocked corner in at least one image comprises using at least one trained model to generate a mobile device mask that includes a smaller number of colors compared to at least one image, extracting a polygonal partial region P of the mobile device mask, determining the convex hull of P, identifying four main edges of the convex hull, Identifying corners by determining the intersections of adjacent major edges. Determining the distance from each angle to P, and This involves comparing each distance to a distance threshold to determine whether any of the angles are blocked in at least one image.

[0237] 37. The apparatus of Embodiment 24, which processes at least one image to determine the integrity status of a mobile device, This includes using at least one trained model to determine whether at least one image includes the front, back, or cover of a mobile device.

[0238] 38. The apparatus of Embodiment 24, for determining whether there is a blocked corner in at least one image, The system includes providing a response for display on a mobile device in real time or near real time in response to the reception of at least one image, the response provided being determined by the integrity status of the determined mobile device.

[0239] 39. The apparatus of Embodiment 24, for determining whether there is a blocked corner in at least one image, The method includes displaying a test pattern on a mobile device that is configured to provide improved accuracy in predicting the properties of at least one image captured when the mobile device is displaying a test pattern, compared to the accuracy in predicting the properties of at least one image captured when the mobile device is displaying a different display pattern.

[0240] 40. The apparatus of Embodiment 24, wherein at least one memory and computer program code are provided to the apparatus using a processor, Identifying a subset of conditions that must be satisfied to determine if the integrity status of a mobile device has been verified. If all conditions within a subset of the conditions are satisfied within a particular image, the image state of that image is set to verified, and The system is further configured to determine that the integrity state of the mobile device has been verified once the image state of each required image has been verified.

[0241] 41. The apparatus of Embodiment 40, wherein at least one of the subset of conditions to be satisfied is performed on a mobile device. 42. The apparatus of Embodiment 24, wherein receiving at least one image includes receiving at least two images captured by a mobile device, the first of the at least two images being of the front of the device, the second of the at least two images being of the back of the device, and processing at least one image to determine the integrity status of the mobile device. To process both the first and second images using at least one trained model, and If the processing of both images verifies the state of each image, this includes determining that the integrity state of the mobile device being considered has been verified.

[0242] 43. The apparatus of Embodiment 24, wherein at least one memory and computer program code are provided to the apparatus using a processor, The apparatus is further configured to train at least one trained model by inputting training images and respective labels describing the characteristics of each training image. 44. The apparatus of Embodiment 24, wherein at least one trained model is a neural network.

[0243] 45. An apparatus for detecting concave occlusion in an image, the apparatus comprising at least one processor and at least one memory containing computer program code, the at least one memory and computer program code using the processor to provide at least, Using at least one trained model, generate a mask containing fewer colors than the image. Extracting the polygonal subregion P of the mask, Determine the convex hull of P. Calculate the difference between P and the convex hull. To eliminate or reduce the fine mismatch between P and at least one edge of the convex hull. Recalculate P as the maximum area of ​​the remaining region, and The system is configured to allow the user to determine the depression as the difference between point P and the convex hull.

[0244] 46. ​​A device for detecting blocked corners of an object in an image, the device comprising at least one processor and at least one memory containing computer program code, the at least one memory and computer program code using the processor to provide the device with at least, Using at least one trained model, generate a mask containing fewer colors than the image. Extracting the polygonal subregion P of the mask, Determine the convex hull of P. Identifying a predetermined number of major edges of the convex hull, Identifying corners by determining the intersections of adjacent major edges. Determining the distance from each angle to P, and The system is configured to compare each distance with a distance threshold to determine whether any corners are blocked within the image.

[0245] 47. A computer program product comprising at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Receiving a device integrity verification request associated with a mobile device, Receiving a mobile device identification data object containing information describing a mobile device, Displaying, on a mobile device, a prompt for causing at least one image of the mobile device to be captured using one or more image sensors and a reflective surface of the mobile device Receiving at least one image captured by one or more image sensors of the mobile device, and Including program code instructions for processing the at least one image using at least one trained model to determine a fully functional state of the mobile device

[0246] 48. The computer program product of embodiment 47, wherein processing at least one image to determine a fully functional state of the mobile device comprises Determining, using at least one trained model, whether there is damage to the mobile device, and In response to determining that there is damage to the mobile device, determining that the fully functional state of the mobile device has not been verified

[0247] 49. The computer program product of embodiment 47, wherein processing at least one image to determine a fully functional state of the mobile device comprises Determining an angle of the mobile device relative to the reflective surface when the at least one image was captured, and Based on the angle, determining that the at least one image includes a mobile device different from the mobile device associated with the mobile device identification data object

[0248] 50. The computer program product of embodiment 47, wherein processing at least one image to determine a fully functional state of the mobile device comprises Determining whether the at least one image includes the mobile device associated with the mobile device identification data object

[0249] 51. A computer program product of Embodiment 50, in which determining whether at least one image includes a mobile device, Identifying a suspicious mobile device in at least one image, To generate predictions for the identification of at least one suspicious mobile device, and to compare a mobile device identification data object with the predictions for the identification of at least one suspicious mobile device to determine whether the suspicious mobile device is that mobile device, and This includes determining that the integrity status of a mobile device has been verified if the suspected mobile device is determined to be that mobile device.

[0250] 52. A computer program product of Embodiment 47, in which the integrity state of the mobile device is determined to be uncertain, and the computer executable program code instructions are, The program code instructions further include a device integrity verification request and a program code instruction for sending at least one image to an internal user computer program product for internal review.

[0251] 53. A computer program product according to Embodiment 49, wherein the computer executable program code instructions are: Based on the angle, in response to the determination that at least one image captures a different mobile device, (a) Display a message on the mobile device instructing the user to recapture the mobile device, and (b) Further includes program code instructions for determining that the integrity status of a mobile device has not been verified.

[0252] 54. The computer program product of Embodiment 47, which processes at least one image to determine the integrity status of a mobile device, Determining the position of a mobile device within at least one image, wherein the position is defined as a bounding box, and The method includes displaying a message on the mobile device instructing it to move closer to the reflective surface if the bounding box has a first predetermined relationship with the threshold ratio of at least one image.

[0253] 55. A computer program product according to Embodiment 54, wherein the computer executable program code instructions are: The code further includes program code instructions for cropping at least one image according to a bounding box, provided that the bounding box has a second predetermined relationship with a threshold ratio of at least one image.

[0254] 56. A computer program product of Embodiment 47, which processes at least one image to determine the integrity status of a mobile device, Using at least one trained model, determine that an object is occluding a mobile device in at least one image, and This includes displaying a prompt to capture the unobstructed image on a mobile device.

[0255] 57. A computer program product of Embodiment 56, wherein determining whether the occlusion of a mobile device is within at least one image is: To determine whether a concave occlusion is present in at least one image, and This includes determining whether the blocked corner is present in at least one image.

[0256] 58. A computer program product of Embodiment 56, wherein determining whether a concave occlusion is present in at least one image is: To generate a mobile device mask containing fewer colors than at least one image, using at least one trained model. Extracting a polygonal subregion P of a mobile device mask, Determine the convex hull of P. Calculate the difference between P and the convex hull. To eliminate or reduce the fine mismatch between P and at least one edge of the convex hull. Identifying the maximum area of ​​the remaining region P, and This includes comparing the maximum area to a threshold to determine whether at least one image contains concave occlusion.

[0257] 59. A computer program product of Embodiment 56, which determines whether there is a blocked corner in at least one image, To generate a mobile device mask containing fewer colors than at least one image, using at least one trained model. Extracting a polygonal subregion P of a mobile device mask, Determine the convex hull of P. Identifying the four main edges of the convex hull, Identifying corners by determining the intersections of adjacent major edges. Determining the distance from each angle to P, and This involves comparing each distance to a distance threshold to determine whether any of the angles are blocked in at least one image.

[0258] 60. A computer program product of Embodiment 47, which processes at least one image to determine the integrity status of a mobile device, This includes using at least one trained model to determine whether at least one image includes the front, back, or cover of a mobile device.

[0259] 61. A computer program product of Embodiment 47, which determines whether there is a blocked corner in at least one image, The system includes providing a response for display on a mobile device in real time or near real time in response to the reception of at least one image, the response provided being determined by the integrity status of the determined mobile device.

[0260] 62. A computer program product of Embodiment 47, which determines whether there is a blocked corner in at least one image, The method includes displaying a test pattern on a mobile device that is configured to provide improved accuracy in predicting the properties of at least one image captured when the mobile device is displaying a test pattern, compared to the accuracy in predicting the properties of at least one image captured when the mobile device is displaying a different display pattern.

[0261] 63. A computer program product according to Embodiment 47, wherein the computer executable program code instructions are: Identifying a subset of conditions that must be satisfied to determine if the integrity status of a mobile device has been verified. If all conditions within a subset of the conditions are satisfied within a particular image, the image state of that image is set to verified, and The program code further includes instructions for determining that the integrity status of the mobile device has been verified, provided that the image status of each required image has been verified.

[0262] 64. A computer program product of Embodiment 63, wherein at least one of the subset of conditions to be satisfied is performed on a mobile device. 65. A computer program product of Embodiment 47, wherein receiving at least one image includes receiving at least two images captured by a mobile device, the first of the at least two images being of the front of the device, the second of the at least two images being of the back of the device, and processing at least one image to determine the integrity status of the mobile device. To process both the first and second images using at least one trained model, and If the processing of both images verifies the state of each image, this includes determining that the integrity state of the mobile device being considered has been verified.

[0263] 66. A computer program product according to Embodiment 47, wherein the computer executable program code instructions are: The software further includes program code instructions for training at least one trained model by inputting training images and respective labels describing the characteristics of each training image.

[0264] 67. A computer program product of Embodiment 47, wherein at least one trained model is a neural network. 68. A computer program product for detecting concave occlusion in an image, wherein the computer program product includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Using at least one trained model, generate a mask containing fewer colors than the image. Extracting the polygonal subregion P of the mask, Determine the convex hull of P. Calculate the difference between P and the convex hull. To eliminate or reduce the fine mismatch between P and at least one edge of the convex hull. Recalculate P as the maximum area of ​​the remaining region, and This includes program code instructions for determining that a depression is the difference between P and the convex hull.

[0265] 69. A computer program product for detecting blocked corners of an object in an image, wherein the computer program product includes at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Using at least one trained model, generate a mask containing fewer colors than the image. Extracting the polygonal subregion P of the mask, Determine the convex hull of P. Identifying a predetermined number of major edges of the convex hull, Identifying corners by determining the intersections of adjacent major edges. Determining the distance from each angle to P, and The program code includes instructions for comparing each distance to a distance threshold to determine whether any corner is blocked within the image.

[0266] 70. Receiving instructions for the target image, and A method for determining whether a target image contains a mobile device, by processing the target image using at least one trained model trained on multiple training images, each labeled as either containing a mobile device or excluding a mobile device.

[0267] 71. Receiving instructions for the target image, The target image is processed using at least one trained model, which is trained on multiple training images each associated with a bounding box indicating the location of a mobile device within the image, to determine the location of the mobile device within the image, and A method that includes cropping a target image based on the determined position of a mobile device within the target image.

[0268] 72. Receiving instructions for the target image on the target mobile device, and A method for determining whether a target image of a mobile device includes a cover on the target mobile device, by processing the target image using at least one trained model trained on multiple training images of mobile devices labeled to include or exclude the cover on each mobile device.

[0269] 73. Receiving instructions for the target image on the target mobile device, and A method for determining whether a target image of a mobile device includes the front or back of the target mobile device, by processing the target image of a mobile device using at least one trained model trained on multiple training images of the mobile device, where each training image is labeled as either including the front of the mobile device or including the back of the mobile device.

[0270] 74. Receiving instructions for the target image on the target mobile device, and A method comprising processing target images of mobile devices using at least one trained model trained on multiple training images of mobile devices, each training image being labeled as either captured by the respective mobile device contained in the image or captured by a different device, to determine whether a target mobile device contained in the target image was captured by the target mobile device or by a different device.

[0271] 75. Receiving instructions for the target image on the target mobile device, and A method comprising processing target images of a mobile device using at least one trained model trained on multiple training images of the mobile device, each training image being labeled with a damage assessment, to compute a damage assessment of the target mobile device within the target images.

[0272] 76. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are used by the processor to provide at least Receiving instructions for the target image, and The system is configured to process the target image using at least one trained model, which has been trained on multiple training images labeled either as containing a mobile device or excluding a mobile device, to determine whether the target image contains a mobile device.

[0273] 77. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are used by the processor to provide at least Receiving instructions for the target image, The target image is processed using at least one trained model, which is trained on multiple training images each associated with a bounding box indicating the location of a mobile device within the image, to determine the location of the mobile device within the image, and The system is configured to crop the target image based on the determined position of the mobile device within the target image.

[0274] 78. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are used by the processor to provide at least Receiving instructions for the target image on the target mobile device, and The system is configured to process target images of mobile devices using at least one trained model, which is trained on multiple training images of mobile devices labeled as either including or excluding the cover on each mobile device, to determine whether the target image includes the cover on the target mobile device.

[0275] 79. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are used by the processor to provide at least Receiving instructions for the target image on the target mobile device, and The system is configured to process target images of mobile devices using at least one trained model trained on multiple training images of mobile devices, each training image being labeled as either containing the front or back of the mobile device, in order to determine whether the target image contains the front or back of the target mobile device.

[0276] 80. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are used by the processor to provide at least Receiving instructions for the target image on the target mobile device, and The system is configured to process target images of mobile devices using at least one trained model trained on multiple training images of mobile devices, each of which is labeled as either captured by the respective mobile device contained within the image, or captured by a different device. This model determines whether the target mobile device contained within the target image was captured by the target mobile device or by a different device.

[0277] 81. A device comprising at least one processor and at least one memory containing computer program code, wherein the at least one memory and the computer program code are used by the processor to provide at least Receiving instructions for the target image on the target mobile device, and The system is configured to process target images of mobile devices using at least one trained model, which is trained on multiple training images of mobile devices, each of which is labeled with a damage assessment, in order to calculate a damage assessment of the target mobile device within the target image.

[0278] 82. A computer program product comprising at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Receiving instructions for the target image, and The program code includes instructions for processing a target image using at least one trained model, which is trained on multiple training images each labeled as either containing a mobile device or excluding a mobile device, to determine whether the target image contains a mobile device.

[0279] 83. A computer program product comprising at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Receiving instructions for the target image, The target image is processed using at least one trained model, which is trained on multiple training images, each associated with a bounding box indicating the location of a mobile device within the image, to determine the location of the mobile device within the image, and Includes program code instructions for cropping a target image based on the determined position of a mobile device within the target image.

[0280] 84. A computer program product comprising at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Receiving instructions for the target image on the target mobile device, and The program code includes instructions for processing a target image of a mobile device using at least one trained model trained on multiple training images of mobile devices labeled as either including or excluding the cover on each mobile device, to determine whether the target image includes the cover on the target mobile device.

[0281] 85. A computer program product comprising at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Receiving instructions for the target image on the target mobile device, and The program includes instructions for processing a target image of a mobile device using at least one trained model trained on multiple training images of a mobile device, where each training image is labeled as either including the front of the mobile device or including the back of the mobile device, to determine whether the target image includes the front or back of the target mobile device.

[0282] 86. A computer program product comprising at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Receiving instructions for the target image on the target mobile device, and The program includes instructions for processing target images of mobile devices using at least one trained model trained on multiple training images of mobile devices, each training image being labeled as either captured by the respective mobile device contained in the image or captured by a different device, in order to determine whether the target mobile device contained in the target image was captured by the target mobile device or by a different device.

[0283] 87. A computer program product comprising at least one persistent computer-readable storage medium in which computer executable program code instructions are stored, Receiving instructions for the target image on the target mobile device, and The program includes instructions for processing target images of mobile devices using at least one trained model trained on multiple training images of mobile devices, each training image being labeled with a damage assessment, to calculate a damage assessment of the target mobile device within the target image.

[0284] Many modifications and other embodiments of the invention described herein will be obvious to those skilled in the art who benefit from the art presented in the foregoing description and accompanying drawings. It is understood that the invention is not limited to the specific embodiments disclosed, and that other embodiments are intended to be included within the scope of the accompanying claims. Furthermore, while the foregoing description and accompanying drawings illustrate exemplary embodiments in the context of combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the accompanying claims. In this regard, for example, combinations of elements and / or functions different from those explicitly described above are intended to be included in some of the accompanying claims. Certain terms are used herein, but they are used only in a general and descriptive sense, and not for restrictive purposes.

Claims

1. Receiving one or more images of the mobile device captured by one or more image sensors of the mobile device via a reflective surface, Processing one or more images to identify one or more depressions using at least one trained model, wherein the one or more depressions are Determining the polygonal subregion P and the convex hull of P, The difference between P and the convex hull of P is defined as one or more depressions, and is identified by this, The one or more depressions identified in the one or more images are compared with a depression threshold, In response to determining that one or more depressions are greater than the depression threshold, a prompt is displayed on the mobile device to capture one or more images that do not have the one or more depressions. In response to determining that one or more depressions are smaller than or less than the depression threshold, the process involves ignoring the one or more depressions, and then processing the one or more images to determine the mobile device integrity state. A method that includes this.

2. Further including outputting the integrity status of the mobile device, The method according to claim 1.

3. Processing one or more of the aforementioned images to determine the mobile device integrity state is, Using the aforementioned trained model, determine whether there is any damage to the mobile device. This includes determining that, in accordance with the determination that there is damage to the mobile device, the integrity status of the mobile device has not been verified. The method according to claim 1.

4. Processing one or more of the aforementioned images to determine the mobile device integrity state is, Determining the angle of the mobile device with respect to the reflective surface when one or more images are captured, The process includes determining, based on the angle, that one or more images include a mobile device different from the mobile device associated with the mobile device identification data object received from the mobile device, The method according to claim 1.

5. Based on the aforementioned angle, in accordance with the determination that one or more images are capturing different mobile devices, (a) Displaying a message on the mobile device instructing the user to reacquire the mobile device, (b) The integrity status of the mobile device is determined to be unverified. Further including, The method according to claim 4.

6. Processing one or more of the aforementioned images to determine the mobile device integrity state is, The process includes determining whether one or more images include the mobile device associated with a mobile device identification data object received from the mobile device, The method according to claim 1.

7. Determining whether one or more of the images include the mobile device is: Identifying at least one suspicious mobile device within one or more of the aforementioned images, To generate a prediction of the identification of at least one suspicious mobile device, and to compare the mobile device identification data object received from the mobile device with the prediction of the identification of the at least one suspicious mobile device to determine whether the at least one suspicious mobile device is the mobile device, If the at least one suspected mobile device is determined to be the mobile device, the determination to verify the integrity status of the mobile device is included. The method according to claim 6.

8. The integrity status of the aforementioned mobile device is determined to be uncertain, Depending on whether the integrity status of the mobile device is determined to be uncertain, the following is further included: transmitting one or more images to an internal user device for internal review: The method according to claim 1.

9. Processing one or more images in order to determine the integrity state of a mobile device is The process involves determining the position within one or more images, wherein the position is defined as a bounding box. If the bounding box has a first predetermined relationship with the threshold ratio of one or more images, the mobile device is instructed to bring the mobile device closer to the reflective surface, including the following: The method according to claim 1.

10. If the bounding box has a second predetermined relationship with the threshold ratio of one or more images, the further includes cropping the one or more images according to the bounding box. The method according to claim 9.

11. An apparatus comprising one or more processors and at least one memory containing computer program code, wherein the at least one memory and the computer program code are used by the one or more processors to provide at least the apparatus, Receiving one or more images of the mobile device captured by one or more image sensors of the mobile device via a reflective surface, Processing one or more images to identify one or more depressions using at least one trained model, wherein the one or more depressions are Determining the polygonal subregion P and the convex hull of P, The difference between P and the convex hull of P is defined as one or more depressions, and is identified by this, The one or more depressions identified in the one or more images are compared with a depression threshold, In response to determining that one or more depressions are greater than the depression threshold, a prompt is displayed on the mobile device to capture one or more images that do not have the one or more depressions. In response to determining that one or more depressions are smaller than or less than the depression threshold, the process involves ignoring the one or more depressions, and then processing the one or more images to determine the mobile device integrity state. A device configured to perform a certain action.

12. The at least one memory and the computer program code are used by the one or more processors to provide at least the device. The mobile device is further configured to output the integrity status of the aforementioned mobile device. The apparatus according to claim 11.

13. Processing one or more of the aforementioned images to determine the mobile device integrity state is, Using the aforementioned trained model, determine whether there is any damage to the mobile device. This includes determining that, in accordance with the determination that there is damage to the mobile device, the integrity status of the mobile device has not been verified. The apparatus according to claim 11.

14. Processing one or more of the aforementioned images to determine the mobile device integrity state is, Determining the angle of the mobile device with respect to the reflective surface when one or more images are captured, The process includes determining, based on the angle, that one or more images include a mobile device different from the mobile device associated with the mobile device identification data object received from the mobile device, The apparatus according to claim 11.

15. The at least one memory and the computer program code are used by the one or more processors to provide at least the device. Based on the aforementioned angle, in accordance with the determination that one or more images are capturing different mobile devices, (a) Displaying a message on the mobile device instructing the user to reacquire the mobile device, (b) The integrity status of the mobile device is determined to be unverified. It is further configured to perform the following: The apparatus according to claim 14.

16. Processing one or more of the aforementioned images to determine the mobile device integrity state is, The process includes determining whether one or more images include the mobile device associated with a mobile device identification data object received from the mobile device, The apparatus according to claim 11.

17. Determining whether one or more of the images include the mobile device is: Identifying at least one suspicious mobile device within one or more of the aforementioned images, To generate a prediction of the identification of at least one suspicious mobile device, and to compare the mobile device identification data object received from the mobile device with the prediction of the identification of the at least one suspicious mobile device to determine whether the at least one suspicious mobile device is the mobile device, If the at least one suspected mobile device is determined to be the mobile device, the determination to verify the integrity status of the mobile device is included. The apparatus according to claim 16.

18. The at least one memory and the computer program code are used by the one or more processors to provide at least the device. The integrity status of the aforementioned mobile device is determined to be uncertain, Depending on whether the integrity status of the mobile device is determined to be uncertain, the system is further configured to transmit one or more images to an internal user device for internal review. The apparatus according to claim 11.

19. Processing one or more images in order to determine the integrity state of a mobile device is The process involves determining the position within one or more images, wherein the position is defined as a bounding box. If the bounding box has a first predetermined relationship with the threshold ratio of one or more images, the mobile device is instructed to bring the mobile device closer to the reflective surface, including the following: The apparatus according to claim 11.

20. A persistent computer-readable storage medium storing computer executable program code instructions, wherein the computer executable program code instructions are Receiving one or more images of the mobile device captured by one or more image sensors of the mobile device via a reflective surface, Processing one or more images to identify one or more depressions using at least one trained model, wherein the one or more depressions are Determining the polygonal subregion P and the convex hull of P, The difference between P and the convex hull of P is defined as one or more depressions, and is identified by this, The one or more depressions identified in the one or more images are compared with a depression threshold, In response to determining that one or more depressions are greater than the depression threshold, a prompt is displayed on the mobile device to capture one or more images that do not have the one or more depressions. In response to determining that one or more depressions are smaller than or less than the depression threshold, the process involves ignoring the one or more depressions, and then processing the one or more images to determine the mobile device integrity state. A persistent computer-readable storage medium containing program code instructions for performing the following actions.

Images