Systems and methods for diagnostics of displays and housings of mobile devices

Abstract

Systems and methods for detecting mobile device fault conditions are presented, particularly for detecting defects in the display or housing of a mobile device. One or more cameras contained within the mobile device are used in conjunction with a reflective surface within a test fixture to capture properly aligned and formatted images of the exterior of the mobile device's housing and display, with interactive guidance assisting in alignment and performing diagnostics. A remote server further analyzes the captured images, after which the status of the defect may be returned to and displayed on the mobile device screen.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims the full benefit of U.S. Provisional Patent Application No. 62 / 924,676, filed October 22, 2019, entitled "SYSTEM AND METHOD FOR MOBILE DEVICE DISPLAY AND HOUSING DIAGNOSIS," the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0002] The present invention relates to a system and method for detecting a fault condition in a mobile device, and more particularly, to a system and method for detecting that a mobile device has a defect, such as a cracked, scratched, chipped, or otherwise damaged screen or housing, and reporting the status of the screen or housing, whether working or not, so that appropriate action can be taken by a third party. [Background technology]

[0003] The use of mobile devices today is widespread, with an increasing number of people using such devices in their daily lives. As used herein, the term "mobile device" generally refers to electronic devices that are capable of moving from place to place. Examples of such mobile devices include, but are not limited to, cellular phones (more commonly referred to as "mobile phones"), smart watches, smart jewelry, personal digital assistants (PDAs), digital cameras, intelligent devices for the "Internet of Things" (or "IOT"), drone devices, mobile subscriber communication devices, tablet computers, media players, smart vehicles, laptop computers, and devices with installed wireless intelligent agents such as Alexa®, Google Voice®, Siri®, and Cortana®. Consumers purchase millions of mobile devices, such as mobile phones, every year for reasons such as improved features and functionality, to replace worn-out, damaged, partially non-functional, or slow-operating devices, or simply to acquire the latest product based on consumer loyalty or to show off the new model to friends and acquaintances. Meanwhile, as more features and functionality are incorporated into newly released mobile devices, the rapidly increasing cost of such devices has created a secondary market for used but working mobile devices, and an industry has developed to process and recycle used mobile devices into a working condition for refurbishment and resale, or to classify inoperable devices into valuable components or scrap.

[0004] Used mobile device processors (also referred to herein as "recyclers") are faced with the challenge of receiving used mobile devices, such as cell phones, whose functionality and operating condition are unknown, assessing the functional condition of such devices, securely wiping all previous user data from the cell phones, and, if necessary, performing any necessary repairs. Due to the narrow profit margins resulting from the competitive market conditions and pricing of reclaimed mobile devices, efficiency in the evaluation, diagnosis, and erasure process is paramount. Recyclers must quickly assess the type of device, its operating condition, and perform any necessary pre-configuration, such as flashing and erasing the mobile device, in the most efficient manner that can ensure that reclaimed devices are returned to the stream of commerce as quickly as possible.

[0005] In traditional mobile phone recycling processes, recyclers often manually inspect the mobile device display and / or housing to assess the condition of the display, the glass on the display, the mobile device housing, and the exterior surface of the mobile device housing. These manual inspections are tedious, error-prone, and require additional manual input to document the current condition of the mobile device, further impeding the recycler's efficiency and warehouse throughput. Automated methods for assessing the condition of mobile devices are error-prone and may require manual alignment of the mobile device with a camera or reflective device, again leading to inefficiencies and delays in the device evaluation process.

[0006] In addition to mobile device recyclers, retail organizations such as mobile phone companies (also known as mobile phone service providers) are also faced with the need to quickly assess, diagnose, and document the condition of mobile devices brought in by customers at their retail stores or kiosks. To provide such services, mobile phone companies are faced with purchasing and configuring expensive hardware to diagnose mobile phones that may be in each retail store or kiosk. Summary of the Invention [Problem to be solved by the invention]

[0007] In summary, there is a need to provide a system and method for performing diagnostics, particularly on the exterior of a mobile device's housing and display, in an efficient manner and providing a documentation and analysis interface to a test system. [Means for solving the problem]

[0008] The following technical disclosure is exemplary and explanatory only and does not necessarily limit the invention as defined in the claims.

[0009] As used herein, the terms "mobile device," "mobile electronic device," or "device" generally refer to electronic devices that are capable of moving from place to place and have components (display, screen, etc.) that may fail or be damaged. A mobile device may be a standalone device such as a laptop computer, desktop computer, mobile subscriber communication device, mobile phone, smart watch, personal digital assistant (PDA), data tablet, digital camera, video camera, video game console, media player, global positioning system (GPS), universal serial bus (USB) key, mobile weapon, smart watch or jewelry, embedded electronics, and combinations thereof. A mobile electronic device may be any electronic device integrated with another system or device. For example, a stereo, global positioning system, or other electronics contained within a vehicle may be utilized in conjunction with the present invention. Software implementing the methods of the present invention may be (1) installed or (2) downloaded to the mobile device by an authorized user at any time, indirectly or directly, via the Internet, SMS text message, wireless communication with an app distribution store, or in any other suitable manner and at any suitable time to perform the methods of the present invention. For example, software may be installed on a device after purchase, downloaded after purchase of the device, or even after the device becomes damaged or otherwise defective. Mobile devices may be insured against loss or theft, and the systems and methods of the present invention may function as part of, or in addition to, a mobile device insurance policy.

[0010] A system and method for detecting a mobile device fault condition is presented, which in particular detects defects in the display or housing of the mobile device. A camera contained within the mobile device is properly aligned to capture the image. arrayThe device may be used with a reflective surface in the test fixture to capture images of the exterior of the mobile device housing and display, with interactive guidance to assist in alignment and performing diagnostics. A remote server may further perform analysis of the captured images, after which defect status may be returned to the mobile device and displayed on the mobile device screen.

[0011] One embodiment of the present invention provides a system for visually diagnosing a mobile device, the system comprising: a test fixture including a mounting platform mechanically connected via a support member to a top cover including a reflective surface; and a server configured to communicate with a provided mobile device, the memory of the mobile device including instructions that when executed by a processor of the mobile device, cause the mobile device to perform the steps of: prompting a user to place the mobile device within the test fixture; determining whether the mobile device is oriented in an acceptable orientation with respect to the reflective surface; and, in response to detecting that the mobile device is oriented in an acceptable orientation, capturing an image of the mobile device reflected from the reflective surface and transmitting the captured image to the server for defect analysis. arrayand determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface. Additionally, in an embodiment, determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface further includes displaying the tracking and positioning guidance fiducial marks on the mobile device, obtaining a tracking image of the mobile device from a camera of the mobile device, identifying an orientation of the mobile device in the tracking image, and determining whether the identified orientation is within a predetermined range. Additionally, displaying the tracking and positioning guidance fiducial marks on the mobile device may further include displaying the tracking fiducial marks as one or more QR codes, one or more Aztec codes, one or more ArUco codes, and one or more graphical tracking elements. The reflective surface may comprise any substantially planar or non-planar surface capable of reflecting an image of the mobile device, such as, for example, a mirror and a one-way mirror.

[0012] A method of visually diagnosing a mobile device is provided, the method including: prompting a user to place a mobile device in a test fixture having a mounting platform and a reflective surface; determining whether the mobile device is acceptably oriented relative to the reflective surface on the mounting platform; capturing an image of the device reflected from the reflective surface in response to detecting that the mobile device is acceptably oriented; analyzing the captured image to determine whether the captured image includes one or more defect indicia; and providing a cosmetic defect rating for the mobile device based on the analysis. The defect indicia may include any desired information, such as one or more indications of whether a housing and / or display of the mobile device includes cracks, chips, tears, blemishes, wear, scratches, abrasions, gouges, stuck pixels, dead pixels, color errors, color bleeding, and deviant marks. The method may further include detecting whether the mobile device is properly positioned relative to the reflective surface by analyzing the captured image including the tracking fiducial marks, providing dynamic guidance by modifying placement guidance fiducial marks displayed on the mobile device, instructing a user regarding placement of the mobile device on the mounting platform, and indicating when the mobile device is properly positioned relative to the reflective surface via the placement guidance fiducial marks.

[0013] In one embodiment, a system is also presented, the system comprising: a test fixture including a mounting platform mechanically connected via a support member to a top cover including a reflective surface, the reflective surface being positioned at a predetermined distance from the mounting platform and at a predetermined angle relative to a top plane of the mounting platform; and a server having a server processor communicatively connected to the database, a server memory communicatively connected to the server processor, a server user interface communicatively connected to the server processor, and a communication interface connected to the server processor and configured to communicate with a communication module of a provided mobile device, the provided mobile device comprising a housing substantially enclosing the mobile device, a mobile processor in communication with the memory of the mobile device, and a user interface including a display and a first camera in communication with the mobile processor. and a communication module configured to communicate with the mobile processor and to communicate with the communication interface of the server, the memory of the mobile device having a memory that, when executed by the processor, causes the mobile device to perform the steps of: prompting a user of the mobile device to initiate a test of one of a display or a housing of the mobile device; presenting an image on the display including tracking fiducial marks and placement guidance fiducial marks including alignment indicia; performing a tracking calibration function to adjust the position of the mobile device and the display for optimal image capture; modifying brightness settings and camera sensitivity settings of the mobile device to optimize image capture; determining that the mobile device is oriented in an acceptable orientation with respect to the reflective surface, capturing an image of the mobile device reflected from the reflective surface; and transmitting the captured image to the server for transmission to the server for defect analysis. arrayand a step of: capturing a first image of the mobile device from the first camera; and capturing a second image of the mobile device from the first camera; and capturing a second image of the mobile device from the first camera. In various embodiments, the mobile device provided includes a second camera, and in alternative embodiments, the reflective surface comprises one or more of a mirror and a one-way mirror. In yet another embodiment, the server is communicatively connected to the mobile device via one of a wired connection or a wireless connection, and in various embodiments, the memory and processor of the mobile device are configured with software that, when executed, performs the steps of initiating a test alignment mode, displaying the tracking fiducial marks on the display, and displaying the placement guidance fiducial marks on the display. A desired image may be output, for example, an image of the mobile device as seen by the camera is displayed on the display. In various embodiments, an image from the front camera captures an image of the display of the mobile device, and the rear camera captures an image of the rear of the housing of the mobile device. Proper alignment of the device relative to the test fixture may include detecting whether the mobile device is in a proper position relative to the reflective surface by analyzing an image from a camera including tracking fiducial marks, providing dynamic guidance by modifying placement guidance fiducial marks, instructing a user regarding placement of the mobile device on the mounting platform, and indicating when the mobile device is correctly positioned relative to the reflective surface via the placement guidance fiducial marks.

[0014] In various embodiments, the systems of the present invention may include a mounting platform that includes an alignment guide, which in optional embodiments may include one or more of optical fiducial marks, illuminated fiducial marks, one or more molded alignment guides, and an electronic display connected to the test processor for displaying alignment information, such as alignment fiducial marks and arrows.

[0015] Various embodiments may include capturing an image of the mobile device from the first camera after the mobile device is properly positioned on the mounting platform, and may further include analyzing the captured image and diagnosing at least one condition of the housing / display of the mobile device. Additionally, embodiments may further include analyzing from the analyzed image whether the positioning of the mobile device requires modification, and providing alternative positioning guidance fiducial marks for further alignment. In some implementations, the support elements are selectable to change the distance between the reflective surface and the mounting platform, and in other embodiments, the support elements can be actuated to obtain a predetermined angle between the reflective surface and the mounting platform.

[0016] Yet another embodiment of the present invention includes a system, the system comprising: a test fixture including a mounting platform mechanically connected to a reflective surface via a support member, the reflective surface being positioned at a predetermined distance from the mounting platform and at a predetermined angle relative to a top plane of the mounting platform; and a server comprising a server processor communicatively connected to a database, a server memory communicatively connected to the server processor, a server user interface communicatively connected to the server processor, and a communication interface connected to the server processor and configured to communicate with a communication module of a provided mobile device, the provided mobile device comprising a housing substantially enclosing the mobile device, a mobile processor in communication with the memory of the mobile device, and a user interface including a display and a first camera in communication with the mobile processor. a mobile device interface and a communications module configured to communicate with the mobile processor and to communicate with the communications interface of the server, the memory of the mobile device having a memory that, when executed by the processor, causes the mobile device to instruct a user of the mobile device to initiate a test of one of the display or housing of the mobile device; presenting an image on the display that is a tracking indicia including indications of alignment; performing a tracking calibration function to adjust the position of the mobile device and the display for optimal image capture; modifying brightness settings and camera sensitivity settings of the mobile device to optimize image capture; determining that the mobile device is oriented in an acceptable direction relative to the mirror, capturing an image of the mobile device reflected from the mirror; and transmitting the captured image to the server for transmission to the server for defect analysis. arrayIn addition to the features described above, the system may include any desired hardware including, for example, one or more cameras, lighting fixtures, microphones, speakers, vibrators, wireless charging pads, NFC circuitry, conveyor belts, and mobile device manipulation gimbals, fixtures, and motors.

[0017] The methods of the present invention may be performed in whole or in part on a mobile device, a host server, external neural network hardware, and a server, and / or a processor communicatively connected thereto.

[0018] A more complete understanding of the present invention may be derived by reference to the detailed description and claims when considered in conjunction with the following illustrative drawings. [Brief description of the drawings]

[0019] [Figure 1A] FIG. 2 is a perspective front view of the test device of the present invention; [Figure 1B] FIG. 1 is a front perspective view of a test fixture of the present invention. [Figure 1C] FIG. 2 is a rear perspective view of the test fixture of the present invention. [Figure 1D] FIG. 2 is a top view of the test fixture of the present invention. [Figure 1E] FIG. 1 illustrates a two-device test fixture that is sized to accommodate two mobile devices. [Figure 1F] FIG. 1 is a perspective view of a larger test fixture configured to accommodate multiple mobile devices. [Figure 1G] FIG. 13 illustrates the top cover / reflective surface closed position configured to allow multiple mobile devices to perform display / housing testing. [Figure 1H] FIG. 13 illustrates an open position that allows easier visual inspection and / or placement of the mobile device within the test fixture. [Figure 1I]FIG. 1 is a perspective view of a test fixture of the present invention configured to include a one-way mirror in the top cover. [Figure 1J] FIG. 1I is a side plan view of the test fixture of FIG. 1I including a one-way mirror in the top cover. [Diagram 2] FIG. 1 is a flow diagram of the process of the present invention. [Diagram 3] FIG. 1 illustrates a mobile device on which a user initiates the process of the present invention. [Figure 4] FIG. 1 is a perspective view of a mobile device including a reference mark of the present invention. [Figure 4A] FIG. 2 is a plan view of a mobile device display presenting two tracking reference marks as a QR code. [Figure 4B] FIG. 1 is a plan view of a mobile device display presenting two tracking reference marks as a QR code together with a second or placement guidance reference mark. [Figure 4C] FIG. 1 is a plan view of a mobile device display presenting two tracking reference marks as a QR code together with a second or placement guidance reference mark. [Figure 4D] FIG. 1 is a plan view of a mobile device display presenting two tracking reference marks as an Aztec code along with a secondary or placement guidance reference mark. [Figure 4E] FIG. 1 is a plan view of a mobile device display presenting two tracking reference marks as an ArUco code together with a secondary or placement guidance reference mark. [Diagram 5] FIG. 1 illustrates the placement of a mobile device within a test fixture of the present invention. [Figure 6] FIG. 1 illustrates a mobile device properly aligned within a test fixture of the present invention. [Figure 7] FIG. 1 illustrates a mobile device indicating pass or fail of a test. [Figure 8] FIG. 1 is a block diagram of an embodiment of a system of the present invention. [Figure 8A] FIG. 2 is an alternative block diagram of an embodiment of the system of the present invention. [Figure 9] FIG. 8 illustrates an embodiment of the present invention in which a mobile device 800 is illuminated externally by a light source such as an LED light strip. [Figure 10] FIG. 8 illustrates an embodiment of the present invention in which a mobile device 800 is illuminated externally by a light source such as an LED light strip. [Figure 11] FIG. 8 illustrates an embodiment of the present invention in which a mobile device 800 is illuminated externally by a light source such as an LED light strip. [Figure 12] FIG. 8 illustrates an embodiment of the present invention in which a mobile device 800 is illuminated externally by a light source such as an LED light strip. [Figure 13] FIG. 8 illustrates an embodiment of the present invention in which a mobile device 800 is illuminated externally by a light source such as an LED light strip. [Figure 14] FIG. 1 illustrates multiple camera options for the test fixture of the present invention. [Figure 15] FIG. 1 illustrates multiple camera options for the test fixture of the present invention. [Figure 15A] FIG. 1 illustrates multiple camera options for the test fixture of the present invention. [Figure 16] FIG. 1 is a process flow diagram of the present invention illustrating image capture and image analysis to identify device defects. [Figure 16A] FIG. 1 is a process flow diagram of the present invention illustrating image capture and image analysis to identify device defects without first installing an app on a mobile device. [Figure 17] FIG. 2 is a process flow diagram of the present invention providing further details of an exemplary image capture process. [Figure 18] FIG. 2 is a process flow diagram of the present invention providing further details regarding an exemplary image pre-processing method. [Figure 19]FIG. 1 illustrates an exemplary display of a mobile device with a defect on the display of the mobile device. [Figure 20] FIG. 20 illustrates the example image of FIG. 19 after a thresholding technique has been applied. [Figure 20A] A diagram showing an example perspective transformed sub-image of a mobile device's display corresponding to the displayed portion of the captured image in Figure 19. [Figure 20B] 20 illustrates an exemplary output of the present invention identifying defects surrounded by a polygon on a display corresponding to the device of FIG. 19. [Figure 20C] FIG. 1 illustrates an exemplary device image for input to a convolutional neural network of the present invention. [Figure 20D] Figure 20C shows the heat map output identifying defect potential areas corresponding to the device image shown in Figure 20C. [Figure 20E] FIG. 2OD shows a defect surrounded by a polygon on the display corresponding to the device image shown in FIG. 20C. [Figure 20F] FIG. 1 illustrates an exemplary device image for input to a convolutional neural network of the present invention. [Figure 20G] Figure 20F shows the heat map output identifying defect potential areas corresponding to the device image shown in Figure 20F. [Figure 20H] FIG. 2C shows a polygonal defect on the display corresponding to the device image shown in FIG. 20F. [Figure 21] FIG. 13 illustrates an exemplary display of a mobile device of the present invention showing corner fiducial markers. [Figure 22] FIG. 22 illustrates template matching correction of the image on the display shown in FIG. 21. [Figure 23] FIG. 22 shows template matched corner markers obtained from the display image in FIG. 21 used to determine edges of a display associated with a mobile device. [Figure 24]FIG. 1 is a block diagram with data flows describing one convolutional neural network configuration for the system of the present invention. [Diagram 25] FIG. 1 is a comprehensive flow diagram describing neural network training from network training data and the network inference process of identifying defects using the trained neural network. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Like reference numbers indicate like structure or function in the various embodiments illustrated herein.

[0021] Test Equipment

[0022] FIG. 1 shows a schematic diagram of an embodiment of the present invention. The test fixture 200 provides a platform for automated testing of the display 1400 and / or housing of a mobile device 800. The test fixture includes a mounting platform 205 with optional alignment guides 225 to assist in positioning the mobile device 800. The mounting platform 205 is mechanically connected to a reflective surface 215 via a support member 210. The reflective surface 215 may be attached to a surface of a top cover 220 such that the reflective surface substantially faces the mounting platform 205, and the top cover 220 may be attached to the support member and allow attachment to the reflective surface 215 via a bottom surface. The reflective surface 215 may comprise a mirror or other surface that reflects an object placed on the mounting platform 205. The reflective surface 215 may be reoriented relative to the mounting platform 205 by changing the distance between the reflective surface 215 and the mounting platform 205 via a height adjustment device 235. Additionally, the reflective surface 215, in a preferred embodiment, is substantially parallel to a plane that substantially includes the top surface of the mounting platform 205, but may be angled relative to the mounting platform, such as via a hinge connecting the support member 210 to the top cover 220 / reflective surface 215, by actuating the reflector angle adjustment fixture 230. Any desired angle may be obtained relative to the mounting platform to improve operation of the display and housing diagnostic aspects of the present invention. Also shown in the figure is a mobile device 800, which is in wired or wireless communication 255 with a server 860. The test fixture 200 is configured to accept insertion 251 of the mobile device 800, and in one embodiment, is configured to assist in alignment of the placement of the mobile device 800 on the mounting platform via optional alignment reference marks 225.The fiducial markings 225 may comprise any visual indicator to assist the user in proper placement of the mobile device, and in some embodiments may comprise a box or other polygon, a raised area or an illuminated area to guide the user to optimal placement of the mobile device 800. Height Adjustment Fixture The height adjustment fixture 235 may comprise any desired mechanism, such as a slide, ratchet, locking bar, rack, selectable pins and holes, or any other adjustment mechanism, to change the distance between the mounting platform 205 and the reflective surface 215.

[0023] The test fixture 200 may be sized in any suitable dimensions, and the dimensions may be selected such that the mobile device's front camera 874 is at a sufficient distance to focus and resolve the reflected image of the mobile device 800 when the image is transmitted from the mobile phone and reflected downward toward the mobile phone. In one embodiment, the distance 277 between the mounting platform 205 and the reflective surface 215 is approximately 160 mm. Additionally, the test fixture 200 may be sized to accommodate any desired mobile device 800 or multiple mobile devices in the horizontal dimension, and in one embodiment, is sized approximately 210 mm deep (direction 276) and 180 mm wide (direction 275). The height adjustment device 235 may be actuated to customize the distance between the mobile device and the reflective surface, and may accommodate a fixed focus device or a variable focus device.

[0024] Positioning the mobile device 800 closer to the reflective mirror 215 may aid in optimal resolution of the image captured by the front-facing camera 874, but because of the typical edge-to-edge location of the front-facing camera 874, the closer the mobile device is moved toward the reflective surface 215, the more tilt needs to be applied to either the reflective surface 215 or the mobile device 800 to capture a complete image of the mobile device 800. In other words, if the mobile device 800 is positioned too close to the reflective surface 215, the camera may not be able to capture the bottom of the screen due to the location of the camera on the phone (e.g., closer to the edge rather than the center), while moving the mobile device 800 farther away may not produce sufficient resolution to detect cracks or other defects in the display. In one aspect, the preferred spacing may be created by positioning the mobile device 800 at least ½ the minimum focal length of the front-facing camera 874, such that the "above and behind" distance correlates with at least one focal length.

[0025] Various other angles further illustrate views of implementations of the present invention. FIG. 1A shows an angled front view looking at test fixture 200, with reflective surface 215 shown positioned under top cover 220. FIG. 1B shows a front perspective view of test fixture 200 with mobile device 800 inserted and positioned within alignment guides. FIG. 1C shows a rear perspective view of test fixture 200. FIG. 1D shows a top view of test fixture 200.

[0026] Various alternative implementations of the present invention are possible that can accommodate multiple mobile devices simultaneously. Such implementations may be advantageous in a warehouse environment that may be used by a device recycler or repair agency. FIG. 1E illustrates a two-device test fixture 200A that is sized to accommodate two mobile devices 800, 801 simultaneously. Note that the display 1400 of the mobile device 800 indicates that the display diagnostic test has passed, and the display 1401 of the mobile device 801 indicates that a crack has been detected on the mobile device. Additionally, FIG. 1F illustrates a perspective view of an even larger test fixture 208 configured to accommodate multiple mobile devices 800 (nine are shown, but any desired number may be used in embodiments of the present invention). Such a configuration may include increasing the width dimensions of the top cover 220, the reflective surface 215, the support member 210, and the mounting platform 205.

[0027] As previously discussed, the top cover 220 and / or the reflective surface 215 may be moved relative to the mounting platform 205. In one embodiment, the top cover 220 and the reflective surface 215 are hinged to the support member such that the top / reflective surface can be rotated between their open and closed positions. FIGS. 1G-1H show such a test fixture 200C, with a hinge 230 connecting the top cover 220 / reflective surface 215 to the support member 210, with FIG. 1G showing a closed position of the top cover / reflective surface configured to allow multiple mobile devices 800 to perform display testing, and FIG. 1H showing an open position allowing easier visual inspection and / or placement of the mobile devices 800 within the test fixture 200C. The test fixture 200C of FIGS. 1G-1H shows legs supporting the mounting platform 205, and the legs shown may or may not be used based on the desired test setup.

[0028] 1I-1J show an embodiment of the test fixture 200D of the present invention further configured with a top cover 220D including a one-way mirror. A one-way mirror is also referred to in the art as a two-way mirror, magic glass, half mirror, or semi-transparent mirror. Such an implementation provides a reciprocal mirror that appears reflective on one side, such as the bottom side (providing a reflective surface 215), and transparent on the other side, such as the top side, as may be seen by an observer viewing (243) the mobile device 800 from a position above the top cover 220D. The mounting platform 205 is mechanically connected to the top cover 220D via the support member 210. The top cover 220D / reflective surface 215D is positioned such that the reflective surface 215D substantially faces the mounting platform 205 and provides a reflection 216D from the reflective surface 215D to the mobile device 800. The top cover 220D may be attached to the support member 210, allowing attachment to the reflective surface 215D of the top cover 220D via its bottom surface. The top cover 220D may be implemented as a continuous one-way mirror, or may include a transparent element such as glass or plexiglas with the one-way mirror attached to its bottom surface, providing the reflective surface 215D with additional rigidity / shatter resistance and / or support provided by the transparent material on the top surface of the top cover 220D. The top cover 220D may include a transparent or opaque frame (not shown) partially or completely surrounding the perimeter of the one-way mirror, providing additional rigidity, strength, and edge chip resistance while allowing an operator to view (243) the mobile device 800 from a position above the top cover 220D. The test fixture 200D of FIGS. 1I-1J shows legs supporting the mounting platform 205, and the legs shown may or may not be used based on the desired test setup.

[0029] FIG. 3 illustrates a method of the present invention utilizing aspects of the test fixture illustrated in FIGS. 1A-1H, and more specifically, intermediate results are illustrated in FIGS. 3-7. Further background to the illustrated method is provided by reviewing the mobile device and server architectures illustrated in FIGS. 1, 8, and 8A. In the illustrated embodiment, a system and method are provided for identifying cracks, housing defects, or other types of defects associated with a mobile device through image analysis. Referring to process 100 illustrated in FIG. 2, an app is installed 110 on a mobile device 800 that is desired to be tested. However, in one embodiment, a user of the mobile device 800 under test activates a web link to a server 860 via a web browser hosted by the operating system of the mobile device. In yet another embodiment, a combination of software installed or operating on the mobile device 800 and software installed and operating on the server 860 may interoperate to perform the described method. When ready, the user of the mobile device 800 initiates the device defect test (115) and, as shown in FIG. 4, the display 1400 of the mobile device 800 displays a first fiducial mark 1430, also called a tracking fiducial mark 1430, and a second fiducial mark 1420, also called a placement guidance fiducial mark, indicating that the mobile device 800 is ready to be placed in the test fixture 200. In a preferred embodiment, the second fiducial mark 1420 includes an indication that placement has not yet been properly sensed (indicated by a red outline in the illustrated photograph), and an image seen by a front first camera 874 is displayed in the second fiducial mark 1420 (see FIGS. 5 and 6, which show an image seen by the camera 874 displayed in the second fiducial mark 1420). Any desired graphics may be used as the first fiducial mark.In an alternative embodiment shown in FIG. 4A, the mobile device display 1400 outputs two "first fiducial marks" 1430, 1430A as a QR code, which need not contain any identifying information, but may be encoded using random information, or may simply contain information that allows optical tracking of the mobile device display 1400. The use of such first fiducial marks 1430, 1430A, together with a predefined placement location on the screen, allows for determination of the screen orientation, and the first fiducial marks 1430, 1430A may allow for extrapolation of edge boundaries of the display 1400 based on the known shape of the device on which the first fiducial marks are displayed. Additionally, FIGS. 4B-4C show plan views of a mobile device display presenting two tracking fiducial marks as a QR code, together with the second fiducial mark 1420, or placement guidance fiducial mark. In FIG. 4B, the second fiducial mark 1420 indicates a position of improper alignment relative to the reflective surface, and in FIG. 4C, the second fiducial mark indicates a position of proper alignment relative to the reflective surface. As shown in FIGS. 4B-C, the second fiducial mark 1420 may include a bounding box placed around the detected preview image of the mobile device 800 to allow the user to further determine the orientation of the mobile device 800 relative to the reflective surface. Additionally, FIG. 4D shows a top view of a mobile device display presenting two tracking fiducial marks as an Aztec code along with a second, or placement guidance, fiducial mark. The Aztec code is known to those skilled in the art as a 2D barcode device detectable and trackable by a camera connected to optical tracking software.

[0030] Moreover, the ArUco marker (or code) is known to those skilled in the art as including a synthetic square marker consisting of a wide black border and an internal binary matrix. The ArUco library is a well-known library for the detection of square fiducial markers, developed by Rafael Munoz and Sergio Garrido, and corresponding software libraries are used by those skilled in the art to detect these fiducial marks. As a first fiducial mark, the ArUco marker enables tracking and orientation calculations for software accessing the rear camera sensor of the mobile device. Thus, FIG. 4E shows a top view of the display of a mobile device presenting two tracking fiducial marks 1430, 1430A as ArUco markers / codes, together with a second or placement guidance fiducial mark 1420.

[0031] Considering Figures 4 and 4A-E, the purpose of the first fiducial mark 1430 (alternatively, along with the second first fiducial mark 1430A) is to provide a frame of reference on the display of the mobile device 800 that is unchanging, allowing tracking and determination of the position of the mobile device 800 in three-dimensional space relative to the reflecting device. The first fiducial mark may include any mark or marking that allows tracking and location determination, and in preferred embodiments need not include encoded information. In Figure 4, the location of the first fiducial mark 1430 is shown at the top center within the display area 1400 of the mobile device 800, but the first fiducial mark may be placed in any desired location to aid in position fixation of the mobile device 800. Although not required in preferred embodiments, in alternative embodiments, the first fiducial mark 1430 may be altered to uniquely identify the particular mobile device 800 on which it is displayed, thus preventing misidentification of other mobile devices that may be displaying the fiducial mark within a multiple test environment. Such unique identification may include a QR code containing a serial number or other identifying indicia of the encoded mobile device. The second fiducial mark 1420 provides guidance to assist the user in moving the mobile device 800 to a proper position relative to the reflecting device so that an ideal image can be captured, to minimize parallax distortion and parallax error, minimize or eliminate noise and undesirable reflections, and ensure that the front-facing camera 874 can capture an image of the entire mobile device 800 in its environment. In one embodiment, the front-facing camera 874 captures images in a real-time live video preview mode, where the image shown near the second fiducial mark 1420 in the display area moves in real-time as the mobile device 800 moves.4, 5 and 6, 14A-D, the second fiducial mark 1420 is shaped as a colored parallelogram such that when the mobile device 800 is properly aligned, the image seen by the front camera 874 is displayed within the second fiducial mark 1420, and as described in more detail below, the fiducial mark changes state (such as by changing color from red to green or other indication including text) to indicate proper alignment. Additionally, in an alternative embodiment as shown in FIGS. 4A-E, the second fiducial mark 1420 may be shaped as a rounded rectangular region with a color indicating the status of the alignment of the mobile device (such as red when not aligned and green when properly aligned), along with an optional preview image as seen from the rear camera of the mobile device 800 (to assist the user in alignment and orientation).

[0032] Alternatively, a marker fiducial mark (not shown) having known geometric dimensions may be output on the display of the mobile device 800 (and a scale such as centimeter scale may be displayed). The mobile device 800 has access to predetermined measurements since it is running its own software, and the fiducial marker allows scaling of the display by the measured fiducial mark, thereby allowing alignment of the bounding box of the display and mobile device, and thus in this approach there is no need to rely on edge detection simply to find the phone / display in the captured image.

[0033] Returning to FIG. 2, the user then places the mobile device 800 on the mounting platform 205 (see also FIG. 5) and the location detection software is executed to locate the mobile device in the image captured by the front first camera 874 and evaluate the placement (e.g., by attempting to detect whether the first fiducial mark 1430 is visible in the camera's image and the mobile device is properly placed). The detection of proper placement may occur by various approaches. If the user is utilizing a web-based server approach, a color (such as a unique color such as bright magenta that is unlikely to appear in a normal image) is selected, the color is output to the display in the four corners of the mobile device's display 1400, and color tracking is then performed in the live video generated by the front camera 874 to locate the color elements in the four corners of the screen; this can be performed, for example, by the use of a color tracking library to locate the color (such as an exemplary bright magenta) in the image, search for those colors, and then identify the location of the corners of the display 1400 in the video image. This information is then used to locate polygons in the image. Alternatively, the image can be analyzed to determine whether the first fiducial mark 1430 is present and to determine the location and orientation of the first fiducial mark 1430 within the image. Given a database of device types, the identity of the mobile device itself further assists in identifying the device's contours and pre- and post-processing of the images.

[0034] After it is determined that the mobile device 800 is properly positioned, the second fiducial mark 1430 changes state to indicate proper placement (see, e.g., the green line in FIG. 6 for the second fiducial mark 1430) and the first camera 874 on the front of the mobile device captures an image (FIG. 3, 120).

[0035] In addition to the wide range of pre-processing options described below with respect to FIG. 17, certain types of checks may be performed on the captured image, including, for example, checking focus, checking brightness, and / or evaluating blurriness of edges. Furthermore, if it is determined that the current image is unsatisfactory, another image can be recaptured using different parameters of the camera (125). Additionally, the captured image may be optionally pre-processed (125) by rotating the image to portrait orientation, and / or finely rotating to reduce distortion, and / or cropping the image to a desired aspect ratio. The captured (and optionally pre-processed) image is then uploaded (130) to a server (such as server 860). Depending on the required compliance of applicable laws such as GDPR, before the image file is transferred to server 860, the user may be prompted to upload the file first, and may optionally be given an opportunity to review the captured image and recapture the image if necessary. After the upload has begun, a progress bar may be output on display 1400 to indicate the progress of the upload.

[0036] As will be further described with respect to FIG. 18, the uploaded images are processed (135) to determine whether cracks or other types of defects are present. Alternatively or in combination, the extent and nature of the screen cracks or defects are characterized by processing step 135 for further analysis and reporting. After the analysis of potential defects 135 is completed, the resulting information may be processed for presentation, storage, or display to any user for any purpose desired, such as evaluating whether the mobile device 800 is in proper operating condition. arrayThe server returns information to the mobile device indicating the results of the analysis (140), and in one embodiment, if no defects are detected, a pass indicator may be displayed on the screen 1400 of the mobile device 800 (see, e.g., "PASS" shown in FIG. 6). Additionally, as described in more detail below, if a defect is detected, a bounding box of the detected defect (crack shown) on the mobile device 801 may be surrounded by a graphical bounding box 1402, such as a red box shown, and an indicator indicating that the diagnosis detected the failure criteria (e.g., "CRACK" shown) is displayed. The process then ends (145).

[0037] The exemplary system illustrated in FIG. 8 includes a mobile device 800 having a processor 810 coupled to a memory 820, which may include volatile memory, non-volatile memory (such as flash memory), or a combination thereof. A communication module 830 includes a wireless transceiver 840 for wireless communication with one or more servers 860 and other entities via an antenna 850, although one skilled in the art will appreciate that a wired connection may be established to provide connectivity instead of or in addition to a wireless connection. The mobile device also includes a user interface 870 coupled to the processor 810. The mobile device 800 may include any suitable power source, such as a battery (not shown). The mobile device 800 may include any other desired components, such as a global positioning system (GPS) to provide geolocation information for locating the mobile device. Some or all of the components of the mobile device 800 may include (or be in communication with) a hardware identification module (not shown), such as a universal subscriber identity module and / or a removable user identity module. A hardware identification module may be coupled to the processor 810 and may comprise an identifier that can be compared to the predefined identifier to determine if the hardware of the mobile device 800 has been altered. The hardware identification module (and predefined identifier) ​​may include any suitable identifier, such as an electronic serial number, a local area ID identifier, an integrated circuit identifier, an international mobile subscriber identifier, an authentication key identifier, and / or an operator-specific emergency number identifier. This identifier may be stored in the memory 820 and transmitted to the host server 860 for comparison against the predefined identifier.

[0038] The functionality of the mobile device 800, including the methods described herein (in whole or in part), may be implemented by the processor 810 executing computer-readable instructions stored in the memory 820 of the mobile device 800. The memory 820 may store any computer-readable instructions and data, including software applications, "apps," installed by the user or by a third party, applets, and embedded operating code.

[0039] Additionally, software applications installed on the mobile device (or interfaced in any manner with the processor 810) may be configured to operate with minimal prerequisite hardware capabilities. For example, such applications may be started before the mobile device establishes a network connection. Such a situation may be provided, for example, when a software application is installed on a SIM card in the mobile device, and the application launches before other software of the mobile device operating system. Alternatively or additionally, a data element such as a link or URL (Universal Resource Locator) may be present on the SIM card, and launching an application such as a browser using the URL or link may cause the application referenced by the link or URL to be loaded from a remote server to the mobile device and / or executed directly from the remote server.

[0040] The software-implemented methods of the present invention may be provided with the device or downloaded to the mobile device by an authorized user. The mobile device 800 functionality may also be implemented by various hardware components storing machine-readable instructions, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and / or complex programmable logic devices (CPLDs). Systems according to aspects of the present invention may operate with any desired combination of software and / or hardware components.

[0041] The processor 810 reads and executes instructions stored in the memory 820 to control the operation of the mobile device 800. Any number and type of processors may be used with the present invention, such as integrated circuit microprocessors, microcontrollers, and / or digital signal processors (DSPs). The memory 820 stores instructions, data, messages sent from (or received by) the mobile device 800, and any other suitable information. The memory 820 that operates with the present invention may include any combination of different memory storage devices, such as a hard drive, random access memory (RAM), read only memory (ROM), flash memory, or any other type of volatile and / or non-volatile memory. Data may be stored in the memory 820 in any desired manner.

[0042] The communication interface 830 communicates with one or more servers 860, such as a host server 860 or other suitable entities. Any suitable communication devices, components, systems, and methods may be used with the present invention. For example, the wireless transceiver 840 may be configured to communicate using any number and type of cellular protocols, such as General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), Personal Communications Services (PCS), Advanced Mobile Phone System (AMPS), Code Division Multiple Access (CDMA), Wideband CDMA (W-CDMA), Time Division Synchronous CDMA (TD-SCDMA), Universal Mobile Telecommunications System (UMTS), and / or Time Division Multiple Access (TDMA). Mobile devices operating with the present invention may alternatively (or additionally) include a wireless transceiver (and associated components) for communicating using any other wireless communication protocol methodology, such as ISO14443 protocol, ISO18000-6 protocol, Bluetooth protocol, ZIGBEE protocol, Wibree protocol, Wi-Fi protocol, IEEE802.15 protocol, IEEE802.11 protocol, IEEE802.16 protocol, Ultra Wideband (UWB) protocol, IrDA protocol, and combinations thereof. Antenna 850 may be configured to transmit and receive any wireless signal having any format, and may include multiple different antennas that transmit and receive using different wireless protocols.

[0043] The communications module 830 may communicate with the server 860 or another device 862 (such that the other device 862 may similarly communicate with the server 860) using any other type of connection, such as a wired Internet connection, a wireless Internet connection, a cellular network connection (including a data link connection), a WLAN connection, a WWAN connection, an optical connection, a USB connection, a Bluetooth® connection, a mobile device sync port connection, a power connection, and / or a security cable. The communications module 830 may be used to communicate with one or more companion devices, and with any number of other devices that aid in tracking / locating a lost or stolen mobile device 800, and to monitor the location or status of the mobile device 800 (e.g., by monitoring whether the communications link between the mobile device and the companion device is healthy).

[0044] Mobile device 800 includes a user interface 870. User interface 870 may include any number of input devices (not shown) for receiving commands, data, and other suitable input from a user, as well as any number of output devices (not shown) for providing data, notifications, and other suitable information from mobile device 800 to the user.

[0045] Any number of input devices, such as a touch pad, touch screen, a display that may or may not be integrated with a touch screen, and / or an alphanumeric keypad, may be provided in the user interface 870 to allow a user to input instructions and data into the mobile device 800. For purposes of this application, the term "touch screen" includes a display that is integrated with or in close proximity to a touch interface that can determine when a user applies a physical connection to a location in close proximity to the display. The touch screen may have sensors that can measure parameters from the user's interaction, such sensors may measure capacitance, resistance, pressure, or differential measurements resulting from the movement of a "touch" to the screen. The user interface 870 may be configured to detect the pressure that a user exerts on the keys of a keypad (either virtually implemented in the display or as a physical arrangement of key switches) and the time interval between multiple key presses to determine whether the current user is authorized to use the device. The user interface may also include a microphone that allows a user to provide voice data to the mobile device 800, and may also include one or more cameras (874, 872) that allow the mobile device to capture still or moving images. In one embodiment, the mobile device 800 includes a front camera 874 that faces the user of the mobile device when the device is in normal operation, and a rear camera 872 on the opposite side of the mobile device, which are connected to the processor 810. The mobile device 800 may also include voice recognition software for processing voice input via the user interface 870. The user interface 870 may also include any number of suitable output devices, such as a display screen for visually displaying information (e.g., video and text) and / or a speaker for providing aural output.The display of the mobile device may be configured to sense touch by a user by any suitable means, such as capacitive sensing, pressure sensing, gel displacement sensing, resistive sensing, or any other suitable or conventional touch transmission technology utilized by those skilled in the art. The mobile device 800 may be configured to provide a word, phrase, tone, recorded music, or any other type of auditory output to the user via a speaker. As previously described, the user interface 870 may be activated to provide information and / or interfere with operation of the mobile device 800 when an unauthorized user attempts to use the mobile device 800. For example, the lighting level of the display may be changed to call attention to the mobile device, and unpleasant and / or annoying sounds may be played through the speaker.

[0046] The mobile device 800 may include one or more biometric authentication devices configured to receive biometric information, such as a fingerprint scanner, an iris scanner, a retina scanner, and / or a breath analyzer. Input devices such as a microphone and a camera may be used to perform biometric analysis, such as voice analysis and facial recognition. Additionally, the mobile device may include a magnetometer (such as that used in an electronic compass) for measuring magnetic fields, a MEMS or other type of gyroscope for measuring orientation, and an accelerometer for measuring changes in the motion of the mobile device.

[0047] Information provided or received by user interface 870 may be in any suitable format. For example, a user interface that conveys information to a user in an audible format may first provide the user with a data header, followed by a data value to identify the data. User interface 870 may provide information in any number of desired languages, regardless of whether the information is provided audibly or visually.

[0048] The user interface may also provide / receive information to the user in a machine-readable format. In one exemplary embodiment of the present invention, for example, the user interface 870 of the mobile device 800 may send and receive messages using dual-tone multi-frequency (DTMF) tones. The mobile device 800 may be configured to send, receive, and process machine-readable data can in any standard format (such as MS Word documents, Adobe PDF files, ASCII text files, JPEG, or other standard formats) and any proprietary format. Machine-readable data to and from the user interface may be encrypted to protect the data from unintended recipients and / or improper use. In an alternative embodiment, the user may be required to enter a passcode to enable use of some or all of the features of the mobile device 800. Any other user interface features may be utilized to enable a human or non-human user to interact with one or more devices operating with the present invention.

[0049] Mobile device 800 may include any other suitable features, components, and / or systems. For example, mobile device 800 may be configured to conserve its battery life by turning off some or all of its components, such as the camera and microphone. Components may be selectively shut down in response to a security breach event and in response to commands from an authorized user or security agency. Alternatively, mobile device 800 may be configured to overuse its components to drain the battery as quickly as possible, e.g., to limit the usefulness of mobile device 800 to unauthorized users.

[0050] The mobile device 800 may be configured to implement one or more security measures to protect data, restrict access, or provide other desired security functionality. For example, the mobile device 800 may encrypt data transmitted and / or data stored on or created by the device itself. Such security measures may be implemented using hardware, software, or a combination thereof. Any method of data encryption or data protection may be utilized with the present invention, such as public / private key encryption systems, data scrambling techniques, hardware and software firewalls, tamper-resistant or tamper-resistant memory storage devices, or any other method or technique for protecting data. Similarly, passwords, biometrics, access cards or other hardware, or any other system, device, and / or method may be used to restrict access to any device operating with the present invention.

[0051] The exemplary system illustrated in FIG. 8A comprises a mobile device 800 including a processor 810 coupled to a memory 820, which may include volatile memory, non-volatile memory (such as flash memory), or a combination thereof. A communication module 830 comprises a wireless transceiver 840 for wirelessly communicating with one or more servers and other entities, such as a host server 860, via an antenna 850, although one skilled in the art would understand that a wired connection may be established to provide connectivity instead of or in addition to a wireless connection. The mobile device also comprises a user interface 870 coupled to the processor 810. The mobile device 800 may comprise any suitable power source, such as a battery (not shown). The mobile device 800 may comprise any other desired components, such as a global positioning system (GPS) for providing geolocation information for locating the mobile device. Some or all of the components of the mobile device 800 may comprise (or be adapted to communicate with) a hardware identification module (not shown), such as a universal subscriber identity module and / or a removable user identity module. A hardware identification module may be coupled to the processor 810 and may include an identifier that can be compared to a predefined identifier to determine if the hardware of the mobile device 800 has been altered. The hardware identification module (and predefined identifier) ​​may include any suitable identifier, such as an electronic serial number, a local area ID identifier, an integrated circuit identifier, an international mobile subscriber identifier, an authentication key identifier, and / or an operator specific emergency number identifier. This identifier may be stored in the memory 820 and transmitted to the host server 860 for comparison to the predefined identifier.

[0052] The functionality of the mobile device 800, including the methods described herein (in whole or in part), may be implemented by the processor 810 executing computer-readable instructions stored in the memory 820 of the mobile device 800. The memory 820 may store any computer-readable instructions and data, including software applications, "apps," installed by the user or by a third party, applets, and embedded operating code.

[0053] Additionally, the software application may be configured to operate with minimal prerequisite hardware capabilities. For example, the application may be launched before the mobile device establishes a network connection. Such a situation may be provided, for example, when the software application is installed on a SIM card in the mobile device, and the application launches before other software of the mobile device operating system. Alternatively or additionally, a data element such as a link or URL (Universal Resource Locator) may be present on the SIM card, and launching an application such as a browser using the URL or link may cause the application referenced by the link or URL to be loaded from a remote server to the mobile device and / or executed directly from the remote server.

[0054] Software-implemented methods of the present invention may be provided with the device or downloaded to the mobile device by an authorized user, and / or may further reside in the memory 16 of the host server 860 and be executable by the server processor 14. The functionality of the mobile device 800 and the host server 860 may also be implemented by various hardware components storing machine-readable instructions, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) and / or complex programmable logic devices (CPLDs), graphics processing units (GPUs), and neural network processing or simulation circuitry. Systems according to embodiments of the present invention may operate with any desired combination of software and / or hardware components.

[0055] The processor 810 reads and executes instructions stored in the memory 820 to control the operation of the mobile device 800. Similarly, the server processor 14 reads and executes instructions stored in the server memory 16 to control the operation of the host server 860. Any number and type of processors, such as integrated circuit microprocessors, microcontrollers, and / or digital signal processors (DSPs), may be used with the present invention. The memory 820 stores instructions, data, messages sent from (or received by) the mobile device 800, and any other suitable information, and the server memory 16 similarly stores instructions, data, messages sent from (or received by) the host server 860, and any other suitable information. The memory 820 and the server memory 16 operating with the present invention may include any combination of different memory storage devices, such as hard drives, random access memory (RAM), read only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory. The data may be stored in memory 820 or server memory 16 in any desired manner.

[0056] The communication interface 830 communicates with one or more servers, such as a host server 860, or other suitable entities. Similarly, the communication interface 18 of the host server is configured to communicate with the mobile device 800, a general network, such as the Internet, or any other suitable entity. Any suitable communication devices, components, systems, and methods may be used with the present invention. For example, the wireless transceiver 840 may be configured to communicate using any number and type of cellular protocols, such as General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), Enhanced Data Rates for GSM Evolution (EDGE), Personal Communications Services (PCS), Advanced Mobile Phone System (AMPS), Code Division Multiple Access (CDMA), Wideband CDMA (W-CDMA), Time Division Synchronous CDMA (TD-SCDMA), Universal Mobile Telecommunications System (UMTS), and / or Time Division Multiple Access (TDMA). A mobile device operating with the present invention may alternatively (or additionally) include a wireless transceiver (and associated components) for communicating using any other wireless communication protocol methodology, such as ISO14443 protocol, ISO18000-6 protocol, Bluetooth® protocol, ZIGBEE® protocol, Wibree protocol, Wi-Fi® protocol, IEEE802.15 protocol, IEEE802.11 protocol, IEEE802.16 protocol, Ultra Wideband (UWB) protocol, IrDA protocol, and combinations thereof, and the communication interface 18 of the host server 860 may be configured to operate with such protocols to communicate with the mobile device 800 or any other device. The antenna 850 may be configured to transmit and receive any wireless signal having any format, and may include multiple different antennas for transmitting and receiving using different wireless protocols.

[0057] The communications module 830 may communicate with the server 860 or another device using any other type of connection, such as a wired Internet connection, a wireless Internet connection, a cellular network connection (including a data link connection), a WLAN connection, a WWAN connection, an optical connection, a FIREWIRE® connection, a THUNDERBOLT® connection, a LIGHTNING® port connection, an e-SATA connection, a USB connection, a mobile device sync port connection, a power connection, and / or a security cable. The communications module 830 may be used to communicate with one or more companion devices, and with any number of other devices that aid in tracking / locating a lost or stolen mobile device 800, and to monitor the location or status of the mobile device 800 (e.g., by monitoring whether the communications link between the mobile device and a companion device is healthy).

[0058] Mobile device 800 includes a user interface 870. User interface 870 may include any number of input devices (not shown) for receiving commands, data, and other suitable input from a user, and any number of output devices (not shown) for providing data, notifications, and other suitable information to a user from mobile device 800. Similarly, host server 860 includes a user interface 15, and may include any number of input devices (not shown) for receiving commands, data, and other suitable input from a user or third party, and any number of output devices (not shown) for providing data, notifications, and other suitable information from host server 860 to a user / third party.

[0059] Any number of input devices, such as a touch pad, touch screen, mouse / trackball / trackpad, microphone and / or alphanumeric keypad, may be provided in the user interface 870,15 to allow a user to input commands and data into the mobile device 800 and the host server 860. For purposes of this application, the term "touch screen" may include a display integrated with or in close proximity to a touch interface that can determine when a user applies a physical connection to a location in close proximity to the display. The touch screen may have sensors capable of measuring parameters from a user's interaction, such sensors may measure capacitance, resistance, pressure or differential measurements resulting from the movement of a "touch" to the screen. The user interface 870 may be configured to detect the pressure a user exerts on a key of a keypad (either virtually implemented in the display or as a physical array of key switches) and the time interval between multiple key presses to determine whether the current user is authorized to use the device. The user interface 870 may also include a microphone that allows the user to provide voice data to the mobile device 800, and may include one or more cameras that allow the mobile device to capture still or moving images. Similarly, the user interface 15 of the host server 860 may also include a microphone that allows the user to provide voice data to the host server 860, and may include one or more cameras that allow the server 860 to capture still or moving images. In one embodiment, the mobile device 800 includes a front camera 874 that faces the user during operation of the device, and a rear camera 872 on the opposite side of the mobile device. The mobile device 800 may include voice recognition software for processing voice input via the user interface 870.The user interface 870, and similarly the server user interface 15, may also include any number of suitable output devices, such as a display screen for visually displaying information (e.g., video and text) and / or a speaker for providing an audible output. The display of the mobile device may be configured to sense touch by a user by any suitable means, such as capacitive sensing, pressure sensing, gel displacement sensing, resistive sensing, or any suitable or conventional touch transmission technique utilized by those skilled in the art. The mobile device 800 may be configured to provide a word, phrase, tone, recorded music, or any other type of audible output to the user via the speaker. As previously mentioned, the user interface 870 may be activated to provide information and / or interfere with the operation of the mobile device 800 when an unauthorized user attempts to use the mobile device 800. For example, the lighting level of the display may be changed to call attention to the mobile device, and unpleasant and / or annoying sounds may be played over the speaker.

[0060] The mobile device 800 may include one or more biometric authentication devices configured to receive biometric information, such as a fingerprint scanner, an iris scanner, a retina scanner, and / or a breath analyzer. Input devices such as a microphone and a camera may be used to perform biometric analysis, such as voice analysis and facial recognition. Additionally, the mobile device may include a magnetometer (such as that used in an electronic compass) for measuring magnetic fields, a MEMS or other type of gyroscope for measuring orientation, and an accelerometer for measuring changes in the motion of the mobile device.

[0061] Information provided or received by the user interface 870,15 may be in any suitable format. For example, a user interface that conveys information to a user in an audible format may first provide the user with a data header, followed by a data value to identify the data. The user interface 870,15 may provide information in any number of desired languages, regardless of whether the information is provided audibly or visually.

[0062] The user interface 870,15 may also provide and receive information to the user in machine-readable form. In one exemplary embodiment of the present invention, for example, the user interface 870 of the mobile device 800 may send and receive messages using dual-tone multi-frequency (DTMF) tones. The mobile device 800 and the host server 860 may be configured to send, receive and process machine-readable data in any standard format (such as MS Word documents, Adobe PDF files, ASCII text files, JPEG or other standard formats) and other proprietary formats. Machine-readable data to and from the user interface 830,15 may be encrypted to protect the data from unintended recipients and / or improper use. In an alternative embodiment, the user may be required to enter a passcode to enable use of some or all of the features of the mobile device 800. Any other user interface features may be utilized to enable a human or non-human user to interact with one or more devices operating with the present invention.

[0063] Mobile device 800 may include any other suitable features, components and / or systems. For example, mobile device 800 may be configured to conserve its battery life by turning off some or all of its components, such as the camera and microphone. Components may be selectively shut down in response to a security breach event and in response to commands from an authorized user or security agency. Alternatively, mobile device 800 may be configured to overuse its components to drain the battery as quickly as possible, e.g., to limit the usefulness of mobile device 800 to unauthorized users.

[0064] The mobile device 800 may be configured to implement one or more security measures to protect data, restrict access, or provide other desired security functionality. For example, the mobile device 800 may encrypt data transmitted and / or data stored on or created by the device itself. Such security measures may be implemented using hardware, software, or a combination thereof. Any method of data encryption or data protection may be utilized with the present invention, such as public / private key encryption systems, data scrambling techniques, hardware and software firewalls, tamper-resistant or tamper-resistant memory storage devices, or any other method or technique for protecting data. Similarly, passwords, biometrics, access cards or other hardware, or any other system, device, and / or method may be used to restrict access to any device operating with the present invention.

[0065] The host server 860 communicates with the mobile device 800, authorized users, unauthorized users, security agencies (particularly insurance agencies) and other entities to monitor and protect the mobile device 800 from unauthorized use and to mitigate damages associated with security breach events or attempted fraud. The host server 860 may include any number of separate computer systems, processors and memory storage devices as well as human operators (e.g., to respond to calls from authorized users reporting lost / stolen mobile devices) and any other suitable entities. The host server 860 may include or be in communication with one or more databases 880 that store information about authorized users and the mobile device 800 to monitor and track the mobile device 800 and provide instructions to the mobile device 800 in the event of a security breach event.

[0066] For example, database 880 may store a usage profile of the mobile device such that software in host server 860 can detect whether continued usage of the mobile device deviates from the usage profile according to a predefined threshold or whether the mobile device has experienced a loss event that leads to a fault condition in mobile device 800. Host server 860 may receive, process, and store (e.g., in database 880) information from mobile device 800. Host server 860 may handle any type of data in any format to accomplish a purpose, such as receiving and processing environmental parameters captured by the mobile device to track the location and position of mobile device 800, as described above. Database 880 may also store location information (e.g., the "whitelist" and "blacklist" described above) that can be used to determine whether mobile device 800 is operating in a valid location. Database 880 may also store data for training a neural network, where example images of defects in the mobile device are associated with identified defect classes, outputs, or conditions accordingly. The neural network training data may be expanded or otherwise modified to improve accuracy of device defect recognition. Additionally, database 880 may store the results of the health and operation analysis of mobile device 800 along with identifying information and historical operational information associated with mobile device 800. In this manner, mobile device operation analysis may be tracked and analyzed over time by server 860 to identify trends that may be indicative of an ongoing failure of at least one component of mobile device 800.

[0067] A database 880 in communication with the host server 860 may also store archived data from the mobile device 800 for recovery if the mobile device 800 is lost or stolen, or if the data on the mobile device 800 is corrupted (e.g., by a virus or other malicious program). The functions of the host server 860 may be performed automatically or semi-automatically, such as via software / hardware operating on one or more computer systems and / or by one or more human operators.

[0068] The host server 860 may include one or more system processors 14 that retrieve and execute computer readable instructions stored in memory 16 to control (at least in part) the operation of the host server 860. Any number and type of conventional computers, computer systems, computer networks, computer workstations, minicomputers, mainframe computers, or computer processors, such as integrated circuit microprocessors or microcontrollers, may be used with the present invention. Computer systems used in accordance with aspects of the present invention may include an operating system 43 (e.g., WINDOWS NT / 95 / 98 / 2000 / XP / Vista / 7 / 8 / 10, OS2, UNIX, Linux, SOLARIS, MacOS, etc.) as well as various conventional support software and drivers typically associated with computers. In some embodiments, specialized applications may be provided or executed, in whole or in part, by the system processor to perform the methods of the present invention.

[0069] The neural network 47 may include a convolutional neural network (CNN) architecture and may be implemented by software stored in the memory device 16, via external hardware connected to the server processor 14 (such as a hardware accelerator or a graphics processing unit (GPU)), or by a combination of stored software and external hardware.

[0070] The host server 860 may be accessed in any desired manner, such as via a website on the Internet and / or via a telephone network. The host server 860 may include any number of human operators, computer systems, mobile phones, mobile computing devices, interactive voice response (IVR) systems, and any other suitable systems and devices for communicating with users, security agencies, computing devices, or other entities. The host server 860 may communicate with unauthorized users of lost or stolen mobile devices, either through the mobile devices or through other communication methods, including directly with a fault condition testing application (or "FSTA" as used herein) installed on the mobile devices. The host server 860 may notify the unauthorized users that the mobile device has been lost or stolen, provide recovery information (such as a shipping address) to the unauthorized users, and forward test results and claim information to third party insurance companies to facilitate the initiation of insurance claims. The host server 860 also communicates with the mobile device 800 to provide software updates, receive data for archiving, identify files and other data to be protected, and perform any other aspects of the present invention.

[0071] The host server 860 may be controlled (locally or remotely) by or operate in conjunction with any third party, such as an authorized user, a telecommunications service provider, a mobile device monitoring / tracking service provider, a security agency, an insurance agency, a mobile device repair agency, a mobile device distributor, an advertising agency, a mobile device warranty reseller, an authorized user of a mobile device, and / or any other desired entity. For example, the authorized user and the security agency may communicate with or through the host server 860 and interact with a fault condition test application (FSTA) installed on the mobile device 800 to verify that the device has developed a fault that may be eligible for an insurance claim. The host server 860 may be configured to provide notifications on how to return a lost / stolen mobile device 800, detect security breach events, detect that a fault condition has occurred in the mobile device, and determine whether the functionality of the mobile device should be altered and, if so, how to alter the functionality of the mobile device 800. The host server 860 may operate in conjunction with any other desired systems, devices, human operators or other entities.

[0072] The FSTA may collect information from the tested mobile device and relay it to a third party or store it in the memory 16 of the server 860 along with the results of the device test (45). The device data 45 may be uniquely associated with one or more images of the mobile device 46 and analyzed by the server processor 14. The device image data may be stored in any format, including but not limited to jpeg, tiff, gif, eps, png, pdf, scalable vector graphics, bitmap or any other conventional or non-conventional image data format. The device data 45 and corresponding device images 46 may also be stored in the database 880. The information relayed to the third party may include any suitable data, such as device type, manufacturer, model number, serial number, date of manufacture, hardware configuration list, memory capacity, software manifest, list of operational features, list of inoperable features, electronic serial number, ESN, IMEI number, International Mobile Equipment Identifier number, IMSI number, International Mobile Subscriber Identity number, UIMID number, and User Identity Module Identifier, test results, fault status, etc. Mobile devices that utilize a SIM card (such as a replaceable subscriber identity module commonly used in mobile phones) may further be used in conjunction with embodiments of the present invention to determine that at least one of the stored device configuration parameters includes an IMSI number within the SIM of the device.

[0073] In an additional embodiment of the present invention, a user of a mobile device suspected of a fault condition (e.g., a cracked screen / display) may be required to install an application (e.g., the fault condition testing application mentioned above) on the suspected faulty mobile device. The FSTA provides a reliable method for assessing the authenticity of the alleged fault condition (e.g., a cracked or broken screen / display) on the mobile device. The FSTA interacts with the mobile device user to obtain sensor measurements and user inputs that are used to assess whether the mobile device is truly faulty or not. The fault / non-fault determination may then be displayed and / or transmitted in a secure manner (such as to an insurance company seeking insurance coverage for the mobile device) to a third party (such as an insurance company seeking insurance coverage for the mobile device) so that tests performed on the mobile device can provide reliable results). In one embodiment, data transmitted by the FSTA in the mobile device is encrypted to prevent tampering or spoofing by the mobile device user and decrypted accordingly by the recipient or software running in the server 860.

[0074] The embodiments of the present invention for testing mobile devices also have broader objectives than testing mobile devices for broken glass. For example, the entire digitizer could be tested when the screen is completely filled (more fully described below). Additionally, a darkened display / screen could be tested by point testing (e.g., the sensor may be working but the LCD may be broken). Similarly, to verify that input from the user is accurate, in one implementation, a normal fill mode could prevent the user from seeing the point at which the test is completed, although the user may attempt to fill the entire screen to cheat the test. Colors could also be tested. For example, the user could be instructed to press two displayed green colors, then two red colors, and finally two blue colors (as an example). Additionally, analysis of images of the mobile device taken, either at the mirror or by a second device, may provide captured image data that can be analyzed to visually determine the grade of the failure, such as by a neural network, such as a convolutional neural network.In this manner, a variety of defects may be detected, including, for example, cracks, scratches on a mobile device display; patterns, cracks, chips, scratches, or bends on a mobile device housing; cracked bezels; worn areas on a mobile device housing; stuck or dead pixels on a mobile device display; LCD bleed on a mobile device display; pink defects on a mobile device display; objects (such as a user's body part, a sticker, or other object) obscuring the mobile device display; a detected aspect ratio of a mobile device screen that does not correspond to an expected aspect ratio for the corresponding mobile device type; wear patterns on a mobile device that do not correspond to an expected usage profile of the mobile device; reflections on a mobile device display; stains on a mobile device display; a logo or other identifier found in an image of a mobile device that does not correspond to an expected manufacturer for the mobile device; separation of seams on a mobile device; missing components on a mobile device; and any other type of defect that may be determined from a captured image of a mobile device.

[0075] Additionally, in one embodiment, a company that sells mobile devices may want to know immediately if an active device has developed a fault condition (such as a cracked display) so that they can offer a new model to the device owner. This may be particularly relevant in situations where a Mobile Network Operator (MNO) offers devices for sale that are discounted by the payment of a monthly service plan fee for a contractually mandated period of time, since the MNO benefits from being able to renew and extend the service agreement, while the device manufacturer also benefits from being able to sell new devices to replace the broken ones. Additional or alternative system implementation forms of embodiments of the present invention include the following. - Mobile Device Insurance: Embodiments of the present invention provide verification of phone condition in the used mobile device insurance market, as well as the ability to support selective or variable insurance clauses related to the provable condition / functionality of the mobile device hardware (e.g., a mobile device with an uncracked screen may be fully insured, whereas a device with a cracked screen but otherwise functional may insure the device except for glass breakage). Similarly, embodiments of the present invention provide verification that the phone being inspected is an insured phone (e.g., by transmitting mobile device identification information such as IMEI information or other data as described herein as part of the test). Additionally, embodiments of the present invention may be used to identify that a mobile device is the correct device to be covered before it is returned for screen repair, in order to reduce false claims. Such a situation may arise, for example, for insured devices when a repair company does not actually repair a broken device, but instead fraudulently splits the insurance money paid with the person who insured the phone. - Transfer of Mobile Devices: Embodiments of the present invention may be utilized to verify that the tested phone is actually handed over in a transfer of possession / ownership. - Lease Return Condition Validation: Aspects of the present invention provide for determining the conditions of the lease return (e.g., if the leased mobile device must be returned at the end of the lease or if the condition of the mobile device falls below a pre-defined agreed-upon condition (i.e., the display must not be cracked upon lease return), then the customer must pay an additional fee for excess wear / breakage). - Purchase / shipment condition verification: FSTA may be used for verification by customers who purchase devices online (e.g., on eBay) when the seller is not fully trusted. FSTA may also be used to determine the current level of damage, i.e., how cracked is the display, if any (in various embodiments, it is expected that the customer or company will pay an additional fee if the screen is not so badly damaged that the known screen condition is not acceptable). Additionally, embodiments of the present invention provide attestation by the warehouse that the mobile device was in acceptable condition when it was shipped. When FSTA is used as a verifier in a situation where a person purchases a used mobile device from another source, the buyer may require the seller to install a trusted screen verifier app, enter the buyer's email address or SMS number into FSTA to verify, and FSTA will run the tests and send the results to the buyer, thereby verifying that the used device does not have a known defect (such as a cracked screen). Additionally, this may be desired by a seller who wants to verify that a device is working as expected before shipping so that the seller cannot be accused of sending a faulty device to the recipient. - Determining Legal Status: Aspects of the present invention may support collaboration with third parties (or police databases) in verifying that a mobile device has not been lost / stolen / subject to outstanding fees.

[0076] Some embodiments provided herein include FSTA performing pressure sensor based tests, magnetometer based tests, or pixel based tests, with some system based usage scenarios presented, to perform fault condition assessment. While each type of FSTA test is designed to be able to determine cracks individually, various types of crack detection techniques may be combined as appropriate for any purpose, such as improving accuracy. In a preferred embodiment, FSTA is installed on a mobile device and runs resident within the mobile device, but one skilled in the art will appreciate that FSTA may be run in other manners, such as deployed via an over-the-air (OTA) application, a website and side-loaded onto a connected mobile device, loaded via in-store hardware (e.g., Raspberry Pi®) or dedicated in-store device, or loaded and run from a vending machine at a mall or store. Additionally, all or part of the FSTA and system implementation embodiments may be included in a library of apps utilized by entities that provide online quotes for insurance, device trade-ins, or mobile device diagnostics.

[0077] 9-13 show an embodiment of the present invention where the mobile device 800 is externally illuminated by a light source such as an LED light strip, and in one embodiment, the color of the light may be selectively changed to illuminate the mobile device and improve crack detection and analysis. In various embodiments, the light needs to be in a position that will cause scattering from the crack if one is present in the glass. In various embodiments, an LED strip with RGB lighting may be utilized so that the color and intensity of the light can be changed as needed when multiple lights are needed depending on the number of cameras being used.

[0078] Experiments that a curved background removes reflections from glass when the camera is aligned at an angle led to the previously described embodiment setup in FIG. 9. The camera is aligned (at about 60°) so that it does not see itself into the glass. In various embodiments, if distortion is an issue, the camera can be calibrated using software such as OpenCV to bring out the inherent value of the lens and not use it to distort the image. The embodiments also use a calibration sheet that is placed in front of each camera to ensure the best image processing results are obtained. In various embodiments, if the cameras are identical, they can be calibrated in-house and the calibration data shipped with the software so that identical settings do not require calibration.

[0079] Figures 9-14 show images taken using various light combinations to confirm the effect of different lighting and the absence of lighting, and that the lighting settings are effective in removing the natural reflections cast from the work area. From the photographs and embossed images, the cracks are visible in all lighting conditions, confirming that lighting is necessary to remove the normal shadows and light reflections on the screen. By inspecting the embossed version of the image, the neural network of the present invention should detect cracks in all illuminated images, regardless of whether a human can easily physically see those cracks or cannot visually see them.

[0080] Further, with regard to the image processing steps on the server 860, certain embodiments include locating the region of interest in the image, performing color masking and / or edge detection to locate the image in the region containing the mobile device, cropping out all parts of the image that are not parts that do not contain the mobile device, and transforming the image to address distortion or skew to obtain a representation of the proper angle. In one optional step, embossing or channel equalization of the image may be performed. After these pre-processing steps are performed, the processed image is sent to a neural model (described in detail below) for further analysis. In summary, the neural model uses a convolutional neural network to detect features in the image by comparison to pre-trained classes, which can be used to analyze LCDs, housings, glass surfaces, and other elements, and returns a score on how well the feature matches each class. For example, in one embodiment, two classes are defined as cracked or non-cracked mobile devices. The image is classified accordingly using a convolutional neural network, as described below. After the image is classified, if it is damaged, class activation mapping is performed and the data is returned through the neural network model. However, instead of the final score layer, layers of the model where the image is output into smaller sub-images are looked at, which gives the parts of the image where cracks or other defects have a higher score and where the model has detected cracks in the image. As a result, an array of scores is obtained, and from this array and from the areas of the image, a heat map is generated, such as a bounding box shown in Figure 7 where damage may be present or a color superimposed on the image, which is displayed on top of the original captured image, and then the server returns the image, and Boolean Damaged==TRUE OR FALSE, if it is TRUE, the image is returned for display as described above.

[0081] In one embodiment, the test results are now stored in database 880 and are made available via a website. A user may log into this website and view the most recent test or search the device and review the original images and any images in between. A customer may then use those images to see if there was any fraud present during a previous diagnostic test, such as someone trying to hide a crack on the screen or somehow concealing a crack.

[0082] During the diagnostic and image capture process, embodiments may display a colored screen instead of displaying a white screen and take multiple images to look for stuck or dead pixels. While white is effective at showing a discolored screen, a white background with stuck pixels may not be very good at identifying individual colored pixels that cannot be erased. For this reason, it may be helpful to consider red, green, and blue images. Additionally, a black background on a mobile device display may make stuck pixels or any type of damage easier to determine. Additionally, there are multiple types of LCD bleed. Type 1 is liquid bleeding and type 2 is LCD light bleeding through the edges, and a black screen may make at least type 2 easier to detect. Thus, in various embodiments, displaying different background colors on the mobile device 800 and capturing multiple images may increase the ability to detect additional defects.

[0083] In additional embodiments, additional elements such as light sources, lighting strips, microphones, speakers, vibrators, and other physical interfaces may be incorporated within the test fixture. For example, vibration may be tested by listening to the mobile device 800's attempts to generate vibrations, and if no vibration sounds are heard, the test may indicate a failure. Further embodiments may assist the mobile device in testing whether it is operating properly or not by detecting using a microphone incorporated into the test fixture or using an accelerometer incorporated into the test fixture / phone. Additionally, a microphone on the mobile device may be tasked with listening to the phone's vibrations. Additionally, an external vibrator may be used to vibrate the mounting platform 205 surface itself and see if the accelerometer and gyro of the mobile device 800 operate and sense what they are intended to. Additionally, if the test fixture 200 is configured to affix and move the mobile device, the gyro / sensor can be detected for a 90 degree flip, which can be analyzed as well.

[0084] Another sensor of the mobile device (ambient light sensor) may be tested. The test fixture may turn on an external light source and check whether the ambient sensor in the mobile device can see the light. Additionally, the proximity sensor of the mobile device may be activated and determine whether its measurement is consistent with the height of a reflective surface above the mobile device 800. Additionally, an NFC field generator and reader may be incorporated into the test fixture, thus testing the NFC interface of the mobile device. A wireless charger of the mobile device may be incorporated into the test fixture, both to power the mobile device during testing and to test the proper operation of the wireless charging circuitry of the mobile device.

[0085] A Wi-Fi® or Bluetooth® connection to the device may be tested, and the test system may broadcast a Wi-Fi® signal and determine whether the mobile device can acquire the SSID of the broadcasted signal. Additionally, the Bluetooth® signal transmitted from the mobile device may be used to control or actuate any function of the test instrument, including actuating or rotating a reinflation surface or an external light source, camera, or other instrument.

[0086] Further, additional features utilize point or spot exposures based on the phone model and its shape. The location of the center of the screen is known and an image of the center of the screen is taken, then the entire image completes that stage. To find the center of the screen, a new mapper is needed and known fiducial marks come into play. Knowing the center of the screen is important because the center of the screen may not be the center of the photo. The phone knows itself as well as its dimensions and color. If the size of the bounding box is known, look up that size on the mobile device for comparison.

[0087] With regard to analyzing the image, the predominant green or blue colors are easiest to remove from the image, and thus analyzing the colors helps determine the appearance and characteristics of the housing or display that may indicate defects. For example, when analyzing the back of a phone, there may be multiple approaches including using a rear-facing phone to capture an image of the mobile device in a reflective surface, and then analyzing the back of the phone from the captured image for broken glass, metal scratches, or other defects.

[0088] Additionally, if the mobile device under test is tethered, it can be determined whether the mobile device is engraved and whether the mobile device is a particular color. Certain smartphone brands may sell engraved phones, and therefore phones sold by manufacturers often have a firmware flag that includes a "yes / no" that the phone is engraved (but does not indicate what is engraved). Through a tethered connection, it can be determined whether the mobile device is engraved and therefore the engraving can be removed from the defect analysis as a suspected crack. Additionally, if the captured image contains engraving or other text, that text can be passed to an OCR analysis tool to determine the actual text on the mobile device (this can be important since personalization can have a significant impact on the resale value of used mobile devices). Additionally, the use of OCR can help determine whether there are images that cannot be detected by the text and therefore need to be ignored. Through this technique, text, serial numbers, etc. may be removed from the crack detection.

[0089] Gravity assistance may also be used to move the mobile device under test into and / or through the test fixture, and a standard conveyor belt may use gravity to flip the device over and capture images of the back of the mobile device. In various embodiments, the device under test may be in motion and moved downstream of the conveyor belt, taking images of itself as it moves along the belt. Continuous or continuous movement may lead to significant gains in processing time within a warehouse / recycler environment. In one embodiment, the conveyor may be paused long enough to take a photo, and in an alternative embodiment, a lever may be thrown into the conveyor belt to flip the mobile device. Additionally, multiple camera setups such as those in FIGS. 14, 15, and 15A may be used to capture multiple images and angles of the mobile device (and in some cases, build a 3D model of the mobile device's appearance). The mobile device may proceed downstream on the conveyor and enter the box shown in Figures 15 and 15A, the mobile device passes through a partition such as a black curtain, the curtain lowers, the light source turns on, the camera takes a picture, then the captured image is uploaded, and all the tests are performed. Those skilled in the art will appreciate that it may be more than one device to accomplish all the tests. As previously mentioned, a conveyor may be used to capture a front facing photo in a reflective surface, the conveyor may drop / flip the mobile device, then a rear facing photo may be taken by the rear facing camera, and the next device may be used. A unique fiducial mark displayed on the mobile device may allow the mobile device to be uniquely tracked within a multi-phone environment. Additionally, a QR code or marker on the screen may be displayed with a value from 1 to 255, giving a larger combination of numbers that can be embedded in the image of the screen to identify which mobile device was proceeding down the conveyor line. An image may then be taken.Additionally, in the conveyor environment (or in the 3D photo box of FIGS. 15-15A), an external camera may capture a side view of the mobile device. External lighting effects may be applied, similar to those described above and shown in the lighting examples of FIGS. 9-13.

[0090] Detecting Device Defects Using Neural Networks Embodiments of the present application utilize a Convolutional Neural Network (CNN) architecture to analyze captured images, video and / or audio files from a mobile device to identify defects and to provide analytical information, such as heat maps to aid in further analysis of defect types. Those skilled in the relevant art will appreciate that many different CNN architectures may be utilized to implement aspects of the present invention. One exemplary CNN architecture may be realized by the Inception model implemented in Tensorflow®, an illustrative example of which is disclosed, for example, at https: / / www.tensorflow.org / tutorials / image_recognition, the disclosure of which is incorporated herein by reference for all purposes. Alternative implementation forms of the present invention may be implemented with Deep Residual Networks (described at https: / / blog.waya.ai / deep-residual-learning-9610bb62c355, which is incorporated herein by reference for all purposes) and / or with convolutional network architectures implemented in environments such as Torch (examples and explanatory text are published at http: / / torch.ch / docs / tutorials.html, the disclosure of which is incorporated herein by reference in its entirety for all purposes). Also, an introductory background on neural network architectures in general, and Convolutional Neural Networks (CNNs) in particular, is provided in Neural Networks and Deep Learning by Michael A. Nielsen (Determination Press, 2015; available at http: / / neuralnetworksanddeeplearning.com, the disclosure of which is incorporated herein by reference in its entirety for all purposes).

[0091] In general, CNN is a type of model architecture that has been successfully used for image classification tasks. CNN applies a series of filters to the raw pixel data of an image to extract and learn higher-level features, which can then be used for classification. CNNs typically contain three components: (1) convolutional layers, (2) pooling layers, and (3) densely / fully connected layers.

[0092] A convolutional layer applies a specified number of convolutional filters to an image. It performs a series of mathematical operations on each identified subregion in the image to generate a single value in the output feature map. The convolutional layer then typically applies a ReLU (rectified linear unit) activation function to the output to introduce nonlinearity into the model, although in various embodiments a logistic sigmoid or hyperbolic tangent activation function may also be used.

[0093] Pooling layers downsample the image data extracted by the convolutional layers to reduce the dimensions of the feature maps in order to reduce processing time. In effect, the pooling function replaces the output of the net at a location by a summary statistic of the neighboring outputs. A commonly used pooling algorithm is max pooling, which extracts subregions of the feature map and keeps their maximum value while discarding all other values, thus reporting the maximum output within a rectangular neighborhood. Other possible pooling functions include the average of the rectangular neighborhood, the L2 norm of the rectangular neighborhood, or a weighted average based on the distance from the central pixel.

[0094] Densely connected (fully connected) layers perform classification on the features extracted by the convolutional layers and the downsampled features by the pooling layers. In a densely connected layer, each node in the layer is connected to each node in the previous layer.

[0095] As introduced above, CNNs are constructed by stacking multiple convolutional modules that perform feature extraction, each consisting of a convolutional layer followed by a pooling layer, and the final convolutional module followed by one or more densely connected layers that perform classification. The final densely connected layer in a CNN contains a single node for each target class in the model (all possible classes that the model can foresee), with an activation function for each node. CNNs allow the interpretation of numerical values ​​for a given image as a relative measurement of how likely the image is to belong to each target class. In various embodiments of the present invention, the target classes may include mobile device defects such as cracks on the display, scratches or cracks on the housing, LCD bleed on the display, pink color appearing on the display, stuck pixels on the display, and various combinations of these.

[0096] In one embodiment using neural network analysis techniques, a system and method is provided for identifying cracks or other types of defects associated with a mobile device through image analysis. Referring to process 1600 depicted in FIG. 16, an FSTA is installed 1610 on a mobile device 800 desired to be tested (FIGS. 14 and 15 also show user prompts and guidance for capturing an image of the mobile device reflected on a reflective surface such as a mirror). Once ready, a user of the mobile device 800 initiates device defect testing 1615. An image of the display of the mobile device 800 is captured 1620, per the process described with respect to FIG. 17, and the captured image is optionally pre-processed 1625 by rotating the image into portrait mode and / or by micro-rotating to reduce skew and / or by cropping the image to a desired aspect ratio. The captured (and optionally pre-processed) image is then uploaded (1630) to a server (such as server 860), which processes (1635) the uploaded image to determine whether any cracks or other types of defects are present, as further discussed with respect to FIG. 18. Alternatively or in combination, the extent and characteristics of the screen cracks or defects are characterized by a processing step 1635 for further analysis and reporting. Such analysis of the captured image (and possibly the processed image) from the mobile may include determining whether the captured image contains one or more defect indicators. Such defect indicators may be predefined within a pre-trained neural network into defect classes such as cracks, chips, tears, blemishes, wear, scratches, abrasions, gouges, stuck pixels, dead pixels, color errors, color bleeding, deviant marks, etc. Based on this analysis, a cosmetic defect grade for the mobile device may be generated, which may indicate the relative visual condition of the mobile device as a whole.Such a rating may be useful in calculating the resale value of a mobile device and / or in determining insurance pricing or insurability standards relating to the mobile device.

[0097] Once any potential defect analysis (1635) is complete, the resulting information may be stored, presented, or displayed to any user for any necessary purposes, such as evaluating whether the mobile device 800 is in proper operating condition. array The process then ends (1645).

[0098] FIG. 17 illustrates a process 1700 of the present invention for capturing an image of a mobile device, as introduced in step 1620 in FIG. 16. The image capture process begins at 1705, where the FSTA analyzes 1707 the ambient lighting from a lighting sensor in the mobile device 800. Then, at 1709, the screen brightness and camera sensitivity of the mobile device 800 are adjusted so that the mobile device's front camera (e.g., FIG. 14, 874) can optimally follow the tracked image displayed on the mobile device's display (e.g., FIG. 14, 1400), and the user is instructed to move the device 800 to a position 1715 with the mobile device's display facing a reflective device such as a mirror. The FSTA's tracking algorithm identifies the mobile device and provides guidance to the user to move / rotate the mobile device 1720 until the mobile device's 800 display determines that the mobile device is in an optimal position to capture an image. As part of this process, the user may be instructed to rotate, tilt, move, or align the mobile device with respect to the surface of the reflector. The FSTA uses data obtained from the device's front-facing camera (e.g., FIG. 14, 874) to determine 1725 when the mobile device is acceptably aligned for image capture, and the FSTA then modifies the front-facing camera sensitivity 1730 for optimal image capture. In one embodiment, the image capture sensitivity may differ from the initial sensitivity used during tracking of the mobile device's placement in front of the reflective surface. Depending on the operating system of the mobile device 800, different parameters may be modified within the mobile device to adjust the camera sensitivity. In one embodiment, the sensitivity is adjusted by changing an exposure compensation value associated with the camera in the mobile device 800. In another embodiment, the shutter speed and ISO value associated with the camera in the mobile device 800 may be adjusted to achieve the desired sensitivity.A captured image, such as a white screen or a screen with predefined fiducial marks, may then be output (1735) to the display of the mobile device 800. An image is then captured by the front camera of the mobile device 800 (e.g., FIG. 14, 874).

[0099] Those skilled in the relevant art will also appreciate that the rear camera 872 of the mobile device 800 of the present invention may be used when facing a reflective surface to record images of other sides of the mobile device 800, including, for example, the backside of the mobile device housing. In this manner, defects in the housing may be analyzed by the optical neural network analysis of the present invention to detect housing defects, wear patterns, chips, cracks, etc.

[0100] Returning to FIG. 16, after the captured image has been pre-processed (1625) and uploaded (1630) to the host server, image analysis may begin (1635) at the server 860. FIG. 18 illustrates an embodiment of image analysis according to the present invention, which may be performed by the server 860. Once the captured image is received (1810) by the server 860, the captured image may be optionally downsized or resampled to reduce the image file size and improve processing efficiency. An exemplary captured image can be seen in FIG. 19, where a mobile device 800 is shown with its display 1400 being manipulated by a user. Edges of the display 1400 of the mobile device 800 are located in the captured image 1815 by an alternative method.

[0101] In a first method for finding edges corresponding to the display 1400, an image thresholding technique is applied to the captured image (an example of which is shown in FIG. 19) to generate a thresholded image (an example of the image of FIG. 19 after the thresholding technique has been applied can be seen in FIG. 20). Optional thresholding techniques are well understood by those skilled in the relevant art, and an exemplary tutorial is disclosed at https: / / docs.opencv.org / 3.3.1 / d7 / d4d / tutorial_py_thresholding.html, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Next, contours are found in the thresholded image, where a contour comprises a defined curve that connects (along a boundary) all consecutive points that have the same color or intensity in the image. Because contours are a useful tool in shape analysis and object detection recognition, contours may be used to aid in identifying areas of the captured image that correspond to the mobile device's display 1400. Contour analysis techniques are well understood by those of skill in the relevant art, and an exemplary tutorial is disclosed at https: / / docs.opencv.org / 3.3.1 / d4 / d73 / tutorial_py_contours_begin.html, the disclosure of which is incorporated herein by reference in its entirety for all purposes. Once contours have been determined in the thresholded image, the four-point contours are analyzed and compared to the known aspect ratio of the mobile device being analyzed. Since device data 45 identifies the type of device, its display aspect ratio may be determined from existing lookup data (or externally from the device manufacturer) and used in comparison to the aspect ratio of the four-point contours found in the thresholded captured image.Once a contour is identified in the thresholded image that is sufficiently similar in aspect ratio to the display aspect ratio defined for the particular mobile device under analysis (e.g., the aspect ratio is within a predefined tolerance of the manufacturer's aspect ratio data), the locations of the vertices 2000 of the matching contour in the thresholded image are obtained, and the edge of the display is thereby identified in the captured image as the area that is within the four vertices that correspond to the corners of the display 1400. Depending on the particular geometric features of the display associated with a particular mobile device manufacturer (e.g., rounded corners, cutouts into the display, etc.), the edge boundary image may be further refined using known geometric tolerances of the manufacturer's device specifications.

[0102] In a second technique for locating edges corresponding to the display 1400, fiducial marks having a predetermined color (as shown by reference numbers 2101-2104 in FIG. 21) are output to the display 1400 of the mobile device 800 immediately prior to image capture (FIG. 17, 1735). An arrangement of fiducial marks 2101-2104 may be displayed adjacent to each corner vertex of the display 1400, and image analysis techniques may be applied to determine the edges of the mobile device's display in the captured image. Portions of the image are extracted using a color range, and since the color of the fiducial marks is known, elements of the image that appear within a predefined tolerance of the color of the fiducial marks will appear in the extracted image. Polygons matching the shape of each of the fiducial marks 2101-2104 can then be found in the image by template matching techniques, for example by locating shapes 2301-2304 in FIG. 24, and the locations of the corresponding corner vertices in the captured image can be determined by the locations of the corner vertices in the captured image (see FIG. 22). Similar to the first edge detection technique just described, the edges of the display in the captured image are thus identified as areas within the four vertices that correspond to the corners of the display 1400. In alternative embodiments, any type of fiducial mark may be used to assist in identifying components of the mobile device, such as displaying a fiducial mark centered on the mobile device display and using the aspect ratio of the display geometry associated with the device (e.g., retrieved from a database of known mobile device manufacturers with associated specifications), and the display edge may then be refined by identifying a search boundary centered on the fiducial mark and extending it to fit the outer boundary using the associated mobile device geometry (and aspect ratio).This method may be further refined and if the fiducial marks printed on the mobile device display are scaled to known dimensions (such as a cross with known dimensions such as 1 centimeter on each side), the edges may be deduced by taking a scale factor with the depicted size of the fiducial marks scale in the image and then applying the scale factor to the manufacturer's specified dimensions for the mobile device's display edges to identify a tolerance close to the outer boundary of the display. Furthermore, the shape of the fiducial marks (such as mutually perpendicular lines) may be used to deskew the image so that edges with orthogonal geometries (such as sides of a rectangle) can be better found.

[0103] Those skilled in the art will further appreciate that while the term "display" or "screen" as used herein generally encompasses the area where a mobile device may output visual images, in other implementations of the invention, the "display" or "screen" may include additional desired sections of the mobile device where visually detectable defects may be found. More specifically, in some mobile phone models, a first area may be adjacent to and above the active display area, and a second area may be adjacent to and below the active display area of ​​the mobile device that is covered by a material that is susceptible to breakage (such as hardened glass). While these adjacent first and second areas may not produce a visual output (e.g., in older iPhone® devices with speaker / camera / sensor orifices in the first area and home button or microphone orifices in the second area), portions of these areas may still be damaged, and it is a feature of the present invention to extend the image analysis of the mobile device's "display" or "screen" to these adjacent first and second areas to detect damage in these areas as well. In other embodiments, a "cut-out area" removed from the active portion of the display may be included in the display analysis, and for that matter, any desired area of ​​the surface of the mobile device where visual defect detection is desired. The additional test area of ​​the mobile device may thus belong to the back of the mobile device, the logo area of ​​the mobile device, the side area of ​​the mobile device, the bezel area of ​​the mobile device, or any other area where visual defect detection is desired.In one embodiment, a number of defects, such as scratches or cracks, found in some proximity to or otherwise covering or abutting a logo area of ​​a mobile device (e.g., the "Apple" logo on an iPhone® model) may be determined and output upon defect analysis. Additionally, in various embodiments of the present invention, certain defects in a bezel of a mobile device, such as cracks, dents, or pits, may be detected by visual analysis of an image of the mobile device. Images of the mobile device may be acquired in any desired manner, such as by holding any surface of the mobile device in front of a reflector and capturing the image with a buddy or companion device having a camera, or by placing / rotating the mobile device in front of a mounted camera or other device that captures an image (and / or video representation) of each surface of the mobile device, and optionally creating a 3D image representation of the mobile device. Further, defects may include any type of visually detectable issue, and in addition to cracks and scratches, display brightness variations, color variations (such as an undesirable pink color), LCD bleed, color defects, and any other desired type of visual defects may also be detected and identified. Additionally, crack / scratch depth, width, and / or other visual appearance criteria may be associated with the identified crack / scratch and reported with an intensity level to assist in classifying the impact of the identified crack / scratch on the value of the mobile device.

[0104] Now that the edges of the display 1400 can be deduced from the locations of the corner vertices in the captured image, the area of ​​the captured image corresponding to the display area can be extracted as a sub-image and transformed into a rectangular shape in a single step. This is accomplished in one embodiment by calculating the moments of the display shape in the image and then perspective transforming the display portion into a rectangular image using techniques known to those skilled in the art (an example of a perspective transformation implementation can be found at https: / / www.pyimagesearch.com / 2014 / 08 / 25 / 4-point-opencv-getperspective-transform-example / , the disclosure of which is incorporated herein by reference in its entirety for all purposes).

[0105] Once a sub-image corresponding to the display 1400 from the captured image has been obtained, additional resizing / rescaling / resampling is performed so that the pixel dimensions of the sub-image are acceptable as input to the neural network 47 of the present invention. The sub-image can be resampled to any desired pixel configuration, and in one embodiment the image is resampled to a 320x544 pixel configuration (width x height in this example, corresponding to the portrait orientation of the sub-image). An example of a perspective transformed sub-image is shown at reference number 1400A in FIG. 20A, which corresponds to the display portion of the captured image 1400 shown in FIG. 19. A defect 1900 appears in both image 1400 and image 1400A.

[0106] Step 1830 illustrates analysis of the rescaled sub-image by a neural network to identify defects in the mobile device 800 and, in particular, defects in the display 1400 of the mobile device. The rescaled and perspective transformed sub-image (e.g., 1400A in FIG. 20A) is applied to the input of a convolutional neural network that has been pre-trained to detect defects using training data that includes the display and associated defect classes. As the network analyzes the image, an output is generated that identifies the defects, as shown by the colored polygon 1900A in FIG. 20B. The colored polygon calls attention to the defects found by the neural network, and, as previously discussed, additional output and analysis, such as heat maps, may be provided based on the particular network architecture. An even more illustrative example is according to Figures 20C to 20E, where an input sub-image (Figure 20C, 1400A) having a defect (Figure 20C, 1900) is input to the neural network 47 of the present invention to generate an output (Figure 20E, 1400B) having a polygonal surrounded defect (Figure 20E, 1900A) and a heat map output version (Figure 20D, 1400D) describing a heat map 1900D corresponding to the activation class for the defect (Figure 20C, 1900) detected in the display sub-image (Figure 20C, 1400A). Another example is provided with a second input sub-image (FIG. 20F, 1400F) that generates an output (FIG. 20G, 1400G), a heat map showing the location of the detected defect class (FIG. 20G, 1900G), and an output (FIG. 20H, 1400H) in which the defects are surrounded by a polygon (FIG. 20H, 1900H). Those skilled in the relevant art will appreciate that many other types of output are possible and that different defect types such as those mentioned above may be detected by an appropriately trained CNN architecture.

[0107] FIG. 16A illustrates another embodiment of the present invention, in which certain analysis steps described with respect to FIG. 16 may be similarly implemented, but the embodiment of the present invention with respect to FIG. 16A utilizes an alternative approach. More specifically, in one embodiment, the present invention allows defects associated with a mobile device to be detected without the need to first install a Fault Condition Test Application (FSTA). More specifically, in step 1610A, contact is initiated between a mobile device under test and a remote diagnostic service (RDS), such as a web server, that includes software and necessary communication circuitry for connecting to the mobile device and is specifically configured to remotely perform tests to determine whether defects can be found within the mobile device. Such contact may be initiated in any desired manner, including by a database connection from the mobile device to a web address, i.e., a web address corresponding to a device diagnostic server, where the mobile device user is prompted to enter identifying information, and a user test profile for the particular mobile device may be created and stored, along with diagnostic results, in a database associated with the web server. The device. Once the connection has been initiated and the requested information has been entered by the mobile device user, the user of the mobile device 800 indicates to the remote diagnostics service that a defect test is to be initiated. In one embodiment, the remote diagnostics service initiates an image capture process 1620A in a manner similar to the steps described with respect to FIG. 16. However, alternative embodiments include activating a webcam on the mobile device or companion device to capture an image of the mobile device under test directly or by the companion device, as well as allowing a browser operating from the RDS to open the rear camera of the device under test to capture images from either the front or rear camera to account for reflective surfaces.If a companion device is used with the mobile device under test, a web socket connection may be initiated between the two devices to allow inter-device communication to command the devices to display and capture images. In this manner, the mobile device under test may be remotely instructed by the RDS via a browser on the mobile device to capture a picture of itself in a reflector. Thus, the RDS may facilitate image capture from a webcam, from either the mobile device's own front and rear cameras, from a companion device in communication with the RDS, and by other methods where the RDS may initiate image capture. The captured image is communicated (1630A) over a communication link between the RDS and the mobile device (such as a pre-established connection to a web server), where the RDS initiates analysis in a manner similar to steps 1635 and 1640 of FIG. 16. Those skilled in the relevant art will appreciate that the RDS may also be implemented with an intelligent agent capable of interacting with the mobile device over a network to perform tests and obtain information to support defect analysis regarding such mobile device.

[0108] Additional defect tests (e.g., 1642A) may be performed using aspects of the mobile device under test, whether associated with an RDS or installed with an FSTA to perform the tests. For example, several tests may be performed (either by the FSTA or remotely via a connection between the mobile device and the RDS) to determine, for example: (1) vibration test (e.g., to determine that the device's vibration agitator is functional), (2) speaker test, (3) screen touch test, (4) front camera test, (5) rear camera test, (6) screen pressure test, (7) accelerometer test, (8) gyroscope test, (9) battery status, including operational capability, charge time, whether the battery is swollen (thus indicating a potential defect / hazardous condition), and (10) mixed model / derived recognition, including binding to a unique fingerprint or mobile device ID. Additionally, aspects of the present invention provide acoustic analysis to determine defects associated with a mobile device.

[0109] With acoustic analysis, a vibration signal (such as an audio signal or a known frequency mechanical vibration) is created and applied to the mobile device, and the mobile device's microphone "listens" to what may be detected as a result. The captured audio signal from the microphone can be further analyzed to determine whether certain defects may exist in the mobile device. For example, a predetermined signal may be output from the mobile device's speaker (or alternatively, by vibrations or vibration patterns caused by an agitator located within the mobile device), and the mobile device then listens to the audio signature received by the mobile device's microphone via a microphone associated with the mobile device (or a companion device). In one version, a simple audio test of various frequencies could identify whether the speaker / microphone pair is functioning in the desired frequency range. Furthermore, a spectral analysis of the audio signal captured by the microphone may identify certain defects in the speaker or microphone. The spectral analysis may further compare the sound propagation to an ideal propagation through a classified manufacturer's identical specimen, and defects such as cracks in the glass attached to the mobile device may cause discontinuities or rattles in the sound propagation that may be deduced from the audio signal received from the microphone. Certain mechanical vibration patterns may also be created by an agitator to try to find resonant frequency rattles due to loose parts within the mobile device that may be sensed by the audio pattern received by the microphone of the device. Such spectral analysis may be performed in concert with a pre-trained neural network implementation to identify certain audio anomalies that correspond to defects that may be associated with the mobile device.

[0110] An image of the display of the mobile device 800 is captured (1620) by the process discussed with respect to FIG. 17, and the captured image is optionally pre-processed (1625) by rotating the image into portrait mode and / or by micro-rotating to reduce skew and / or by cropping the image to a desired aspect ratio. The captured (and optionally pre-processed) image is then uploaded (1630) to a server (such as server 860), which processes the uploaded image to determine whether any cracks or other types of defects are present (1635), as further discussed with respect to FIG. 18. Alternatively or in combination, the extent and characteristics of the screen cracks or defects are characterized by a processing step 1635 for further analysis and reporting. Once the analysis 1635 of any potential defects is completed, the resulting information may be stored or displayed for presentation to any user for any necessary purposes, such as evaluating whether the mobile device 800 is in proper operating condition. array The process then ends (1645).

[0111] FIG. 24 illustrates one exemplary data flow according to a block diagram 2400 of a convolutional neural network (CNN) architecture of the present invention. Prior to presentation of device images or data 2405, components of the CNN are trained to recognize specific device defect classes, including defects such as scratches, cracks, and creases in the device display. In one embodiment, the downscaled and resampled device images (e.g., image files resulting from step 1820 of FIG. 18) are presented 2405 to an input layer 2410 of the neural network along with any optional device data that may be used to process the image associated with the device (such as a unique identifier associated with the device, including, but not limited to, an IMEI associated with the device). The input layer is communicatively coupled to block 2420 and includes at least one convolutional layer, a pooling layer, and a fully connected layer, as conventionally implemented in a CNN. The trained CNN operates on the input data to generate outputs corresponding to outputs indicative of any detected defect classes 2430. Operation may end at the analysis of the output of block 2430 or may continue for further processing by input to a second input layer 2440. The defects in block 2430 may be presented to block 2450 where deconvolution, depooling and fully connected layers further process and enhance features of the input image corresponding to where the defects were located. Output layer 2460 provides output in the form of a diagnostic image 2470 which may be stored in a database (e.g., FIG. 8A, 880) and may also be associated in database 880 with device data identifying the device images analyzed by network 2400.

[0112] In general, the number of different network layers in a CNN can be selectively increased to increase the performance and learning ability of the CNN model. The number of distinct intermediate layers between the input layer and the output layer can be very large, thereby increasing the architectural complexity of the CNN model. The present disclosure utilizes a learning CNN model to identify device defects and / or create a visual map that indicates the presence of defects located within an image of a mobile device. Consistent with the disclosed embodiments, the CNN system 2400 may be configured to generate an output indicative of the defects (such as the identified defects 1900A in the device display 1400B of FIG. 20E or the identified defects 1900H in the display 1400H of FIG. 20H). Alternatively, embodiments of the present invention contemplate that the CNN system 2400 may be configured to generate an output indicative of a heat map (such as the identified heat map 1900D in the device display 1400D of FIG. 20D or the identified heat map 1900G in the display 1400G of FIG. 20G) corresponding to areas of the image corresponding to the likelihood of the detected defect class. For further explanation, the image 1400A in FIG. 20C may be presented as the device image / data 2405 of the network 2400, and FIG. 20D and / or FIG. 20E may be obtained from the output of the CNN 2400. Similarly, the image 1400F in FIG. 20F may be presented as the device image / data 2405 of the network 2400, and FIG. 20G and / or FIG. 20H may be obtained from the output of the CNN 2400. As discussed above, depending on the particular embodiment, the output may be obtained from either block 2430 or block 2470 of network 2400.

[0113] As used herein, and as previously described, a CNN model used by the disclosed systems and methods of the present invention may refer to any neural network model formulated, adapted, or modified based on the framework of a convolutional neural network. For example, a CNN model used to identify device defects in embodiments of the present disclosure may optionally include intermediate layers between the input layer and the output layer, such as one or more convolution / deconvolution layers, nonlinear operator layers, pooling or subsampling layers, fully connected layers, final loss layers, deconvolution layers, and / or inverse pooling or upsampling layers.

[0114] In an alternative embodiment, the output from the CNN model may be trained to generate either a heat map or a specific defect identification without further processing after block 2430 and without a deconvolution layer. Thus, in one embodiment, the output is generated by a class activation map by training an output layer of a neural network of the present invention, and in one embodiment, the heat map may be output as described above. In one aspect, the heat map is generated by obtaining a mapping function for each section of the image being output by the trained neural network of the present invention. The mapping function within each subsection of the image may use the score for the subsection of the image to identify a specific defect class when the trained network detects a defect in the image data. In one implementation of the present invention, the heat map may be obtained from a specific layer in the CNN model itself. As will be appreciated by those skilled in the art of neural network training, for example, a softmax function may be used in the final layer of the CNN defect classifier and the resulting layer element scores may be calculated to generate the heat map, and the output may be generated in the form of image data that may be overlaid on a corresponding area of ​​the test image to identify the heat map. Alternatively, in one embodiment, the CNN may reduce or remove aspects of the input data image, apply different convolutional layers, capture the output of one of the convolutional layers (or output across the entire image), and generate a score for each portion of the image, which in turn generates a defect activation map. Areas that exceed a certain level of activation may be converted to a particular color in the output image data, for example, red, to identify defects found on the test image (e.g., reading the color applied to areas of the image that correspond to display cracks in the test image).As a further aspect, at the network output layer, a threshold (compared to a predetermined color threshold) may be applied to the image data to find the red portions of the image, and a minimum bounding rectangle may be placed around the red areas to more specifically identify the defective areas.

[0115] FIG. 25 provides an overview of a flow diagram 2500 for one embodiment of the present invention, including three stages: a training preparation stage 2525, a network training stage 2526, and a network inference stage 2527. As briefly described above, the disclosed system and method for identifying device defects includes several stages that are generally described sequentially, including, for example, a training stage in which a CNN model is "trained" and "learned" using a training dataset, and a prediction or inference stage in which the trained CNN model inputs data to identify device defects (and / or defect classes) using the training dataset. As mentioned above, the aforementioned training stages generally correlate with network training 2526 as described in FIG. 25. As used herein, "training" a CNN model refers to determining one or more parameters of at least one layer in the CNN model. For example, a convolutional layer of the CNN model may include at least one filter or kernel. One or more parameters, such as weights, size, shape, and structure of the kernel of the at least one filter, may be determined, for example, by a backpropagation-based training process. Moreover, in general, the discussion of Figure 24 above correlates with network inference 2527 as shown in Figure 25. As used with respect to the embodiments described herein, "predicting" from the CNN model refers, in one embodiment, to identifying device defects from images of the device using any desired type of processed image of the device 2510, including images of the device's display.

[0116] In a training preparation stage 2525, training inputs 2520 are prepared and processed for use in training a CNN neural network component (e.g., FIG. 8A, 47). A network model is selected (2505) for use in training. Such model selection may include, for example, identifying a convolutional neural network architecture and appropriate processing layer configuration. The neural network component model (FIG. 8A, 47) is initialized with an initial layer configuration, an initial connection configuration, a set of weights, and a set of biases. In addition to the device images 2510 and each predicted outcome (also known as "ground truth" data by those skilled in the relevant art) 2515, other parameters (described in more detail below), such as identification information for each respective device image, may be set for input as training data 2520. In one embodiment, the training data 2520 may also include data for real or fake defective devices. In another embodiment, training data 2520 may be created synthetically, for example as computer-generated images based on theoretically plausible defect type scenarios, and may be devised for model testing purposes.

[0117] In various embodiments, and in a manner similar in context to process 1800 of FIG. 18, preparation step 2525 includes resampling all imaging data in training data 2520 to a common grid size and grid spacing. Image pixel values ​​can then optionally be resampled to a common scale across all image data in training data 2720, thereby improving learning performance and convergence based on the training data provided to the network. After data preparation, the training data and test data represent device images and each corresponds to identified defects including ground truth data.

[0118] The network training stage 2526 begins by presenting training data 2520 to the neural network model (without inputs) configured and initialized in step 2505. The neural network component (FIG. 8A, 47) estimates 2535 the results from the patient image data to generate estimated results, e.g., identified defect classes. A comparison is then made between the estimated results from step 2535 and the expected results 2515 corresponding to the device images 2510, and an error map ("training error") is generated based on the difference between the expected results 2515 and the estimated results 2535. The error map is compared to an evaluation criterion 2545. For example, a loss function such as a mean absolute error function (MAE) or a mean squared error (MSE) function can be used to determine whether the error criterion is met. Those skilled in the art will recognize that the mean absolute error (MAE) or the L1 norm J(θ * )=arg min θ ||Y=Y * ||1 or mean square error (MSE) or L2 norm J(θ * )=arg min θ ||Y=Y * You are familiar with various loss functions, such as ||2, where θ * is Y and Y * The backpropagation algorithm may be used to calculate the gradient of the error function with respect to the model parameters or weights. Then, θ is chosen as * The stochastic gradient descent algorithm may be iteratively updated to converge to

[0119] If the error does not meet the error threshold criterion, then the model parameters (e.g., weights and biases) of the neural network (FIG. 8A, 47) are updated (2550) to minimize the error, e.g., according to a learning algorithm (e.g., a regularized gradient descent optimization approach), and the training data is re-submitted (2530) to the neural network (FIG. 8A, 47) with the newly assigned model parameters of the neural network.

[0120] In one embodiment, determining whether the criteria are met (2545) includes presenting one or more test device images that differ from the training images (but do not have the pre-identified defects) to the network model as currently configured to generate estimates of defects corresponding to each such test device image. The resulting defect classifications can then be compared to the ground truth / expected defect classifications to assess the quality of the network parameter model. The difference between the predicted defects and the estimated defects for this test data is the "test error." In any case, the repetition of steps 2550 through 2530, 2535 and 2540, and 2545 continues until the error threshold criteria is met, at which point the training process 2525 is terminated (2555) and the trained neural network component model is ready for use with captured images of real devices and corresponding data (FIGS. 8A, 45, 46).

[0121] In alternative embodiments, the neural network (FIG. 8A, 47) may be trained in multiple steps, with different portions of the neural network being trained with separate training data sets to achieve specific defect classifications. For example, the convolutional neural network of the present invention may be pre-trained to recognize a number of generic visual object classes, and the final layers of the network may then be retrained to identify specific visual items, such as screen defects (e.g., cracked or damaged displays of mobile devices). In one version of the present invention, characteristics related to defects (e.g., display cracks) may be provided in the training data associated with the training images, and in various embodiments may be provided without coordinates of where such cracks exist in the training data images, and in yet other embodiments, training data images showing defects (e.g., display cracks) are annotated with coordinates identifying cracked areas in the images such that training of the output layer is optimized. Using multiple training passes, such as training a network with multiple visual classes and then retraining the final layer of the network with the specific defects to be detected, may provide an embodiment that has efficiency advantages over retraining the entire network when training data 2560 is updated or modified with new defect types or an updated amount of training examples.

[0122] Once trained, the neural network component model (FIG. 8, 47) may be stored in a memory, such as a server memory (e.g., FIG. 8, 16), in a non-transient memory of the network component (FIG. 8, 47), or in a database (e.g., FIG. 8A, 880). The trained network model may then be used in a network inference step 2527 to compute useful output (e.g., classified defects associated with the device) when presented with mobile device images and / or data (FIG. 8A, 45, 46) for which defect analysis is desired. The trained neural network is configured (2565), for example, by loading the stored weights and biases for a particular trained network configuration. Patient data inputs are presented to the trained and constructed neural network component (FIG. 8A, 47) to obtain predicted results. In a preferred embodiment, the predicted results include one of defect classification using the identified defects and device classification using the displayed heat map. Thus, for a device of interest, device images and / or data 2560 are presented 2570 to the trained and constructed neural network model to generate 2575 a predicted outcome. In a further embodiment, the predicted outcome may subsequently be reviewed 2580 by any desired entity, such as for use in determining insurability of the mobile device or validity of an insurance claim for the mobile device.

[0123] The particular implementation forms described above are illustrative of the present invention and its best mode, and are not otherwise intended to limit the scope of the present invention in any manner. Indeed, for the sake of brevity, conventional data storage, data transmission, and other functional aspects of the system may not have been described in detail. The methods illustrated in the various figures may include more steps, fewer steps, or other steps. Additionally, steps may be performed in any suitable order without departing from the scope of the present invention. Furthermore, the connecting lines shown in the various figures are intended to represent example functional relationships and / or physical connections between the various elements. Many alternative or additional functional relationships or physical connections may be present in an actual system.

[0124] Changes and modifications can be made to the disclosed embodiments without departing from the scope of the invention. These and other changes or modifications are intended to be included within the scope of the invention, as set forth in the appended claims.

Claims

1. A mobile device; a test fixture comprising a mounting platform mechanically connected via a support member to a top cover including a reflective surface; a server configured to communicate with the deployed mobile device, The mobile device memory may include instructions that, when executed by a processor of the mobile device, cause the mobile device to: indicating that the mobile device is ready to be placed in the test fixture; determining whether the mobile device is oriented in an acceptable manner relative to the reflective surface; responsive to detecting that the mobile device is oriented in an acceptable orientation, capturing an image of the mobile device reflected from the reflective surface and arranging the captured image for transmission to the server for defect analysis; said instructions causing the determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface; Displaying tracking and positioning guidance fiducial marks on the mobile device; and acquiring a tracking image of the mobile device from a camera of the mobile device; Displaying the tracking image within the placement guidance reference mark; determining an orientation of the mobile device within the tracking image based on a position and an orientation of an image of the tracking reference mark within the tracking image; determining whether the identified orientation is within a predetermined range; Including, the system.

2. Displaying tracking fiducial marks and placement guidance fiducial marks on the mobile device includes displaying the tracking fiducial marks on the mobile device, the display including: one or more QR codes; one or more Aztec codes; one or more ArUco codes, or The system of claim 1 , further comprising displaying as one or more graphical tracking elements.

3. The system of claim 1 , wherein the reflective surface comprises one or more of a mirror and a one-way mirror.

4. 1. A method for visually diagnosing a mobile device, comprising: indicating that the mobile device is ready to be placed in a test fixture comprising a mounting platform, a reflective surface; determining whether the mobile device is positioned on the mounting platform in an acceptable orientation with respect to the reflective surface; In response to detecting that the mobile device is oriented in an acceptable orientation, capturing an image of the mobile device reflected from the reflective surface; analyzing the captured image to determine whether the captured image includes a defect indicia indicative of whether the captured image includes one or more predefined defects; and providing a cosmetic defect rating for the mobile device based on the analysis. determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface; Displaying tracking and positioning guidance fiducial marks on the mobile device; and acquiring a tracking image of the mobile device from a camera of the mobile device; Displaying the tracking image within the placement guidance reference mark; determining an orientation of the mobile device within the tracking image based on a position and an orientation of an image of the tracking reference mark within the tracking image; determining whether the identified orientation is within a predetermined range; A method comprising:

5. 5. The method of claim 4, wherein the one or more defect indicia include indications of one or more of cracks, chips, tears, blemishes, wear, scratches, abrasions, gouges, stuck pixels, dead pixels, color errors, and color bleeding.

6. The method of claim 4 , wherein the reflective surface comprises one or more of a mirror and a one-way mirror.

7. a test fixture including a mounting platform mechanically connected via a support member to a top cover including a reflective surface, the reflective surface being positioned a predetermined distance from the mounting platform and at a predetermined angle relative to a top plane of the mounting platform; a server processor communicatively connected to the database; a server memory communicatively connected to the server processor; a server user interface communicatively connected to the server processor; and a server connected to the server processor and having a communication interface configured to communicate with a communication module of a mobile device disposed thereon, the located mobile device, a housing that substantially encloses the mobile device; and a mobile processor in communication with a memory of the mobile device; a user interface including a display and a first camera in communication with the mobile processor; a communication module configured to communicate with the mobile processor and to communicate with the communication interface of the server; The memory of the mobile device may include instructions that, when executed by the mobile processor, cause the mobile device to: indicating a readiness to begin testing of one of the display or the housing of the mobile device; Tracking fiducial marks, and Placement guidance fiducial marks including alignment indication presenting on said display an image comprising: adjusting the position and display of the mobile device; modifying brightness and camera sensitivity settings of the mobile device to optimize image capture; determining that the mobile device is acceptably oriented with respect to the reflective surface; capturing an image of the mobile device reflected from the reflective surface; sequencing the captured images for transmission to the server for defect analysis; said instructions causing the determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface; displaying the tracking fiducial marks and the positioning guidance fiducial marks on the mobile device; acquiring a tracking image of the mobile device from a camera of the mobile device; Displaying the tracking image within the placement guidance reference mark; determining an orientation of the mobile device within the tracking image based on a position and an orientation of an image of the tracking reference mark within the tracking image; determining whether the identified orientation is within a predetermined range; Including, the system.

8. The system of claim 7 , wherein the provided mobile device includes a second camera.

9. The system of claim 7 , wherein the reflective surface comprises one or more of a mirror and a one-way mirror.

10. The system of claim 7 , wherein the server is communicatively connected to the mobile device via one of a wired connection or a wireless connection.

11. The memory and the processor of the mobile device 1. Software that, when executed, performs the following: The tracking reference mark is displayed on the display; The system of claim 7 configured with the software, wherein the placement guidance reference marks are displayed on the display.

12. The system of claim 7 , wherein an image of the mobile device as seen by the camera is displayed on the display.

13. The system of claim 12, wherein a front camera captures an image of the display of the mobile device and a rear camera captures an image of the rear of the housing of the mobile device.

14. The system of claim 7 , wherein the mounting platform includes an alignment guide.

15. The alignment guide is An optical fiducial mark; an illuminated fiducial mark; one or more molded alignment guides; an electronic display connected to the test processor for displaying the placement fiducial marks and the alignment information; The system of claim 14 , comprising one or more of:

16. 8. The system of claim 7, further comprising capturing an image of the mobile device from the first camera after the mobile device is properly positioned on the mounting platform.

17. The system of claim 7 , further comprising: analyzing the captured image; and diagnosing at least one condition of the housing / display of the mobile device.

18. The system described in claim 7, wherein a height adjustment device enables the distance between the reflective surface and the mounting platform to be changed.

19. The system of claim 7 , wherein the support member is actuable to obtain a predetermined angle between the reflective surface and the mounting platform.

Images