Systems and methods for diagnostics of mobile device displays and housings

Abstract

Systems and methods for detecting mobile device fault conditions are presented, particularly for detecting defects in the display or housing of a mobile device. One or more cameras contained within the mobile device are used in conjunction with a reflective surface within a test fixture to capture properly aligned and formatted images of the exterior of the mobile device's housing and display, with interactive guidance assisting in alignment and performing diagnostics. A remote server further analyzes the captured images, after which the status of the defect may be returned to and displayed on the mobile device screen.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims full benefit under U.S. Provisional Patent Application No. 62 / 924,676, filed on 22 October 2019, entitled “System and Method for Diagnostics of Displays and Housings of Mobile Devices,” and the disclosure thereof is fully incorporated herein by reference for all purposes.

[0002] The present invention relates to a system and method for detecting a malfunction in a mobile device. More specifically, the present invention provides a system and method for detecting defects in a mobile device, such as a cracked, scratched, chipped, or otherwise damaged screen or casing, and for reporting the status of the screen or casing, whether it is operational or not, so that appropriate action can be taken by a third party. [Background technology]

[0003] The use of mobile devices today is widespread, and more and more people use such devices in their daily lives. As used herein, the term “mobile device” generally refers to an electronic device that is capable of being moved from place to place. Examples of such mobile devices include, but are not limited to, cellular phones (also more commonly called “mobile phones”), smartwatches, smart jewelry, personal digital assistants (PDAs), digital cameras, intelligent devices for the “Internet of Things” (or “IoT”), drone devices, mobile subscriber communication devices, tablet computers, media players, smart vehicles, laptop computers, and devices equipped with installed wireless intelligent agents such as Alexa®, Google Voice®, Siri®, and Cortana®. Consumers purchase millions of mobile devices, such as cell phones, every year for reasons such as wanting improved features and functions, replacing worn-out, damaged, partially non-functional, or slow-performing devices, or simply acquiring the latest product based on consumer loyalty, or wanting to show off a new model to friends and acquaintances. On the other hand, as more features and functions were incorporated into newly released mobile devices, the rapid increase in the cost of such devices created a secondary market for used but working mobile devices. As a result, industries developed to process and refurbish used mobile devices, making them operational again for refurbishment and resale, or to classify non-functional devices into valuable components or scrap.

[0004] Used mobile device processors (also known as “recyclers” in this specification) face the challenge of receiving used mobile devices, such as cell phones, whose functionality and operational status are unknown, evaluating the functional condition of such devices, ensuring that all data from the previous user is securely erased from the cell phones, and performing necessary repairs where necessary. Due to the narrow profit margins resulting from the highly competitive market conditions and pricing of refurbished mobile devices, efficiency in the evaluation, diagnosis, and erasure processes is paramount. Recyclers must quickly evaluate the type and operational status of the devices and re-perform necessary pre-configurations, such as flashing and erasing the mobile devices, in the most efficient way possible to ensure that the refurbished devices are returned to the commercial flow as quickly as possible.

[0005] In traditional mobile phone recycling processes, recyclers often manually inspect the mobile device's display and / or casing to assess the condition of the display, the glass on the display, the mobile device's casing, and the exterior of the mobile device's casing. These manual inspections are cumbersome, error-prone, and require additional manual input to document the current condition of the mobile device, further hindering the recycler's efficiency and warehouse throughput. Automated methods for assessing the condition of mobile devices are also error-prone and may require manual alignment of the mobile device with a camera or reflective device, again leading to inefficiencies and delays in the device evaluation process.

[0006] In addition to mobile device recyclers, retail organizations such as mobile phone companies (also known as mobile phone service providers) also face the need to quickly assess, diagnose, and document the condition of mobile devices brought in by customers at their retail stores or kiosks. To provide such services, mobile phone companies have faced the challenge of purchasing and configuring expensive hardware to diagnose mobile phones that may be in each retail store or kiosk. [Overview of the project] [Problems that the invention aims to solve]

[0007] In short, there is a need to provide systems and methods for efficiently performing diagnostics, particularly on the exterior of mobile device casings and displays, and for providing documentation and analysis interfaces to test systems. [Means for solving the problem]

[0008] The following technical disclosures are illustrative and descriptive, and do not necessarily limit the inventions described in the claims.

[0009] As used herein, the terms “mobile device,” “mobile electronic device,” or “device” generally refer to an electronic device that is movable and has components (such as displays and screens) that are subject to failure or damage. A mobile device may be a standalone device such as a laptop computer, desktop computer, mobile subscriber communication device, mobile phone, smartwatch, personal digital assistant (PDA), data tablet, digital camera, video camera, video game console, media player, Global Positioning System (GPS), Universal Serial Bus (USB) key, mobile weapon, smartwatch or jewelry, embedded electronic equipment, or a combination thereof. A mobile electronic device may be any electronic device integrated with another system or device. For example, a stereo, Global Positioning System, or other electronic equipment contained in a vehicle can be used in conjunction with the present invention. Software implementing the method of the present invention may be (1) installed or (2) downloaded to a mobile device indirectly or directly at any time by an authorized user via the Internet, SMS text message, wireless communication with an app distribution store, or any other suitable method and at any suitable time to perform the method according to the present invention. For example, the software may be installed on the device after purchase, or downloaded after the device has been purchased, or even after the device is damaged or otherwise malfunctioning. Mobile devices can be insured against loss or theft, and the systems and methods of the present invention may function as part of or in addition to the mobile device insurance policy.

[0010] A system and method for detecting mobile device failure conditions are presented, which in particular detects defects in the display or housing of a mobile device. A camera contained within the mobile device is used, along with a reflective surface in the test apparatus, to acquire appropriately aligned and formatted images of the housing and the outside of the display of the mobile device, and interactive guidance assists in the placement and execution of the diagnosis. A remote server may perform further analysis of the captured images, after which the defect status may be returned to the mobile device and displayed on the mobile device screen.

[0011] One embodiment of the present invention provides a system for visually diagnosing a mobile device, the system comprising a test instrument having a mounting platform mechanically connected to an upper cover including a reflective surface via a support member, and a server configured to communicate with a provided mobile device, the mobile device's memory including, when executed by the mobile device's processor, instructions to cause the mobile device to perform the following steps: instruct a user to place the mobile device within the test instrument; determine whether the mobile device is oriented in an acceptable orientation to the reflective surface; and, in response to detecting that the mobile device is oriented in an acceptable orientation, capture an image of the mobile device reflected from the reflective surface and format the captured image for transmission to the server for defect analysis. Furthermore, in one embodiment, determining whether the mobile device is oriented in an acceptable orientation to the reflective surface further includes displaying tracking reference marks and positioning guidance reference marks on the mobile device, acquiring a tracking image of the mobile device from the mobile device's camera, identifying the orientation of the mobile device in the tracking image, and determining whether the identified orientation is within a predetermined range. Additionally, displaying tracking criterion marks and placement guidance criterion marks on a mobile device may further include displaying the tracking criterion marks as one or more QR codes®, one or more Aztec codes, one or more ArUco codes, and one or more graphical tracking elements. The reflective surface may comprise any substantially planar or non-planar surface capable of reflecting the image of the mobile device, such as mirrors and one-way mirrors.

[0012] A method is provided for visually diagnosing a mobile device, the method comprising: instructing a user to place the mobile device in a test apparatus having a mounting platform, a reflective surface; determining whether the mobile device is positioned on the mounting platform in an acceptable orientation relative to the reflective surface; capturing an image of the device reflected from the reflective surface in response to detecting that the mobile device is oriented in an acceptable orientation; analyzing the captured image to determine whether the captured image contains one or more defect marks; and providing a surface defect rating of the mobile device based on the aforementioned analysis. The defect marks may include any desired information, such as one or more indications of whether the housing and / or display of the mobile device contains cracks, chips, tears, scratches, wear, abrasion, scratches, fraying, abrasions, stuck pixels, dead pixels, color errors, color bleeding, and deviant marks. Furthermore, this method may include detecting whether the mobile device is properly positioned relative to the reflective surface by analyzing a captured image that includes tracking reference marks, providing dynamic guidance by changing the placement guidance reference marks displayed on the mobile device, instructing the user regarding the placement of the mobile device on the mounting platform, and indicating by the placement guidance reference marks when the mobile device is correctly positioned relative to the reflective surface.

[0013] In one embodiment, a system is also presented, which includes a test instrument comprising a mounting platform mechanically connected to an upper cover including a reflective surface via a support member, wherein the reflective surface is positioned at a predetermined distance from the mounting platform and at a predetermined angle with respect to the upper plane of the mounting platform; and a server comprising a server processor communicatively connected to a database, a server memory communicatively connected to the server processor, a server user interface communicatively connected to the server processor, and a communication interface connected to the server processor and configured to communicate with a communication module of a provided mobile device, wherein the provided mobile device comprises a housing substantially surrounding the mobile device, a mobile processor communicating with the memory of the mobile device, a user interface including a display and a first camera communicating with the mobile processor, and a mobile The mobile device's memory includes a communication module configured to communicate with a data processor and a server's communication interface, and when executed by the processor, the mobile device's memory includes instructions to cause the mobile device to perform the following steps: instruct the user of the mobile device to start testing one of the mobile device's displays or housings; present an image on the display that includes tracking reference marks and alignment guidance reference marks including alignment indicators; perform a tracking calibration function to adjust the position of the mobile device and the display for optimal image capture; adjust the brightness and camera sensitivity settings of the mobile device to optimize image capture; determine that the mobile device is oriented in an acceptable orientation to the reflective surface, capture an image of the mobile device reflected from the reflective surface; and format the captured image for transmission to the server for defect analysis. In various embodiments, the provided mobile device includes a second camera, and in alternative embodiments, the reflective surface includes one or more of a mirror and a one-way mirror.Furthermore, in yet another embodiment, the server is communicably connected to the mobile device via either a wired or wireless connection, and in various embodiments, the mobile device's memory and processor are configured with software that, if executed, performs the step of initiating a test alignment mode, a tracking reference mark is displayed on the display, and a placement guidance reference mark is displayed on the display. A desired image may be output, for example, an image of the mobile device as seen from a camera is displayed on the display. In various embodiments, the image from the front camera captures an image of the mobile device's display, and the rear camera captures an image of the back of the mobile device's housing. Proper alignment of the device with respect to the test fixture may include detecting whether the mobile device is properly positioned relative to the reflective surface by analyzing the image from the camera containing the tracking reference mark, providing dynamic guidance by changing the placement guidance reference mark, instructing the user regarding the placement of the mobile device on the mounting platform, and indicating by the placement guidance reference mark when the mobile device is correctly positioned relative to the reflective surface.

[0014] In various embodiments, the system of the present invention may include a mounting platform that includes an alignment guide. In an optional embodiment, the alignment guide may include one or more of the following: an optical reference mark, an illuminated reference mark, one or more molded alignment guides, and an electronic display connected to a test processor for displaying alignment information such as the alignment reference marks and arrows.

[0015] Various embodiments may include capturing an image of the mobile device from a first camera after the mobile device has been properly positioned on the mounting platform, further including analyzing the captured image and diagnosing at least one state of the mobile device's housing / display. Additionally, embodiments may further include analyzing from the analyzed image whether the positioning of the mobile device requires change and providing alternative positioning guidance reference marks for further alignment. In some implementations, support elements are selectable to change the distance between the reflective surface and the mounting platform, and in other embodiments, support members can be actuated to obtain a predetermined angle between the reflective surface and the mounting platform.

[0016] A further embodiment of the present invention includes a system comprising a test instrument comprising a mounting platform mechanically connected to a reflective surface via a support member, wherein the reflective surface is positioned at a predetermined distance from the mounting platform and at a predetermined angle to the upper plane of the mounting platform; and a server comprising a server processor communicatively connected to a database, a server memory communicatively connected to the server processor, a server user interface communicatively connected to the server processor, and a communication interface connected to the server processor and configured to communicate with a communication module of a provided mobile device, wherein the provided mobile device comprises a housing substantially surrounding the mobile device, a mobile processor communicating with the memory of the mobile device, and a user interface including a display and a first camera communicating with the mobile processor. The mobile device includes a communication module configured to communicate with a mobile processor and a server's communication interface, and the mobile device's memory includes instructions, when executed by the processor, to cause the mobile device to perform the following steps: instruct the user of the mobile device to start testing one of the mobile device's displays or housings; present on the display an image that is a tracking indicator containing signs of alignment; perform a tracking calibration function to adjust the position of the mobile device and the display for optimal image capture; change the brightness setting and camera sensitivity setting of the mobile device to optimize image capture; determine that the mobile device is oriented in an acceptable orientation to the mirror and capture an image of the mobile device reflected from the mirror; and format the captured image for transmission to the server for defect analysis.In addition to the features described above, the system may include any desired hardware, for example, one or more cameras, lighting fixtures, microphones, speakers, vibrators, wireless charging pads, NFC circuits, conveyor belts, and mobile device operation gimbals, jigs, and motors.

[0017] The method of the present invention may be executed in whole or in part on a mobile device, a host server, external neural network hardware, and a server, and / or a processor communicably connected thereto.

[0018] A more complete understanding of the present invention can be derived by referring to the detailed description and the claims when considered in connection with the following exemplary drawings.

Brief Description of the Drawings

[0019] [Figure 1A] It is a diagonal front view of viewing the test instrument of the present invention. [Figure 1B] It is a front perspective view of the test instrument of the present invention. [Figure 1C] It is a rear perspective view of the test instrument of the present invention. [Figure 1D] It is a top view of the test instrument of the present invention. [Figure 1E] It is a view showing a two-device test instrument sized to accommodate two mobile devices. [Figure 1F] It is a perspective view of a larger test instrument configured to accommodate a plurality of mobile devices. [Figure 1G] It is a view showing a closed position of an upper cover / reflection surface configured such that a plurality of mobile devices can perform a display / housing test. [Figure 1H] It is a view showing an open position that enables easier visual inspection and / or placement of mobile devices within the test instrument. [Figure 1I] It is a perspective view of the test instrument of the present invention configured to include a magic mirror in the upper cover. [Figure 1J] Figure 1I is a side plan view of a test instrument including a magic mirror in an upper cover. [Figure 2] It is a flow chart of the process of the present invention. [Figure 3] It is a diagram showing a mobile device where a user has started the process of the present invention. [Figure 4] It is a perspective view of a mobile device including a reference mark of the present invention. [Figure 4A] It is a plan view of a display of a mobile device presenting two tracking reference marks as QR codes (registered trademarks). [Figure 4B] It is a plan view of a display of a mobile device presenting two tracking reference marks as QR codes (registered trademarks) together with a second or placement guidance reference mark. [Figure 4C] It is a plan view of a display of a mobile device presenting two tracking reference marks as QR codes (registered trademarks) together with a second or placement guidance reference mark. [Figure 4D] It is a plan view of a display of a mobile device presenting two tracking reference marks as Aztec codes together with a second or placement guidance reference mark. [Figure 4E] It is a plan view of a display of a mobile device presenting two tracking reference marks as ArUco codes together with a second or placement guidance reference mark. [Figure 5] It is a diagram showing the placement of a mobile device within a test instrument of the present invention. [Figure 6] It is a diagram showing a mobile device properly aligned within a test instrument of the present invention. [Figure 7] It is a diagram showing a mobile device indicating pass or fail of a test. [Figure 8] It is a block diagram of an embodiment of the system of the present invention. [Figure 8A] It is an alternative block diagram of an embodiment of the system of the present invention. [Figure 9] This figure shows one embodiment of the present invention in which a mobile device 800 is illuminated from the outside by a light source such as an LED lighting strip. [Figure 10] This figure shows one embodiment of the present invention in which a mobile device 800 is illuminated from the outside by a light source such as an LED lighting strip. [Figure 11] This figure shows one embodiment of the present invention in which a mobile device 800 is illuminated from the outside by a light source such as an LED lighting strip. [Figure 12] This figure shows one embodiment of the present invention in which a mobile device 800 is illuminated from the outside by a light source such as an LED lighting strip. [Figure 13] This figure shows one embodiment of the present invention in which a mobile device 800 is illuminated from the outside by a light source such as an LED lighting strip. [Figure 14] This figure shows a selection of cameras for the test apparatus of the present invention. [Figure 15] This figure shows a selection of cameras for the test apparatus of the present invention. [Figure 15A] This figure shows a selection of cameras for the test apparatus of the present invention. [Figure 16] This is a process flow diagram illustrating the present invention, which describes image capture and image analysis for identifying device defects. [Figure 16A] This is a process flow diagram illustrating the present invention's process of image capture and image analysis to identify device defects without first installing an application on the mobile device. [Figure 17] This is a process flow diagram of the present invention, providing further details of an exemplary image capture process. [Figure 18] This is a process flow diagram of the present invention, providing further details on an exemplary image preprocessing method. [Figure 19] This is a diagram illustrating an exemplary mobile device display with a defect on the mobile device's screen. [Figure 20]This figure shows an example image of Figure 19 after thresholding technology has been applied. [Figure 20A] This figure shows an exemplary perspective-transformed sub-image of a mobile device's display, corresponding to the display portion of the captured image in Figure 19. [Figure 20B] This figure shows an exemplary output of the present invention, identifying a defect enclosed by a polygon on the display corresponding to the device in Figure 19. [Figure 20C] This figure shows an exemplary device image for input to a convolutional neural network according to the present invention. [Figure 20D] This figure shows a heat map output that identifies potential defect areas corresponding to the device image shown in Figure 20C. [Figure 20E] Figure 20C shows a defect enclosed by a polygon in the display corresponding to the device image shown. [Figure 20F] This figure shows an exemplary device image for input to a convolutional neural network according to the present invention. [Figure 20G] This figure shows a heat map output that identifies potential defect areas corresponding to the device image shown in Figure 20F. [Figure 20H] This figure shows a defect enclosed by a polygon in the display corresponding to the device image shown in Figure 20F. [Figure 21] This figure shows an exemplary display of the mobile device of the present invention, indicating corner reference markers. [Figure 22] Figure 21 illustrates the template matching correction for the display image shown. [Figure 23] This figure shows template-matched corner markers obtained from the display image in Figure 21, which are used to determine the edges of the display associated with a mobile device. [Figure 24] This is a block diagram with a data flow describing one convolutional neural network configuration relating to the system of the present invention. [Figure 25]This is a comprehensive flowchart describing the neural network training process using training data and the network inference process that uses the trained neural network to identify defects. [Modes for carrying out the invention]

[0020] Similar reference numerals indicate similar structures or functions in the various embodiments shown herein.

[0021] Test equipment

[0022] Figure 1 shows a schematic diagram of an embodiment of the present invention. The test fixture 200 provides a platform for automated testing of the display 1400 and / or housing of a mobile device 800. The test fixture includes a mounting platform 205, along with an optional alignment guide 225 to assist in the positioning of the mobile device 800. The mounting platform 205 is mechanically connected to a reflective surface 215 via a support member 210. The reflective surface 215 may be mounted on the surface of an upper cover 220 such that the reflective surface substantially faces the mounting platform 205, and the upper cover 220 may be mounted on a support member and allow mounting to the reflective surface 215 via its bottom surface. The reflective surface 215 may comprise a mirror or other surface that reflects an object placed on the mounting platform 205. The reflective surface 215 may be reoriented relative to the mounting platform 205 by changing the distance between the reflective surface 215 and the mounting platform 205 using a height adjustment device 235. Furthermore, in a preferred embodiment, the reflective surface 215 is substantially parallel to a plane substantially including the upper surface of the mounting platform 205, but the angle may be changed with respect to the mounting platform by operating the reflector angle adjustment device 230, via a hinge or the like that connecting the support member 210 to the upper cover 220 / reflective surface 215. Any desired angle with respect to the mounting platform may be obtained to improve the operation of the diagnostic aspects of the display and housing of the present invention. A mobile device 800 is also shown in the figure, which communicates with the server 860 by wire or wireless (255). The test device 200 is configured to accept the insertion 251 of the mobile device 800 and, in one embodiment, is configured to assist in the alignment of the placement of the mobile device 800 on the mounting platform via an optional alignment reference mark 225.The reference mark 225 may include any visual indicators to assist the user in the proper placement of the mobile device, and in some embodiments, it may include a box or other polygon, raised area or illuminated area to guide the user to the optimal placement of the mobile device 800. The height adjustment device 235 may include any desired mechanism, such as a slide, ratchet, locking bar, rack, selectable pins and holes, or any other adjustment mechanism, to change the distance between the mounting platform 205 and the reflective surface 215.

[0023] The test apparatus 200 may be of any appropriate size, and the dimensions may be selected so that the front camera 874 of the mobile device 800 is at a sufficient distance to focus on and resolve the reflected image of the mobile device 800 when the image of the mobile device 800 is sent from the mobile phone and reflected downward toward the mobile phone. In one embodiment, the distance 277 between the mounting platform 205 and the reflective surface 215 is about 160 mm. Furthermore, the test apparatus 200 may be sized in the horizontal direction to accommodate any desired mobile device 800 or multiple mobile devices, and in one embodiment, it is about 210 mm deep (direction 276) and 180 mm wide (direction 275). A height adjustment device 235 may be operated to customize the distance between the mobile device and the reflective surface and may accommodate a fixed-focus device or a variable-focus device.

[0024] Placing the mobile device 800 closer to the reflective mirror 215 may help optimize the resolution of the image captured by the front camera 874. However, due to the front camera 874's standard near-edge position, the closer the mobile device is moved to the reflective surface 215, the greater the tilt applied to either the reflective surface 215 or the mobile device 800 in order to capture a full image of the mobile device 800. In other words, if the mobile device 800 is placed too close to the reflective surface 215, the camera may not be able to capture the bottom of the screen due to the camera's position on the phone (e.g., closer to the edge rather than the center). On the other hand, if the mobile device 800 is moved too far away, it may not produce sufficient resolution to detect cracks or other defects in the display. In one embodiment, a preferred spacing may be created by positioning the mobile device 800 at at least half of the front camera 874's minimum focal length such that the “up and behind” distances correlate with at least one focal length.

[0025] Various other angles further illustrate the implementation form of the present invention. Figure 1A shows an oblique front view of the test apparatus 200, with the reflective surface 215 located beneath the upper cover 220. Figure 1B shows a front perspective view of the test apparatus 200, with the mobile device 800 inserted and positioned within the alignment guide. Figure 1C shows a rear perspective view of the test apparatus 200. Figure 1D shows a top view of the test apparatus 200.

[0026] Various alternative implementations of the present invention are also possible that can accommodate multiple mobile devices simultaneously. Such implementations may be advantageous in warehouse environments that may be used by device recyclers or repair agencies. Figure 1E shows a two-device test apparatus 200A that is sized to accommodate two mobile devices 800, 801 simultaneously. Note that the display 1400 of mobile device 800 indicates that it has passed the display diagnostic test, and the display 1401 of mobile device 801 indicates that a crack has been detected on the mobile device. Additionally, Figure 1F shows a perspective view of a larger test apparatus 208 configured to accommodate multiple mobile devices 800 (nine are shown in the figure, but any desired number may be used in embodiments of the present invention). Such a configuration may include increasing the width dimensions of the top cover 220, reflective surface 215, support member 210, and mounting platform 205.

[0027] As previously mentioned, the upper cover 220 and / or reflective surface 215 may be moved relative to the mounting platform 205. In one embodiment, the upper cover 220 and reflective surface 215 are hinged to a support member so that the upper surface / reflective surface can rotate between their open and closed positions. Figures 1G-1H show such a test apparatus 200C, where a hinge 230 connects the upper cover 220 / reflective surface 215 to the support member 210. Figure 1G shows the closed position of the upper cover / reflective surface, configured to allow multiple mobile devices 800 to perform display testing, and Figure 1H shows the open position, allowing for easier visual inspection and / or placement of the mobile devices 800 within the test apparatus 200C. The test apparatus 200C in Figures 1G-1H shows legs supporting the mounting platform 205, which may or may not be used depending on the desired test setup.

[0028] Figures 1I–1J show embodiments of the test apparatus 200D of the present invention, further configured using an upper cover 220D containing a magic mirror. A magic mirror is also known in the art as a two-way mirror, magic glass, half-mirror, or semi-transparent mirror. Such a mounting configuration provides a reciprocal mirror that appears transparent on one side, such as the bottom (providing a reflective surface 215), and on the opposite side, such as the top, so that it can be seen by an observer viewing the mobile device 800 from a position above the upper cover 220D (243). The mounting platform 205 is mechanically connected to the upper cover 220D via a support member 210. The upper cover 220D / reflective surface 215D is positioned such that the reflective surface 215D substantially faces the mounting platform 205 and provides a reflection 216D from the reflective surface 215D to the mobile device 800. The top cover 220D may be attached to the support member 210 and may be attached to the reflective surface 215D of the top cover 220D via its bottom surface. The top cover 220D may be implemented as a continuous magic mirror, or the magic mirror may include a transparent element such as glass or plexiglass attached to its bottom surface, providing a reflective surface 215D given additional rigidity / break resistance and / or support by the transparent material on the top surface of the top cover 220D. The top cover 220D may include a transparent or opaque frame (not shown) that partially or completely surrounds the magic mirror, providing additional rigidity, strength, and edge chip resistance while allowing the operator to view the mobile device 800 (243) from a position on top of the top cover 220D. The test fixture 200D in Figures 1I-1J shows legs supporting the mounting platform 205, which may or may not be used depending on the desired test setup.

[0029] Figure 3 illustrates the method of the present invention utilizing the configuration of the test equipment shown in Figures 1A-1H, and more specifically, Figures 3-7 show intermediate results. Further consideration of the mobile device and server architectures shown in Figures 1, 8, and 8A provides further context for the described method. In the described embodiments, image analysis provides a system and method for identifying cracks, housing defects, or other types of defects related to the mobile device. Referring to process 100 shown in Figure 2, an application is installed on the mobile device 800 to be tested (110). However, in one embodiment, the user of the mobile device 800 under test activates a web link to the server 860 via a web browser hosted by the mobile device's operating system. In yet another embodiment, a combination of software installed on or running on the mobile device 800 and software installed on and running on the server 860 may interoperate to perform the described method. When ready, the user of the mobile device 800 initiates a device defect test (115), and the display 1400 of the mobile device 800 displays a first reference mark 1430, also called a tracking reference mark 1430, and a second reference mark 1420, also called a placement guidance reference mark, as shown in Figure 4, indicating that the mobile device 800 is ready to be placed on the test apparatus 200. In a preferred embodiment, the second reference mark 1420 includes an indication that the placement has not yet been correctly sensed (indicated by a red outline in the shown photograph), and the image seen by the first front camera 874 is displayed within the second reference mark 1420 (see Figures 5 and 6 showing the image seen by camera 874 displayed within the second reference mark 1420). Any desired graphics may be used as the first reference mark.In the alternative embodiment shown in Figure 4A, the mobile device display 1400 outputs two “first reference marks” 1430,1430A as QR codes®, which do not need to contain identification information and may be encoded using random information, or simply contain information that enables optical tracking of the mobile device display 1400. The use of such first reference marks 1430,1430A, along with a predetermined placement position on the screen, enables the determination of the screen orientation, and the first reference marks 1430,1430A may enable the extrapolation of the edge boundaries of the display 1400 based on a known shape of the device on which the first reference marks are displayed. Furthermore, Figures 4B-4C show plan views of the mobile device display presenting the two tracking reference marks as QR codes®, along with a second reference mark 1420, or a placement guidance reference mark. In Figure 4B, the second reference mark 1420 indicates an improper alignment position relative to the reflective surface, and in Figure 4C, the second reference mark indicates a proper alignment position relative to the reflective surface. As shown in Figures 4B-4C, the second reference mark 1420 may include a bounding box positioned around the detected preview image of the mobile device 800, allowing the user to further determine the orientation of the mobile device 800 relative to the reflective surface. Additionally, Figure 4D shows a plan view of the mobile device display presenting the two tracking reference marks as an Aztec code, along with the second, or alignment guidance, reference mark. An Aztec code is known to those skilled in the art as a 2D barcode device detectable and traceable by a camera connected to optical tracking software.

[0030] Furthermore, ArUco markers (or codes) are known to those skilled in the art as containing a composite square marker consisting of a wide black border and an internal binary matrix. The ArUco library, developed by Rafael Munoz and Sergio Garrido, is a well-known library for detecting square reference markers, and corresponding software libraries are used by those skilled in the art to detect these reference marks. As a first reference mark, the ArUco marker enables tracking and orientation calculations for software accessing the rear camera sensor of a mobile device. Thus, Figure 4E shows a plan view of a mobile device display presenting two tracking reference marks 1430, 1430A as ArUco markers / codes, along with a second, or orientation guidance reference mark 1420.

[0031] Considering Figures 4 and 4A-E, the purpose of the first reference mark 1430 (alternatively, together with a second first reference mark 1430A) is to provide a frame of reference on the display of the mobile device 800 that is constant, enabling tracking and determination of the position of the mobile device 800 in three-dimensional space relative to a reflective device. The first reference mark may include any mark or marking that enables tracking and positioning, and in a preferred embodiment, it is not necessary to include encoded information. In Figure 4, the position of the first reference mark 1430 is shown at the top center within the display area 1400 of the mobile device 800, but the first reference mark may be placed at any desired position to assist in fixing the position of the mobile device 800. Although not required in a preferred embodiment, in an alternative embodiment, the first reference mark 1430 may be modified to uniquely identify a particular mobile device 800 on which the first reference mark 1430 is displayed, thus preventing misidentification of other mobile devices that may be displaying the reference mark in multiple test environments. Such unique identification may include a serial number or a QR code (registered trademark) which includes other identifying features of the encoded mobile device. The second reference mark 1420 provides guidance to help the user move the mobile device 800 to a suitable position relative to reflective devices so that an ideal image can be captured, minimizing parallax distortion and visual errors, minimizing or eliminating noise and undesirable reflections, and ensuring that the front camera 874 captures an image of the entire mobile device 800 in its environment. In one embodiment, the front camera 874 captures an image in real-time live video preview mode, and the image shown near the second reference mark 1420 in the display area moves in real time as the mobile device 800 moves.As shown in Figures 4, 5, and 6, 14A-D, the second reference mark 1420 is shaped as a colored parallelogram so that when the mobile device 800 is properly aligned, the image seen by the front camera 874 is displayed within the second reference mark 1420, and as will be described in more detail below, the reference mark changes its state to indicate proper alignment (such as by changing its color from red to green, or by other indications including text). Furthermore, in alternative embodiments as shown in Figures 4A-E, the second reference mark 1420 may be shaped as a rounded rectangular area having a color indicating the alignment status of the mobile device (such as red if not aligned, and green if properly aligned), along with an optional preview image seen from the rear camera of the mobile device 800 (to assist the user in alignment and orientation).

[0032] Alternatively, a marker reference mark (not shown) with known geometric dimensions may be output to the display of the mobile device 800 (furthermore, a scale such as a centimeter scale may be displayed). Since the mobile device 800 is running its own software, it can access the given measurement results, and the reference marker allows the display to be zoomed in and out by the measured reference mark, thereby allowing the positioning of the display and the bounding box of the mobile device, and thus in this approach, there is no need to rely on edge detection to simply find the phone / display in the captured image.

[0033] Returning to Figure 2, the user then places the mobile device 800 on the mounting platform 205 (see also Figure 5), and position detection software is executed to determine the position of the mobile device in the image acquired by the first front camera 874 and evaluate its position (for example, by attempting to detect whether the first reference mark 1430 is visible in the camera image and whether the mobile device is properly positioned). Detection of proper positioning may occur by various approaches. If the user is using a web-based server approach, a color (such as a unique color like bright magenta that is unlikely to appear in a normal image) is selected and that color is output to the display in the four corners of the mobile device's display 1400, and then color tracking is performed in the live video generated by the front camera 874 to find the position of the color element in the four corners of the screen, which can be done by using a color tracking library to find the position of the color (such as an exemplary bright magenta) in the image, search for those colors and then identify the position of the corners of the display 1400 in the video image. This information is then used to identify the position of polygons in the image. Alternatively, the image can be analyzed to determine whether a first reference mark 1430 exists and to determine the position and orientation of the first reference mark 1430 within the image. Based on a database of device types, identification information of the mobile device itself further assists in identifying the device's contours and in image pre-processing and post-processing.

[0034] After it is determined that the mobile device 800 is properly positioned, the second reference mark 1430 changes its state to indicate proper positioning (see, for example, the green line in Figure 6 with respect to the second reference mark 1430), and the first camera 874 on the front of the mobile device captures an image (Figure 3, 120).

[0035] With respect to Figure 17, in addition to the wide range of preprocessing options described below, certain types of checks may be performed on the captured image, including, for example, focus checks, brightness checks, and / or evaluation of contour blurriness. Furthermore, if the current image is determined to be insufficient, another image may be captured again using different camera parameters (125). Additionally, the captured image may be optionally preprocessed by rotating the image vertically and / or finely rotating it to reduce distortion and / or cropping the image to a desired aspect ratio (125). The captured (and optionally preprocessed) image is then uploaded to a server (such as server 860) (130). Depending on the required compliance with applicable laws such as GDPR, the user may be instructed to upload the file first before the image file is transferred to server 860, and the user may optionally be given the opportunity to review the captured image and recapture the image if necessary. After the upload has started, a progress bar to indicate the upload progress may be output to display 1400.

[0036] As further illustrated with respect to Figure 18, the uploaded image is processed to determine whether cracks or other types of defects are present (135). Alternatively or in combination, the extent and characteristics of cracks or defects on the screen are characterized by processing step 135 for further analysis and reporting. After the analysis of potential defects 135 is complete, the resulting information is formatted for presentation, storage, or display to any user for any necessary purpose, such as evaluating whether the mobile device 800 is in proper working order (140). In one embodiment, information to indicate the results of the analysis is returned from the server to the mobile device, and if no defects are detected, a pass indicator may be displayed on the screen 1400 of the mobile device 800 (see, for example, “Pass” shown in Figure 6). Furthermore, as further illustrated in detail below, if a defect is detected, the bounding box of the detected defect (shown as a crack) on the mobile device 801 may be surrounded by a graphical bounding box 1402, such as the red box shown, and an indicator (e.g., “Crack” shown) indicating that the diagnosis has detected a failure criterion is displayed. The process then terminates (145).

[0037] The exemplary system shown in Figure 8 comprises a mobile device 800 having a processor 810 connected to memory 820 which may include volatile memory, non-volatile memory (such as flash memory), or a combination thereof. A communication module 830 comprises a radio transceiver 840 for wirelessly communicating with one or more servers 860 and other entities via an antenna 850, although those skilled in the art will understand that a wired connection may be established to provide connectivity in lieu of or in addition to the wireless connection. The mobile device also comprises a user interface 870 connected to the processor 810. The mobile device 800 may have any suitable power source, such as a battery (not shown). The mobile device 800 may have any other desired components, such as a Global Positioning System (GPS), to provide geolocation information for determining the location of the mobile device. Some or all components of the mobile device 800 may include (or communicate with) hardware identification modules (not shown), such as a general-purpose subscriber identification module and / or a removable user identification module. The hardware identification module may be connected to the processor 810 and may include an identifier that can be compared to a predetermined identifier to determine whether the hardware of the mobile device 800 has been modified. The hardware identification module (and the predetermined identifier) ​​may include any appropriate identifier, such as an electronic serial number, a local area ID identifier, an integrated circuit identifier, an international mobile subscriber identifier, an authentication key identifier, and / or an operator-specific emergency number identifier. This identifier may be stored in memory 820 and transmitted to the host server 860 for comparison with the predetermined identifier.

[0038] The functions of the mobile device 800, including the methods described herein (in whole or in part), may be implemented by the processor 810 executing computer-readable instructions stored in the memory 820 of the mobile device 800. The memory 820 may store any computer-readable instructions and data, including software applications, user-installed or third-party-installed "apps," applets, and embedded operating code.

[0039] Additionally, software applications installed on a mobile device (or interface with processor 810 in any way) may be configured to operate with minimal prerequisite hardware capabilities. For example, such applications may be started before the mobile device establishes a network connection. This situation may be provided, for example, when a software application is installed on a SIM card in the mobile device, and the application starts before other software of the mobile device operating system. Alternatively or additionally, data elements such as links or URLs (universal resource locators) may be present on the SIM card, and by launching an application such as a browser using such URLs or links, the applications referenced by such links or URLs may be loaded from a remote server to the mobile device and / or executed directly from such remote server.

[0040] The software execution method of the present invention may be provided with the device or downloaded to the mobile device by an authorized user. The mobile device 800 functionality may also be implemented by various hardware components that store machine-readable instructions, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or complex programmable logic devices (CPLDs). Systems according to aspects of the present invention may operate with any desired combination of software and / or hardware components.

[0041] Processor 810 reads and executes instructions stored in memory 820 to control the operation of mobile device 800. Any number and type of processors, such as integrated circuit microprocessors, microcontrollers, and / or digital signal processors (DSPs), may be used in conjunction with the present invention. Memory 820 stores instructions, data, messages sent from (or received by) mobile device 800, and any other appropriate information. Memory 820 operating with the present invention may include any combination of different memory storage devices, such as hard drives, random access memory (RAM), read-only memory (ROM), flash memory, or any other type of volatile and / or non-volatile memory. Data can be stored in memory 820 in any desired manner.

[0042] The communication interface 830 communicates with one or more servers 860 or other suitable entities, such as the host server 860. Any suitable communication devices, components, systems, and methods may be used in conjunction with the present invention. For example, the radio transceiver 840 may be configured to communicate using any number and types of cellular protocols, such as General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), GSM Evolution High Speed ​​Data Rate (EDGE), Personal Communication Service (PCS), Advanced Mobile Phone Systems (AMPS), Code Division Multiple Access (CDMA), Wideband CDMA (W-CDMA), Time Division Synchronized CDMA (TD-SCDMA), Universal Mobile Communications System (UMTS), and / or Time Division Multiple Access (TDMA). Mobile devices operating with the present invention may optionally (or additionally) include radio transceivers (and associated components) for communicating using any other radio communication protocol, such as ISO 14443 protocol, ISO 18000-6 protocol, Bluetooth® protocol, ZIGBEE® protocol, Wibree protocol, Wi-Fi® protocol, IEEE 802.15 protocol, IEEE 802.11 protocol, IEEE 802.16 protocol, ultra-wideband (UWB) protocol, IrDA protocol, and combinations thereof. The antenna 850 may be configured to transmit and receive any radio signal of any format, and may include multiple different antennas that transmit and receive using different radio protocols.

[0043] The communication module 830 can communicate with server 860 or another device 862 using any other connection type such as a wired internet connection, wireless internet connection, cellular network connection (including data link connection), wireless LAN connection, wireless WAN connection, optical connection, USB connection, Bluetooth® connection, mobile device sync port connection, power connection, and / or security cable (so that other devices 862 can similarly communicate with server 860). The communication module 830 can be used to monitor the location or status of mobile device 800 (for example, by monitoring whether the communication link between the mobile device and the companion devices is functioning correctly) by communicating with one or more companion devices and any number of other devices that help track / locate a lost or stolen mobile device 800.

[0044] The mobile device 800 includes a user interface 870. The user interface 870 may include any number of input devices (not shown) for receiving commands, data, and other appropriate inputs from the user, as well as any number of output devices (not shown) for providing the user with data, notifications, and other appropriate information from the mobile device 800.

[0045] The user interface 870 may be provided with any number of input devices, such as a touchpad, a touchscreen, a display which may or may not be integrated with a touchscreen, and / or an alphanumeric keypad, so that the user can input commands and data into the mobile device 800. For the purposes of this application, the term “touchscreen” includes a display which is integrated with or adjacent to a touch interface which can determine when the user applies a physical connection to a location adjacent to the display. The touchscreen may have sensors which can measure parameters from the user’s interaction which may measure measurements of capacitance, resistance, pressure, or difference resulting from the movement of “touch” to the screen. The user interface 870 may be configured to detect the pressure the user applies to the keys of a keypad (which may be virtually implemented on the display or as a physical array of key switches) and the time interval between multiple key presses in order to determine whether the current user is authorized to use the device. The user interface may also include a microphone that enables the user to provide voice data to the mobile device 800, and may also include one or more cameras (874, 872) that enable the mobile device to capture still or moving images. In one embodiment, the mobile device 800 includes a front camera 874 that faces the user of the mobile device when the device is operating normally, and a rear camera 872 located on the opposite side of the mobile device, the cameras of which are connected to the processor 810. The mobile device 800 may also include speech recognition software for processing voice input via the user interface 870. The user interface 870 may also include any number of suitable output devices, such as a display screen for visually displaying information (such as images and text) and / or speakers for providing auditory output.The mobile device's display may be configured to sense user touch by any suitable means, such as capacitive sensing, pressure sensing, gel displacement sensing, resistance sensing, or any other suitable or conventional touch transmission technology available to those skilled in the art. The mobile device 800 may be configured to provide the user with words, phrases, tones, recorded music, or any other type of auditory output via a speaker. As previously stated, the user interface 870 can be activated to provide information when an unauthorized user attempts to use the mobile device 800 and / or to interfere with the operation of the mobile device 800. For example, the display's illumination level may be changed to draw attention to the mobile device, and unpleasant and / or loud sounds may be played through the speaker.

[0046] The mobile device 800 may include one or more biometric authentication devices configured to receive biometric information, such as a fingerprint scanner, iris scanner, retinal scanner, and / or breath analyzer. Input devices such as microphones and cameras may be used to perform biometric analysis, such as voice analysis and facial recognition. Furthermore, the mobile device may include a magnetometer (of the type used in electronic compasses) for measuring magnetic fields, a MEMS or other type of gyroscope for measuring orientation, and an accelerometer for measuring changes in the movement of the mobile device.

[0047] The information provided or received by the user interface 870 may be in any appropriate format. For example, a user interface that conveys information to a user in an auditory format may first provide the user with a data header, followed by data values ​​to identify the data. The user interface 870 may provide information in any number of desired languages, regardless of whether the information is provided auditorily or visually.

[0048] The user interface can also provide / receive information to the user in a machine-readable format. In one exemplary embodiment of the present invention, for example, the user interface 870 of the mobile device 800 may send and receive messages using dual-tone multi-frequency (DTMF) tones. The mobile device 800 can be configured to transmit, receive, and process machine-readable data cans in any standard format (such as MS Word documents, Adobe PDF files, ASCII text files, JPEG, or other standard formats) and any proprietary format. Machine-readable data to and from the user interface may be encrypted to protect the data from unintended recipients and / or improper use. In an alternative embodiment, the user may be required to enter a passcode to enable the use of some or all of the functions of the mobile device 800. Any other user interface functions may be utilized to enable human or non-human users to interact with one or more devices operating with the present invention.

[0049] The mobile device 800 may have any other appropriate functions, components, and / or systems. For example, the mobile device 800 may be configured to conserve its battery life by turning off some or all of its components, such as a camera or microphone. Components can be selectively shut down in response to a security breach event and to a command from an authorized user or security authority. Alternatively, for example, to limit the usefulness of the mobile device 800 to an unauthorized user, the mobile device 800 may be configured to overuse its components and drain the battery as quickly as possible.

[0050] The mobile device 800 may be configured to implement one or more security measures to protect data, restrict access, or provide other desired security features. For example, the mobile device 800 may encrypt transmitted data and / or data stored in or created by the device itself. Such security measures may be implemented using hardware, software, or a combination thereof. Any method of data encryption or data protection, such as public-key / private-key cryptography systems, data scrambling techniques, hardware and software firewalls, tamper-resistant or tamper-proof memory storage devices, or any other method or technique for protecting data, may be used in conjunction with the Invention. Similarly, passwords, biometric authentication, access cards or other hardware, or any other system, device and / or method may be used to restrict access to any device operating with the Invention.

[0051] The exemplary system shown in Figure 8A comprises a mobile device 800 which includes a processor 810 connected to memory 820 which may include volatile memory, non-volatile memory (such as flash memory), or a combination thereof. The communication module 830 comprises a radio transceiver 840 for wireless communication via antenna 850 with one or more servers and other entities, such as a host server 860, but those skilled in the art will understand that a wired connection may be established to provide a connection in lieu of or in addition to the wireless connection. The mobile device also comprises a user interface 870 connected to the processor 810. The mobile device 800 may have any suitable power source, such as a battery (not shown). The mobile device 800 may have any other desired components, such as a Global Positioning System (GPS) for providing geolocation information for determining the location of the mobile device. Some or all components of the mobile device 800 may include (or be configured to communicate with) hardware identification modules (not shown), such as a general-purpose subscriber identification module and / or a removable user identification module. The hardware identification module may be connected to the processor 810 and may include an identifier that can be compared against a predetermined identifier for determining whether the hardware of the mobile device 800 has been modified. The hardware identification module (and the predetermined identifier) ​​may include any appropriate identifier such as an electronic serial number, a local area ID identifier, an integrated circuit identifier, an international mobile subscriber identifier, an authentication key identifier, and / or an operator-specific emergency number identifier. This identifier may be stored in memory 820 and transmitted to the host server 860 for comparison against the predetermined identifier.

[0052] The functions of the mobile device 800, including those described herein (in whole or in part), may be implemented by the processor 810 executing computer-readable instructions stored in the memory 820 of the mobile device 800. The memory 820 may store any computer-readable instructions and data, including software applications, user-installed or third-party-installed "apps," applets, and embedded operating code.

[0053] Additionally, a software application may be configured to operate with minimal prerequisite hardware capabilities. For example, the application may be started before the mobile device establishes a network connection. Such a situation may be provided, for example, when a software application is installed on a SIM card in a mobile device, and the application starts before other software of the mobile device operating system. Alternatively or additionally, data elements such as links or URLs (Universal Resource Locators) may exist on the SIM card, and by launching an application such as a browser using such URLs or links, the application referenced by such links or URLs may be loaded from a remote server to the mobile device and / or executed directly from such remote server.

[0054] The software execution method of the present invention may be provided with the device or downloaded to the mobile device by an authorized user, and / or may further reside in the memory 16 of the host server 860 and be executable by the server processor 14. The functions of the mobile device 800 and the host server 860 may also be implemented by various hardware components that store machine-readable instructions, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) and / or complex programmable logic devices (CPLDs), graphics processing units (GPUs), and neural network processing or simulation circuits. Systems according to aspects of the present invention may operate with any desired combination of software and / or hardware components.

[0055] Processor 810 reads and executes instructions stored in memory 820 to control the operation of mobile device 800. Similarly, server processor 14 reads and executes instructions stored in server memory 16 to control the operation of host server 860. Any number and type of processors, such as integrated circuit microprocessors, microcontrollers, and / or digital signal processors (DSPs), may be used with the present invention. Memory 820 stores instructions, data, messages sent from (or received by) mobile device 800, and any other appropriate information, and server memory 16 similarly stores instructions, data, messages sent from (or received by) host server 860, and other appropriate information. Memory 820 and server memory 16 operating with the present invention may include any combination of different memory storage devices, such as hard drives, random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory. The data can be stored in memory 820 or server memory 16 in any desired manner.

[0056] The communication interface 830 communicates with one or more servers or other suitable entities, such as the host server 860. Similarly, the host server's communication interface 18 is configured to communicate with a mobile device 800, a general network such as the Internet, or any other suitable entity. Any suitable communication device, component, system, and method may be used in conjunction with the present invention. For example, the radio transceiver 840 may be configured to communicate using any number and types of cellular protocols, such as General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), GSM Evolution High Speed ​​Data Rate (EDGE), Personal Communication Service (PCS), Advanced Mobile Phone Systems (AMPS), Code Division Multiple Access (CDMA), Wideband CDMA (W-CDMA), Time Division Synchronized CDMA (TD-SCDMA), Universal Mobile Communications System (UMTS), and / or Time Division Multiple Access (TDMA). A mobile device operating with the present invention may optionally (or additionally) include a radio transceiver (and associated components) for communicating using any other radio communication protocol, such as the ISO 14443 protocol, ISO 18000-6 protocol, Bluetooth® protocol, ZIGBEE® protocol, Wibree protocol, Wi-Fi® protocol, IEEE 802.15 protocol, IEEE 802.11 protocol, IEEE 802.16 protocol, ultra-wideband (UWB) protocol, IrDA protocol, and combinations thereof. Furthermore, the communication interface 18 of the host server 860 may be configured to operate with such protocols for communicating with the mobile device 800 or any other device. The antenna 850 may be configured to transmit and receive any radio signal of any format, and may include multiple different antennas that transmit and receive using different radio protocols.

[0057] The communication module 830 can communicate with server 860 or another device using any other connection type, such as wired internet connection, wireless internet connection, cellular network connection (including data link connection), wireless LAN connection, wireless WAN connection, optical connection, FIREWIRE® connection, THUNDERBOLT® connection, LIGHTNING® port connection, e-SATA connection, USB connection, mobile device sync port connection, power connection, and / or security cable. The communication module 830 can be used to monitor the location or status of mobile device 800 (for example, by monitoring whether the communication link between the mobile device and the companion devices is functioning correctly) by communicating with one or more companion devices and any number of other devices that help track / locate a lost or stolen mobile device 800.

[0058] The mobile device 800 has a user interface 870. The user interface 870 may have any number of input devices (not shown) for receiving commands, data, and other appropriate inputs from the user, and any number of output devices (not shown) for providing data, notifications, and other appropriate information from the mobile device 800 to the user. Similarly, the host server 860 has a user interface 15, which may have any number of input devices (not shown) for receiving commands, data, and other appropriate inputs from the user or a third party, and any number of output devices (not shown) for providing data, notifications, and other appropriate information from the host server 860 to the user / third party.

[0059] The user interface 870,15 may be provided with any number of input devices, such as a touchpad, touchscreen, mouse / trackball / trackpad, microphone and / or alphanumeric keypad, so that the user can input commands and data to the mobile device 800 and the host server 860. For the purposes of this application, the term “touchscreen” may include a display integrated with or adjacent to a touch interface that can determine when the user applies a physical connection to a location adjacent to the display. The touchscreen may have sensors that can measure parameters from the user’s interaction, such sensors may measure measurements of capacitance, resistance, pressure or difference resulting from the movement of “touch” to the screen. The user interface 870 may be configured to detect the pressure a user applies to the keys of a keypad (which may be virtually implemented on the display or as a physical array of key switches) and the time intervals between multiple key presses in order to determine whether the current user is authorized to use the device. The user interface 870 may also include a microphone that allows the user to provide voice data to the mobile device 800, and may also include one or more cameras that allow the mobile device to capture still or moving images. Similarly, the user interface 15 of the host server 860 may include a microphone that allows the user to provide voice data to the host server 860, and may also include one or more cameras that allow the server 860 to capture still or moving images. In one embodiment, the mobile device 800 includes a front camera 874 that faces the user during device operation and a rear camera 872 located on the opposite side of the mobile device. The mobile device 800 may also include speech recognition software for processing voice input via the user interface 870.User interface 870 and similarly server user interface 15 may also include a display screen for visually displaying information (such as images and text) and / or any number of suitable output devices such as speakers for providing auditory output. The mobile device's display may be configured to sense user touch by any suitable means such as any suitable or conventional touch transmission technology available to those skilled in the art, such as capacitive sensing, pressure sensing, gel displacement sensing, or resistance sensing. Mobile device 800 may be configured to provide the user with words, phrases, tones, recorded music, or any other type of auditory output via speakers. As previously stated, user interface 870 can be activated to provide information and / or interfere with the operation of mobile device 800 when an unauthorized user attempts to use mobile device 800. For example, the display's illumination level may be changed to draw attention to the mobile device, and unpleasant and / or loud sounds may be played through the speakers.

[0060] The mobile device 800 may include one or more biometric authentication devices configured to receive biometric information, such as a fingerprint scanner, iris scanner, retinal scanner, and / or breath analyzer. Input devices such as microphones and cameras may be used to perform biometric analysis, such as voice analysis and facial recognition. Furthermore, the mobile device may include a magnetometer (of the type used in electronic compasses) for measuring magnetic fields, a MEMS or other type of gyroscope for measuring orientation, and an accelerometer for measuring changes in the movement of the mobile device.

[0061] The information provided or received by the user interface 870,15 may be in any appropriate format. For example, a user interface that conveys information to a user in an auditory format may first provide the user with a data header, followed by data values ​​to identify the data. The user interface 870,15 may provide information in any number of desired languages, regardless of whether the information is provided auditorily or visually.

[0062] User interfaces 870,15 may also provide and receive information to the user in a machine-readable format. In one exemplary embodiment of the present invention, for example, the user interface 870 of the mobile device 800 may send and receive messages using dual-tone multi-frequency (DTMF) tones. The mobile device 800 and the host server 860 can be configured to transmit, receive, and process machine-readable data in any standard format (such as MS Word documents, Adobe PDF files, ASCII text files, JPEG, or other standard formats) and other proprietary formats. Machine-readable data between the user interfaces 830,15 may be encrypted to protect the data from unintended recipients and / or improper use. In an alternative embodiment, the user may be required to enter a passcode to enable the use of some or all of the functions of the mobile device 800. Any other user interface functions may be utilized to enable human or non-human users to interact with one or more devices operating with the present invention.

[0063] The mobile device 800 may have any other appropriate functions, components, and / or systems. For example, the mobile device 800 may be configured to conserve its battery life by turning off some or all of its components, such as a camera or microphone. Components can be selectively shut down in response to a security breach event and to a command from an authorized user or security authority. Alternatively, for example, to limit the usefulness of the mobile device 800 to an unauthorized user, the mobile device 800 may be configured to overuse its components and drain the battery as quickly as possible.

[0064] The mobile device 800 may be configured to implement one or more security measures to protect data, restrict access, or provide other desired security features. For example, the mobile device 800 may encrypt transmitted data and / or data stored in or created by the device itself. Such security measures may be implemented using hardware, software, or a combination thereof. Any method of data encryption or data protection, such as public-key / private-key cryptography systems, data scrambling techniques, hardware and software firewalls, tamper-resistant or tamper-proof memory storage devices, or any other method or technique for protecting data, may be used in conjunction with the Invention. Similarly, passwords, biometric authentication, access cards or other hardware, or any other system, device and / or method may be used to restrict access to any device operating with the Invention.

[0065] The host server 860 communicates with the mobile device 800, authorized users, unauthorized users, security agencies (particularly insurance agents), and other entities to monitor the mobile device 800 and protect it from misuse, and to mitigate damage associated with security breach events or attempted misconduct. The host server 860 may include any number of individual computer systems, processors, and memory storage devices, as well as human operators (e.g., to respond to calls from authorized users reporting the loss / theft of a mobile device) and any other appropriate entities. The host server 860 may have, or communicate with, one or more databases 880 that store information about authorized users and the mobile device 800 in order to monitor and track the mobile device 800 and provide instructions to the mobile device 800 in the event of a security breach event.

[0066] For example, the database 880 may store usage profiles of mobile devices so that the host server 860 software can detect whether the continued use of the mobile device deviates from the usage profile according to a predetermined threshold, or whether the mobile device has caused a lost event that leads to a fault condition within the mobile device 800. The host server 860 may receive, process, and store information from the mobile device 800 (e.g., in the database 880). The host server 860 may handle any type of data in any format to achieve its purpose, such as receiving and processing environmental parameters captured by the mobile device to track the location and position of the mobile device 800, as described above. The database 880 may also store location information (e.g., the "whitelist" and "blacklist" described above) that can be used to determine whether the mobile device 800 is operating in a valid location. The database 880 may also store neural network training data, in which case exemplary images of defects in the mobile device are appropriately associated with identified defect classes, outputs, or states. The neural network training data may be expanded or otherwise modified to improve the accuracy of device defect recognition. Furthermore, the database 880 may store the results of health and operation analysis of the mobile device 800, along with identification information and operation history information related to the mobile device 800. In this way, the operation analysis of the mobile device may be tracked and analyzed over time by the server 860 to identify trends indicating ongoing failures of at least one component of the mobile device 800.

[0067] The database 880, which communicates with the host server 860, may also store archived data from the mobile device 800 for recovery purposes in the event that the mobile device 800 is lost or stolen, or that the data on the mobile device 800 is corrupted (for example, by a virus or other malicious program). The functions of the host server 860 may be performed automatically or semi-automatically via software / hardware running on one or more computer systems, and / or by one or more human operators.

[0068] The host server 860 may include one or more system processors 14 that search for and execute computer-readable instructions stored in memory 16 in order to (at least partially) control the operation of the host server 860. Any number and type of conventional computers, computer systems, computer networks, computer workstations, minicomputers, mainframe computers, or computer processors such as integrated circuit microprocessors or microcontrollers can be used in conjunction with the present invention. Computer systems used according to aspects of the present invention may include operating systems 43 (e.g., WINDOWS® NT / 95 / 98 / 2000 / XP / Vista / 7 / 8 / 10, OS2, UNIX®, Linux®, SOLARIS®, MacOS®, etc.) as well as various conventional support software and drivers typically associated with computers. In some embodiments, a dedicated application may be provided or executed by the system processor, all or partly, to perform the method of the present invention.

[0069] The neural network 47 may include a convolutional neural network (CNN) architecture, and may be implemented by software stored in the memory device 16 via external hardware (such as a hardware accelerator or graphics processing unit (GPU)) connected to the server processor 14, or by a combination of the stored software and external hardware.

[0070] The host server 860 may be accessed in any desired manner, such as via a website on the Internet and / or via a telephone network. The host server 860 may include any number of human operators, computer systems, mobile phones, mobile computing devices, interactive voice response (IVR) systems, and any other suitable systems and devices for communicating with users, security agencies, computing devices, or other entities. The host server 860 may communicate with an unauthorized user of a lost or stolen mobile device, either via the mobile device or via other means of communication, including direct communication with the fault condition test application (or "FSTA" as used herein) installed on the mobile device. The host server 860 may notify such unauthorized user that the mobile device is lost or stolen, provide the unauthorized user with recovery information (such as a shipping address), and forward test results and claim information to a third-party insurance company to facilitate the initiation of an insurance claim. The host server 860 also communicates with the mobile device 800 to provide software updates, receive data for archiving, identify files and other data to be protected, and perform any other aspects of the present invention.

[0071] The host server 860 may be controlled (locally or remotely) by or operate with any third party, such as an authorized user, a telecommunications service provider, a mobile device monitoring / tracking service provider, a security agency, an insurance agent, a mobile device repair agent, a mobile device vendor, an advertising agency, a mobile device warranty reseller, an authorized user of a mobile device, and / or other desired entity. For example, an authorized user and a security agency may communicate with or through the host server 860 to interact with a Fault Condition Test Application (FSTA) installed on the mobile device 800 to confirm that the device has experienced a failure that may qualify for an insurance claim. The host server 860 may be configured to provide notifications on how to return a lost / stolen mobile device 800, to detect security breach events, to detect that a failure condition has occurred on the mobile device, and to determine whether the functionality of the mobile device should be changed, and (if so) to determine how to change the functionality of the mobile device 800. The host server 860 may operate with any other desired systems, devices, human operators, or other entities.

[0072] The FSTA may collect information from the tested mobile device and relay it to a third party, or store it in the server 860's memory 16 (45) along with the device test results. The device data 45 may be uniquely associated with one or more images of the mobile device 46 and may be analyzed by the server processor 14. The device image data may be stored in any format, including but not limited to jpeg, TIFF, GIF, EPS, PNG, PDF, scalable vector graphics, bitmap, or any other conventional or non-conventional image data format. The device data 45 and the corresponding device images 46 may also be stored in the database 880. Information relayed to a third party may include any appropriate data, such as device type, manufacturer, model number, serial number, manufacturing date, hardware configuration list, memory capacity, software manifest, list of operable functions, list of non-operable functions, electronic serial number, ESN, IMEI number, International Mobile Device Identifier number, IMSI number, International Mobile Subscriber ID number, UIMID number, and user ID module identifier, test results, and failure status. The present invention can be further used in conjunction with embodiments of the present invention when a mobile device using a SIM card (such as a replaceable subscriber identification module commonly used in mobile phones) determines that at least one of the stored device configuration parameters contains an IMSI number within the device's SIM.

[0073] In an additional embodiment of the present invention, a user of a mobile device suspected of having a fault condition (such as a cracked screen / display) may be required to install an application (such as the fault condition test application mentioned above) on the suspected faulty mobile device. The FSTA provides a reliable method for evaluating the authenticity of the alleged fault condition (such as a cracked or damaged screen / display) on the mobile device. The FSTA interacts with the user of the mobile device to obtain sensor measurements and user input used to evaluate whether the mobile device is truly faulty. The fault condition / non-fault condition determination can then be displayed and / or transmitted in a secure manner (so that tests performed on the mobile device can provide reliable results) to a third party (such as an insurance company seeking insurance coverage for the mobile device). In one embodiment, data transmitted by the FSTA within the mobile device is encrypted to prevent tampering or impersonation by the user of the mobile device and is appropriately decrypted by the recipient or software running on the server 860.

[0074] Embodiments of the present invention for testing mobile devices also have a broader purpose than testing a mobile device for broken glass. For example, the entire digitizer could be tested when the screen is completely filled (this point will be described in more detail below). Furthermore, a darkened display / screen could also be tested by a point test (e.g., the sensor is working but the LCD is broken). Similarly, in order to verify that user input is accurate, in one implementation, in the normal fill mode, the user might try to fill the entire screen to cheat the test, but the user could not see the point at which the test is completed. Testing colors could also be considered. For example, the user could be instructed to press two displayed greens, then two reds, and finally two blues (as an example). Furthermore, analysis of images of the mobile device taken, either in a mirror or by a second device, may provide captured image data that can be analyzed for visual determination of the degree of damage, such as by a neural network, including a convolutional neural network.In this manner, various defects may be detected, including, for example, cracks or scratches on the mobile device display; patterns, cracks, chips, scratches or breaks on the mobile device casing; cracks in the bezel; worn areas on the mobile device casing; stacked or dead pixels on the mobile device display; LCD bleed on the mobile device display; pink defects on the mobile device display; objects obstructing the mobile device display (such as user body parts, stickers or other items); detected aspect ratios of the mobile device screen that do not correspond to the expected aspect ratio for the corresponding mobile device type; wear patterns on the mobile device that do not correspond to the expected usage profile for the mobile device; reflections on the mobile device display; smudges on the mobile device display; logos or other identifiers found in images of the mobile device that do not correspond to the expected manufacturer for the mobile device; separation of seams on the mobile device; missing components of the mobile device; and any other types of defects that can be determined from captured images of the mobile device.

[0075] Furthermore, in one embodiment, a company selling mobile devices may want to know immediately whether a currently used device has developed a malfunction (such as a cracked display) so that it can offer a new model to the device owner. This may be particularly relevant when a mobile network operator (MNO) provides devices for sale at a reduced price through the payment of monthly service plan fees over a contractually mandated period, for the MNO benefits from being able to renew and extend the service contract, while the device manufacturer also benefits from being able to sell new devices to replace the faulty ones. Additional or alternative system implementations of embodiments of the present invention include the following: - Mobile Device Insurance: Embodiments of the present invention provide verification of the condition of a phone in the used mobile device insurance market and the ability to support selective or variable insurance clauses relating to the verifiable condition / function of mobile device hardware (e.g., full insurance coverage for a mobile device with an uncracked screen, but coverage for a device with a cracked screen but otherwise functional, excluding glass breakage). Similarly, embodiments of the present invention provide verification that a phone being inspected is an insured phone (e.g., by transmission of mobile device ID information such as IMEI information or other data described herein as part of a test). Furthermore, embodiments of the present invention may be used to identify a mobile device as the correct insured device before it is returned for screen repair, in order to reduce false claims. Such a situation may occur with an insured device, for example, if a repair company does not actually repair the broken device and instead fraudulently splits the insurance money paid with the person who insured the phone. - Transfer of mobile devices: Embodiments of the present invention may be used to verify that the tested phone was actually handed over in the transfer of possession / ownership. - Verification of lease return condition: An aspect of the present invention provides for determining the conditions of lease return (for example, if a leased mobile device must be returned at the end of the lease, or if the condition of the mobile device falls below a predetermined agreed condition (i.e., the display must not be cracked at the time of lease return), the customer must pay an additional fee for the excess wear / damage). - Verification of Purchase / Shipping Condition: FSTA may also be used for verification by customers purchasing devices online (e.g., on eBay®) if the seller is not fully trusted. FSTA may also be used to determine the current level of damage, i.e., to what extent the display is cracked (in various embodiments, it is expected that the customer or company will pay an additional fee if the known screen condition is not so severely damaged as to be unacceptable). Additionally, embodiments of the present invention provide warehouse certification that the mobile device was in acceptable condition when it was shipped. When FSTA is used as a verifier in a situation where one person purchases a used mobile device from another source, the buyer may require the seller to install a trusted screen verifier app, enter the buyer's email address or SMS number into FSTA for verification, and FSTA performs a test and sends the results to the buyer, thereby verifying that the used device does not have any known defects (e.g., a cracked screen). Furthermore, this may also be desirable for sellers who want to verify that the device is working as expected before shipping, so that the seller will not be blamed for sending a faulty device to the recipient. - Determination of legal status: Aspects of the present invention may support cooperation with a third party (or police database) in verifying that a mobile device is not subject to loss / theft / unpaid charges.

[0076] Some embodiments provided herein, along with several system-based usage scenarios presented, include FSTAs for performing fault condition assessments, including FSTAs that perform pressure sensor tests, magnetometer tests, or pixel tests. While each type of FSTA test is designed to individually determine cracks, various types of crack detection techniques may be combined as appropriate for any purpose, such as improving accuracy. In preferred embodiments, the FSTA is installed on a mobile device and resides and runs within the mobile device, but those skilled in the art will understand that the FSTA may be executed in other ways, such as being deployed via an OTA (over-the-air) application, a website and sideloaded onto a connected mobile device, loaded via in-store hardware (e.g., Raspberry Pi®) or a dedicated in-store device, or loaded and executed from a vending machine in a mall or store. Furthermore, all or part of embodiments of the FSTA and system implementations may be provided in a library of apps used by entities providing online quotes for insurance, device trade-ins, or mobile device diagnostics.

[0077] Figures 9-13 illustrate one embodiment of the present invention in which a mobile device 800 is illuminated from the outside by a light source such as an LED light strip, in which the color of the light may be selectively changed to illuminate the mobile device and improve crack detection and analysis. In various embodiments, the light needs to be positioned to cause scattering from the cracks if the cracks are present in the glass. In various embodiments, an LED strip with RGB illumination may be used so that the color and intensity of the light can be changed as needed when multiple lights are required depending on the number of cameras used.

[0078] Experiments in which a curved background eliminates reflections from glass when the camera is aligned at a certain angle led to the configuration of the embodiment described above in Figure 9. The camera is aligned (at approximately 60°) so that the camera does not see itself in the glass. In various embodiments, if deformation is a problem, the camera can be calibrated using software such as OpenCV to bring out the inherent value of the lens and use it without distorting the image. The embodiments also use a calibration sheet placed in front of each camera to obtain the best image processing results. If the camera is the same in various embodiments, the same settings can be calibrated in-house and the calibration data can be shipped with the software so that the same settings do not require calibration.

[0079] Figures 9-14 show images taken using various light combinations to observe the effects of different lighting and the absence of lighting, demonstrating that lighting settings are effective in eliminating projections of natural reflections from the work area. From the photographs and embossed images, cracks can be seen under all lighting conditions, confirming that lighting is necessary to eliminate normal shadows and light reflections on the screen. By examining the embossed version of the image, the neural network of the present invention should be able to detect cracks in all illuminated images, regardless of whether humans can easily see those cracks or not.

[0080] Furthermore, with respect to the image processing steps on server 860, a particular embodiment includes the steps of: finding a target region in the image; finding the image of the region containing the mobile device by performing color masking or edge detection or both; cutting out all parts of the image that do not contain the mobile device; and transforming the image to address distortion or skew and obtain an appropriate angular representation. One optional step may be performed to emboss the image or equalize the channels. After these preprocessing steps are performed, the processed image is sent to a neural model (described in detail below) for further analysis. In summary, the neural model can be used to detect features in the image by comparison with pre-trained classes and analyze the LCD, housing, glass surface, and other elements, with the feature match returning a score on how well it matches each class. For example, in one embodiment, two classes are defined as a cracked mobile device or a non-cracked mobile device. Accordingly, the image is classified using a convolutional neural network as described below. After the image is classified, if it is damaged, a class-activating mapping is performed and data is returned via a neural network model. However, the model layers are examined, not the final score layer, but the layers to which the image is output into smaller sub-images, thereby providing the parts of the image where cracks or other defects have higher scores and where the model detected cracks in the image. As a result, an array of scores is obtained, and from this array and the regions of the image a heat map is generated, producing an overlay that will be displayed on top of the original captured image, such as a bounding box where damage may exist or colors superimposed on the image as shown in Figure 7, and then the server returns the image and Boolean Damaged == TRUE OR FALSE, and if TRUE, the image is returned for display as described above.

[0081] In one embodiment, the test results are currently stored in database 880 and made available via a website. Users may log in to this website to view the latest test or search for a device and view the original image and any intermediate images. A customer may then use these images to determine if any fraud occurred during the previous diagnostic test, such as someone attempting to conceal a crack on the screen or somehow concealing a crack.

[0082] During the diagnostic and image capture process, the embodiment may display a colored screen instead of a white screen to look for stuck pixels or dead pixels, and multiple images may be captured. While white is effective for displaying discolored screens, a white background containing stuck pixels may not be very suitable for identifying individual colored pixels that cannot be erased. For this reason, considering red, green, and blue images may be helpful. Furthermore, a black background on the mobile device's display may facilitate the determination of stuck pixels or any type of damage. In addition, there are multiple types of LCD bleeding. In type 1, liquid is leaking, and in type 2, the LCD light is bleeding through the edges, and a black screen may make it easier to detect at least type 2. Therefore, in various embodiments, displaying different background colors on the mobile device 800 and capturing multiple images may enhance the ability to detect additional defects.

[0083] In additional embodiments, additional elements such as light sources, lighting strips, microphones, speakers, vibrators, and other physical interfaces may be incorporated into the test apparatus. For example, vibration may be tested by a mobile device attempting to generate vibration listening to the 800's attempt, and if no vibration sound is heard, the test may indicate a failure. Further embodiments may assist the mobile device in testing whether it is functioning correctly by using a microphone incorporated into the test apparatus or by using an accelerometer incorporated into the test apparatus / phone for detection. Furthermore, a microphone on the mobile device may be assigned the role of listening to the phone's vibrations. Additionally, an external vibrator may be used to vibrate the surface of the mounting platform 205 itself to verify whether the accelerometer and gyroscope of the mobile device 800 are working and sensing what is intended. Furthermore, if the test apparatus 200 is configured to attach and move the mobile device, the gyroscope / sensor can be detected with respect to a 90-degree inversion, which can be analyzed in the same manner.

[0084] Another sensor of the mobile device (ambient light sensor) may be tested. The test apparatus may turn on an external light source and check whether the ambient sensor in the mobile device can see the light. Furthermore, the proximity sensor of the mobile device may be activated and it may be determined whether its measurement matches the height of the reflective surface on the mobile device 800. Furthermore, an NFC field generator and reader may be incorporated into the test apparatus so that the NFC interface of the mobile device can be tested. A wireless charger for the mobile device may be incorporated into the test apparatus for both supplying power to the mobile device during the test and for testing the proper operation of the mobile device's wireless charging circuit.

[0085] Wi-Fi® or Bluetooth® connectivity to the device may be tested, and the test system may broadcast a Wi-Fi® signal and determine whether the mobile device can acquire the SSID of the broadcasted signal. Furthermore, Bluetooth® signals transmitted from the mobile device may be used to control or activate any function of the test equipment, including the operation or rotation of a re-expansion surface or external light source, camera, or other equipment.

[0086] Furthermore, additional features include utilizing point exposure or spot exposure based on the phone model and its shape. The location of the screen's center is known, and an image of the screen's center is captured; the entire image then completes the stage. A new mapper is needed to find the screen's center, and known reference marks work. Knowing the screen's center is important because it may not be the center of the photograph. The phone itself, as well as its dimensions and color, is known. If the size of the bounding box is known, its size on the mobile device is examined for comparison.

[0087] Regarding image analysis, the primary components, green or blue, are the easiest to remove from an image, and therefore color analysis is useful in determining the appearance and characteristics of a casing or display that may indicate defects. For example, when analyzing the back of a phone, there may be multiple approaches, including using a backward-facing phone to capture an image of the mobile device in a reflective surface, and then analyzing the captured image to determine the back of the phone for glass breaks, metal scratches, or other defects.

[0088] Furthermore, if the mobile device under test is tethered, it can be determined whether the mobile device is engraved and whether it is a specific color. Certain smartphone brands may sell engraved phones, and therefore phones sold by manufacturers often have a firmware flag that includes a "yes / no" to whether the phone is engraved (but does not indicate what is engraved). It can be determined whether the mobile device is engraved via the tethered connection, and therefore the engraving can be excluded from defect analysis as a suspicious crack. Furthermore, if the captured image contains an engraving or other text, that text can be passed to an OCR analysis tool to determine the actual text on the mobile device (this can be important because personalization can significantly impact the resale price of used mobile devices). In addition, the use of OCR can help determine whether there are images that should be ignored because they cannot be detected by text. This technique may exclude text, serial numbers, etc., from crack detection.

[0089] Furthermore, gravity may be used to move the mobile device under test to and / or through the test equipment, and a standard conveyor belt may be used to use gravity to flip the device over and obtain an image of the back of the mobile device. In various embodiments, the device under test may be in motion and moved downstream on the conveyor belt, and an image of itself may be taken as it moves along the belt. Continuous or continuous movement can lead to significant benefits in processing time in the warehouse / recycler environment. In one embodiment, the conveyor may be paused for a sufficiently long time to take a photograph, and in an alternative embodiment, a lever may be inserted into the conveyor belt to flip the mobile device over. Furthermore, multiple camera settings, such as those shown in Figures 14, 15, and 15A, may be used to capture multiple images and angles of the mobile device (and in some cases, a 3D model of the mobile device's appearance may be constructed). The mobile device may move downstream on the conveyor and enter the box shown in Figures 15 and 15A, the mobile device passing through a partition such as a black curtain, the curtain lowering, the light source turning on, the camera taking a picture, and then the captured image being uploaded, and all tests being performed. Those skilled in the art will understand that it may be two or more devices to perform all tests. As previously mentioned, the conveyor may be used to capture a forward-facing photograph within a reflective surface, the conveyor may drop / turn the mobile device, and then a rear-facing photograph may be taken by a rear-facing camera, and the next device may be used. A unique reference mark displayed on the mobile device allows the mobile device to be uniquely tracked within a multi-phone environment. Furthermore, a QR code® or marker on the screen may be displayed with a value between 1 and 255, thereby giving a large combination of numbers that can be embedded in the image on the screen to identify which mobile device was moving along the conveyor line. The image can then be taken.Furthermore, in the conveyor environment (or in the 3D photo box in Figures 15-15A), an external camera may capture a side view of the mobile device. External lighting effects may be applied, as described above and shown in the lighting examples in Figures 9-13.

[0090] Device defect detection using neural networks Embodiments of this application utilize a convolutional neural network (CNN) architecture to analyze captured image, video, and / or audio files from a mobile device to identify defects and to provide analytical information, such as heat maps, to assist in further analysis of defect types. Those skilled in the art in the relevant field will understand that many different CNN architectures may be used to implement aspects of the present invention. One exemplary CNN architecture may be implemented by an Inception model implemented in Tensorflow®, and an explanatory example is disclosed, for example, at https: / / www.tensorflow.org / tutorials / image_recognition, which is incorporated herein by reference for all purposes. Alternative implementations of the present invention may be implemented by a Deep Residual Network (explanation available at https: / / blog.waya.ai / deep-residual-learning-9610bb62c355, which is incorporated herein by reference for all purposes) and / or by a convolutional network architecture implemented in an environment such as Torch (examples and explanatory text are available at http: / / torch.ch / docs / tutorials.html, which disclosure is fully incorporated herein by reference for all purposes). Furthermore, an introductory background on neural network architectures in general and in particular on convolutional neural networks (CNNs) is provided in "Neural Networks and Deep Learning" by Michael A. Nielsen (Determination Press, 2015; available at http: / / neuralnetworksanddeeplearning.com, which disclosure is fully incorporated herein by reference for all purposes).

[0091] Generally, CNNs are a type of model architecture that has been successfully used for image classification tasks. CNNs apply a series of filters to raw pixel data of an image to extract and learn higher-level features, and this model can then be used for classification. CNNs typically contain three components: (1) convolutional layers, (2) pooling layers, and (3) tightly coupled / fully coupled layers.

[0092] The convolutional layer applies a specified number of convolutional filters to the image. The convolutional layer performs a series of mathematical operations on each identified sub-region in the image, generating a single value in the output feature map. The convolutional layer then typically applies a ReLU (Normalized Linear Unit) activation function to the output to introduce nonlinearity into the model, although in various embodiments, a logistic-sigmoid function or a hyperbolic tangent activation function may be used.

[0093] Pooling layers reduce the size of feature maps by downsampling image data extracted by convolutional layers to shorten processing time. In practice, pooling replaces the net output at a given location with a summary statistic of neighboring outputs. A commonly used pooling algorithm is max pooling, which extracts subregions of the feature map, discards all other values ​​while retaining their maximums, thereby reporting the maximum output within a rectangular adjacency. Other possible pooling functions include the mean of rectangular adjacencies, a weighted mean based on the L2 norm of rectangular adjacencies or distance from the center pixel.

[0094] Tightly coupled (fully coupled) layers perform classification on features extracted by convolutional layers and those downsampled by pooling layers. In tightly coupled layers, each node in the layer is connected to each node in the preceding layer.

[0095] As introduced above, a CNN is constructed by stacking multiple convolutional modules that perform feature extraction, each module consisting of a convolutional layer followed by a pooling layer, with the final convolutional module followed by one or more tightly coupled layers that perform classification. The final tightly coupled layer in a CNN contains a single node for each target class in the model (all possible classes that the model can foresee), and each node has an activation function. A CNN allows a numerical value for a given image to be interpreted as a relative measure of how likely the image is to belong to each target class. In various embodiments of the present invention, target classes may include defects in mobile devices such as cracks in the display, scratches or cracks in the casing, LCD bleeding on the display, pink discoloration appearing on the display, stacked pixels on the display, and various combinations thereof.

[0096] In one embodiment using a neural network analysis technique, a system and method are provided for identifying cracks or other types of defects associated with mobile devices by image analysis. Referring to process 1600 described in Figure 16, the FSTA is installed on the mobile device 800 that is to be tested (1610) (Figures 14 and 15 also show user prompts and guidance for capturing an image of the mobile device reflected on a reflective surface such as a mirror). Once ready, the user of the mobile device 800 starts the device defect test (1615). An image of the display of the mobile device 800 is captured by the process described with respect to Figure 17 (1620), and the captured image is optionally preprocessed by rotating the image to portrait mode and / or by fine-tuning to reduce skew and / or by cropping the image to a desired aspect ratio (1625). The captured (and optionally pre-processed) image is then uploaded to a server (e.g., server 860) (1630), and the uploaded image is processed to determine whether any cracks or other types of defects are present, as will be further discussed with respect to Figure 18 (1635). Alternatively or in combination, for further analysis and reporting, the degree and characteristics of screen cracks or defects are characterized by processing step 1635. Such analysis of the image captured from the mobile (and optionally processed image) may include determining whether the captured image contains one or more defect markers. Such defect markers may be predefined in a pre-trained neural network into defect classes such as cracks, chips, tears, scratches, wear, abrasion, scratches, fraying, abrasion, stuck pixels, dead pixels, color errors, color bleeding, and deviated marks. Based on this analysis, a surface defect rating of the mobile device may be generated, which may indicate the relative visual condition of the mobile device as a whole.Such ratings may be useful in calculating the resale value of a mobile device and / or determining the insurable price or insurability criteria associated with the mobile device.

[0097] Once the analysis of any potential defects (1635) is complete, the resulting information is formatted for presentation, storage, or display to any user for any necessary purpose, such as evaluating whether the mobile device 800 is in proper working order (1640). The process then terminates (1645).

[0098] Figure 17 shows the process 1700 of the present invention for capturing an image of a mobile device, as introduced in step 1620 in Figure 16. The image capture process begins at 1705, where the FSTA analyzes ambient lighting from the lighting sensor in the mobile device 800 (1707). Then, at 1709, the screen brightness and camera sensitivity of the mobile device 800 are adjusted so that the mobile device's front camera (e.g., Figure 14, 874) can optimally track the tracking image displayed on the mobile device's display (e.g., Figure 14, 1400), and the user is instructed to move the device 800 to position 1715 so that the mobile device's display faces a reflective device such as a mirror. The FSTA's tracking algorithm identifies the mobile device and provides guidance to the user to move / rotate the mobile device 1720 from the mobile device 800's display until it is determined that the mobile device is in the optimal position to capture an image. As part of this process, the user may be instructed to rotate, tilt, move, or align the mobile device with respect to the surface of the reflective device. The FSTA uses data acquired from the device's front camera (e.g., Figure 14, 874) to determine when the mobile device is in an alignment acceptable for image capture (1725), and then the FSTA adjusts the sensitivity 1730 of the front camera for optimal image capture. In one embodiment, the image capture sensitivity may differ from the initial sensitivity used during tracking the position of the mobile device in front of the reflective surface. Depending on the operating system of the mobile device 800, different parameters within the mobile device may be modified to adjust the camera sensitivity. In one embodiment, the sensitivity is adjusted by changing the exposure compensation value associated with the camera in the mobile device 800. In another embodiment, the shutter speed and ISO value associated with the camera in the mobile device 800 may be adjusted to achieve the desired sensitivity.Subsequently, a captured image, such as a white screen or a screen with predetermined reference marks, may be output to the display of the mobile device 800 (1735). The image is then captured by the front camera of the mobile device 800 (for example, Figure 14, 874).

[0099] Those skilled in the art in a relevant field will also understand that the rear camera 872 of the mobile device 800 of the present invention may be used when facing a reflective surface to record an image of another surface of the mobile device 800, including, for example, the back surface of the mobile device housing. In this way, defects in the housing may be analyzed by the optical neural network analysis of the present invention to detect defects, wear patterns, chips, cracks, etc. in the housing.

[0100] Returning to Figure 16, after the captured image has been preprocessed (1625) and uploaded to the host server (1630), image analysis may be initiated (1635) on server 860. Figure 18 shows an embodiment of image analysis according to the present invention, and such analysis may be performed by server 860. Once the captured image is received by server 860 (1810), the captured image may be optionally downsized or resampled to reduce the size of the image file and improve processing efficiency. An exemplary captured image can be seen in Figure 19, where a mobile device 800 is shown, its display 1400 being handled by a user. The edges of the display 1400 of the mobile device 800 are located in the captured image 1815 by an alternative method.

[0101] In a first method for discovering edges corresponding to display 1400, an image thresholding technique is applied to a captured image (an example shown in Figure 19) to generate a thresholded image (an example of the image in Figure 19 after the thresholding technique has been applied can be seen in Figure 20). Arbitrary thresholding techniques are well understood by those skilled in the art in the relevant fields, and an exemplary tutorial is disclosed at https: / / docs.opencv.org / 3.3.1 / d7 / d4d / tutorial_py_thresholding.html, the disclosure of which is fully incorporated herein by reference for all purposes. Next, contours are discovered in the thresholded image, where contours include a defined curve having the same color or intensity in the image, formed by joining all continuous points (along the boundary). Since contours are a useful tool in shape analysis and object detection and recognition, contours may be used to assist in identifying areas of the captured image corresponding to the display 1400 of a mobile device. Contour analysis techniques are widely understood by those skilled in the art in the relevant fields, and an exemplary tutorial is disclosed at https: / / docs.opencv.org / 3.3.1 / d4 / d73 / tutorial_py_contours_begin.html, the disclosure of which is fully incorporated herein by reference for all purposes. Once contours are determined in the thresholded image, the four-point contour is analyzed and compared to the known aspect ratio of the mobile device being analyzed. Since device data 45 identifies the type of device, its display aspect ratio can be determined from existing lookup data (or externally from the device manufacturer) and may also be used in comparison to the aspect ratio of the four-point contour found in the thresholded captured image.Once a contour is identified in a thresholded image that is sufficiently similar in aspect ratio to the display aspect ratio defined for the specific mobile device under analysis (e.g., the aspect ratio is within a predetermined tolerance of the manufacturer's aspect ratio data), the positions of the matching contour vertices 2000 in the thresholded image are obtained, and the display edge in the captured image is identified as an area within the four vertices corresponding to the corners of the display 1400. Depending on the specific geometric features of the display associated with a particular mobile device manufacturer (e.g., rounded corners, notches that cut into the display, etc.), the edge boundary image may be further refined using known geometric tolerances from the manufacturer's device specifications.

[0102] In a second technique for identifying the edge position corresponding to the display 1400, reference marks having a predetermined color (as indicated by reference numerals 2101 to 2104 in Figure 21) are output to the display 1400 of the mobile device 800 immediately before image capture (Figures 17, 1735). The reference marks 2101 to 2104 are positioned close to the vertices of each corner of the display 1400, and image analysis techniques may be applied to determine the edges of the mobile device's display in the captured image. Multiple parts of the image are extracted using a color range, and since the color of the reference marks is known, elements of the image that appear within a predefined tolerance for the color of the reference marks appear in the extracted image. Next, polygons matching each shape of reference marks 2101 to 2104 are found in the image by a template matching method, for example, by locating the positions of shapes 2301 to 2304 in Figure 24, and the positions of the corresponding corner vertices in the captured image can be determined by the positions of each corner vertex in the captured image (see Figure 22). Similar to the first edge detection technique described above, the edges of the display in the captured image are thus identified as areas within the four vertices corresponding to the corners of the display 1400. In an alternative embodiment, any type of reference mark may be used to assist in identifying components of the mobile device, such as displaying a reference mark placed in the center of the mobile device's display and using it in conjunction with the aspect ratio of the display geometry associated with the device (for example, retrieved from a database of known mobile device manufacturers with associated specifications), and the display edges may be refined by identifying a search boundary centered on the reference mark and extending it to fit the outer boundary using the associated mobile device geometry (and aspect ratio).This method may be further refined. If a reference mark output on the mobile device's display is scaled to known dimensions (such as a cross with known dimensions, like 1 centimeter on each side), the edges may be deduced by obtaining a scale factor having the drawn size of the reference mark scale in the image. Then, by applying this scale factor to the manufacturer's specified dimensions for the mobile device's display edges, tolerances close to the outer boundary of the display can be determined. Furthermore, the shape of the reference mark (such as mutually orthogonal lines) may be used to desk the image so that edges with orthogonal geometric shapes (such as the sides of a rectangle) can be more preferably found.

[0103] Those skilled in the art will further understand that, as used in this application, the terms “display” or “screen” generally encompass the area on which a mobile device may output a visual image, but in other implementations of the present invention, “display” or “screen” may also include additional desired sections of the mobile device on which visually detectable defects may be found. More specifically, in some mobile phone models, the first area may be adjacent to and above the active display area, and the second area may be adjacent to and below the active display area of ​​a mobile device covered by a fragile material (such as hardened glass). While these adjacent first and second areas may not produce a visual output (for example, in older iPhone® devices where the first area has a speaker / camera / sensor orifice and the second area has a home button or microphone orifice), parts of these areas may still be damaged, and one feature of the present invention is to extend the image analysis of the mobile device's “display” or “screen” to these adjacent first and second areas to detect damage in these areas as well. In other embodiments, a “notch area” removed from the active portion of the display may be included in the display analysis, and in that sense, any desired area on the surface of the mobile device where visual defect detection is desired may be included. Additional test areas of the mobile device may therefore belong to the back of the mobile device, the logo area of ​​the mobile device, the side area of ​​the mobile device, the bezel area of ​​the mobile device, or any other area where visual defect detection is desired.In one embodiment, a number of defects, such as scratches and cracks, found to be located to a certain extent in close proximity to, or otherwise covering or touching, the logo area of ​​the mobile device (e.g., the "Apple" logo on an iPhone® model), may be identified and output as part of the defect analysis. Furthermore, in various embodiments of the present invention, specific defects in the bezel of the mobile device, such as cracks, dents, or holes, may be detected by visual analysis of an image of the mobile device. The image of the mobile device may be acquired in any desired manner, such as by capturing an image using a buddy or companion device equipped with a camera while holding any surface of the mobile device in front of a reflective device, or by placing / rotating the mobile device in front of a fixed camera or other device that captures images (and / or video representations) of each surface of the mobile device, and a 3D image representation of the mobile device may be optionally created. Furthermore, defects may include any type of visually detectable problem, and in addition to cracks and scratches, variations in display brightness, color variations (e.g., undesirable pink), LCD bleed, color defects, and any other desired type of visual defect may also be detected and identified. Additionally, the depth, width, and / or other visual appearance criteria of cracks / scratches may be associated with the identified cracks / scratches and may be reported with intensity levels to help classify the impact of the identified cracks / scratches on the value of the mobile device.

[0104] Since the edges of the display 1400 can be deduced from the positions of the corner vertices in the captured image, the area of ​​the captured image corresponding to the display area can be extracted as a sub-image and transformed into a rectangular shape in a single step. In one embodiment, this is achieved by calculating the moment of the display shape in the image by art known to those skilled in the art and then perspective-transforming the display portion into a rectangular image (an example of an implementation of the perspective transformation can be found at https: / / www.pyimagesearch.com / 2014 / 08 / 25 / 4-point-opencv-getperspective-transform-example / , the disclosure thereof being fully incorporated herein by reference for all purposes).

[0105] Once a subimage corresponding to display 1400 is obtained from the captured image, additional resizing / rescaling / resampling is performed so that the pixel dimensions of the subimage are acceptable as input to the neural network 47 of the present invention. The subimage can be resampled to any desired pixel configuration, and in one embodiment, the image is resampled to a 320 × 544 pixel configuration (in this example, width × height, corresponding to the portrait orientation of the subimage). An example of a perspective-transformed subimage is shown by reference numeral 1400A in Figure 20A, corresponding to the display portion of the captured image 1400 shown in Figure 19. Defect 1900 appears in both image 1400 and image 1400A.

[0106] Step 1830 describes the analysis of sub-images rescaled by a neural network to identify defects in the mobile device 800 and, in particular, defects in the mobile device's display 1400. The rescaled and perspective-transformed sub-images (e.g., 1400A in Figure 20A) are applied as input to a convolutional neural network pre-trained to detect defects using training data including the display and associated defect classes. As the network analyzes the images, it generates an output that identifies defects such as the colored polygon 1900A shown in Figure 20B. The colored polygons draw attention to the defects found by the neural network, and, as previously mentioned, additional outputs and analyses such as heat maps may be provided based on a particular network architecture. A more exemplary example follows Figures 20C to 20E, where an input sub-image (Figure 20C, 1400A) with a defect (Figure 20C, 1900) is input to the neural network 47 of the present invention, and an output (Figure 20E, 1400B) is generated which has a defect enclosed by a polygon (Figure 20E, 1900A) and a heat map output version (Figure 20D, 1400D) that describes a heat map 1900D corresponding to the activation class for the defect (Figure 20C, 1900) detected in the display sub-image (Figure 20C, 1400A). Another example is provided with a second input sub-image (Figure 20F, 1400F) that generates the output (Figure 20G, 1400G), a heat map showing the location of the detected defect class (Figure 20G, 1900G), and the output (Figure 20H, 1400H) with the defects enclosed by polygons (Figure 20H, 1900H). Those skilled in the art in the relevant field will understand that many other types of output are possible and that different defect types, as described above, may be detected by a well-trained CNN architecture.

[0107] Figure 16A illustrates another embodiment of the present invention, in which certain analysis steps described with respect to Figure 16 may be similarly implemented, but the embodiment of the present invention with respect to Figure 16A utilizes alternative methods. More specifically, in one embodiment, the present invention enables the detection of defects associated with a mobile device without the need to first install a Fault State Test Application (FSTA). More specifically, in step 1610A, contact is initiated between the mobile device under test and a remote diagnostic service (RDS), such as a web server, which has software and necessary communication circuits for connecting to the mobile device and is specifically configured to remotely perform tests to determine whether defects can be found within the mobile device. Such contact may be initiated in any desired manner, including by connecting from the mobile device to a web address, i.e., a database to a web address corresponding to the device diagnostic server, the mobile device user may be instructed to enter identification information, a user test profile for the specific mobile device may be created and stored in a database associated with the web server, along with the diagnostic results. Once the connection is established and the requested information is entered by the mobile device user, the user of mobile device 800 indicates to the remote diagnostic service that a defect test should be initiated. In one embodiment, the remote diagnostic service initiates the image capture process 1620A in a manner similar to the steps described with respect to Figure 16. However, an alternative embodiment includes activating a webcam on the mobile device or a companion device to capture an image of the mobile device under test directly or by the companion device, and similarly, for the mobile device under test, a browser running from RDS may open the device's rear camera and capture an image from either the front or rear camera, taking into account the reflective surface.When a companion device is used with the mobile device under test, a WebSocket connection may be initiated between the two devices to enable inter-device communication to instruct the device to display and capture images. In this way, the mobile device under test may be remotely instructed by the RDS via a browser on the mobile device to capture a photograph of itself reflected in a reflective device. Thus, the RDS can facilitate image capture from a webcam, from either the front or rear camera of the mobile device itself, from a companion device communicating with the RDS, and by other means by which the RDS can initiate image capture. The captured images are transmitted (1630A) via a communication link between the RDS and the mobile device (such as a connection to a pre-established web server), where the RDS initiates analysis in an embodiment similar to the steps shown in Figures 1635 and 1640. Those skilled in the art will understand that the RDS may also be implemented with an intelligent agent that can interact with the mobile device over a network, thereby enabling the test to obtain information to support defect analysis of such mobile device.

[0108] Additional defect tests (e.g., 1642A) may be performed using the mobile device under test, whether associated with the RDS or with an FSTA installed to perform the tests. For example, several tests may be performed (either by the FSTA or remotely via a connection between the mobile device and the RDS) to determine, for example: (1) vibration testing (e.g., determining if the device's vibration agitator is functioning), (2) speaker testing, (3) screen touch testing, (4) front camera testing, (5) rear camera testing, (6) screen pressure testing, (7) accelerometer testing, (8) gyroscope testing, (9) battery status including operating capacity, charging time, whether the battery is swollen (and therefore indicating a potential defect / hazardous condition), and (10) mixed model / origin recognition including association with a unique fingerprint or mobile device ID. Furthermore, aspects of the present invention provide acoustic analysis for determining defects associated with the mobile device.

[0109] Acoustic analysis allows a vibration signal (such as an audio signal or a known frequency mechanical vibration) to be generated, applied to a mobile device, and then "listened to" by the mobile device's microphone, which may consequently detect it. The captured audio signal from the microphone can then be further analyzed to determine whether any defects may exist in the mobile device. For example, a given signal may be output from the mobile device's speaker (or alternatively, by vibrations or vibration patterns induced by an agitator placed within the mobile device), and the mobile device then listens to the audio signature received by the mobile device's microphone via a microphone associated with the mobile device (or companion device). In one version, simple audio tests at various frequencies may determine whether the speaker / microphone pair is functioning in the desired frequency range. Furthermore, spectral analysis of the audio signal captured by the microphone may identify any defects in the speaker or microphone. The spectral analysis may further compare sound propagation to ideal propagation through a classified manufacturer-identified sample, and defects such as cracks in glass attached to the mobile device may generate discontinuities or rattles in sound propagation that can be deduced from the audio signal received from the microphone. Certain mechanical vibration patterns may also be generated by an agitator to attempt to discover resonant frequency rattles caused by loose parts within the mobile device that can be perceived by the audio pattern received by the device's microphone. Such spectral analysis may be performed in conjunction with a pre-trained neural network implementation to identify certain audio anomalies corresponding to defects that may be associated with the mobile device.

[0110] The process described in relation to Figure 17 captures an image of the display of the mobile device 800 (1620), and the captured image is optionally preprocessed by rotating it to portrait mode and / or by fine-tuning to reduce skew and / or by cropping it to a desired aspect ratio (1625). The captured (and optionally preprocessed) image is then uploaded to a server (server 860, etc.) (1630), and the uploaded image is processed to determine whether any cracks or other types of defects are present, as will be further discussed in relation to Figure 18 (1635). Alternatively or in combination, for further analysis and reporting, the degree and characteristics of any cracks or defects on the screen are characterized by processing step 1635. Once the analysis of any potential defects 1635 is complete, the resulting information is formatted for presentation, storage or display to any user for any necessary purpose, such as evaluating whether the mobile device 800 is in proper working order (1640). The process then terminates (1645).

[0111] Figure 24 shows one exemplary data flow relating to block diagram 2400 of the convolutional neural network (CNN) architecture of the present invention. Prior to the presentation of a device image or data 2405, components of the CNN are trained to recognize specific device defect classes, including defects such as scratches, cracks, and breaks on the device display. In one embodiment, a downscaled and resampled device image (e.g., an image file obtained by step 1820 in Figure 18) is presented to the input layer 2410 of the neural network (2405) along with any optional device data that can be used to process the image associated with the device (including, but not limited to, a device-associated IMEI, such as a device-associated unique identifier). The input layer is commutatically connected to block 2420 and includes at least one convolutional layer, a pooling layer, and a fully connected layer, as conventionally implemented in CNNs. The trained CNN operates on the input data to produce outputs corresponding to outputs indicating any detected defect class 2430. The operation may end at the point of analyzing the output of block 2430, or it may continue with additional processing by input to a second input layer 2440. Defects in block 2430 may be presented to block 2450, where the deconvolution layer, inverse pooling layer, and fully connected layer further process and enhance the features of the input image corresponding to the location where the defect is identified. The output layer 2460 provides the output in the form of a diagnostic image 2470, which may be stored in a database (e.g., Figures 8A, 880), and may also be associated in database 880 with device data that identifies the device image analyzed by network 2400.

[0112] Generally, to increase the performance and learning ability of a CNN model, the number of different network layers in a CNN can be selectively increased. The number of separate intermediate layers between the input and output layers can be very large, thereby increasing the complexity of the CNN model architecture. This disclosure utilizes a trained CNN model to identify device defects and / or create a visual map indicating the presence of localized defects within images of a mobile device. Consistent with the disclosed embodiments, the CNN system 2400 may be configured to produce an output indicating a defect (e.g., identified defect 1900A on device display 1400B in Figure 20E or identified defect 1900H on display 1400H in Figure 20H). Alternatively, embodiments of the present invention assume that the CNN system 2400 may be configured to generate an output showing a heat map corresponding to an area of ​​the image corresponding to the likelihood of the detected defect class (e.g., the identified heat map 1900D on the device display 1400D in Figure 20D or the identified heat map 1900G on the display 1400G in Figure 20G). For further explanation, image 1400A in Figure 20C may be presented as device image / data 2405 of network 2400, and Figures 20D and / or 20E may be obtained from the output of CNN 2400. Similarly, image 1400F in Figure 20F may be presented as device image / data 2405 of network 2400, and Figures 20G and / or 20H may be obtained from the output of CNN 2400. As described above, in certain embodiments, the output may be obtained from either block 2430 or block 2470 of network 2400.

[0113] As used herein, and as stated above, the CNN model used by the systems and methods of the disclosed invention may mean any neural network model formulated, adapted or modified based on the framework of a convolutional neural network. For example, a CNN model used to identify device defects in embodiments of this disclosure may selectively include intermediate layers between the input layer and the output layer, such as one or more convolution / deconvolution layers, nonlinear operator layers, pooling or subsampling layers, fully connected layers, final loss layers, deconvolution layers, and / or inverse pooling or upsampling layers.

[0114] In alternative embodiments, the output from the CNN model may be trained to generate either a heat map or identification of specific defects without a deconvolution layer requiring further processing after block 2430. Thus, in one embodiment, the output is created by a class activation map obtained by training the output layer of the neural network of the present invention, and in one embodiment, the heat map may be output as described above. In one embodiment, the heat map is generated by obtaining a mapping function for each section of the image output by the trained neural network of the present invention. The mapping function within each subsection of the image may use a score for the subsection of the image to identify a specific defect class when the trained network detects a defect in the image data. In one implementation of the present invention, the heat map may be obtained from a specific layer in the CNN model itself. As those skilled in the art will understand, for example, a softmax function may be used in the final layer of the CNN defect classifier, and the resulting layer element score may be calculated to generate a heat map, and the output may be generated in the form of image data, which may be superimposed on the area corresponding to the test image to identify the heat map. Alternatively, in one embodiment, the CNN may reduce or remove aspects of the input data image, apply different convolutional layers, capture the output of one of the convolutional layers (which may be output across the entire image), and generate a score for each part of the image, which in turn generates a defect activation map. Areas exceeding a predetermined level of activation may be converted to a specific color in the output image data to identify defects found on the test image, for example, by converting them to red (e.g., by reading the color assigned to the area of ​​the image corresponding to a display crack in the test image).In a further embodiment, a threshold (contrasted with a predetermined color threshold) may be applied to the image data in the network output layer to find the red portion of the image, and a minimum bounding rectangle may be placed around the red area to more specifically identify the defect area.

[0115] Figure 25 shows an overview of a flowchart 2500 for one embodiment of the present invention, which includes three stages: a training preparation stage 2525, a network training stage 2526, and a network inference stage 2527. As briefly described earlier, the disclosed systems and methods for identifying device defects generally include several stages described sequentially, for example, a training stage in which a CNN model is “trained” and “learned” using a training dataset, and a prediction or inference stage in which the trained CNN model is input with data to identify device defects (and / or defect classes) using the training dataset. As mentioned above, the aforementioned training stage generally correlates with network training 2526 as shown in Figure 25. As used herein, “training” a CNN model means determining one or more parameters of at least one layer in the CNN model. For example, the convolutional layers of a CNN model may include at least one filter or kernel. One or more parameters of at least one filter kernel, such as weights, size, shape, and structure, may be determined, for example, by a backpropagation-based training process. Furthermore, generally speaking, the description relating to Figure 24 above correlates with network inference 2527 as shown in Figure 25. When used in relation to the embodiments described herein, “predicting” from a CNN model means, in one embodiment, identifying device defects from images of the device using any desired type of processed images of the device 2510, including images of the device's display.

[0116] In the training preparation phase 2525, the training input 2520 is prepared and processed for use in training a CNN neural network component (e.g., Figure 8A, 47). A network model is selected for use in training (2505). Such model selection may include, for example, the identification of a convolutional neural network architecture and an appropriate processing layer configuration. The neural network component model (Figure 8A, 47) is initialized with an initial layer configuration, initial connection configuration, a set of weights, and a set of biases. In addition to the device images 2510 and each expected result (also known as "ground truth" data to those skilled in the art in the relevant field) 2515, other parameters (detailed below), such as identification information for each respective device image, may be set for input as training data 2520. In one embodiment, the training data 2520 may also include data about actually defective devices or disguised defective devices. In another embodiment, the training data 2520 may be synthetically created as computer-generated images based on theoretically possible defect types, for example, and may be designed for model testing purposes.

[0117] In various embodiments, and in embodiments similar to process 1800 in Figure 18, preparation step 2525 includes resampling all imaging data in training data 2520 to a common grid size and grid spacing. The image pixel values ​​can then be optionally resampled to a common scale across all image data in training data 2720, thereby improving learning performance and convergence based on the training data provided to the network. After data preparation, the training data and test data represent device images and correspond to identified defects, each including ground truth data.

[0118] The network training stage 2526 starts when the training data 2520 is presented to the neural network model (input not set) that has been configured and initialized from step 2505. The neural network component (Figure 8A, 47) estimates the result from the patient image data (2535) and generates an estimated result, e.g., a specified defect class. Then, a comparison is made between the result estimated from step 2535 and the expected result 2515 corresponding to the device image 2510, and an error map ("training error") is generated based on the difference between the expected result 2515 and the estimated result 2535. The error map is compared with the evaluation criterion 2545. For example, a loss function such as the mean absolute error function (MAE) or the mean squared error (MSE) function can be used to determine whether the error criterion is met. Those skilled in the art are proficient in various loss functions such as the mean absolute error (MAE) or L1 norm J(θ * ) = arg min θ ||Y = Y * ||1 or the mean squared error (MSE) or L2 norm J(θ * ) = arg min θ ||Y = Y * ||2, etc., and in the formula, θ * involves the selection of parameters that minimize the difference between Y and Y * . The backpropagation algorithm may be used to calculate the gradient of the error function with respect to the model parameters or weights. Then, θ may be updated by repeatedly using a stochastic gradient descent algorithm that converges to θ * .

[0119] If the error does not meet the error threshold criterion, for example, in order to minimize the error according to a learning algorithm (e.g., a regularization gradient descent optimization approach), the model parameters (e.g., weights and biases) of the neural network (Figure 8A, 47) are updated (2550), and the training data is re-presented to the neural network (Figure 8A, 47) together with the newly assigned model parameters of the neural network (2530).

[0120] In one embodiment, determining whether the criterion is met (2545) involves presenting one or more test device images different from the training images (but without the pre-identified defects) to the network model, which is currently configured to generate estimates of the defects corresponding to each of these test device images. The resulting defect classification can then be compared to the ground truth / predicted defect classification to evaluate the quality of the network parameter model. The difference between the predicted defects and the predicted defects for this test data is the "test error". In any case, the iteration from step 2550 to step 2530, to steps 2535 and 2540, and to step 2545 continues until the error threshold criterion is met, at which point the training process 2525 is terminated (2555), and the trained neural network component model is ready for use with captured images of real devices and corresponding data (Figure 8A, 45, 46).

[0121] In alternative embodiments, the neural network (Figure 8A, 47) may be trained in multiple steps, and different parts of the neural network may be trained using separate training datasets to achieve specific defect classifications. For example, the convolutional neural network of the present invention may be pre-trained to recognize a number of generic visual object classes, in which case the final layer of the network may be retrained to identify specific visible items such as screen defects (e.g., cracked or damaged displays on mobile devices). In one version of the present invention, the characteristics of a defect (e.g., a cracked display) may be provided in training data associated with training images, and in various embodiments, they may be provided without coordinates of the location where such cracks exist in the training data images, and in further other embodiments, the training data images showing defects (e.g., cracked displays) are annotated with coordinates that identify the cracked area in the image so as to optimize the training of the output layer. Utilizing multiple training passes, such as training a network with a large number of visual classes and then retraining the final layer of the network with specific defects to be detected, may provide an embodiment that offers efficiency advantages over retraining the entire network when the training data 2560 is updated or modified by new defect types or the amount of updated training examples.

[0122] Once trained, the neural network component model (Figure 8, 47) may be stored in memory such as server memory (e.g., Figure 8, 16), in non-temporary memory of the network component (Figure 8, 47), or in a database (e.g., Figure 8A, 880). The trained network model may then be used in network inference step 2527 to compute useful outputs (e.g., classified defects associated with the device) when presented with mobile device images and / or data (Figure 8A, 45, 46) for which defect analysis is desired. The trained neural network is constructed, for example, by loading stored weights and biases for a particular trained network configuration (2565). Patient data input is presented to the trained and constructed neural network component (Figure 8A, 47) to obtain prediction results. In a preferred embodiment, the prediction results include one of the defect classification using identified defects and one of the device classification using a displayed heat map. Thus, for a device of interest, device images and / or data 2560 are presented to a trained and constructed neural network model (2570) to generate prediction results (2575). In a further embodiment, the prediction results are sequentially reviewed by any desired entity, such as for determining the insurance eligibility of a mobile device or the validity of an insurance claim for a mobile device (2580).

[0123] The specific implementations described above illustrate the present invention and its best mode, and are not intended to limit the scope of the invention in any other way. In fact, for the sake of brevity, other functional aspects of conventional data storage, data transmission, and systems may not be described in detail. The methods shown in the various figures may include more steps, fewer steps, or other steps. In addition, the steps may be performed in any suitable order without departing from the scope of the invention. Furthermore, the connecting lines shown in the various figures are intended to represent exemplary functional relationships and / or physical connections between various elements. Many alternative or additional functional relationships or physical connections may exist in actual systems.

[0124] Modifications and alterations can be made to the embodiments disclosed without departing from the scope of the present invention. These and other modifications are intended to be included within the scope of the present invention as set forth in the appended claims.

Claims

1. A mobile device; a test fixture comprising a mounting platform mechanically connected via a support member to a top cover including a reflective surface; a server configured to communicate with the deployed mobile device, The mobile device memory may include instructions that, when executed by a processor of the mobile device, cause the mobile device to: indicating that the mobile device is ready to be placed in the test fixture; determining whether the mobile device is oriented in an acceptable manner relative to the reflective surface; responsive to detecting that the mobile device is oriented in an acceptable orientation, capturing an image of the mobile device reflected from the reflective surface and arranging the captured image for transmission to the server for defect analysis; said instructions causing the determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface; Displaying tracking and positioning guidance fiducial marks on the mobile device; and acquiring a tracking image of the mobile device from a camera of the mobile device; Displaying the tracking image within the placement guidance reference mark; determining an orientation of the mobile device within the tracking image based on a position and an orientation of an image of the tracking reference mark within the tracking image; determining whether the identified orientation is within a predetermined range; Including, the system.

2. Displaying tracking fiducial marks and placement guidance fiducial marks on the mobile device includes displaying the tracking fiducial marks on the mobile device, the display including: one or more QR codes; one or more Aztec codes; one or more ArUco codes, or The system of claim 1 , further comprising displaying as one or more graphical tracking elements.

3. The system of claim 1 , wherein the reflective surface comprises one or more of a mirror and a one-way mirror.

4. 1. A method for visually diagnosing a mobile device, comprising: indicating that the mobile device is ready to be placed in a test fixture comprising a mounting platform, a reflective surface; determining whether the mobile device is positioned on the mounting platform in an acceptable orientation with respect to the reflective surface; In response to detecting that the mobile device is oriented in an acceptable orientation, capturing an image of the mobile device reflected from the reflective surface; analyzing the captured image to determine whether the captured image includes a defect indicia indicative of whether the captured image includes one or more predefined defects; and providing a cosmetic defect rating for the mobile device based on the analysis. determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface; Displaying tracking and positioning guidance fiducial marks on the mobile device; and acquiring a tracking image of the mobile device from a camera of the mobile device; Displaying the tracking image within the placement guidance reference mark; determining an orientation of the mobile device within the tracking image based on a position and an orientation of an image of the tracking reference mark within the tracking image; determining whether the identified orientation is within a predetermined range; A method comprising:

5. 5. The method of claim 4, wherein the one or more defect indicia include indications of one or more of cracks, chips, tears, blemishes, wear, scratches, abrasions, gouges, stuck pixels, dead pixels, color errors, and color bleeding.

6. The method of claim 4 , wherein the reflective surface comprises one or more of a mirror and a one-way mirror.

7. a test fixture including a mounting platform mechanically connected via a support member to a top cover including a reflective surface, the reflective surface being positioned a predetermined distance from the mounting platform and at a predetermined angle relative to a top plane of the mounting platform; a server processor communicatively connected to the database; a server memory communicatively connected to the server processor; a server user interface communicatively connected to the server processor; and a server connected to the server processor and having a communication interface configured to communicate with a communication module of a mobile device disposed thereon, the located mobile device, a housing that substantially encloses the mobile device; and a mobile processor in communication with a memory of the mobile device; a user interface including a display and a first camera in communication with the mobile processor; a communication module configured to communicate with the mobile processor and to communicate with the communication interface of the server; The memory of the mobile device may include instructions that, when executed by the mobile processor, cause the mobile device to: indicating a readiness to begin testing of one of the display or the housing of the mobile device; Tracking fiducial marks, and Placement guidance fiducial marks including alignment indication presenting on said display an image comprising: adjusting the position and display of the mobile device; modifying brightness and camera sensitivity settings of the mobile device to optimize image capture; determining that the mobile device is acceptably oriented with respect to the reflective surface; capturing an image of the mobile device reflected from the reflective surface; sequencing the captured images for transmission to the server for defect analysis; said instructions causing the determining whether the mobile device is oriented in an acceptable orientation relative to the reflective surface; displaying the tracking fiducial marks and the positioning guidance fiducial marks on the mobile device; acquiring a tracking image of the mobile device from a camera of the mobile device; Displaying the tracking image within the placement guidance reference mark; determining an orientation of the mobile device within the tracking image based on a position and an orientation of an image of the tracking reference mark within the tracking image; determining whether the identified orientation is within a predetermined range; Including, the system.

8. The system of claim 7 , wherein the provided mobile device includes a second camera.

9. The system of claim 7 , wherein the reflective surface comprises one or more of a mirror and a one-way mirror.

10. The system of claim 7 , wherein the server is communicatively connected to the mobile device via one of a wired connection or a wireless connection.

11. The memory and the processor of the mobile device 1. Software that, when executed, performs the following: The tracking reference mark is displayed on the display; The system of claim 7 configured with the software, wherein the placement guidance reference marks are displayed on the display.

12. The system of claim 7 , wherein an image of the mobile device as seen by the camera is displayed on the display.

13. The system of claim 12, wherein a front camera captures an image of the display of the mobile device and a rear camera captures an image of the rear of the housing of the mobile device.

14. The system of claim 7 , wherein the mounting platform includes an alignment guide.

15. The alignment guide is An optical fiducial mark; an illuminated fiducial mark; one or more molded alignment guides; an electronic display connected to the test processor for displaying the placement fiducial marks and the alignment information; The system of claim 14 , comprising one or more of:

16. 8. The system of claim 7, further comprising capturing an image of the mobile device from the first camera after the mobile device is properly positioned on the mounting platform.

17. The system of claim 7 , further comprising: analyzing the captured image; and diagnosing at least one condition of the housing / display of the mobile device.

18. The system described in claim 7, wherein a height adjustment device enables the distance between the reflective surface and the mounting platform to be changed.

19. The system of claim 7 , wherein the support member is actuable to obtain a predetermined angle between the reflective surface and the mounting platform.

Images