Image device error detector
Automated analysis of image data using transformations and threshold comparisons addresses the challenge of diagnosing faults in portable electronic devices, enhancing repair efficiency and device reliability by accurately identifying issues in image capture and display devices.
Patent Information
- Application Number
- PCT/US2023/068452
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2023-06-14
- Publication Date
- 2024-12-05
AI Technical Summary
Portable electronic devices, such as mobile phones, face challenges in diagnosing faults in image capture and display devices due to fragile components like lenses and screens, which can lead to time-consuming and inaccurate manual repairs, and hinder efficient identification of defects such as cracks, blurriness, and color casts.
The implementation of automated systems that analyze image data using transformations like Fast Fourier Transform (FFT) and comparisons to threshold data to detect faults in image capture and display devices, including cracks, blurriness, and color casts, facilitating efficient and accurate diagnostic reporting.
Enables rapid and precise identification of faults in image capture and display devices, reducing repair time and improving device reliability by automating the diagnostic process, thus aiding in efficient maintenance and future device design improvements.
Smart Images

Figure US2023068452_05122024_PF_FP_ABST
Abstract
Description
IMAGE DEVICE ERROR DETECTORBACKGROUND
[0001] Users depend on image devices in their mobile phones and other portable electronic devices. For example, users depend on display devices that present images to be able to access information and operate their devices. Users similarly depend on image capture devices to take photographs, scan codes, and capture other image data with image capture devices incorporated in their portable devices. However, portable devices are often bumped or dropped, which may damage the display devices or image capture devices. The operability of these devices may be undermined by fragile glass components, such as lenses, lens covers, and screens, being broken, as well as by damage to electronic components that display or capture images. When a display device or image capture device is not functioning properly, it may be difficult and / or timeconsuming to accurately diagnose the fault to determine how to repair the device or whether to replace the device. Accurate diagnoses of faults also may aid manufacturers in trying to improve devices to reduce faults in later-generation devices.SUMMARY
[0002] This document describes systems and techniques for evaluating image data to identify a potential fault in an image device, such as an image capture device that captures image data or a display device presenting the image data. The systems and techniques apply one or more automated methods to the image data to detect cracks in the image capture device or the display device, detect blurriness in image data captured by the image capture device, detect a defect in an infrared cutoff filter of the image capture device, and / or detect a color sensing defect in an image sensor of the image capture device or a color cast defect in an image generator of the display device that causes discolored areas on the display device.
[0003] For example, a method is described that accesses image data from a device, where the image data includes an image of a reference object captured by an image capture device associated with the device or an output of a display device associated with the device. At least one transformation is applied to the image data to generate diagnostic data. The diagnostic data is compared to threshold data, where the threshold data represents an empirically determined indication of a fault in the device. Responsive to the comparing of the diagnostic data to the threshold data indicating the fault in the device, a report is generated communicating a presence of the fault in the device. For another example, an apparatus is described that includes at least one processor and computer-readable storage media storing instructions that, responsive to execution by the at least one processor, directs the apparatus to perform a method such as that previouslydescribed. For an additional example, a non-transitory computer-readable memory that stores instructions which, when executed by one or more processors, causes the one or more processors to perform a method such as previously described.
[0004] This Summary is provided to introduce systems and techniques for evaluating image data to identify a potential fault in an image capture device that captured the image data or in a display device displaying the image, as further described below in the Detailed Description and Drawings. This Summary' is not intended to identify essential features of the claimed subject matter, nor is it intended for use in determining the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS
[0005] The details of one or more aspects of systems and techniques for evaluating image data to identify a potential fault in an image device are described in this document with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:
[0006] FIGS. 1 and 2 are block diagrams of systems to evaluate image data to identify' a potential fault in an image device;
[0007] FIG. 3 is an exploded diagram of an image capture device of the system of FIG. 1 and potential faults associated with components of the image capture device;
[0008] FIG. 4 is a block diagram of the system of FIG. 1 configured to identify one or more cracks in a cover glass of the image capture device of FIG. 3;
[0009] FIG. 5 is a block diagram of the system of FIG. 1 configured to identify blurriness resulting from a flaw in components of the image capture device of FIG. 3;
[0010] FIG. 6 is a block diagram of the system of FIG. 1 configured to identify a defect in an infrared cutoff filter of the image capture device of FIG. 3;
[0011] FIG. 7 is a block diagram of the system of FIG. 1 configured to identify a defect in an image sensor of the image capture device of FIG. 3;|0012| FIG. 8 shows multiple sets of image data of a display device of the system of FIG. 2 representing potential faults associated with the display device;
[0013] FIG. 9 is a block diagram of the system of FIG. 2 configured to identify one or more cracks in the display device of FIG. 8;
[0014] FIG. lO is ablock diagram ofthe system of FIG. 2 configured to identify one ormore discolored areas in the display device of FIG. 8;
[0015] FIG. 11 is a block diagram of the system of FIG. 2 configured to identify a color cast defect in the display device of FIG. 8;
[0016] FIG. 12 is a block diagram of an example computing device configured to perform the diagnostic processes of FIGS. 4-7 and 9-11;
[0017] FIG. 13 is a block diagram of networked computing devices configured to perform the diagnostic processes of FIGS. 4-7 and 9-11;
[0018] FIG. 14 is a flow diagram of an example method of identifying one or more faults as described with reference to FIGS. 4-7 and 9-11;
[0019] FIG. 15 is a flow diagram of an example method of detecting cracks represented in image data as described with reference to the systems of FIGS. 4 and 9;
[0020] FIG. 16 is a flow diagram of an example method of detecting blurriness in the image capture device as described with reference to the system of FIG. 5;
[0021] FIG. 17 is a flow diagram of an example method of detecting an image capture device capturing image data with discolored areas as described with reference to the system of FIG. 7 or a defect in the display device as described with reference to the system of FIG. 10; and
[0022] FIG. 18 is a flow diagram of an example method of detecting a color cast defect in a display device as described with reference to the system of FIG. 11.DETAILED DESCRIPTIONOVERVIEW
[0023] Image capture devices or display devices in portable electronic devices may exhibit defects as a result of any number of faults in the devices. An image capture device may generate poor-quality images as a result of cracks in a cover glass, misalignment of lenses and / or a lens actuator, flaking of coatings in an infrared cutoff filter (IRCF), or failure of an image sensor. Similarly, a display device may present unclear images because of a cracked component, such as a cracked cover plate, backlight, color filter, or liquid crystal, or the display device may generate fully or completely miscolored images as a result of a damaged filter or liquid crystal.
[0024] To be able to efficiently and accurately repair or replace defective components, or to gather device reliability data, users, technicians, and / or manufacturers need to be able to readily diagnose faults in the image capture device or the display device. Manual diagnoses performed by technicians may be time-consuming. Also, a technician may mistakenly attribute a defect to a fault in one component when the fault may be in a different component. Thus, in the interest of efficiency and accuracy, it may be desirable to automate diagnostic procedures used to identify potential faults.
[0025] In various implementations, for example, a method comprises accessing image data from a device, where the image data includes an image of a reference object captured by an image capture device associated with the device or an output of a display device associated with thedevice. At least one transformation is applied to the image data to generate diagnostic data. The diagnostic data is compared to threshold data, where the threshold data represents an empirically determined indication of a fault in the device. Responsive to the comparing of the diagnostic data to the threshold data indicating the fault in the device, a report is generated communicating a presence of the fault in the device.
[0026] As further described below, in diagnosing an image capture device associated with a device, the device may be configured to operate as its own diagnostic system. Image data captured by the image capture device may be diagnosed by an application executing on the device. Alternatively, the image data may be transmitted over one or more networks to a separate diagnostic system for analysis of the image data. In diagnosing a display device associated with a device, image data of the display device may be captured by a secondary image capture device because the image capture device of the device may not be configured or manipulated to capture an image of the display device. The image data may be presented to the device and diagnosed on the device itself. Alternatively, the image data may be transmitted over one or more networks to a separate diagnostic system for analysis of the image data.
[0027] This document describes systems and techniques for evaluating image data to identify a potential fault in an image capture device that captured the image data or in a display device from which the image data was captured.EXAMPLE SYSTEMS
[0028] FIG. 1 illustrates a system 100 in which a diagnostic system 130 analyzes image data 120 of a device 110 to identify a potential fault in the device 110. As further explained below, the image data 120 may be captured by the device 110 or the image data 120 captured may represent an image of the device 110. A transformer 140 of the diagnostic system 130 applies one or more transformations to the image data 120 to generate diagnostic data 150. A comparator 160 of the diagnostic system 130 compares the diagnostic data 150 with threshold data 170. The threshold data 170 is empirically determined to identify an indication of a fault in the device 110. By comparing the diagnostic data 150 with the threshold data 170, the comparator 160 determines whether a fault is present in the device 110. Responsive to the comparator 160 finding an indication of the fault in the device 110, a report 180 is generated to communicate a presence of the fault in the device 110. Thus, by analyzing the image data 120, the diagnostic system 130 may identify' the presence of the fault in the device 110 to facilitate repair of the device 110 or to support a decision to replace the device 110.
[0029] FIG. 2 illustrates a system 200 depicting collection of the image data 120 (see FIG. 1) either from an image capture device 210 of the device 110 being tested or from a secondaryimage capture device 216 external to the device 110. Using the image data 120 captured by the image capture device 210 associated with the device 110, the diagnostic system 130 is able to analyze the image data 120 to identify one or more potential faults in the image capture device 210, as further described below. To facilitate analysis of the image data 120, the image capture device 210 may be used to capture an image of a reference object 212 having known visual properties, as described further below. Because the visual properties of the reference object 212 are known, variations of the diagnostic data 150 from expected data as indicated by the threshold data 170 may signify one or more potential faults in the image capture device 210, as further described below.
[0030] Alternatively, when the image data 120 is captured using the secondary image capture device 216 trained on a display device 214 of the device 110, the image data 120 may be analyzed to identify one or more potential faults in the display device 214. The secondary image capture device 216 may be part of the diagnostic system 130 or may be included in another device in communication with the diagnostic system 130 to provide the image capture data 120 to the diagnostic system 130, as further described below.
[0031] In various implementations, the diagnostic sy stem 130 may be included in the device 110 being evaluated. Thus, the diagnostic system 130 executing on the device 110 itself may be used to identify one or more faults in the image capture device 210 of the device 110. Similarly, if a separate sy stem (not shown in FIG. 2) associated with the secondary image capture device 216 communicates the image data 120 to the device 110, the diagnostic system 130 of the device 110 may be used to diagnose one or more faults in the display device 214 of the device 110. Alternatively, the diagnostic system 130 may be supported by a separate system (not shown in FIG. 2) that may receive the image data 120 from the device 110 being analyzed and / or that directly captures the image data 120 of the display device 214. In either case, the diagnostic system 130 may analyze the image data 120 and, upon identifying a presence of a fault represented in the image data 120, may generate the report 180 indicating the presence of the fault in the device 110. The report 180 may include a message displayed on the display device 214, displayed on the separate system, or transmitted to or generated by another system.DETECTION OF FAULT IN AN IMAGE CAPTURE DEVICE
[0032] FIG. 3 illustrates an exploded view of the image capture device 210 of the device 110 (see FIGS. 1 and 2) being analyzed and sets of image data 301, 321, 331, 341, and 351 that may be generated by the image capture device 210 depending on a condition of one or more components of the image capture device 210. The example image capture device 210 includes a cover glass 320, a lens system 330 that includes a lens actuator 332 and one or more lenses 334,an infrared cutoff filter (IRCF) 340, and an image sensor 350. Implementations herein described may be used with image capture systems that include fewer, additional, or different components.
[0033] For reference, when the image capture device 210 is functioning properly, the image data 301 is not corrupted by any faults in any of the components of the image capture device. However, different faults in one or more of the components of the image capture device 210 may result in image data that may be corrupted.
[0034] For example, when the cover glass 320 includes one or more cracks 322, the image data 321 may include one or more lines 323 that may corrupt the image data 321. If the lens system 330 includes a fault, such as the lens actuator 332 not being properly calibrated or a lens element 335 becoming dislodged or displaced to become a misaligned lens element 336 (as represented in dotted lines in FIG. 3), the image data 331 may be blurred. Also, if the IRCF 340 is damaged, such as if a portion of coating 342 has bubbled or flaked off, the image data 341 may include one or more corresponding discolored regions 343. If the image sensor 350 is faulty, the image data 351 may exhibit a color cast 353 (represented by hashed lines) that incorrectly records all or some of the colors in the image data 351.
[0035] Various implementations of the diagnostic system 130 (see FIGS. 1 and 2) may be configured to apply one or more transformations and comparisons to the threshold data 170 to particularly identify faults in one or more components of the image capture system 210. By identifying a particular fault, a user or technician (not shown) may receive the report 180 indicating which of the components may need to be repaired and / or replaced, without a user or technician having to spend time manually attempting to identify (or guess at) the source of the fault in the image capture device 210. The report 180 also may be useful to a manufacturer in improving components used in the image capture system 210 or in future designs.
[0036] FIGS. 4-7 depict tests that may be applied by the diagnostic system 130 (see FIGS. 1 and 2) using the transformer 140 and the comparator 160 to identify potential faults in the image capture device 210. The transformer 140 and comparator 160 may perform one or more of the tests - or all of the tests - described with reference to FIGS. 4-7 to identify one or more faults in the image capture system 210. In each of FIGS. 4-7, versions of a component under test without a fault and corresponding diagnostic data generated by the transformer 140 are shown toward the top of the corresponding figure; versions of the component under test exhibiting a fault and the corresponding diagnostic data generated by the transformer are shown toward the bottom of the corresponding figure. In the tests described with reference to each of FIGS. 4-7, for contrast, the tests performed are first described with reference to how the test would operate with a component of the image capture device 210 operating without a fault: the tests are next described with reference to how the test would operate with that same component exhibiting a fault. In each ofFIGS. 4-7, solid lines ending in arrows are used to represent a flow of data between elements, such as between the transformer 140 and the comparator 160, while dashed lines are used to show a correspondence in illustrative output of the transformer 140 or the comparator 160.
[0037] The tests illustrated in each of the examples of FIGS. 4-7 are performed using image data captured by the image capture device 210 of the reference object 212 in the form of a reference image 400. As shown in FIG. 4, the reference image 400 includes juxtaposed dark areas 412 and light areas 414 arranged, for example, in a black-and-white checkerboard pattern. Known attributes of the reference image 400 may be useful in identifying faults in the image capture device 210 that may manifest in image data captured by the image capture device 210. For example, contrast between the dark areas 412 and light areas 414 of the reference image 400 may aid in identifying cracks in a component of the image capture device 210, such as the cracks 322 in the cover glass 320. Also, a regular, known pattern of the reference image 400 may aid in identify ing when image data generated by the image capture device 210 is blurry as a result of a fault in the lens system 330. Known colors of the reference image 400 may be useful in identifying discolored areas that may result from a damaged IRCF 340 or a faulty image sensor 350. Although the reference image 400 is only expressly shown in FIG. 4, for the sake of example, it should be appreciated that image data captured by the image capture device 210 in each of the tests depicted in FIGS. 4-7 is based on image data captured by the image capture device 210 of the reference image 400.
[0038] As previously described, in testing the image capture device 210 of a device (not shown in FIGS. 4-7), it is assumed that the transformer 140 and the comparator 160 execute on the same device in which the image capture device 210 is included or that the image capture data captured by the image capture device 210 is communicated to a separate device (also not shown in FIGS. 4-7) where the transformer 140 and the comparator 160 operate on the image data. Thus, for example, the image capture device 210, the transformer 140, and the comparator 160 all may be included within a mobile phone or another device (not shown in FIGS. 4-7). Alternatively, the mobile phone in which the image capture device 210 is incorporated may capture image data of the reference image 400 and transmit the image data to a remote computing sy stem, such as a system at a service location, at the manufacturer, or at another location where the transformer 140 and comparator 160 may operate on the image data to diagnose a potential fault in the image capture device 210.
[0039] FIG. 4 depicts a diagnostic system 430 configured to identify whether a fault exists in a cover glass 420 or 421 in the form of one or more cracks 422 in the cover glass 421. No cracks exist in the cover glass 420. Thus, when the image capture device 210 with the cover glass 421 captures image data 424 of the reference image 400, the image data 424 should not includeany detectable faults as a result of the undamaged cover glass 420. The image data 424 should include an even distribution of dark and light pixels representative of an even distribution of dark areas 412 and light areas 414 in the reference image 400.
[0040] The image data 424 is presented to the transformer 140. The transformer 140 is configured to apply a transformation to the image data 424 in the nature of a Fast Fourier Transform (FFT) to generate diagnostic data 450 The resulting diagnostic data 450 includes a frequency domain rendering of the image data 424. Because the diagnostic data 450 is in a frequency domain of an even distribution of dark areas 412 and light areas 414 in the image data 424, the diagnostic data 450 includes two lines 455 that represent uniformity in the frequency distribution of dark and light pixels. Any one of a number of standard line tracking algorithms may be used to count a line quantity represented in the diagnostic data 450. The diagnostic data 450, including the line quantity, is presented to the comparator 160. The comparator 160 accesses threshold data 470 to evaluate whether the line quantity indicates a presence of a fault in the cover glass 420. Empirically, as previously described, there being only two lines in the diagnostic data450 indicates that there is no fault in the cover glass 420 that disrupts the even frequency distribution in the diagnostic data 450. Thus, for example, the threshold data 570 may include a line threshold of two. Thus, if the comparator 160 finds that the line quantity is not greater than two, the comparator 160 generates a report 480 indicating that there are no cracks and, thus, no fault in the cover glass 420.
[0041] By contrast, when the cover glass 421 includes the one or more cracks 422, the image capture device 210 generates image data 426 that includes representations 427 of the cracks 422 in the cover glass 421. The image data 426 is presented to the transformer 140, where application of an FFT yields diagnostic data 451 that, once again, is a frequency domain rendenng of the image data 426. The diagnostic data 451 includes the lines 455 that represent general uniformity in the frequency distribution of dark and light pixels. However, the diagnostic data 451 also includes four additional lines 457 that represent nonuniformity in the frequency domain distribution of dark and light pixels as a result of the cracks 422 in the cover glass 421 disrupting the uniform distribution of dark and light pixels in the diagnostic data 451. The diagnostic data451 is presented to the comparator 160 that accesses the threshold data 470, which, as previously described, indicates the presence of a fault when the line quantity included in the diagnostic data 451 is greater than two. The line quantity in the diagnostic data 451 includes six lines 455 and 457, including the two lines 455 representing the general uniformity of the dark and light pixels and the four additional lines 457. Thus, the comparator 160 generates a report 481 indicating that there is a fault in the cover glass 421 because the line quantity in the diagnostic data 451 is greater than two.
[0042] FIG. 5 depicts a diagnostic system 530 configured to identify whether a fault exists in the lens system 330 (see FIG. 3), such as in a case in which a lens element 535 becomes displaced to become a misaligned lens element 536 that may result in the image capture device 210 generating blurry images. Although the misaligned lens element 536 is used as a cause of blurring images in the example of FIG. 5, it will be appreciated that any fault in the lens system 330, such as a fault in the lens actuator 332 or any one or more of the lenses 334, may cause blurry images that may be identified by the test described with reference to FIG. 5.
[0043] When the lens system 330 includes the properly aligned lens element 535, the image capture device 210 captures image data 524 of the reference image 400 (see FIG. 4). For purposes of the example of FIG. 5, the image data 524 is considered to be in focus and, thus, non-blurry within design parameters of the lens system 330. The image data 524 is presented to the transformer 140, which is configured to first apply a Fast Fourier Transform (FFT) to the image data 524 to generate first-phase diagnostic data 550, similar to the FFT applied by the transformer 140 to the image data 424 in the example of FIG. 4. (The result of the FFT generated by the transformer 140 may be used in both the test of FIG. 4 and the test of FIG. 5 if the processes of both FIGS. 4 and 5 are applied to test the image capture device 210.) The transformer 140 is then configured to next apply a high-pass filter (HPF) to the first-phase diagnostic data 550 to generate second-phase diagnostic data 552. The HPF enhances edge detection of objects included in the image data 524 represented in the first-phase diagnostic data 550. The transformer 140 is then configured to apply an inverse Fast Fourier Transform (IFFT) to convert the second-phase diagnostic data 552 into third-phase diagnostic data 554. The resulting third-phase diagnostic data 554 represents the image data 524 having enhanced edges as compared to the image data 524.
[0044] The comparator 160 receives the third-phase diagnostic data 554 and calculates a mean blurriness score of the third-phase diagnostic data 554. The comparator 160 compares the mean blurriness score to a blurriness threshold in threshold data 570, which includes an empirically determined value of a mean blurriness score that differentiates between acceptably non-blurred image data and blurred data that indicates a fault in the lens system 330. When the mean blurriness score exceeds the blurriness threshold included in the threshold data 570, the comparator 160 generates a report 580 indicating that there is no fault in the lens system 330. Thus, for example, if the mean value is 34.2 and the threshold value is 20, the report 580 will indicate the lens system 330 is not generating blurry images and, thus, there is no fault in the lens system 330.
[0045] When the lens system 330 includes the misaligned lens element 536, the image capture device 210 captures image data 526 of the reference image 400. The image data 526 is presented to the transformer 140, which, as previously described, applies an FFT to generate first-phase diagnostic data 551, applies the HPF to the first-phase diagnostic data 551 to generate second-phase diagnostic data 553, and applies an IFFT to generate third-phase diagnostic data 555. As previously described, the resulting third-phase diagnostic data 555 represents the image data 526 having enhanced edges as compared to the image data 526 that, in a blurry image, may have blurred edges.
[0046] The comparator 160 receives the third-phase diagnostic data 555 and calculates a mean blurriness score of points in the third-phase diagnostic data 555. The comparator 160 compares the mean blurriness to the blurriness threshold in the threshold data 570. It will be appreciated that the mean blurriness score will be lower when the third-phase diagnostic data 555 is derived from a blurred image because contrast between points is diminished, as compared to when the third-phase diagnostic data 554 of the image data 524 is collected with the properly aligned lens element 535. When the mean value does not reach the blurriness threshold included in the threshold data 570, the comparator 160 generates a report 581 indicating that there is a fault in the lens system 330. Therefore, for example, if the mean value is 2 and the blurriness threshold is 20, the report 581 will indicate the lens system 330 is generating blurry images and, thus, there is a fault in the image capture device 210.
[0047] FIG. 6 depicts a diagnostic system 630 configured to identify whether a fault exists in the IRCF 340 (see FIG. 3) in the image capture device 210, such as in a case in which a flaked or an otherwise damaged coating results in one or more discolored portions in images captured by the image capture device 210. When the image capture device 210 includes an undamaged IRCF 640, the image capture device 210 captures image data 624 of the reference image 400 (see FIG. 4). For purposes of the example of FIG. 6, the image data 624 is assumed not to include discolored areas as a result of flaws in the IRCF 640. The image data 624 is presented to the transformer 140, which is configured to convert red, green, and blue (RGB) values to Hue Saturation Values (HSV) to generate first-phase diagnostic data 650. The transformer 140 then applies one or more masks to the first-phase diagnostic data 650 to mask off red and green areas in second-phase diagnostic data 654 (of which none exist in the second-phase diagnostic data 654) to identify one or more potential areas of defects in the IRCF 640. In various implementations, the transformer 140 is also configured to disregard any individual red and / or green areas in the second-phase diagnostic data 654 that are not of a specified minimum area in generating third-phase diagnostic data 658 to isolate one or more potential remaining areas of defect of a selected minimum size in the IRCF 640.
[0048] The comparator 160 receives the third-phase diagnostic data 658 and calculates a total area of any red and / or green areas in the third-phase diagnostic data 658. When the total area of red and / or green areas in the third-phase diagnostic data 658 does not meet a threshold areaspecified in threshold data 670, the comparator 160 generates a report 680 indicating that there is not a fault in the IRCF 640. For example, if the threshold area specified in the threshold data 570 is 5% of the area covered by the image data 624, because there are no red and / or green areas in the third-phase diagnostic data 658, the report 680 generated by the comparator 160 indicates that there is no fault in the IRCF 640.
[0049] However, when an IRCF 641 includes areas where the coating is damaged, including small damaged areas 643 and a large damaged area 645, image data 626 will include corresponding small discolored areas 627 and a large discolored area 629, respectively. The image data 626 is presented to the transformer 140, which converts RGB values to HSV in first-phase diagnostic data 651 that includes small areas 652 and a large area 653 corresponding to the discolored areas 627 and 629, respectively, in the image data 626. The transformer 140 then applies one or more masks to the first-phase diagnostic data 651 to identify corresponding small and red and / or green areas 656 and / or a large red and / or green area 657 in second-phase diagnostic data 655. In various implementations, the transformer 140 is also configured to disregard any individual red and / or green areas in the second-phase diagnostic data 655 that are not of a specified minimum area in generating third-phase diagnostic data 659. Thus, the third-phase diagnostic data 659 includes a large area 661 corresponding to the large red and / or green area 657 in the second-phase diagnostic data 655. It should be appreciated that, in various implementations, the small red and / or green areas 656 in the second-phase diagnostic data 655 may be disregarded by the comparator 160 instead of being masked off by the transformer 140.
[0050] The comparator 160 receives the third-phase diagnostic data 659 and calculates a total area of any remaining red and / or green areas in the third-phase diagnostic data 659, such as the large red and / or green area 661 signifying a fault in the IRCF 641. When the total area of the remaining red and / or green areas in the third-phase diagnostic data 659 meets the threshold area specified in the threshold data 670, the comparator 160 generates a report 681 indicating that there is a fault in the IRCF 641. In the example of FIG. 6, the area of the large red and / or green area 661 resulting from the IRCF defect encompasses more than 5% of the area encompassed by the image data 626 and, thus, meets the threshold area specified in the threshold data 670. Thus, the report 681 generated by the comparator 160 indicates that there is a fault in the IRCF 641.
[0051] FIG. 7 depicts a diagnostic system 730 configured to identify whether a fault exists in the image sensor 350 (see FIG. 3) in the image capture device 210 that results in captured images being discolored or having a color cast that is skewed from actual colors in an obj ect being imaged. When the image capture device 210 includes an undamaged image sensor 745, the image capture device 210 captures image data 724 of the reference image 400 (see FIG. 4). The image data 724 is presented to the transformer 140, which is configured to generate diagnostic data 750in the form of histograms for red, green, and blue (RGB) and grayscale values. The comparator 160 receives the diagnostic data 750 and determines whether black-white (B-W) peaks of the grayscale histogram differ from B-W peaks of the RGB histograms differ by more than a threshold amount as specified in threshold data 770. In the example of FIG. 7, the threshold amount included in the threshold data 770 is 10%.
[0052] The diagnostic data 750 shows that the B-W peaks of the grayscale histogram and the B-W peaks of the RGB histograms generally correspond. For example, B-W peaks of a red value histogram 752 and 754 (as well as green and blue histograms) are close to B-W peaks of the grayscale histogram 756 and 758 and, thus, the difference does not exceed the 10% difference threshold amount specified by the threshold data 770. Thus, the comparator 160 generates a report 780 indicating that there is not a color cast problem with the image capture device 210.
[0053] When the image capture device 210 includes a faulty image sensor 747, the image capture device 210 captures image data 726 of the reference image 400. The image data 726 is presented to the transformer 140, which generates diagnostic data 751 in the form of RGB histograms and a grayscale histogram. As shown in FIG. 7, in contrast to the diagnostic data 750 resulting from the functional image sensor 745, because the image sensor 747 has a color cast fault, the grayscale histogram and the RGB histograms do not generally correspond. The diagnostic data 751 is presented to the comparator 160. Again using red pixel values as an example, the comparator 160 determines that B-W peaks of the red value histogram 792 and 794 vary by more than the 10% threshold amount from B-W peaks of the grayscale histogram 796 and 798. Accordingly, the comparator 160 generates a report 781 indicating that there is a color cast fault in the image sensor 747.
[0054] In vanous embodiments, the diagnostic system 130 (see FIGS. 1 and 2) performs some or all of the tests described with reference to FIGS. 4-7 to determine whether faults exist in the cover glass 320, lens system 339, IRCF 340, or image sensor 350 (see FIG. 3) to diagnose each of these components of the image capture device 210 (see FIGS. 2 and 3). The diagnostic system 130 thus may diagnose each of the components of the image capture device 210 from a single set of the image data 120 captured from the reference image 400 (see FIG. 4).DETECTION OF FAULT IN A DISPLAY DEVICE
[0055] Similar to some of the tests used in diagnosing the image capture device 210 (see FIGS 1-7), one or more tests may be used by the diagnostic system 130 to identify faults in the display device 214 of the device 110.
[0056] FIG. 8 illustrates sets of image data 800, 810, 820, and 830 of the display device 214, some of which exhibit potential faults in the display device 214 that may be evaluated using thediagnostic system 130. As described with reference to FIG. 2 and as further described below, the image data 800, 810, 820, and 830 may be captured by the secondary image capture device 216 for evaluation by the diagnostic system 130.
[0057] For reference, the image data 800 represents the display device 214 in fully operational condition and exhibiting no faults. The image data 810 depicts one or more cracks 812 in the display device 214. The cracks 812 may be present in a top surface of the display device 214, in a liquid crystal display or another image generation layer, or in another layer of the display device 214. The image data 820 depicts a number of discolored areas 822 and 824, including small discolored areas 822 and a large discolored area 824, that may result from faults in the liquid crystal display or another image generation layer. The image data 830 presents a mostly or fully discolored screen 832 that may result from a fault in the liquid crystal display or another image generation layer.
[0058] Similar to the tests previously described with reference to FIGS. 4-7, FIGS. 9-11 depict tests that may be applied by the diagnostic system 130 using the transformer 140 and the comparator 160 to identify potential faults in the display device 214. The transformer 140 and comparator 160 may perform one or more of the tests - or all of the tests - described with reference to FIGS. 9-11 to identify one or more faults in the display device 214. In each of FIGS. 9-11, screen displays of the display device 214 under test without a fault and corresponding diagnostic data generated by the transformer 140 are shown toward the top of the corresponding figure; screen displays of the display device 214 under test exhibiting a fault and the corresponding diagnostic data generated by the transformer 140 are shown toward the bottom of the corresponding figure. In the tests described with reference to each of FIGS. 9-11, the tests performed are first descnbed with reference to how the test would operate with the display device 214 not exhibiting a fault; the tests are next described with reference to how the test would operate with the display device 214 exhibiting a fault.
[0059] As with the examples of FIGS. 4-7, in the examples of FIGS. 9-11, solid lines ending in arrows are used to represent a flow of data between elements, such as between the transformer 140 and the comparator 160, while dashed lines are used to show a correspondence in illustrative output of each of the components. In contrast to the examples of FIGS. 4-7, the diagnostic system 130 operates using image data of the display device 214 of the device 110 captured by the secondary image capture device 216, rather than image data of the reference image 400 captured by the image capture device 210 (see FIGS. 2-7) within the device 110. The transformer 140 and the comparator 160 may execute on the same device in which the secondary image capture device 216 is included, or the image capture data captured by the secondary image capture device 216may be communicated to the device 110 whose display device 214 is being tested or to another computing device for application of the tests described.
[0060] FIG. 9 depicts a diagnostic system 930 configured to identify whether cracks exist in the display device 214. The image data 800 represents the display device 214 without any cracks. The image data 800 is presented to the transformer 140, which, as in the example of FIG. 4, applies an FFT to the image data 800. Resulting diagnostic data 950 includes a frequency domain rendering of the pixels in the image data 924. The diagnostic data 950 includes two lines 955 that represent uniformity in a frequency distribution of dark and light areas as previously described with reference to FIG. 4. Any one of a number of standard line tracking algorithms may be used to determine a line quantity included in the diagnostic data 950. The diagnostic data 950, including the line quantity, is presented to the comparator 160, which accesses threshold data 970. In this test, for the sake of example, the threshold data 970 indicates a presence of a fault when the line quantity included in the diagnostic data 950 is greater than two. Because the line quantify included in the diagnostic data 950 signifies only two lines 955, the comparator 160 generates a report 980 indicating that there is no fault in the display device 214 represented in the image data 800.
[0061] When the display device 214 includes one or more cracks, the secondary image capture device 216 generates the image data 810 including the cracks 812. The image data 810 is presented to the transformer 140, where application of an FFT yields diagnostic data 951, which again is a frequency domain rendering of the pixels in the image data 810. The diagnostic data 951 includes lines 955 that represent general uniformity in the frequency distribution of dark and light pixels and lines 957 that represent nonuniformify in the frequency domain distribution of dark and light pixels as a result of the cracks 812. The diagnostic data 951 is presented to the comparator 160 that accesses the threshold data 970, which, as previously described, indicates a presence of a fault when a line quantify included in the diagnostic data 951 is greater than two. The diagnostic data 951 reflects a line quantify of six lines, including the lines 955 representing the general uniformity of the dark and light pixels and four additional lines 957 signifying a nonuniformify resulting from the cracks 812 present in the image data 810. Thus, the comparator 160 generates a report 981 indicating that there is a fault in the display device 214 because the number of lines found in the diagnostic data 951 is greater than two.
[0062] FIG. 10 depicts a diagnostic system 1030 configured to identify whether discolored areas exist in the display device 214 (see FIG. 2). The test for discolored areas on the display device 214 is similar to the test previously described with reference to FIG. 5 for identifying discolored areas captured by the image capture device 210 (see FIGS 1-7) when the image capture device 210 includes a fault on the IRCF 340. When the display device 214 does not includediscolored areas, the image data 800 is presented to the transformer 140, which is configured to convert RGB values to HSV to generate first-phase diagnostic data 1050. The transformer 140 then applies one or more masks to the first-phase diagnostic data 1050 to identify red and / or green areas in second-phase diagnostic data 1054 (of which none exist in the second-phase diagnostic data 1054). In various implementations, the transformer 140 is also configured to disregard any individual red and / or green areas in the second-phase diagnostic data 1054 that are not of a specified minimum area in generating third-phase diagnostic data 1058.
[0063] The comparator 160 receives the third-phase diagnostic data 1058 and calculates a total area of any red and / or green areas in the third-phase diagnostic data 1058. When the total area of red and / or green areas in the third-phase diagnostic data 1058 does not meet a threshold area specified in the threshold data 1070, the comparator 160 generates a report 1080 indicating that there is not a fault in the display device 214. For example, if the threshold area specified in the threshold data 1070 is 5% of the area covered by the image data 800, because there are no red and / or green areas in the third-phase diagnostic data 1058, the report 1080 generated by the comparator 160 indicates that there is no fault detected in the image data 800 of the display device 214.
[0064] By contrast, the image data 820 of the display device 214 may include the discolored areas 822 and 824, including the small discolored areas 822 and the large discolored area 824, when the display device 214 exhibits a fault. The image data 820 is presented to the transformer 140, which converts RGB values to HSV in first-phase diagnostic data 1051 that includes small areas 1052 and a large area 1053 corresponding to the discolored areas 822 and 824, respectively, in the image data 820. The transformer 140 then applies one or more masks to the first-phase diagnostic data 1051 to identify corresponding small red and / or green areas 1056 and a large red and / or green area 1057 in second-phase diagnostic data 1055. In various implementations, the transformer 140 is also configured to disregard any individual red and / or green areas in the second- phase diagnostic data 1055 that are not of a specified minimum area in generating third-phase diagnostic data 1059. Thus, the third-phase diagnostic data 1059 includes a large area 1061 corresponding to the large red and / or green area 1057 in the second-phase diagnostic data 1055. It should be appreciated that, in various implementations, the small red and / or green areas 1056 in the second-phase diagnostic data 1055 may be disregarded by the comparator 160 instead of being masked off by the transformer 140.
[0065] The comparator 160 receives the third-phase diagnostic data 1059 and calculates a total area of any remaining red and / or green areas in the third-phase diagnostic data 1059, such as the large red and / or green area 1061. When the total area of the remaining red and / or green areas in the third-phase diagnostic data 1059 meets the threshold area specified in the threshold data1070, the comparator 160 generates a report 1081 indicating that there is a fault in the display device 214. In the example of FIG. 10, the area of the large red and / or green area 1061 encompasses more than 5% of the area encompassed by the image data 820 and, thus, meets the threshold area specified in the threshold data 1070. Thus, the report 1081 generated by the comparator 160 indicates that there is a fault in the display device 214.
[0066] FIG. 11 depicts a diagnostic system 1130 configured to identify whether a fault exists in the display device 214 (see FIG. 2) that results in the display device 214 being miscolored, for example, because of a fault in the liquid crystal display or another image generation layer of the display device 214. When the display device 214 does not generate a miscolored image, the secondary image capture device 216 captures the image data 830. The image data 830 is presented to the transformer 140, which is configured to generate diagnostic data 1150 in the form of RGB histograms and a grayscale histogram. The comparator 160 receives the diagnostic data 1150 and determines an extent to which one or more B-W peaks of the RGB histograms differ from B-W peaks of the grayscale histogram. When the B-W peaks of the RGB histograms differ from the B-W peaks of the grayscale histogram by more than a threshold amount as determined in threshold data 1170, the comparator 160 determines there is a color fault in the display device 214. In the example of FIG. 11, the threshold included in the threshold data 1170 is 10%.
[0067] The diagnostic data 1150 shows that the RGB histograms and the grayscale histogram generally correspond. Thus, using red pixel values as an example, B-W peaks of the red histogram 1152 and 1154 are close to B-W peaks of the grayscale histogram 1156 and 1158 and thus do not exceed the 10% difference threshold amount specified by the threshold data 1170. Thus, the comparator 160 generates a report 1180 indicating that there is not a color cast problem with the display device 214.
[0068] When the display device 214 is faulty, the secondary image capture device 216 captures the image data 830, which includes the discolored screen 832. The image data 830 is presented to the transformer 140, which generates diagnostic data 1151 in the form of RGB histograms and a grayscale histogram. As shown in FIG. 11, in contrast to the diagnostic data 1150 resulting from the functional display device 214, because the display device 214 has a color cast fault, the RGB histograms and the grayscale histogram of the diagnostic data 1151 do not generally correspond. The diagnostic data 1151 is presented to the comparator 160. Again using the red pixel values as an example, B-W peaks of the red histogram 1192 and 1194 vary from B- W peaks of the grayscale histogram 1196 and 1198 by more than the 10% threshold amount included in the threshold data 1170. Accordingly, the comparator 160 generates a report 1181 indicating that there is a color cast fault in the display device 214.
[0069] In various embodiments, the diagnostic system 130 performs some or all of the tests described with reference to FIGS. 9-11 to determine whether faults exist in the display device 214 (see FIGS. 1 and 2). The diagnostic system 130 thus may diagnose one or more aspects of the display device 214 from a single set of the image data 120 captured from the display device 214.EXAMPLE OPERATING ENVIRONMENT
[0070] In various embodiments, the diagnostic system 130 (see FIGS. 1 and 2) is implemented in one or more computing devices operable to execute computer-executable instructions configured to perform the functions of the diagnostic system 130 as previously described. The one or more computing devices thus access the image data 120 and perform the functions of the transformer 140 in generating the diagnostic data 150 and perform the functions of the comparator 160, using the threshold data 170, in order to generate the report 180.
[0071] Referring to FIG. 12, and given by way of example only and not of limitation, some form of a computing system 1200 may be used to implement the functions of the diagnostic system 130 (see FIGS. 1 and 2). In various embodiments, the computing device 1200 typically includes at least one processor 1220 and a system memory 1230. Depending on the exact configuration and type of computing device, the system memory 1230 may be volatile memory, such as randomaccess memory (“RAM”), non-volatile memory, such as read-only memory (“ROM”), flash memory, and the like, or some combination of volatile memory and non-volatile memory . The system memory 1230, in various embodiments, includes instructions 1231, stored in a non- transitory computer-readable memory' that maintains computer-executable instructions for an operating system 1232 and one or more applications 1234, as well as program data 1236. The operating system 1232 may include any number of operating systems executable on desktop or portable devices, including, but not limited to, Android®, Apple OS®, Linux, Microsoft Windows®, or a proprietary operating system. The one or more applications 1234 may include a diagnostic application 1235 that performs the function of the diagnostic system 130, including a transformer module 1240 and a comparator 1260 that operate using threshold data 1270 maintained in the program data 1236. Image data and diagnostic data (not shown in FIG. 12) as previously described also may be maintained, temporarily or otherwise, in the program data 1236. The computing device 1200 also may include other removable storage 1245 or nonremovable storage 1255 for storing other application or program data.
[0072] The computing system 1200 may also have one or more input devices 1265 such as a keyboard, mouse, stylus, voice input device, touchscreen input device, etc., as well as the image capture device 210 orthe secondary image capture device 216. One or more output devices 1275 such as a display, speakers, a printer, etc. may also be included. The computing system 1200 alsomay include one or more communication systems 1285 that allow the computing system 1200 to communicate with other computing systems 1290 via wired or wireless communications, including, but not limited to, short-range transceivers such as a Bluetooth transceiver or a Wi-Fi transceiver, a cellular network transceiver, an Ethernet interface, or other communications devices. By way of illustrative example only and not of limitation, communication media may include wired media such as a wired network or direct-wired connection and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.
[0073] Referring to FIG. 13, the functions of the diagnostic system 130 (see FIGS. 1 and 2) may be performed in a networked environment 1300 across one or more computing devices 1310 and 1320. For example, if the device 110 is a mobile phone with a potentially defective image capture device 210, the diagnostic system 130 may reside and execute on the mobile phone. In such a case, the mobile phone may be a local computing device 1310 where all functions of the diagnostic system 130 are performed. Results, such as the report 180, may be transmitted via a network 1330 from the local computing device 1310 to a remote computing device 1320, which may be maintained, for example, by a manufacturer of the mobile phone, for tracking diagnostic procedures of hardware components. Alternatively, when a mobile phone is the local computing device 1310, the local computing device 1310 may capture the image data 120 to be used by the diagnostic system 130, which operates on the remote computing device 1320. The image data 120 may be communicated via the network 1330, such as a local wired or wireless communications network and / or another communications network, such as the Internet, between the local computing device 1310 and the remote computing device 1320. Results of diagnostics performed on the remote computing device 1320 may be communicated back to the local computing device 1310 via the network 1330, as well as communicated to other computing devices (not shown in FIG. 13).
[0074] Alternatively, the local computing device 1310 may include the secondary image capture device 216 that is used to capture the image data 120 of the device 110 with a potentially defective display device 214. The local computing device 1310 may then communicate the image data 120 via the network 1330 to the remote computing device 1320 to perform diagnostics. In this case, the local computing device 1310 may be a second mobile phone or another device used to capture the image data 120 and communicate the image data 120 via the network 1330. The remote computing device 1320 may be the device 110 under test performance of its own diagnostics (because the device 110 may not have the facility to capture the image data 120 of its own display device 214). Alternatively, the device 110 under test may represent neither the local computing device 1310 nor the remote computing device 1320. The local computing device 1310 may capture the image data 120 from the device 110 or receive the image data 120 from the device110. The local computing device 1310 may perform the diagnostics or may communicate the image data 120 via the network 1330 to another remote computing device 1320 to perform diagnostics as previously described.EXAMPLE METHODS
[0075] Example methods 1400, 1500, 1600, 1700, and 1800 are described with reference to FIGS. 14, 15, 16, 17, and 18, respectively, in accordance with processes used to test for potential faults in an image capture device or a display device as previously described with reference to tire foregoing figures. Generally, the methods 1400, 1500, 1600, 1700, and 1800 illustrate sets of operations (or acts) that may be performed in, but are not necessarily limited to, the order or combinations in which the operations are shown herein. Further, any of one or more of the operations may be repeated, combined, reorganized, skipped, or linked to provide a wide array of additional and / or alternate methods. In portions of the following discussion, reference may be made to the systems of FIG. 1 through FIG. 13, reference to which is made for example only. The techniques and systems described in this disclosure are not limited to an implementation or performance by one entity or multiple entities operating on one device or those described with reference to the figures.
[0076] FIG. 14 illustrates an example method 1400 of testing to determine presence of one or more faults in an image capture device or a display device of a device based on image data captured from the image capture device or of the display device, respectively, as described with reference to FIGS. 4-7 and 9-11. At a block 1402, image data from a device is accessed, where the device includes an image capture device and the image data includes an image of a reference object captured by the image capture device or the device includes a display device and the image data includes an output of the display device as described with reference to FIGS. 4 and 9, respectively.
[0077] At a block 1404, at least one transformation is applied to the image data to generate diagnostic data, as previously described with reference to FIGS. 4-7 and 9-11. As previously described, the one or more transformations may include applying an FFT, an HPF, or an IFFT, rendering a histogram of the image data, or one or more other transformations.
[0078] At a block 1406, the diagnostic data is compared to threshold data, the threshold data representing an empirically determined indication of a fault in the device. As previously described, such comparisons may include comparing a line quantity in diagnostic data to threshold data that that may indicate a presence of one or more cracks, comparing B-W peaks of histograms to determine color cast issues, etc.
[0079] At a block 1408, responsive to the comparing of the diagnostic data to the threshold data indicating a fault in the device, a report is communicated to indicate a presence of the fault in the device. As previously explained, if the comparison of the diagnostic data to the threshold data does not indicate the presence of a fault in the device, the report will not indicate the presence of a fault in the device. It will be appreciated that the example method 1400 may be applied multiple times to determine the presence of any number of faults as described with reference to FIGS. 4-7 and 9-11.
[0080] FIG. 15 illustrates an example method 1500 of determining whether one or more cracks exist in an image capture device or a display device. At a block 1502, image data from an image capture device is accessed, where the image data includes an image of a reference object captured by the image capture device (as described with reference to FIG. 4) or the image data represents an output of a display device (as described with reference to FIG. 9) and where a fault includes at least one crack in a cover glass of the image capture device or in the display device, respectively. At a block 1504, a Fast Fourier Transformation (FFT) is applied to the image data to generate diagnostic data. As previously described with reference to FIGS. 4 and 9, a line quantity represented in the resulting diagnostic data includes two lines for an undamaged cover glass or display screen and additional lines if there are one or more cracks in the cover glass or the display screen. At a block 1506, the line quantity in the diagnostic data is counted and presence of the fault is indicated when the line quantity exceeds a line threshold, such as maintained in the threshold data 470 or 970 as described with reference to FIGS. 4 and 9, respectively.
[0081] FIG. 16 illustrates an example method 1600 of determining whether an image generated by an image capture device is blurry, indicating a fault in a component of a lens system, such as a lens or lens actuator, as described with reference to FIG. 5. At a block 1602, image data from an image capture device is accessed, where the image data includes an image of a reference obj ect captured by the image capture device, wherein a fault includes a blurriness in data generated by the image capture device. At a block 1604, a Fast Fourier Transformation (FFT) is applied to the image data to generate diagnostic data. At a block 1606, a high-pass filter (HPF) is applied to the diagnostic data. As described with reference to FIG. 5, the result of the FFT may be regarded as first-phase diagnostic data and the result of the HPF may be regarded as second-phase diagnostic data. At a block 1608, an inverse Fast Fourier Transform (IFFT) is applied to the diagnostic data (so as to generate third-phase diagnostic data, as described with reference to FIG. 5) to generate an edge-enhanced representation of the original image data. At a block 1610, a mean blurriness score is determined from the diagnostic data and compared to a blurriness threshold, such as the blurriness threshold stated in the threshold data 570 (FIG. 5), wherein presence of the fault is indicated by the mean blurriness score exceeding the blurriness threshold.
[0082] FIG. 17 illustrates an example method 1700 of determining whether an image capture device or a display device presents one or more discolored areas as a result of a fault in an IRCF in the image capture device or in the display device generating an image, as previously described with reference to FIGS . 6 and 10, respectively . At a block 1702, image data from an image capture device is accessed, where the image data includes an image of a reference object captured by the image capture device (as described with reference to FIG. 6) or the image data represents an output of a display device (as described with reference to FIG. 10) and wherein a fault includes one or more discolored areas represented in the image data. At a block 1704, red, green, and blue (RGB) levels in the image data are converted to hue saturation values (HSV) in diagnostic data. At a block 1706, red and green areas in the HSV image are masked off to identify the one or more discolored areas in the diagnostic data. At a block 1708, in various implementations, any of the one or more discolored areas below a selected minimum size are disregarded to isolate one or more remaining areas. At a block 1710, a total size of one or more remaining areas is compared to a threshold area as stated in the threshold data 670 (FIG. 6) or 1070 (FIG. 10), and presence of the fault is indicated by the total size of the one or more remaining areas exceeding the threshold area.
[0083] FIG. 18 illustrates an example method 1800 of determining whether an entire image is miscolored as a result of a fault in an image sensor in an image capture device or a fault in a display device, as previously described with reference to FIGS. 7 and 11, respectively. At a block 1802, image data from a device is accessed, where the device includes an image capture device and the image data includes an image of a reference object captured by the image capture device and wherein a fault includes a color sensing fault in the image capture device or a color cast fault in the display device. At a block 1804, histograms for red, green, and blue (RGB) and grayscale values in the image data are generated as diagnostic data. At a block 1806, B-W peaks of the grayscale histogram are compared to B-W peaks of the RGB histograms to determine if the B-W peaks differ by more than athreshold amount as stated in the threshold data 770 (FIG. 7) and 1170 (FIG. 11).
[0084] The preceding discussion describes systems and techniques for testing image capture devices and display devices to automatically identify one or more faults. These systems and techniques may be realized using one or more of the entities or components shown in FIGS. 1-13, which may be further divided, combined, and so on. Thus, these figures illustrate some of the many possible systems capable of employing the described techniques.
[0085] Unless context dictates otherwise, use herein of the word “or” may be considered use of an “inclusive or,” or a term that permits inclusion or application of one or more items that are linked by the word “or” (e.g., a phrase “A or B” may be interpreted as permitting just “A,” aspermitting just “B,” or as permitting both “A” and “B”). Also, as used herein, a phrase referring to “at least one of’ a list of items refers to any combination of those items, including single members. For instance, “at least one of a, b, or c” can cover a, b, c, a-b, a-c, b-c, and a-b-c, as well as any combination with multiples of the same element (e.g., a-a, a-a-a, a-a-b, a-a-c, a-b-b, a-c-c, b-b, b-b-b, b-b-c, c-c, and c-c-c, or any other ordering of a, b, and c). Further, items represented in the accompanying figures and terms discussed herein may be indicative of one or more items or terms, and thus reference may be made interchangeably to single or plural forms of the items and terms in this written description.ADDITIONAL EXAMPLES
[0086] In the following section, additional examples are provided.
[0087] Example 1: A method comprising: accessing image data from a device, the image data including an image of a reference object captured by an image capture device associated with the device; or an output of a display device associated with the device; applying at least one transformation to the image data to generate diagnostic data; comparing the diagnostic data to threshold data, the threshold data representing an empirically determined indication of a fault in the device; and responsive to the comparing of the diagnostic data to the threshold data indicating the fault in the device, generating a report communicating a presence of the fault in the device.
[0088] Example 2: The method of example 1, wherein the fault includes at least one crack in a cover glass of the image capture device or in the display device.
[0089] Example 3: The method of example 2, wherein the applying of the at least one transformation to generate the diagnostic data includes applying a Fast Fourier Transform (FFT) to the image data.
[0090] Example 4: The method of example 2, wherein the comparing of the diagnostic data includes counting a line quantity in the diagnostic data and determining the presence of the fault is indicated by the line quantity exceeding a line threshold of the threshold data.|0091| Example 5: The method of example 1, wherein the fault includes a blurriness in output of the image capture device.
[0092] Example 6: The method of example 5, wherein the applying of the at least one transformation to generate the diagnostic data includes: applying a Fast Fourier Transform (FFT) to the image data; applying a high-pass filter (HPF) to the FFT of the image data; and applying an Inverse Fast Fourier Transform (IFFT) to a result of the HPF.
[0093] Example 7 : The method of example 6, wherein the comparing of the diagnostic data includes determining a mean blurriness score of the diagnostic data and comparing the meanblurriness score to a blurriness threshold, and determining the presence of the fault is indicated by the mean blurriness score not reaching the blurriness threshold.
[0094] Example 8: The method of example 1, wherein the fault includes an infrared cutoff filter (IRCF) defect in the image capture device.
[0095] Example 9: The method of example 8, wherein the applying of the at least one transformation to generate the diagnostic data includes: converting red, green, and blue (RGB) levels in the image data to a hue saturation value (HSV) image; masking off red and green areas in the HSV image to identify one or more areas of the IRCF defect; and disregarding any of the one or more areas of the IRCF defect below a selected minimum size to isolate one or more remaining areas.
[0096] Example 10: The method of example 9, wherein the comparing of the diagnostic data includes comparing a total size of the one or more remaining areas to a threshold area of the threshold data and determining the presence of the fault is indicated by the total size of the one or more remaining areas exceeding the threshold area.
[0097] Example 11 : The method of example 1 , wherein the fault includes a color cast defect in an image sensor included in the image capture device or an image generator in the display device.
[0098] Example 12: The method of example 11, wherein the applying of the at least one transformation to generate the diagnostic data includes generating red, green, and blue (RGB) histograms and a grayscale histogram.
[0099] Example 13: The method of example 12, wherein the comparing of the diagnostic data includes comparing black-white (B-W) peaks of each of the RGB histograms and the grayscale histogram and determining the presence of the fault is indicated by B-W peaks of the grayscale histogram differing from B-W peaks of the RGB histograms by more than a threshold amount.
[0100] Example 14: The method of example 1, further comprising executing an application on the computing device, wherein the application includes instructions executable on the computing device for the accessing of the image data from the device, the applying of the at least one transformation to the image data to generate the diagnostic data, the comparing of the diagnostic data to the threshold data, and, responsive to the comparing of the diagnostic data to the threshold data, the generating of the report communicating the presence of the fault in the image capture device of the computing device.
[0101] Example 15: The method of example 1, further comprising transmitting the image data from the computing device to a remote computing device, wherein the remote computing device is configured to execute instructions for the applying of the at least one transformation tothe image data to generate the diagnostic data, the comparing of the diagnostic data to the threshold data, and, responsive to the comparing of the diagnostic data to the threshold data, the generating of the report communicating the presence of the fault in the image capture device of the computing device.
[0102] Example 16: A system comprising: a transformer configured to receive image data including an image of a reference object captured by an image capture device associated with a device or an output of a display device associated with the device and to apply at least one transformation to the image data to generate diagnostic data; and a comparator configured to compare the diagnostic data to threshold data representing an empirically determined indication of a fault in the device and, responsive to the comparing of the diagnostic data to the threshold data indicating the fault in the device, to generate a report communicating a presence of the fault in the device.
[0103] Example 17: The system of example 16, wherein the fault includes at least one crack in a cover glass of the image capture device or the display device.
[0104] Example 18: The system of example 17, wherein the at least one transformation includes a Fast Fourier Transform (FFT).
[0105] Example 19: The system of example 17, wherein the comparator is configured to count a line quantity in the diagnostic data and determine the presence of the fault when the line quantity exceeds a line threshold of the threshold data.
[0106] Example 20: The system of example 16, wherein the fault includes a blurriness in output of the image capture device.
[0107] Example 21: The system of example 20, wherein the at least one transformation includes: a Fast Fourier Transform (FFT); a high-pass filter (HPF) applied to an output of the FFT; and an Inverse Fast Fourier Transform (IFFT) applied to an output of the HPF.
[0108] Example 22: The system of example 21, wherein the comparator is configured to determine a mean blurriness score of the diagnostic data and compare the mean blurriness score to a blurriness threshold, wherein the presence of the fault is indicated by the mean blurriness score not reaching the blurriness threshold.
[0109] Example 23: The system of example 16, wherein the fault includes an infrared cutoff filter (IRCF) defect in the image capture device.
[0110] Example 24: The system of example 23, wherein the at least one transformation includes: conversion of red, green, and blue (RGB) levels in the image data to a hue saturation value (HSV) image; masking off of red and green areas in the HSV image to identify one or more areas of the IRCF defect; and disregard of any of the one or more areas of the IRCF defect below a selected minimum size to isolate one or more remaining areas.
[0111] Example 25: The system of example 24, wherein the comparator is configured to compare a total size of the one or more remaining areas to a threshold area of the threshold data and the presence of the fault is indicated by the total size of the one or more remaining areas exceeding the threshold area.
[0112] Example 26: The system of example 16, wherein the fault includes a color cast defect in an image sensor included in the image capture device and an image generator in the display device.
[0113] Example 27: The system of example 26, wherein the applying of the at least one transformation to generate the diagnostic data includes generating red, green, and blue (RGB) histograms and a grayscale histogram.
[0114] Example 28: The system of example 27, wherein the comparing of the diagnostic data includes comparing black-white (B-W) peaks of each of the RGB histograms and the grayscale histogram and determining the presence of the fault is indicated by B-W peaks of the grayscale histogram differing from B-W peaks of the RGB histograms by more than a threshold amount.
[0115] Example 29: The system of example 16, wherein the system includes at least one computing device configured to execute a first set of computer-executable instructions to perform the functions of the transformer and to execute a second set of computer-executable instructions to perform the functions of the transformer.
[0116] Example 30: A non-transitory computer-readable memory storing instructions executable by one or more processors to: access image data from a device, the image data including an image of a reference object captured by an image capture device associated with the device; or an output of a display device associated with the device; apply at least one transformation to the image data to generate diagnostic data; compare the diagnostic data to threshold data, the threshold data representing an empirically determined indication of a fault in the device; and responsive to the comparing of the diagnostic data to the threshold data indicating the fault in the device, generate a report communicating a presence of the fault in the device.
[0117] Example 31: The non-transitory computer-readable memory of example 30, wherein the fault includes at least one crack in a cover glass of the image capture device or in the display device.
[0118] Example 32: The non-transitory computer-readable memory of example 31, wherein the instructions to apply the at least one transformation to generate the diagnostic data includes instructions to apply a Fast Fourier Transform (FFT) to the image data.
[0119] Example 33: The non-transitory computer-readable memory of example 31, wherein the instructions to compare the diagnostic data includes instructions to count a line quantity in thediagnostic data and to determine the presence of the fault when the line quantity exceeding a line threshold of the threshold data.
[0120] Example 34: The non-transitory computer-readable memory of example 30, wherein the fault includes a blurriness in output of the image capture device.
[0121] Example 35: The non-transitory computer-readable memory of example 34, wherein the instructions to apply the at least one transformation to generate the diagnostic data includes instructions to: apply a Fast Fourier Transform (FFT) to the image data; apply a high-pass filter (HPF) to the FFT of the image data; and apply an Inverse Fast Fourier Transform (IFFT) to a result of the HPF.
[0122] Example 36: The non-transitory computer-readable memory of example 35, wherein the instructions to compare the diagnostic data includes instructions to determine a mean blurriness score of the diagnostic data and to compare the mean blurriness score to a blurriness threshold, and to determine the presence of the fault when the mean blurriness score not reaching the blurriness threshold.
[0123] Example 37: The non-transitory computer-readable memory of example 30, wherein the fault includes an infrared cutoff filter (IRCF) defect in the image capture device.
[0124] Example 38: The non-transitory computer-readable memory of example 37, wherein the instruction to apply the at least one transformation to generate the diagnostic data includes instructions to: convert red, green, and blue (RGB) levels in the image data to a hue saturation value (HSV) image; mask off red and green areas in the HSV image to identify one or more areas of the IRCF defect; and disregard any of the one or more areas of the IRCF defect below a selected minimum size to isolate one or more remaining areas.
[0125] Example 39: The non-transitory computer-readable memory of example 38, wherein the instructions to the compare the diagnostic data includes instructions to compare a total size of the one or more remaining areas to a threshold area of the threshold data and to determine the presence of the fault when the total size of the one or more remaining areas exceeding the threshold area.
[0126] Example 40: The non-transitory computer-readable memory of example 30, wherein the fault includes a color cast defect in an image sensor included in the image capture device or an image generator in the display device.
[0127] Example 41: The non-transitory computer-readable memory of example 40, wherein the instructions to apply the at least one transformation to generate the diagnostic data includes instructions to generate red, green, and blue (RGB) histograms and a grayscale histogram.
[0128] Example 42: The non-transitory computer-readable memory of example 41, wherein the instructions to compare the diagnostic data includes instructions to compare black-white (B-W) peaks of each of the RGB histograms and the grayscale histogram and to determine the presence of the fault when B-W peaks of the grayscale histogram differ from B-W peaks of the RGB histograms by more than a threshold amount.
[0129] Example 43: The non-transitory computer-readable memory of example 30, further comprising instructions to transmit the image data to a remote computing device, wherein the remote computing device is configured to execute instructions to apply the at least one transformation to the image data to generate the diagnostic data, to compare the diagnostic data to the threshold data, and, responsive to the comparing of the diagnostic data to the threshold data, to generate the report communicating the presence of the fault in the image capture device of the computing device.
[0130] Example 44: a system comprising at least one processor and computer-readable storage media comprising instructions, responsive to execution by the at least one processor, for directing the apparatus to perform a method as recited in any one of examples 1 to 15.CONCLUSION
[0131] Although implementations for evaluating image data to identify a potential fault in an image capture device that captured the image data or in a display device that presents the image data have been described in language specific to certain features and / or methods, the subject of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations for implementing evaluation of image data to identify a potential fault in an image capture device or a display device.
Claims
CLAIMSWhat is claimed is:
1. A method comprising: accessing image data from a device, the image data including: an image of a reference object captured by an image capture device associated with the device; or an output of a display device associated with the device; applying at least one transformation to the image data to generate diagnostic data; comparing the diagnostic data to threshold data, the threshold data representing an empirically determined indication of a fault in the device; and responsive to the comparing of the diagnostic data to the threshold data indicating the fault in the device, generating a report communicating a presence of the fault in the device.
2. The method of claim 1 , wherein the fault includes at least one crack in a cover glass of the image capture device or in the display device.
3. The method of claim 2, wherein the apply ing the at least one transformation to generate the diagnostic data includes applying a Fast Fourier Transform (FFT) to the image data.
4. The method of claim 2, wherein the comparing of the diagnostic data includes counting a line quantity in the diagnostic data and determining the presence of the fault is indicated by the line quantity exceeding a line threshold of the threshold data.
5. The method of claim 1, wherein the fault includes a blurriness in output of the image capture device.
6. The method of claim 5, wherein the applying of the at least one transformation to generate the diagnostic data includes: applying a Fast Fourier Transform (FFT) to the image data; applying a high-pass filter (HPF) to the FFT of the image data; and applying an Inverse Fast Fourier Transform (IFFT) to a result of the HPF.
7. The method of claim 6, wherein the comparing of the diagnostic data includes determining a mean blurriness score of the diagnostic data and comparing the mean blurriness score to a blurriness threshold, and determining the presence of the fault is indicated by the mean blurriness score not reaching the blurriness threshold.
8. The method of claim 1, wherein the fault includes an infrared cutoff filter (IRCF) defect in the image capture device.
9. The method of claim 8, wherein the applying of the at least one transformation to generate the diagnostic data includes: converting red, green, and blue (RGB) levels in the image data to a hue saturation value (HSV) image; masking off red and green areas in the HSV image to identify one or more areas of the IRCF defect; and disregarding one or more areas of the IRCF defect below a selected minimum size to isolate one or more remaining areas.
10. The method of claim 9, wherein the comparing of the diagnostic data includes comparing a total size of the one or more remaining areas to a threshold area of the threshold data and determining the presence of the fault is indicated by the total size of the one or more remaining areas exceeding the threshold area.
11. The method of claim 1, wherein the fault includes a color cast defect in an image sensor included in the image capture device or an image generator in the display device.
12. The method of claim 11 , wherein the applying of the at least one transformation to generate the diagnostic data includes generating red, green, and blue (RGB) histograms and a grayscale histogram.
13. The method of claim 12, wherein the comparing of the diagnostic data includes comparing black-white (B-W) peaks of each of the RGB histograms and the grayscale histogram and determining the presence of the fault is indicated by B-W peaks of the grayscale histogram differing from B-W peaks of the RGB histograms by more than a threshold amount.
14. An apparatus comprising: at least one processor; and computer-readable storage media comprising instructions, responsive to execution by the at least one processor, for directing the apparatus to perform a method as recited in any one of claims 1 to 13.
15. A non-transitory computer readable memory storing instructions which, when executed by one or more processors, cause the one or more processors to execute any one of the methods of claims 1 to 13.
Citation Information
Patent Citations
Video quality diagnosis method and system
CN106993185A
Camera imaging defect detection method, display cabinet and storage medium
CN114202537A
Information processing system, information processing method and program
JP7220972B1
Detecting camera conditions to initiate camera maintenance
US20150356371A1
Computer apparatus and method for detecting defects in near-eye display
US20220207664A1
