Dynamic Interface Intervention to Improve Sensor Performance

Abstract

The computer-implemented method includes obtaining a physical location of a sensor on a device; determining that quality of data captured by the sensor physically located on the device is degraded based on a comparison of the captured data to a quality threshold; and displaying a visual indicator on a display of the device in response to determining that quality of the data captured by the sensor is degraded, the visual indicator including at least one non-textual component that indicates to a user the physical location of the sensor on the device.

Description

[Technical Field]

[0001] Users are increasingly able to capture video, images, and audio using their computing devices. For example, many personal devices (e.g., handheld and wearable devices) have microphones, cameras, and other sensors that can capture content. In addition, laptop computers, desktop computers, tablets, and the like are often also equipped with such sensors that can capture content. In addition, such electronic devices enable multiple forms of communication, including, for example, voice calls and video conferencing. However, one or more of the sensors on a device may be inadvertently covered or blocked by an obstacle, or a combination thereof, while capturing content (e.g., audio data, video data, or a combination thereof), resulting in suboptimal captured content. Summary of the Invention [Means for solving the problem]

[0002] Aspects of the present disclosure may include computer-implemented methods, computer program products, and systems. One example of the method includes obtaining a physical location of a sensor on a device, determining that quality of data captured by the sensor physically located on the device has deteriorated based on comparing the captured data to a quality threshold, and displaying a visual indicator on a display of the device in response to determining that quality of the data captured by the sensor has deteriorated. The visual indicator includes at least one non-textual component that indicates to a user the physical location of the sensor on the device.

[0003] The exemplary embodiments will be described with additional specificity and detail through the use of the accompanying drawings, with the understanding that the drawings illustrate exemplary embodiments only and are therefore not to be considered limiting in scope. [Brief explanation of the drawings]

[0004] [Figure 1] FIG. 1 is a block diagram of one embodiment of an exemplary device. [Figure 2] FIG. 2 is a block diagram of one embodiment of an exemplary computing device. [Figure 3] FIG. 3 is a flow chart diagram illustrating one embodiment of an exemplary method for improving sensor performance.

[0005] In accordance with common practice, the various features described are not drawn to scale, but are drawn to highlight particular features relevant to the exemplary embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0006] In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and in which specific exemplary embodiments are shown by way of illustration. However, it is to be understood that other embodiments may be utilized and that logical, mechanical, and electrical changes may be made. Furthermore, the methods illustrated in the accompanying drawings and the specification should not be construed as limiting the order in which individual steps may be performed. Therefore, the following detailed description is not to be construed in a limiting sense.

[0007] As used herein, "multiple" when used in conjunction with a reference item means one or more of the item. For example, "multiple different types of networks" refers to one or more different types of networks.

[0008] Furthermore, the phrases "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" means A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C. In other words, "at least one," "one or more," and "and / or" refer to any combination of items, and while a number of items may be used from a list, not all items in the list need be. The items may be particular objects, things, or categories. Additionally, the amount or number of each item in a combination of listed items need not be the same. For example, in some illustrative examples, "at least one of A, B, and C" may be, for example, without limitation, 2 items A; 1 item B; and 10 items C; or 0 items A, 4 items B, and 7 items C; or other suitable combinations.

[0009] Additionally, the word "a" or "an" entity refers to one or more of that entity. Thus, the words "a" ("an"), "one or more," and "at least one" can be used interchangeably herein. It should also be noted that the words "including," "comprising," and "having" can be used interchangeably.

[0010] Furthermore, as used herein, the term "automatic" and variations thereof refer to any process or operation that is performed without material human input when the process or operation is performed. However, a process or operation can be automatic if the performance of the process or operation uses material or non-material human input, but the input is received before the process or operation is performed. Human input is considered significant if it affects how the process or operation is performed. Human input consenting to the performance of the process or operation is not considered "critical." Additionally, the terms "in communication" or "communicatively connected" encompass the use of electrical or optical connections, whether wireless or wired, that enable two or more systems, components, modules, devices, etc., to exchange data, signals, or other information using any protocol or format.

[0011] As described above, many computing devices have sensors, such as cameras and microphones, that allow users to use the portable device to capture data, such as media content (e.g., images, video, audio), temperature data, light data, proximity data, and pressure data. In fact, many such devices include multiple sensors, such as those described above that include multiple sensors of the same type (e.g., multiple cameras or multiple microphones, or a combination thereof). However, such sensors may be blocked by an obstruction. For example, one or more sensors may be blocked by the user of the device (e.g., a hand or part of clothing may partially or completely cover a sensor). In addition, sensors may be blocked by other external elements. For example, dust may accumulate on a sensor, such as a microphone, thereby reducing the quality of the captured data. As used herein, the terms “blocked” and “obstructed” and variations thereof may be used interchangeably. Furthermore, as used herein, a sensor is considered to be blocked or obstructed if at least a portion of the sensor is covered or obstructed. As noted above, the sensor may be covered by clothing, a user's finger, dirt on the sensor, etc.

[0012] In addition, a sensor may be obstructed by a physical object, such as another person positioned in front of the sensor (e.g., a person standing in front of a camera obstructing the view of the scene captured by the camera). Such blockage or obstruction of the sensor may degrade the quality of the captured media, which may result in reduced user satisfaction. Additionally, the quality of the captured media may degrade if the user is too far from a sensor, such as a microphone in a video conference, or if the user speaks away from the microphone. Such actions may also degrade the quality of the captured media. The embodiments described herein enable proactive communication with the user to facilitate mitigating such causes of degradation in the quality of the captured data. As used herein, the term "captured data" refers to data obtained from one or more sensors on the device. Such data may be media data (e.g., audio, image, video), temperature data, light data, pressure data, proximity data, location data, etc. In the embodiments described below, specific reference is made to different types of media data for ease of explanation. However, it should be understood that the captured data can include data other than audio and visual data, and that the embodiments described herein can be implemented to facilitate compensation for the degraded quality of data captured by other types of sensors in addition to microphones and cameras.

[0013] FIG. 1 is a depiction of one embodiment of an exemplary device 100. While device 100 is depicted as a tablet or smartphone, it should be understood that embodiments of the present invention are not limited thereto. For example, in other embodiments, device 100 can be implemented as a desktop computer, a laptop computer, a wearable device, or the like. Device 100 includes a housing 104, a display 102, a camera 106, and a microphone 108. While only a single camera 106 and a single microphone 108 are shown in this example, it should be understood that in other embodiments, two or more cameras or two or more microphones, or combinations thereof, can be implemented. Similarly, in some embodiments, two or more displays can be implemented, such as a desktop computer with two or more attached display units.

[0014] Moreover, in embodiments in which device 100 has multiple sensors of the same type (e.g., multiple cameras or multiple microphones), each instance of the sensor can differ from the other instances. For example, a mobile device can be equipped with two or more camera sensors, where the characteristics of each camera sensor differ. For example, a camera sensor can have a higher pixel density than another on the same device, can be located on a different part of the device (e.g., the front side or the back side), can have a different focal length than another camera sensor, etc. Similarly, a device with multiple microphone sensors can have microphone sensors with different characteristics from each other.

[0015] Device 100 is configured to determine when the quality of data captured by a sensor falls below a predetermined level or is predicted to fall below that level. For example, device 100 can determine that a sensor capturing images, video, or audio, or a combination thereof, is blocked, or predict when such blockage will occur, or a combination thereof. For example, in some embodiments, device 100 can obtain data from one or more proximity sensors 132 to determine that a camera sensor or a microphone sensor, or a combination thereof, is blocked. Such proximity sensors 132 can be implemented as any suitable type of proximity sensor, including, but not limited to, a capacitive sensor, an inductive sensor, a magnetic sensor, or an optical proximity sensor, or a combination thereof. Such proximity sensors 132 can be positioned near a microphone or camera of device 100 such that they can detect when an object is proximate to the microphone or camera and potentially blocking the respective microphone or camera. Additionally, device 100 can use data from proximity sensor 132 to identify how far a physical object or obstacle is from the camera or microphone and whether the physical object may block the camera or microphone.

[0016] Additionally, in some embodiments, device 100 is configured to analyze image data and audio data as they are being captured to identify degradation in quality. For example, degradation in the quality of the captured data can indicate possible blockage of the corresponding sensor. Additionally, degradation in audio quality can indicate that a speaker or audio source is too far from device 100's microphone 108 for the microphone 108 to capture the audio at a high quality or volume. Accordingly, in some embodiments, device 100 is configured to detect changes in the quality of the recorded data. For example, device 100 can detect changes in the amplitude of audio data. Additionally, such degradation can be determined by comparing characteristics of the captured data to a baseline developed over time based on multiple previous data captured by the same sensor. In other words, if a sensor is blocked at the start of data capture, the blockage can be detected by comparing the data to the baseline; thus, the blockage can be detected even if there has been no significant change in the data since the start of data capture.

[0017] Moreover, the baseline can be used to account for variations in usage by different users. For example, a first user may speak relatively louder than a second user. Thus, a different respective baseline can be calculated for each of the first and second users. In some such embodiments, device 100 is configured to identify which user is currently using device 100. For example, device 100 can perform voice recognition analysis, image analysis, fingerprint analysis, or a unique passcode associated with each user, or a combination thereof, to identify which user is using device 100. Thus, device 100 can store different profiles for multiple users of device 100, where such profiles include, but are not limited to, respective baselines used to detect quality degradation, historical data associated with the user, etc. Moreover, in addition to or instead of using data collected from device 100, in some embodiments, baselines can also be calculated based on crowdsourced data obtained from other devices.

[0018] Additionally, historical data collected from sensors, such as the camera 106 and microphone 108, as well as other sensors, can be used to help predict when sensor blockage will occur. For example, based on historical data, device 100 can determine the likelihood that a given user will obstruct the microphone 108 when making a phone call by analyzing the way the user holds the device and the resulting pattern of obstruction of the microphone 108. In other words, device 100 can predict that when the user is answering a call or making a phone call, the user will typically hold device 100 in a manner that obstructs the microphone. Similar historical patterns can be identified for other sensors and activities, such as, but not limited to, taking pictures, participating in video conferences, etc. Moreover, in some embodiments, device 100 can be equipped with other sensors to help identify patterns in the historical data. For example, the device 100 may be equipped with one or more pressure sensors 130 that can provide data regarding a user's grip pattern that can be used to predict when the sensor will be obstructed.

[0019] Additionally, in some embodiments, the historical data can be further used to help identify potential causes of degradation in the quality of the sensor data. For example, if the historical data indicates that the quality of audio received via microphone 108 has gradually deteriorated over a period of time, this may indicate that lint or dust has accumulated on microphone 108. In contrast, if the deterioration occurs suddenly, this may indicate that microphone 108 is being covered by an object, such as clothing or a hand. Thus, by analyzing the historical data, device 100 can determine the relative likelihood of different causes of quality degradation.

[0020] Device 100 is further configured to provide guidance to a user of device 100 to mitigate detected degradation in quality. In particular, device 100 is configured to display one or more visual indicators, such as indicators 110, 112, and 114. Visual indicators 110, 112, and 114 are selected to help guide the user in mitigating the cause of the reduced or degraded data quality. For example, many devices include multiple components, such as multiple cameras, microphones, speakers, etc. The user may not know or be able to easily identify where each camera, microphone, etc. is located or which of multiple sensors is receiving degraded quality data. Furthermore, in some cases, the user may not be aware of the degraded quality. For example, while speaking in a video conference, the user may not be aware that their video is not being captured and transmitted to other participants in the video conference. This may occur, for example, if the user does not remove a sliding privacy shield covering the camera. Similarly, a user may not be aware that the audio being transmitted to other participants in a video conference or to other participants in a phone call is of degraded quality (e.g., muted, low volume, distorted, etc.) Accordingly, one or more of visual indicators 110, 112, and 114 can be used to alert the user to the detection of degraded quality and to instruct the user on mitigating its cause.

[0021] For example, visual indicator 110 includes multiple concentric partial circles 122 and icon 120. In this example, multiple concentric partial circles 122 are approximately centered near microphone 108. Similarly, icon 120 is positioned near the physical location of microphone 108. By positioning icon 120 and circle 122 near the physical location of microphone 108, visual indicator 110 can guide the user to the location of microphone 108. Moreover, icon 120 in this example depicts a generic microphone to indicate to the user that visual indicator 110 refers to the location of microphone 108. As described above, the user may not know or may not be able to easily identify where sensors are located or which sensor is obstructed. Therefore, by positioning indicator 110 near the physical location of microphone 108, the user can easily locate the sensor that is detected as inputting degraded data. 1 as a static visual representation, in some embodiments, visual indicator 110 is not a static visual representation. That is, visual indicator 110 may be animated, such as by changing color, color intensity, or brightness in a pattern, such as a pulsating or wave pattern, to orient the user to the location of microphone 108. It should be understood that visual indicator 110 is provided by way of example only. Notably, in other embodiments, visual indicator does not include icon 108 or concentric circles 122, or a combination thereof. In still other embodiments, the visual indicator may also include other components in addition to or in place of icon 108 or concentric circles 122, or a combination thereof.

[0022] For example, visual indicator 112 includes icon 116 and arrow 118. Thus, visual indicator 112 does not include concentric circles 122. Rather, icon 116 and arrow 118 are positioned proximate to the physical location of the sensor (i.e., camera 106 in this example). Additionally, arrow 118 points to the location of camera 106. Like visual indicator 110, visual indicator 112 can also be animated, changing color, intensity, brightness, etc., to orient the user to the physical location of the sensor. Moreover, the implementation of a visual indicator need not be located on display 102 proximate to the physical location of the corresponding sensor. For example, visual indicator 114 is a depiction of device 100 with labels 126 and 128 to indicate the location of each sensor.

[0023] In some embodiments, visual indicators 110, 112, or 114, or a combination thereof, can be displayed on display 102 by overlaying visual indicators 110, 112, and 114 on other content displayed on display 102. In other embodiments, content displayed on display 102 is minimized so that a new screen or window is opened to display one or more of visual indicators 110, 112, or 114. Additionally, multiple visual indicators can be displayed simultaneously. For example, indicator 114 showing a diagram of device 100 can be displayed simultaneously with visual indicator 110 or visual indicator 112, or a combination thereof. Furthermore, other information can be displayed along with one or more visual indicators. For example, a text list of possible causes or possible steps a user should take, or a combination thereof, can be displayed on display 102 along with one or more visual indicators that orient the user to the location of a given sensor or provide other visual guidance to the user. Such a text list can be ordered based on the probability or likelihood of the cause of the degradation as determined by device 100 based on historical data and data from other sensors, such as sensor 130 or sensor 132, or sensors 130 and 132. In another example, the text information can confirm or orient a user to the location of a sensor. For example, if a given sensor is located on a different side of device 100 than display 102, the text information can convey this information to the user.

[0024] Additionally, other variations of visual indicators may provide other guidance to the user in addition to or instead of the location of a given sensor. For example, a picture of a hand indicating for the user to move closer may be displayed on display 102 so that the user knows that device 100 is too far from the user and that the user needs to move closer to device 100. Thus, visual indicators 110, 112, and 114 illustrated in FIG. 1 are provided by way of example only and to illustrate that other visual indicators may be implemented in other embodiments to guide the user in mitigating possible causes of detected degradation in the quality of media data captured by one or more sensors.

[0025] Information about the location of sensors on device 100 can be obtained in various ways. For example, in some embodiments, an application running on device 100 can collect identification information from device 100, such as the model number, manufacturer, etc. In some embodiments, the application can communicate over a network to collect device information from the manufacturer or other repository based on the collected identification information (e.g., model number, serial number, etc.), such as the sensor type, sensor location, display size, display type, sensor features, etc. In other embodiments, a database or repository of device information for multiple devices can be stored on device 100, and the corresponding device information can be located based on the identification information collected by the application running on device 100. The device information, such as the sensor location and type, can be used to place visual indicators 110, 112, or 114, or a combination thereof, in appropriate locations on display 102. Furthermore, it should be understood that in some embodiments, the functionality of the embodiments described herein may be implemented at the application level, while in other embodiments, the functionality of the embodiments described herein may be implemented at the operating system level, or at a combination of the application level and the operating system level.

[0026] It should be understood that the components of device 100 illustrated in FIG. 1 are provided by way of example only, and that other implementations may include other components in addition to or instead of those shown in FIG. 1. For example, device 100 may include one or more haptic vibration motors that can provide haptic feedback to the user in addition to the visual indicators. For example, if a user is talking on the phone and is not looking at display 102, a haptic vibration motor can be used to notify the user that they should look at display 102. Additionally, device 100 may include one or more speakers, which in some embodiments can provide audio instructions or guidance to the user in addition to the visual guidance provided on display 102.

[0027] Figure 2 is a block diagram of one embodiment of an exemplary computing device 200 configured to implement the functionality of device 100 described above. The components of computing device 200 shown in Figure 2 include one or more processors 202, memory 204, storage interface 216, input / output ("I / O") device interface 212, and network interface 218, all of which are communicatively coupled, directly or indirectly, for communication between these components via memory bus 206, I / O bus 208, bus interface unit ("IF") 209, and I / O bus interface unit 210.

[0028] In the embodiment shown in FIG. 2, computing device 200 includes one or more general-purpose programmable central processing units (CPUs) 202A and 202B, generally referred to herein as processors 202. In some embodiments, computing device 200 includes multiple processors. However, in other embodiments, computing device 200 is a single-CPU system. Each processor 202 executes instructions stored in memory 204. Additionally, while embodiments are described with respect to a central processing unit chip (CPU chip), it will be understood that the embodiments described herein can also be applied to computer systems that utilize digital signal processor (DSP) chips or graphic processing unit (GPU) chips, or combinations thereof, in addition to or instead of a CPU chip. Accordingly, references herein to a processor or processing unit can refer to a CPU chip, a GPU chip, or a DSP, or combinations thereof.

[0029] In some embodiments, memory 204 includes a random-access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing or encoding data and programs. For example, memory 204 stores instructions 211, historical data 213, and device information 215. When executed by a processor, such as processor 202, instructions 211 cause processor 202 to perform the functions and calculations discussed herein with respect to detecting degradation in the quality of captured media data (e.g., images, video, audio) and providing visual indicators to guide a user in mitigating sources of the degradation. Additionally, although not shown in FIG. 2 , in some embodiments, memory 204 further stores profile data for multiple users, as discussed herein. The profile data can be used to personalize the detection of degradation and to provide personalized guidance in mitigating causes of the detected degradation, as discussed herein.

[0030] In some embodiments, memory 204 represents the entire virtual memory of computing device 200 and may also include virtual memory of other computer devices connected to computing device 200 via a network. In some embodiments, memory 204 is a single monolithic entity, while in other embodiments, memory 204 includes a hierarchy of caches and other memory devices. For example, memory 204 may exist in multiple levels of caches, and these caches may be further divided by function, such that one cache holds instructions while another cache holds non-instruction data used by the processor. Memory 204 may be further distributed and associated with different processing units or sets of processing units, for example, as known in any of a variety of so-called non-uniform memory access (NUMA) computer architectures. Thus, while for purposes of illustration, in the example shown in FIG. 2 , instructions 211, history data 213, and device information 215 are stored on the same memory 204, it should be understood that other embodiments may be implemented differently. For example, the instructions 211, the historical data 213, or the device information 215, or a combination thereof, may be distributed across multiple physical media.

[0031] The computing device 200 of the embodiment shown in FIG. 2 also includes a bus interface unit 209 for handling communications between the processor 202, memory 204, display system 224, and I / O bus interface unit 210. The I / O bus interface unit 210 is connected to an I / O bus 208 for transferring data to and from various I / O units. In particular, the I / O bus interface unit 210 can communicate with multiple I / O interface units 212, 216, and 218 (also known as I / O processors (IOPs) or I / O adapters (IOAs)) through the I / O bus 208. The display system 224 includes a display controller, display memory, or both. The display controller can provide video, still images, audio, or a combination thereof to a display device 226. The display memory can be dedicated memory for buffering video data. The display system 224 is connected to the display device 226. In some embodiments, the display device 226 also includes one or more speakers for rendering audio. Alternatively, the one or more speakers for rendering audio may be connected to the I / O interface unit. In alternative embodiments, one or more functions provided by the display system 224 are implemented on an integrated circuit that also includes the processor 202. Additionally, in some embodiments, one or more functions provided by the bus interface unit 209 are implemented on an integrated circuit that also includes the processor 202.

[0032] The I / O interface unit supports communication with various storage and I / O devices. For example, the I / O device interface unit 212 supports the connection of one or more user I / O devices 220, which may include user output devices and user input devices (e.g., a keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing device). A user can manipulate the user input devices using a user interface to provide input data and commands to the user I / O devices 220. In addition, a user can receive output data via the user output devices. For example, a user interface may be presented via the user I / O devices 220, e.g., displayed on a display device or played through speakers.

[0033] Storage interface 216 supports the connection of one or more storage devices 228, such as flash memory. The contents of memory 204, or any portion thereof, may be stored to and retrieved from storage device 228 as needed. Network interface 218 provides one or more communication paths from computing device 200 to other digital and computer devices. For example, in some embodiments, computing device 200 can communicate with servers to request and receive device information via network interface 218.

[0034] 2 depicts a particular bus structure providing direct communication paths between processor 202, memory 204, bus interface unit 209, display system 224, and I / O bus interface unit 210, in alternative embodiments, computing device 200 may include different buses or communication paths, which may be arranged in any of a variety of configurations, such as point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. Moreover, while I / O bus interface unit 210 and I / O bus 208 are depicted as single respective units, computing device 200 may include multiple I / O bus interface units 210 or multiple I / O buses 208, or a combination thereof, in other embodiments. Although multiple I / O interface units are shown separating the I / O bus 208 from the various communication paths running to the various I / O devices, in other embodiments, some or all of the I / O devices are directly connected to one or more system I / O buses.

[0035] As described above, in some embodiments, one or more of the components and data shown in Figure 2 include instructions or statements that execute on processor 202 or that are interpreted by instructions or statements that execute on processor 202 to perform functions as described herein. In other embodiments, one or more of the components shown in Figure 2 are implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, or other physical hardware devices, or combinations thereof, in place of or in addition to a processor-based system. Additionally, in other embodiments, some of the components shown in Figure 2 can be omitted, or other components can be included, or some of the components shown in Figure 2 can be omitted and other components can be included.

[0036] 3 is a flowchart illustrating one embodiment of an exemplary method 300 for improving sensor performance. Method 300 can be implemented by a computing device, such as computing device 200 described above. It should be understood that the order of operations in exemplary method 300 is provided for illustrative purposes, and that the method can be performed in a different order in other embodiments. It should also be understood that in other embodiments, some operations can be omitted or additional operations can be included. Furthermore, it should be understood that a user can opt in to data collection before method 300 is performed on a device, such as device 100 or 200. For example, the user can be presented with a dialog requesting the user to select whether or not to allow data collection and analysis.

[0037] In step 302, device information is obtained. For example, as described above, a repository of device information can be searched based on the device's identification information in some embodiments. The device information can include sensor type, sensor characteristics, or the physical location of one or more sensors on the device, or a combination thereof. Moreover, the repository of device information can be stored locally on the device or accessed over a network, e.g., the Internet. In addition, the identification information can, in some embodiments, be obtained by an application running on the device that obtains identification information from the device, as described above.

[0038] In step 304, data is captured by one of multiple sensors on the device. For example, a camera can capture video or images. Similarly, a microphone can capture audio data. In addition, other sensors, such as a light sensor that captures data about ambient light or a pressure sensor that captures data about applied pressure, can be used to capture corresponding data. In step 306, it is determined whether the quality of the captured data has deteriorated. For example, the captured data can be compared to a quality threshold to determine whether the captured data has deteriorated. In some embodiments, a sensor profile can be developed for each sensor to be analyzed based on the sensor's historical performance data. For example, a profile for each camera, each microphone, etc. For audio data from a microphone, because audio quality tends to follow a sinusoidal format, in some embodiments, a mathematical approximation of the sound profile for the microphone can be developed, for example, using a Fourier series. For a camera that captures image data, in some embodiments, the image data can be evaluated in terms of pixels. For example, in some such embodiments, a Convolutional Neural Network (CNN) image classifier can be implemented to identify camera blockages or obstructions that degrade the quality of the resulting image data (e.g., still photos or video) by identifying what certain pixels look like.

[0039] Convolutional neural networks are a class of feed-forward artificial neural networks that have been successfully applied to analyze visual images. Convolutional neural networks consist of artificial structures, or neuron-like structures, with learnable weights and biases. Each neuron receives several inputs and performs a dot product. Convolutional neural network architectures typically include a stack of layers that accept an input (e.g., a single vector) and transform it through a series of hidden layers. Each hidden layer consists of a set of neurons, where each neuron has learnable weights and biases, and each neuron can be fully connected to all neurons in the previous layer, or neurons in a layer can function independently without shared connections. The final layer is a fully connected output layer, which, in a classification setting, represents the class score, which can be any real value or a real-valued target (e.g., in the case of regression). However, it should be understood that the embodiments described herein are not limited to convolutional neural networks and can be implemented using other types of neural networks, such as recurrent neural networks (RNNs), deep convolutional inverse graphics networks (DCIGNs), etc., to analyze images and videos.

[0040] Additionally, determining a quality threshold for the data obtained for the sensor can be based, at least in part, on a respective user profile for each of one or more users, as described above. For example, each user profile can be based on respective historical device usage data for the user. The historical device usage data can include, but is not limited to, items such as how the device is held, time intervals, actions being performed, applications being used, and the volume of the user's voice. Thus, in some embodiments, the quality threshold can be adapted to account for differences in how each user interacts with the device to determine what constitutes abnormal or degraded performance for the sensor.

[0041] If, at step 306, it is determined that data quality has not deteriorated, method 300 returns to step 304, where subsequent data is captured. If, at step 306, it is determined that the data quality has deteriorated, method 300 proceeds to step 308, where a visual indicator is displayed on the device's display. The visual indicator includes a non-text component that guides the user in mitigating potential causes of the deteriorated quality of the data captured by the sensor. For example, in some embodiments, the non-text component directs the user to the physical location of the sensor on the device. Additionally, the non-text component can instruct the user on an action to take, such as moving closer to the sensor.

[0042] In some embodiments, the visual indicator is displayed at a location on the display proximate the physical location of the sensor. For example, as described above, in some embodiments, the non-text component can include an icon displayed near the physical location of the sensor, concentric partial circles centered near the physical location of the sensor, an arrow displayed near and pointing toward the physical location of the sensor, etc. Additionally, in some embodiments, the non-text component of the visual indicator can include a diagram of the device displayed on the device's display. The diagram can explain the physical location of the sensor, for example, by highlighting the location of the sensor on the device (e.g., by including a label indicating the location of the sensor on the device, etc.). Moreover, in some embodiments, the visual indicator can be overlaid on content already displayed on the display, similar to a pop-up message. In other embodiments, content displayed on the device is replaced by the visual indicator. Additionally, as described above, the visual indicator can also include text components, such as a label and a list of one or more potential causes of the corrupted data. In some such embodiments, the list may order the potential causes in order of the likelihood that a given potential cause is the actual cause of the degraded performance.

[0043] In step 310, a success notification can optionally be output or provided to the user. The notification can be a visual notification, an audible notification, or tactile feedback, or a combination thereof. The success notification is configured to provide feedback to the user that the quality of the degraded data has improved, thus indicating that the comparison with the quality threshold no longer determines the data to be degraded. For example, the device can detect and monitor usage actions taken after receiving the degraded data notification. In addition, the device can continue to monitor the quality of data received from the sensor. Thus, in response to detecting a user action and determining that data captured by the sensor following the user action has improved quality, the device can output the success notification.

[0044] At step 312, historical data can be updated based on information about the degraded data, any user actions taken to mitigate the degraded data, device usage data at the time the degraded data was detected, etc. This feedback can be used to further improve the device's ability to detect degraded data, for example, to update the quality threshold. Additionally, in some embodiments, at step 314, the device is configured (e.g., a processor on the device executes instructions) to predict when the quality of the data captured by the sensors is likely to degrade. For example, if it is detected that a given user is using the device to make a phone call, the device can use the historical data to predict, based on the historical data, how the user will handle and / or use the device to determine whether the user is likely to obstruct the microphone, speak too quietly, etc. Thus, in some such embodiments, the device can analyze historical device usage data to identify patterns indicative of the likelihood of sensor obstruction, and then compare current device usage data to the identified patterns. In response to determining that the current device usage matches the identified pattern, the device can provide a notification along with displaying the visual indicator before detecting a degradation in the quality of data captured by the sensor. Thus, to reduce the likelihood that the sensor will be blocked or otherwise capture degraded quality data, the device can provide a proactive warning. The notification can be visual, audible, or tactile feedback.

[0045] In some embodiments, the device uses machine learning techniques to determine the spatiotemporal coordinates,

number

number

[0046] Thus, the embodiments described herein enable the detection of sensor anomalies or degraded sensor performance and provide visual guidance for the user to locate the sensor on the device and to correct the underlying cause of the abnormal performance. Moreover, the embodiments described herein can proactively predict when degraded performance will occur along with the underlying cause and provide a warning or notification to the user to help them prevent the quality of the data being collected from being reduced.

[0047] The present invention may be a system, method, or computer program product, or combination thereof, at any level of technical detail that may be integrated. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0048] The computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a ridge structure in a groove in which instructions are recorded, or any suitable combination thereof. As used herein, the computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0049] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing device / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer-readable program instructions from the network and transmits the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing device / processing device.

[0050] The computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, e.g., Smalltalk, C++, etc., or conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0051] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer-readable program instructions.

[0052] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowchart diagrams or the block diagrams, or a combination thereof, to produce a machine. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer-programmable data processing apparatus or other device, or a combination thereof, to function in a particular manner, such that a computer-readable storage medium having stored instructions includes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowchart diagrams or the block diagrams, or a combination thereof.

[0053] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on a computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0054] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step performed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, depending on the functionality involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, and combinations of multiple blocks in the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special-purpose hardware and computer instructions.

[0055] While specific embodiments have been shown and described herein, it will be understood by those skilled in the art that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. Therefore, it is manifestly intended that this invention be limited only by the claims and the equivalents thereof.

Claims

1. 1. A computer-implemented method comprising: obtaining, by the device, the physical location of a sensor on the device; determining, by the device, that quality of data captured by the sensor physically located on the device is degraded based on a comparison of the captured data to a quality threshold, the quality threshold being based at least in part on historical usage data associated with a device user, the historical usage data indicating a pattern of likelihood of interference with the sensor by the device user; and displaying a visual indicator on a display of the device in response to determining that the quality of the data captured by the sensor has deteriorated, the visual indicator including at least one non-text component that indicates to the device user the physical location of the sensor on the device. The method comprising:

2. 1. A computer-implemented method, the method comprising: obtaining, by the device, the physical location of a sensor on the device; determining a quality threshold based at least in part on a respective user profile for each of one or more users, the respective user profile being based on historical device usage data for one of the one or more users; determining, by the device, that the quality of data captured by the sensor physically located on the device has deteriorated based on a comparison of the captured data to the quality threshold; and displaying a visual indicator on a display of the device in response to determining that the quality of the data captured by the sensor has deteriorated, the visual indicator including at least one non-text component that indicates to the device user the physical location of the sensor on the device. wherein the method comprises: analyzing the respective historical device usage data for each of the one or more users to identify patterns indicative of a likelihood of sensor interference; comparing current device usage data of one of the one or more users with the identified patterns; and displaying the visual indicator and providing a notification prior to detecting a degradation in the quality of data captured by the sensor in response to determining that the current device usage matches the identified pattern. The method further comprises:

3. 1. A computer-implemented method, the method comprising: obtaining, by the device, the physical location of a sensor on the device; determining, by the device, that the quality of data captured by the sensors physically located on the device has deteriorated based on a comparison of the captured data to a quality threshold; and displaying a visual indicator on a display of the device in response to determining that the quality of the data captured by the sensor has deteriorated, the visual indicator including at least one non-text component that indicates to the device user the physical location of the sensor on the device. Including, the non-textual component of the visual indicator includes a diagram of the device displayed on the display, the diagram illustrating the physical location of the sensor; The method.

4. 4. The method of claim 1, further comprising determining the quality threshold based at least in part on a sensor profile for the sensor developed based on historical performance data for the sensor.

5. 4. The method of claim 1, further comprising determining the quality threshold based at least in part on a respective user profile for each of one or more users, wherein the respective user profile is based on historical device usage data for one of the one or more users.

6. analyzing the respective historical device usage data for each of the one or more users to identify patterns indicative of a likelihood of sensor interference; comparing current device usage data of one of the one or more users with the identified patterns; and displaying the visual indicator and providing a notification prior to detecting a degradation in the quality of data captured by the sensor in response to determining that the current device usage matches the identified pattern. The method of claim 5 further comprising:

7. The method of any one of claims 1 to 3, wherein the visual indicator is displayed on the display of the device in a location proximate to the physical location of the sensor.

8. The method of claim 7 , wherein the non-text component of the visual indicator comprises a plurality of concentric partial circles centered near the physical location of the sensor.

9. The method of any one of claims 1 to 3, wherein the non-text component is animated to orient the user to the physical location of the sensor.

10. The method of claim 1 or 2, wherein the non-textual component of the visual indicator comprises a diagram of the device displayed on the display, the diagram illustrating the physical location of the sensor.

11. 4. The method of claim 1, further comprising: detecting a user action; and providing a success notification to the device user in response to determining that data captured by the sensor following the user action has improved quality.

12. 1. A computing device, comprising: display; a sensor configured to capture data; and a processor communicatively connected to the sensor and to the display It is equipped with the processor: Identifying a physical location of a sensor on the computing device; determining that the quality of data captured by the sensor is degraded based on a comparison of the captured data to a quality threshold, the quality threshold being based at least in part on historical usage data associated with a device user, the historical usage data indicating a pattern of likelihood of interference with the sensor by the device user; and displaying a visual indicator on the display in response to determining that the quality of the data captured by the sensor has deteriorated. wherein the visual indicator includes at least one non-text component configured to guide the user to mitigate potential causes of the degraded quality of the data captured by the sensor. The computing device.

13. 1. A computing device, comprising: display; a sensor configured to capture data; and a processor communicatively connected to the sensor and to the display It is equipped with the processor: Identifying a physical location of a sensor on the computing device; determining a quality threshold based at least in part on a respective user profile for each of the one or more users, wherein the respective user profile is based on historical device usage data for one of the one or more users; determining that the quality of the data captured by the sensor has deteriorated based on comparing the captured data to a quality threshold; and displaying a visual indicator on the display in response to determining that the quality of the data captured by the sensor has deteriorated. wherein the visual indicator includes at least one non-text component configured to guide the user to mitigate potential causes of the degraded quality of the data captured by the sensor; the processor: analyzing the respective historical device usage data for each of the one or more users to identify patterns indicative of a likelihood of sensor interference; comparing current device usage data of one of the one or more users with the identified patterns; and displaying the visual indicator and providing a notification prior to detecting a degradation in the quality of data captured by the sensor in response to determining that the current device usage matches the identified pattern. further configured as follows: The computing device.

14. 1. A computing device, comprising: display; a sensor configured to capture data; and a processor communicatively connected to the sensor and to the display It is equipped with the processor: Identifying a physical location of a sensor on the computing device; determining that the quality of the data captured by the sensor has deteriorated based on comparing the captured data to a quality threshold; and displaying a visual indicator on the display in response to determining that the quality of the data captured by the sensor has deteriorated. wherein the visual indicator includes at least one non-text component configured to guide the user to mitigate potential causes of the degraded quality of the data captured by the sensor; the at least one non-textual component of the visual indicator includes a diagram of the computing device displayed on the display, the diagram illustrating the physical location of the sensor; The computing device.

15. the processor: developing a sensor profile for the sensor based on historical performance data for the sensor; and Determining the quality threshold based at least in part on a sensor profile. A computing device according to any one of claims 12 to 14, configured to:

16. the processor:

15. The computing device of claim 12, wherein the computing device is configured to determine the quality threshold based at least in part on a respective user profile for each of one or more users, wherein the respective user profile is based on historical device usage data for one of the one or more users.

17. The computing device of any one of claims 12 to 14, wherein the visual indicator is displayed on the display of the device in a location proximate to the physical location of the sensor.

18. 14. The computing device of claim 12 or 13, wherein the at least one non-textual component of the visual indicator comprises a diagram of the computing device displayed on the display, the diagram illustrating the physical locations of the sensors.

19. The computing device of claim 12, wherein the at least one non-text component of the visual indicator is animated to orient the user to the physical location of the sensor.

20. A computer program comprising: The computer program causing a processor to carry out the method according to any one of claims 1 to 11.

Images