Dynamic intervention through interfaces to improve sensor performance

Abstract

A computer-implemented method (300) comprising: retrieving (302) by a unit (100) the physical location of a sensor (106, 108) on the unit; establishing a quality threshold at least partially based on a corresponding user profile for each of one or more users, each corresponding user profile being based on corresponding historical unit usage data for one of the one or more users; determining (306) by the unit that the quality of the data acquired by the sensor (304) physically located on the unit has deteriorated based on a comparison of the acquired data with the quality threshold;in response to the detection that the quality of the data captured by the sensor has deteriorated, display (308) a visual indicator (110) on a display (102) of the unit, the visual indicator comprising at least one non-text component that shows a unit user where the physical location of the sensor is on the unit; and analyze the relevant historical unit usage data for each of the one or more users to identify a pattern indicating a probability of the sensor being blocked; compare the current unit usage data of one of the one or more users with the identified pattern; and in response to the detection that the current unit usage matches the identified pattern, provide a notification along with the display of the visual indicator before any deterioration in the quality of the data captured by the sensor is detected.

Description

BACKGROUNDUsers are increasingly able to take video, image and audio images with their data processing units. Many personal devices (e.g., handheld devices and wearable devices) include, for example, microphones, cameras, and other sensors that can capture content. Moreover, laptop computers, desktop computers, tablets, etc. are also often equipped with such sensors that can capture content. In addition, these electronic devices allow for multiple forms of data transmission, including audio telephone calls and video conferencing. However, one or more sensors in a unit may be accidentally obscured and / or blocked by obstacles when capturing content (e.g., audio and / or video data), resulting in suboptimal captured content.US 2015 / 0 304 786 A1 relates to an apparatus comprising: a detector configured to determine that at least one microphone is impaired by analyzing at least one audio signal from at least one microphone; and a controller configured to determine an indicator based on the determination of the impairment of the at least one microphone; and configured to apply the indicator based on the determination of the impairment of the at least one microphone such that the at least one audio signal is processed based on the indicator.US 10 073 541 B1 relates to a computer implemented method comprising: generating image data using a camera of a computing device; determining from the image data that a field of view of the camera is at least partially obstructed, preventing the camera from accurately detecting gesture input; and in response to the field of view of the camera being at least partially obstructed, displaying a display notification on a portion of a display screen of the computing device corresponding to a physical location of the camera of the computing device to indicate that the field of view of the camera is at least partially obstructed.SUMMARYAspects of the disclosure may include a computer implemented method, a computer program product and a system, the features of which are set forth in the respective claims. An example of the method includes retrieving a physical location of a sensor on a device; determining that the quality of the data captured by the sensor physically on the device has degraded based on a comparison of the captured data to a quality threshold, and in response to determining that the quality of the data captured by the sensor has degraded, displaying a visual indicator on a display of the device. The visual indicator includes at least one non-text component that indicates to a user where the physical location of the sensor is on the device.DRAWINGSAssuming that the drawings represent only exemplary embodiments and are therefore not to be taken as limiting the scope of application, the exemplary embodiments will be described with greater accuracy and additional details through the use of the accompanying drawings, in which: FIG. 1 is a block diagram of an embodiment of an example device. FIG. 2 is a block diagram of an embodiment of an example computing device. FIG. 3 is a flow chart showing an embodiment of an example method for improving sensor performance.In accordance with common practice, the various features described are not drawn to scale, but rather serve to emphasize certain features that are important to the example embodiments.DETAILED DESCRIPTIONIn the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and which show illustrative specific illustrative embodiments. It should be understood that other embodiments may be used and logical, mechanical, and electrical changes may be made. Moreover, the method illustrated in the drawing figures and in the description should not be understood as limiting the order in which the individual steps can be carried out. The following detailed description is therefore not to be considered as limiting.Similarly, "a number of" with respect to reference elements means one or more of these elements. For example, "a number of different types of networks" is one or more different types of networks.The terms "at least... and / or", "one / one or more" and "and / or" are moreover indeterminate terms which can be used both in a connecting and a disconnecting manner. Each of the terms "at least one of A, B and C", "at least one of A, B or C", "one or more of A, B and C", "one or more of A, B or C", and "A, B and / or C" means, for example, A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. In other words, "at least... and / or," "one / more of," and "and / or" means that any combination of elements and any number of elements from the list may be used, but not all elements of the list are required. The element may be a particular item, a ping, or a category. Moreover, the amount or number of each element in a combination of the listed elements need not be the same. In some illustrative examples, "at least one of A, B, and C" may mean, for example, but not limited to, two of element A, one of element B and ten of element C, or 0 of element A, four of element B and seven of element C, or other suitable combinations.Further, the term "a" entity refers to one or more of these entities. The terms "a / an", "one / more" and "at least... and / or" may therefore be used interchangeably herein. It should also be noted that the terms "comprise", "comprise" and "have" may be used interchangeably.As used herein, the term "automatic" and variations thereof further refers to any processes or operations performed without substantial human intervention when the process or operation is performed. However, a process or operation may be automatic, even if the process or operations are performed with substantial or non-substantial human intervention, if the intervention is performed prior to performing the process or operation. Human intervention is considered to be essential when this intervention has an influence on how the process or operation is performed. Human intervention in consensus with performing the process or operation is not considered "essential.". The terms "communication" or "connected for purposes of data transmission" include electrical or optical connections, whether wireless or wired, that enable two or more systems, components, modules, units, etc. to communicate data, signals, or other information using any protocol or format.As discussed above, many computing devices have sensors such as cameras and microphones that allow a user to capture data such as media content (e.g., images, videos, audio), temperature data, light data, proximity data, print data, etc., with the wearable device. Many of these devices have multiple sensors, including multiple sensors of the same type (e.g., multiple cameras and / or multiple microphones). However, these sensors may be blocked by obstacles. One or more of the sensors may be blocked, for example, by the user of the unit (e.g., a portion of the hand or a garment may totally or partially occlude a sensor). A sensor may also be blocked by other external factors. For example, dust may collect on a sensor such as a microphone, degrading the quality of the captured data. As used herein, the terms "blocked" and "hidden", and variations thereof, may be used interchangeably. As used herein, a sensor is further considered blocked or obscured when at least a portion of the sensor is blocked or obscured. As discussed above, the sensor may be obscured by a garment, a user's finger, dust on the sensor, etc.A sensor may be further obscured by a physical object, such as another person located in front of the sensor (e.g., a person standing in front of a camera and obstructing vision of a scene captured by the camera). Such blockage or occlusion of sensors may degrade the quality of the captured media, which may result in lower user satisfaction. The quality of the captured media may also be degraded if the user is too far from the sensor, e.g., from a microphone during a videoconferencing, or if the user is speaking in a direction away from the microphone. These actions may also degrade the quality of the captured media. The embodiments described herein enable proactive data exchange with the user to mitigate such causes for degrading the quality of the captured data. As used herein, the term "sensed data" refers to data retrieved from one or more sensors on the device. These data may be media data (e.g., audio, image, or video), temperature data, light data, print data, proximity data, location data, and the like. The embodiments described below relate to different types of media data for simplicity. It should be appreciated, however, that captured data may include data other than audio and visual data, and that the embodiments described herein may be implemented to enable the degraded quality of data captured by other types of sensors in addition to microphones and cameras.FIG. 1 is an illustration of an embodiment of an example unit 100. While the device 100 is shown as a tablet or smartphone, it should be understood that embodiments of the invention are not so limited. In other embodiments, the device 100 may be implemented as, for example, a desktop computer, laptop computer, or wearable device, etc. The unit 100 includes a housing 104, a display 102, a camera 106, and a microphone 108. It should be appreciated that while only one camera 106 and only one microphone 108 are shown in this example, it should be appreciated that in other embodiments, more than one camera and / or more than one microphone may be implemented. In some embodiments, more than one display may also be implemented, e.g., in a desktop computer with more than one connected display unit.In embodiments where the unit 100 further includes a plurality of sensors of the same type (e.g., a plurality of cameras or a plurality of microphones), each instance of the sensor may be different from other instances. For example, a mobile unit may include two or more camera sensors, wherein the characteristics of the individual camera sensors are different. For example, a camera sensor may have a higher pixel density than another camera sensor on the same unit, located on a different portion of the unit (e.g., on the front or back), have different focal lengths than another camera sensor, etc. Similarly, a unit with multiple microphone sensors may have microphone sensors with different characteristics.The unit 100 is configured to determine if the quality of the data detected by a sensor falls below a predetermined level or predict that it falls below this level. For example, the unit 100 may determine that a sensor capturing images, video and / or audio is blocked and / or predict when such blocking will occur. For example, in some embodiments, the unit 100 may retrieve data from one or more proximity sensors 132 to determine that a camera sensor and / or a microphone sensor is / are blocked. These proximity sensors 132 may be implemented as any suitable type of proximity sensors, including, but not limited to, capacitive, inductive, magnetic, and / or optical proximity sensors. These proximity sensors 132 may be mounted proximate a microphone or camera of the unit 100 to enable the proximity sensor 132 to detect that an object is proximate the microphone or camera and may block the respective microphone or camera. The unit 100 may also use the data of the proximity sensors 132 to determine how far the physical object or obstacle is from the camera or microphone and whether the object could block the camera or microphone.In some embodiments, the unit 100 is further configured to analyze image and audio data already during the acquisition to detect degradation of quality. A deterioration in the quality of the captured data may indicate, for example, a possible blockage of the corresponding sensor. Degradation of audio quality may further indicate that the speaker or audio source is too far from the microphone 108 of the unit 100 so that the microphone 108 cannot capture the sound with high quality or volume. In some embodiments, therefore, the unit 100 is configured to detect a change in the quality of the recorded data. For example, the unit 100 may recognize changes in the amplitude of audio data. Such degradation may be further determined by comparing characteristics of the captured data to output values that have resulted over time based on a plurality of past data captured by the same sensor. In other words, if the sensor is blocked at the beginning of data acquisition, the blocking can be detected by comparing the data with output values, so that blocking can be detected even if the data has not changed significantly since the beginning of data acquisition.The output values may also be used to account for differences in usage by different users. For example, a first user may speak relatively louder than a second user. In this way, different initial values can be calculated for the first user and the second user, respectively. In some of these embodiments, the unit 100 is configured to recognize which user is currently using the unit 100. For example, the unit 100 may perform speech recognition analysis, image analysis, fingerprint analysis, and / or unique identification codes associated with each user, etc., to determine which user the unit 100 is using. The device 100 may thus store various profiles for a plurality of users of the device 100, including, but not limited to, profiles including corresponding output values used to detect degradation in quality, historical data associated with the user, etc. Moreover, in addition to or instead of using data collected from the unit 100, in some embodiments, output values may be calculated based on crowdsourced data collected from other units.Moreover, historical data collected from sensors such as camera 106 and microphone 108, as well as other sensors, may be used to predict when blockage of the sensors will occur. For example, the unit 100 may determine a probability that a particular user obscures a microphone when telephony based on historical data by analyzing patterns in the manner the user holds the unit and a resultant occlusion of the microphone 108. In other words, the unit 100 may predict that the user, when receiving or carrying a telephone call, will typically hold the unit 100 so that the microphone is obscured. Similar historical patterns may be determined for other sensors and activities, including, but not limited to, capturing photos, participating in video conferencing, etc. The unit 100 may further include other sensors to detect patterns in the historical data in some embodiments. For example, the unit 100 may include one or more pressure sensors 130, which may provide data about the user's grip pattern, which may be used to predict when a sensor is blocked.In some embodiments, the historical data may further be used to determine possible causes for the degradation of the quality of the sensor data. For example, if the historical data indicates that the quality of the sound received via the microphone 108 has gradually degraded over a period of time, this may be an indication that lint or dust has accumulated on the microphone 108. On the other hand, if the degradation occurs suddenly, it may be an indication that the microphone 108 has been obscured by an object such as a garment or hand. Thus, by analyzing the historical data, the unit 100 is able to determine the relative probability of various causes for the degradation of quality.The unit 100 is further configured to provide guidance to a user of the unit 100 to remedy the detected degradation of quality. In particular, the unit 100 is configured to display one or more visual indicators such as indicators 110, 112, and 114. The visual indicators 110, 112, and 144 are selected to assist a user in eliminating a cause of the degraded data quality. For example, many devices have a plurality of components, such as multiple cameras, microphones, speakers, etc. The user may not know or may not readily determine where the individual cameras, microphones, etc. are located or which of the plurality of sensors is the sensor that receives degraded data. In addition, in some situations, the user may not notice the degraded quality. For example, if the user is speaking in a video conference, he may be unaware that the user's video is not being recorded and sent to the other participants of the video conference. This may be the case, for example, when the user has not removed a displaceable screen that obscures a camera. Likewise, a user may not notice that the sound transmitted to the other participants of the videoconferencing or a telephone call is of degraded quality (e.g., muted, quiet, distorted, etc.). One or more indicators 110, 112, and 114 may thus be used to alert a user to a detection of degraded quality and to show to him how the cause may be eliminated.The visual indicator 110 includes, for example, a plurality of concentric reference circles 122 and a symbol 120. The concentric pitch circles 122 are approximately centered on a location near the microphone 108 in this example. Similarly, the icon 120 is located near the physical location of the microphone 108. By arranging the icon 120 and the circles 122 near the physical location of the microphone 108, the visual indicator 110 may show a user where the location of the microphone 108 is located. Icon 120 also represents a general microphone in this example to indicate to the user that visual indicator 110 is referencing a location of microphone 108. As discussed above, the user may not know or may not readily determine where a sensor is located or which sensor is obscured. By placing the indicator 110 near the physical location of the microphone 108, the user can easily locate the sensor that has been detected to input degraded data. Moreover, while the visual indicator 110 is shown as a static visual representation in FIG. 1, in some embodiments it is not a static visual representation. That is, the visual indicator 110 may be animated, e.g., by changing colors, color intensity, or brightness in a pattern, e.g., by a pulsing or wave-shaped pattern, to show the user where the location of the microphone 108 is located. It should be appreciated that the visual indicator 110 is provided by way of example only. In other embodiments, visual indicators do not include, in particular, a symbol 108 and / or concentric circles 122. In other embodiments, the visual indicators may also include other components in addition to or in place of the symbol 108 and / or the concentric circles 122.The visual indicator 112 includes, for example, a symbol 116 and an arrow 118. The visual indicator 112 thus does not include concentric circles 122. Rather, the icon 116 and arrow 118 are near the physical location of a sensor (i.e., the camera 106 in this example). The arrow 118 also points to the location of the camera 106. Similar to the visual indicator 110, the visual indicator 112 may also be animated to change colors, intensity, brightness, etc., and to show the user where the physical location of the sensor is. Moreover, it is not necessary for visual indicator implementations on the display 102 to be in close proximity to the physical location of the corresponding sensor. The visual indicator 114 is, for example, a representation of the unit 100 including the indicia 126 and 128 to indicate the location of the respective sensors.The visual indicators 110, 112, and / or 114 may be displayed on the display 102 by superimposing the visual indicators 110, 112, 114 on other content displayed on the display 102, in some embodiments. In other embodiments, the content displayed on the display 102 is minimized such that a new screen or window is opened to display one or more of the visual indicators 110, 112, or 114. Further, more than one visual indicator may be displayed simultaneously. For example, the indicator 114 showing a representation of the unit 100 may be displayed simultaneously with the visual indicator 110 and / or the visual indicator 112. Other information may also be displayed with one or more of the visual indicators. For example, a text list of possible causes and / or action to be taken by the user may be displayed on the display 102, along with one or more visual indicators that show the user where the location of a particular sensor is located, or that provide other visual instructions to the user. Such a text list may be ordered based on a likelihood of the cause of degradation determined by the unit 100 based on historical data and data from other sensors such as sensors 130 and / or 132, for example. In another example, text information may confirm the location of a sensor or show it to the user. For example, if a particular sensor is located on another side of the device 100 from the display 102, the text information may provide that information to the user.Moreover, other variations of visual indicators may provide other indicia to the user in addition to or instead of a location of a particular sensor. For example, an image of a hand indicating the user to come closer may be displayed on the display 102, so that the user knows that the unit 100 is too far away from it and that he needs to come closer to the unit 100. The visual indicators 110, 112, and 114 shown in FIG. 1 are therefore provided by way of example only, and other visual indicators may be implemented in other embodiments to show the user how a possible cause of a detected degradation in the quality of the media data captured by one or more sensors may be resolved.The information about the location of the sensors on the unit 100 can be retrieved in various ways. In some embodiments, for example, an application executing in the unit 100 may acquire identification information on the model number, the manufacturer, etc. from the unit 100. In some embodiments, the application may communicate over a network to collect information about the entity regarding the sensor type, location of the sensors, size of the display, type of display, characteristics of the sensors, etc., from the manufacturer or other repository based on the collected identification information (e.g., model number, serial number, etc.). In other embodiments, a database or data repository including device information for a plurality of devices may be stored in the device 100, and the corresponding device information may be found based on the identification information collected from the application executing on the device 100. The unit information such as location and sensor type may be used to determine the visual indicators 110, 112, and / or 114 at the corresponding location on the display 102. Further, while the functionality of the embodiments described herein may be implemented at the application level in some embodiments, it should be appreciated that the functionality of the embodiments described herein may be implemented at the operating system level or a combination of application and operating system levels in other embodiments.It should be appreciated that the components of the unit 100 shown in FIG. 1 are provided by way of example only and that other implementations may include other components in addition to or in place of those shown in FIG. 1. The unit 100 may include, for example, one or more haptic vibration motors that may provide haptic feedback to a user in addition to the visual indicators. For example, if a user is telephony and not looking at the display 102, haptic vibration motors may be used to alert the user that he should look at the display 102. The unit 100 may also include, in some embodiments, one or more speakers that may provide audio instructions or instructions to the user in addition to the visual cues provided on the display 102.FIG. 2 is a block diagram of an embodiment of an example computing device 200 configured to implement the functionality of device 100 described above. The components of computing device 200 shown in FIG. 2 include one or more processors 202, memory 204, memory interface 216, input / output ("I / O") device interface 212, and network interface 218, all of which are directly or indirectly connected for exchanging data between the components via memory bus 206, I / O bus 208, bus interface device ("IF") 209, and I / O bus interface device 210.In the embodiment shown in FIG. 2, computing device 200 includes one or more general purpose programmable central processing units (CPUs) 202A and 202B, referred to generally as processor 202. In some embodiments, computing device 200 includes multiple processors. However, in other embodiments, the computing device 200 is a single CPU system. Each processor 202 executes instructions stored in memory 204. Further, while the embodiments are described with respect to chips of central processing units, it should be appreciated that the embodiments described herein may be applied to a computer system that uses digital signal processors (DSPs) and / or graphics processing unit (GPU) chips in addition to or in place of CPU chips. Reference herein to a processor or processing unit may therefore refer to CPU chips, GPU chips, and / or a DSP.In some embodiments, memory 204 includes a random access semiconductor memory, a storage device, or a storage medium (either volatile or nonvolatile) for storing or encoding data and programs. For example, memory 204 stores instructions 211, historical data 213, and information about device 215. When executed by a processor such as processor 202, instructions 211 cause processor 202 to perform the functions and calculations discussed herein with respect to detecting a degraded quality of the captured media data (e.g., images, video, audio) and provide visual indicators to show a user as discussed herein how the cause of the degradation may be remedied. Although not shown in FIG. 2, in some embodiments as discussed herein, memory 204 continues to store profile data for a plurality of users. The profile data may be used to personalize the detection of a degradation as discussed herein and to provide personalized guidance to remedy a cause of the detected degradation.In some embodiments, memory 204 represents all virtual memory of computing device 200, and may also include virtual memory from other computing devices connected to computing device 200 via a network. In some embodiments, memory 204 is a single monolithic entity, but in other embodiments, memory 204 includes a hierarchy of caches and other storage devices. For example, memory 204 may consist of multiple levels of caches, and these caches may be further divided by functions such that one cache contains instructions while another cache contains non-instruction data used by the processor. Memory 204 may be further distributed and associated with different processing units or sets of processing units, as is known from various so-called NUMA computer architectures (NUMA= Non Memory Access). Thus, while the instructions 211, historical data 213, and information about the device 215 are stored in the same memory 204 in the example shown in FIG. 2 for purposes of explanation, it should be appreciated that other embodiments may be implemented in other ways. For example, the instructions 211, the historical data 213, and / or the information about the device 215 may be distributed across multiple physical media.The computing device 200 in the embodiment shown in FIG. 2 also includes a bus interface device 209 for performing data exchange between the processor 202, memory 204, display system 224, and I / O bus interface device 210. The I / O bus interface unit 210 is connected to the I / O bus 208 to transfer data from and to the various I / O units. Specifically, the I / O bus interface unit 210 may communicate with multiple I / O interface units 212, 216, and 218, also referred to as I / O processors (IOPs) or I / O adapters (IOAs), via the I / O bus 208. The display system 224 includes a display controller, a display memory, or both. The display controller may provide video, still images, audio, or a combination thereof to a display unit 226. The display memory may be a special memory for buffering video data. The display system 224 is connected to the display unit 226. In some embodiments, display unit 226 also includes one or more speakers for audio playback. Alternatively, one or more speakers for audio reproduction may be connected to an I / O interface unit. In alternative embodiments, one or more functions provided by display system 224 are located on an integrated circuit that also includes processor 202. Moreover, in some embodiments, one or more of the functions provided by the bus interface unit 209 are located on an integrated circuit that also includes the processor 202.The I / O interface units support data transfer with a variety of memory and I / O units. For example, the I / O device interface unit 212 facilitates connection of one or more user I / O devices 220, which may include user output devices and user input devices (e.g., keyboard, mouse, keypad, touchpad, trackball, buttons, light pen, or other pointing devices). A user may operate the user input devices via a user interface to provide input data and commands to the user I / O device 220. A user may further receive output data via the user output units. For example, a user interface may be presented via the user I / O unit 220, e.g., displayed on a display or played back via a speaker.The memory interface 216 supports the connection of one or more memory units 228, e.g., flash memory. The content of the memory 204 or a portion thereof may be stored in the storage unit 228 and retrieved therefrom as needed. Network interface 218 provides one or more communication paths from data processing unit 200 to other digital and computer units. In some embodiments, for example, computing device 200 may communicate with a server to request and receive information about the device via network interface 218.Although the computing device 200 illustrated in FIG. 2 illustrates a single bus structure providing a direct communication path between the processors 202, memory 404 and bus interface device 209, display system 224 and I / O bus interface device 210, in alternative embodiments, the computing device 200 may include multiple different buses or communication paths that may be arranged in various forms such as point-to-point connections in hierarchical, star or network configurations, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. Although I / O bus interface unit 210 and I / O bus 208 are each shown as individual units, in other embodiments, computing device 200 may include multiple I / O bus interface units 210 and / or multiple I / O buses 208. Although multiple I / O interface units are shown separating I / O bus 208 from the various data transmission paths leading to the various I / O units, in other embodiments, some or all of the I / O units are directly connected to one or more I / O system buses.As discussed above, in some embodiments, one or more of the components and data shown in FIG. 2 include instructions or instructions executed on processor 202, or instructions or instructions configured by instructions or instructions executed on processor 202 to perform the functions described herein. In other embodiments, one or more of the components shown in FIG. 2 are implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. Additionally, in other embodiments, some of the components shown in FIG. 2 may be omitted and / or other components may be added.FIG. 3 is a flow chart showing an embodiment of an example method 300 for improving sensor performance. The method 300 may be implemented by a computing device such as the computing device 200 discussed above. It should be appreciated that the order of action of the example method 300 is provided for illustration only and that the method may be performed in a different order in other embodiments. Similarly, it should be appreciated that in other embodiments, some actions may be omitted or additional actions may be taken. It should also be appreciated that a user may agree on data acquisition before method 300 is performed in a device such as device 100 or 200. For example, a dialog can be displayed to the user by requesting it to decide whether to agree with the acquisition and analysis of data.At 302, information about the unit is retrieved. As discussed above, in some embodiments, a data repository may be searched with information about the device based on, for example, identification information of the device. The information about the unit may include the sensor types, characteristics of the sensors, and / or the physical location of one or more sensors in the unit. The repository of information about the device may further be stored locally in the device or accessible via a network such as the Internet. The identification information may also be retrieved from an application executing on the device, which retrieves the identification information from the device as described above, in some embodiments.At 304, data is collected from one of the sensors on the unit. A camera may capture video or images, for example. Similarly, a microphone can capture audio data. Further, additional sensors may be used to acquire corresponding data, e.g., light sensors that determine data about the ambient light, or pressure sensors that determine data about the applied pressure. At 306, it is determined whether the quality of the captured data has degraded. The captured data may be compared to, for example, a quality threshold to determine whether the captured data has degraded. In some embodiments, for each sensor to be analyzed, a sensor profile may be created based on historical performance data of the sensor. For example, a profile for each camera, microphone, etc. With respect to audio data from a microphone, for example, in some embodiments, a mathematical approximation of the sound profile for a microphone may be created using a Fourier series because the audio quality tends to follow a sinusoidal waveformate. With respect to a camera capturing image data, in some embodiments, the image data may be evaluated based on pixels. In some of these implementations, for example, a convolutional neural network (CNN) image classifier may be implemented to determine blockage or impairment of the camera that degrades the quality of the obtained image data (e.g., still photo or video) by determining how certain pixels look.Convolutional neural networks are a class of deep forward artificial neural networks that are successfully used to analyze visual images. Convolutional neural networks consist of artificial or neuron-like structures that have trainable weights and distortions. Each neuron receives some inputs and executes a dot product. A convolutional neural network architecture typically includes a stack of layers that serve to receive an input (e.g., a single vector) and transform it through a series of hidden layers. Each hidden layer consists of a set of neurons, each neuron having trainable weights and distortions, each neuron being fully connectable to all neurons in the previous layer, and neurons in a single layer being able to function independently of each other without shared connections. The last layer is the fully connected output layer, which in classification settings represents the class values, which may be random real value numbers or real value targets (e.g., in regression). It should be appreciated, however, that the embodiments described herein are not limited to convolutional neural networks, but may also be implemented with other types of neural networks such as recurrent neural networks (RNNs), deep convolutional inverse graphics networks (DCIGNs), etc., to analyze images and videos.Setting a quality threshold for data captured for the sensor may also be based, as discussed above, at least in part on a corresponding user profile for each of one or more users. Each respective user profile may be based on, for example, the respective historical device usage data for the users. Historical device usage data may include, but is not limited to, elements such as how the device is held, time of day, action taken, applications used, volume of the user's voice, etc. Thus, in some embodiments, the quality threshold may be adjusted to account for the differences in how each user interacts with the unit to determine what abnormal or degraded performance is for the sensor.If it is determined at 306 that the data quality has not degraded, the method 300 returns to 304 where the subsequent data is acquired at 304. If it is determined at 306 that the data quality has degraded, the method 300 proceeds to 308 where a visual indicator is displayed on a display of the unit. The visual indicator includes a non-text component that indicates to a user how a possible cause of the degraded quality of the data captured by the sensor can be resolved. For example, in some embodiments, the non-text component shows the user where the physical location of the sensor is on the device. The non-text component may also provide instructions to the user, e.g., to approach the sensor.In some embodiments, the visual indicator is displayed at a location on the display that is proximate to the physical location of the sensor. As discussed above, in some embodiments, the non-text component may include, for example, a symbol displayed near the physical location of the sensor, concentric circles centered near the physical location of the sensor, an arrow displayed near and pointing at the physical location of the sensor, etc. In some embodiments, the non-text component of the visual indicator may further include a representation of the unit displayed on the display of the unit. The representation may illustrate the physical location of the sensor, e.g., by highlighting the location of the sensor on the unit, including indicia indicating a location of the sensor on the unit, etc. The visual indicator may also, in some embodiments, be superimposed on the content already displayed on the display, similar to a pop-up message. In other embodiments, the content displayed on the device is replaced with the visual indicator. As discussed above, the visual indicator may also include textual components such as labels and a list of one or more possible causes for the degraded data. In some of these embodiments, the list may order the possible causes in the order of the probability that a particular possible cause is the actual cause of the degraded performance.At 310, a success notification may optionally be issued or provided to the user. The notification may be a visual notification, an audible notification, and / or haptic feedback. The success notification is configured to provide feedback to the user that the quality of the degraded data has improved, such that a comparison with the quality threshold indicates that the data is no longer deemed degraded. For example, the unit may detect and monitor usage actions performed after receiving a report of the degraded data. The unit may also further monitor the quality of the data received from the sensor. Thus, in response to detecting a measure of the user and determining that the quality of the data acquired by the sensor after the measure of the user has improved, the unit may output the success notification.At 312, the historical data may be updated based on the information about the degraded data, any actions of the user to remedy the degraded data, device usage data at the time of detecting the degraded data, etc. This feedback may be used to further improve the capability of the deteriorated data recognition unit, e.g., to be used to update the quality threshold. In some embodiments, the unit is further configured (e.g., a processor on the unit executing instructions) to predict when the quality of the data captured by a sensor is likely to degrade at 314. For example, if a particular user is determined to use the unit for telephony, the unit may use the historical data to predict how the user will handle the unit based on historical data and / or use it to determine whether the user is likely to obscure the microphone, talk too quiet, etc. Thus, in some of these embodiments, the unit may analyze historical unit usage data to determine a pattern that indicates a likelihood of sensor blockage, and compare current unit usage data to the determined pattern. In response to determining that the current device usage matches the determined pattern, the device may provide a notification along with the display of the visual indicator before detecting a degradation in the quality of the data captured by the sensor. The unit may thus provide proactive warnings to reduce the likelihood that the sensor will be blocked or otherwise data of degraded quality will be captured. The notification may be visual, audible, or haptic feedback.The unit may be configured, in some embodiments, to use machine learning methods to use space-time coordinates (i.e., x, y, z, t ∈R 3, T) to generate paths in the historical data that represent circumstances or conditions where the quality of the sensor-detected data has degraded. If a current predicted path resembles a previous path in the historical data where degraded data quality has been determined, the unit may predict future degradation in the quality of the data captured by a sensor (e.g., camera or microphone). P1, p2,... pn is intended to represent, for example, a set of paths in the historical data in which a deteriorated data quality has been determined. Each path pn is well defined by points in the space-time coordinates. Let Ki be a list of paths describing the current use of the mobile unit by a user. Each path ki is defined by points in the space-time coordinates. When a difference between the current paths ki and the historical paths pn is less than a predetermined threshold ε, the unit may predict a future degradation in the quality of data captured by a sensor (e.g., a camera or a microphone). For example, if |ki-p1|<ε, |kip2|<ε,..., |ki-pn|<ε for a certain number of n, the unit predicts a future degradation in the quality of the acquired data.The embodiments described herein thus enable abnormal or degraded performance of a sensor to be detected and provide visual instructions to the user to locate the sensor on the unit and remedy the possible causes of the abnormal performance. The embodiments described herein are also capable of proactively predicting when performance degradation occurs and determining the possible causes to provide a user with a warning or notification to prevent the quality of the captured data from degrading.The present invention can be a system, a method and / or a computer program product at any possible technical level of detail of the integration. The computer program product may comprise computer readable storage medium(s) on / on which computer readable program instructions are / are stored for causing a processor to carry out aspects of the present invention.The computer readable storage medium may be a physical device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage unit, a magnetic storage unit, an optical storage unit, an electromagnetic storage unit, a semiconductor storage unit, or any suitable combination thereof. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions stored thereon, and any suitable combination thereof. A computer readable storage medium, as used herein, is not intended to be transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through a wire.Computer readable program instructions described herein may be downloaded from a computer readable storage medium to respective data processing / processing devices or via a network such as the Internet, a local area network, a wide area network, and / or a wireless network to an external computer or storage device. The network may include copper transmission cables, lightwave transmission conductors, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing unit receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the corresponding computing / processing unit.Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, integrated circuit configuration data, or either source code or object code written in any combination of one or more programming languages, including object oriented programming languages such as Smalltalk, C++, or the like, and procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, via the Internet using an Internet Service Provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuits to perform aspects of the present invention.Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It is noted that each block of the flowcharts and / or the block diagrams, as well as combinations of blocks in the flowcharts and / or the block diagrams, may be executed by computer readable program instructions.These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed via the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions / steps specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored on a computer readable storage medium that can control a computer, programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored thereon comprises an article of manufacture including instructions that implement aspects of / the function / step specified in the flowchart and / or block diagram block or blocks.The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of process steps to be performed on the computer or other programmable apparatus or other device to produce a computer executed process such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / steps specified in the flowchart and / or block diagram block or blocks.The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that include one or more executable instructions for executing the particular logical function(s). In some alternative implementations, the functions specified in the block may occur in a different order than shown in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order depending on the appropriate functionality. It is further noted that each block of the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented by special purpose hardware-based systems that perform the specified functions or steps, or perform combinations of special purpose hardware and computer instructions.

Claims

A computer implemented method (300) comprising: retrieving (302), by a device (100), a physical location of a sensor (106, 108) on the device; setting a quality threshold based at least in part on a corresponding user profile for each of one or more users, each corresponding user profile based on corresponding historical device usage data for one of the one or more users; determining (306), by the device, that the quality of the data captured (304) by the sensor physically located on the device has degraded based on a comparison of the captured data with the quality threshold; in response to determining that the quality of the data captured by the sensor has degraded, displaying (308) a visual indicator (110) on a display (102) of the device, the visual indicator comprising at least one non-text component that indicates to a device user where the physical location of the sensor is on the device; and analyzing the corresponding historical device usage data for each of the one or more users to determine a pattern that indicates a likelihood of blockage of the sensor; comparing the current device usage data of one of the one or more users to the determined pattern; and in response to determining that the current device usage matches the determined pattern, providing a notification along with the display of the visual indicator before detecting a degradation in the quality of the data captured by the sensor.The method of claim 1, wherein the quality threshold is further based at least in part on a sensor profile for the sensor created based on historical performance data for the sensor.The method of claim 1, wherein the visual indicator is displayed on the display of the unit at a location proximate to the physical location of the sensor.The method of claim 3, wherein the non-text component of the visual indicator includes a plurality of concentric sub-circles centered near the physical location of the sensor.The method of claim 1, wherein the non-text component is animated to show the user where the physical location of the sensor is.The method of claim 1, wherein the non-text component of the visual indicator includes a representation of the entity displayed on the display, the representation illustrating the physical location of the sensor.The method of claim 1, further comprising providing (310) a success notification to the user in response to detecting a user action and determining that the data captured by the sensor after the user action has an improved quality.A computing device (100; 200) comprising: a display (102; 226); a sensor (106, 108) configured to acquire data; and a processor (202) coupled to the sensor and the display for purposes of communication of data, the processor configured to: determine a physical location of the sensor on the computing device; determine a quality threshold based at least in part on a corresponding user profile for each of one or more users, each corresponding user profile based on corresponding historical device usage data for one of the one or more users; determine that the quality of the data acquired by the sensor has degraded based on a comparison of the acquired data with the quality threshold; in response to determining that the quality of the data captured by the sensor has degraded, displaying a visual indicator (110) on the display of the device, the visual indicator comprising at least one non-text component configured to show a device user how to remedy a possible cause of the degraded quality of the data captured by the sensor; analyzing the corresponding historical device usage data for each of the one or more users to determine a pattern indicating a likelihood of blockage of the sensor; comparing the current device usage data of one of the one or more users to the determined pattern; and in response to determining that the current device usage matches the determined pattern, providing a notification along with the display of the visual indicator before detecting a degradation in the quality of the data captured by the sensor.The computing device of claim 8, wherein the processor is configured to: generate a sensor profile for the sensor based on historical performance data for the sensor; and set the quality threshold based in part on the sensor profile.The computing device of claim 8, wherein the visual indicator is displayed on the display of the computing device at a location proximate to the physical location of the sensor.The computing device of claim 8, wherein the at least one non-text component of the visual indicator includes a representation of the computing device displayed on the display, the representation illustrating the physical location of the sensor.The computing device of claim 8, wherein the at least one non-text component of the visual indicator is animated to show the user where the physical location of the sensor is.A computer program product comprising a computer readable storage medium having a computer readable program stored thereon, the computer readable program, when executed by a processor (202), causing the processor to perform a method according to any one of claims 1 to 7.

Images