Dynamic interface intervention to improve sensor performance

Abstract

A computer-implemented method comprising: obtaining a physical location of a sensor on a device; determining, based on a comparison of captured data to a quality threshold, that a quality of data captured by the sensor physically located on the device is degraded; and in response to determining that the quality of the data captured by the sensor is degraded, displaying a visual indicator on a display of the device. The visual indicator comprises at least one non-text component that points a user to the physical location of the sensor on the device.

Description

Background Technology

[0001] Users are increasingly able to capture video, images, and audio with their computing devices. For example, many personal devices (e.g., handheld and wearable devices) have microphones, cameras, and other sensors capable of capturing content. Furthermore, laptops, desktop computers, tablets, and the like are often equipped with such sensors capable of capturing content. Additionally, such electronic devices enable various forms of communication, including audio phone calls and video conferencing. However, one or more of the sensors on a device may be unintentionally covered and / or blocked by obstacles when capturing content (e.g., audio and / or video data), resulting in suboptimal content capture. Summary of the Invention

[0002] Aspects of the invention may include computer-implemented methods, computer program products, and systems. Examples of the methods include: obtaining the physical location of a sensor on a device; determining, based on a comparison of captured data with a quality threshold, that the quality of data captured by the sensor physically located on the device has been degraded; and, in response to determining that the quality of the data captured by the sensor has been degraded, displaying a visual indicator on a display of the device. The visual indicator includes at least one non-text component that directs the user to the physical location of the sensor on the device. Attached Figure Description

[0003] It should be understood that the accompanying drawings depict only exemplary embodiments and are therefore not intended to limit the scope. The exemplary embodiments will be described with additional features and details using the drawings, in which:

[0004] Figure 1 This is a block diagram of one embodiment of the example device.

[0005] Figure 2 This is a block diagram of one embodiment of an example computing device.

[0006] Figure 3 This is a flowchart illustrating one embodiment of an example method for improving sensor performance.

[0007] By convention, features described differently are not drawn to scale, but rather to emphasize specific features relevant to the exemplary embodiments. Detailed Implementation

[0008] In the following detailed description, reference is made to the accompanying drawings, which form a part of the description, and specific illustrative embodiments are shown by way of illustration. However, it should be understood that other embodiments may be utilized, and logical, mechanical, and electrical changes may be made. Furthermore, the methods presented in the drawings and description should not be construed as limiting the order in which the various steps can be performed. Therefore, the following detailed description should not be considered restrictive.

[0009] As used in this article, when used with reference items, "multiple" means one or more items. For example, "multiple different types of networks" means one or more different types of networks.

[0010] Furthermore, the phrases “at least one,” “one or more,” and “and / or” are open-ended expressions that are both combined and disjunctive in operation. For example, each of the expressions “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, and C,” “one or more of A, B, or C,” and “A, B, and / or C” means a single A, a single B, a single C, A and B together, A and C together, B and C together, or A, B, and C together. In other words, “at least one,” “one or more,” and “and / or” means that any combination of items and multiple items in the list can be used, but not all items in the list are required. Items can be specific objects, things, or categories. Furthermore, the quantity or number of each item in the combination of listed items does not need to be the same. For example, in some illustrative instances, “at least one of A, B, and C” could be, for example, but not limited to, two items A; one item B; and ten items C; or 0 of items A; four of items B and seven of items C; or other suitable combinations.

[0011] Furthermore, the term "a" or "an" entity refers to one or more of the entities. Therefore, the terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" are used interchangeably.

[0012] Furthermore, as used herein, the term "automatic" and its variations refer to any process or operation performed without substantial human input. However, a process or operation can be automatic if input is received prior to its execution, even if substantial or insignificant human input is used in its execution. Human input is considered substantial if it influences how the process or operation will be performed. Human input consenting to the execution of a process or operation is not considered "material." Additionally, the terms "communication" or "communicatively coupled" include the use of any electrical or optical connection (whether wireless or wired) that allows two or more systems, components, modules, devices, etc., to exchange data, signals, or other information using any protocol or format.

[0013] As discussed above, many computing devices have sensors (such as cameras and microphones) that enable users to capture data such as media content (e.g., images, videos, audio) from portable devices, temperature data, light data, proximity data, pressure data, etc. In fact, many such devices have multiple sensors, including multiple sensors of the same type (e.g., multiple cameras and / or multiple microphones). However, such sensors can be obstructed by obstacles. For example, one or more of these sensors can be obstructed by the user of the device (e.g., a hand or part of clothing can partially or completely cover the sensor). Additionally, sensors can be obstructed by other external factors. For example, dust can accumulate on the sensor (e.g., microphone), which degrades the quality of the captured data. As used herein, the terms “obstructed” and “blocked” and their variations are used interchangeably. Furthermore, as used herein, a sensor is considered covered or blocked when at least a portion of it is covered or blocked. As mentioned above, sensors can be covered by clothing, the user's fingers, dust on the sensor, etc.

[0014] Furthermore, sensors can be obstructed by physical objects such as another person located in front of the sensor (e.g., a person standing in front of a camera, obstructing the view of the scene being captured by the camera). Such obstruction or blockage of the sensor can degrade the quality of the captured media, which can lead to reduced user satisfaction. Additionally, if the user is too far from the sensor (e.g., a microphone during a video conference), or if the user speaks in a direction away from the microphone, the quality of the captured media can also be degraded. Such actions can further degrade the quality of the captured media. The embodiments described herein implement active communication with the user to facilitate mitigation of such causes of quality degradation in captured data. As used herein, the term "captured data" refers to data obtained from one or more sensors on a device. Such data can be media data (e.g., audio, images, video), temperature data, light data, pressure data, proximity data, location data, etc. For ease of explanation, the embodiments described below specifically refer to different types of media data. However, it should be understood that captured data can include data other than audio and visual data, and the embodiments described herein can be implemented to help facilitate correction of degraded quality of data captured by sensors other than microphones and cameras.

[0015] Figure 1This is a depiction of one embodiment of example device 100. Although device 100 is depicted as a tablet or smartphone, it should be understood that embodiments of the invention are not limited thereto. For example, in other embodiments, device 100 may be implemented as a desktop computer, laptop computer, or wearable device, etc. Device 100 includes a housing 104, a display 102, a camera 106, and a microphone 108. It should be understood that although only a single camera 106 and a single microphone 108 are depicted in this example, it should be understood that more than one camera and / or more than one microphone may be implemented in other embodiments. Similarly, in some embodiments, more than one display may be implemented, such as utilizing a desktop computer having, for example, more than one attached display unit.

[0016] Furthermore, in embodiments where device 100 has multiple sensors of the same type (e.g., multiple cameras or multiple microphones), each instance of a sensor can differ from the others. For example, a mobile device may include two or more camera sensors, each with different characteristics. For instance, a camera sensor may have a higher pixel density than another camera sensor on the same device, or a sensor located on a different part of the device (e.g., the front or rear side) may have a different focal length than another camera sensor, etc. Similarly, a device with multiple microphone sensors can have microphone sensors with different characteristics from each other.

[0017] Device 100 is configured to determine when the quality of data captured by a sensor degrades below a predetermined level or is predicted to degrade below a predetermined level. For example, device 100 may determine that a sensor capturing images, video, and / or audio is being obstructed and / or predict when such obstruction will occur. For example, in some embodiments, device 100 may obtain data from one or more proximity sensors 132 to determine that a camera sensor and / or microphone sensor is obstructed. Such proximity sensors 132 may be implemented as any suitable type of proximity sensor, including but not limited to capacitive, inductive, magnetic, and / or optical proximity sensors. Such proximity sensors 132 may be placed near a microphone or camera of device 100 such that the proximity sensors 132 can detect an object near the microphone or camera and that it may be obstructing the corresponding microphone or camera. Furthermore, device 100 may use data from proximity sensors 132 to identify how far a physical object or obstacle is from a camera or microphone and whether that object may be obstructing the camera or microphone.

[0018] Additionally, in some embodiments, device 100 is configured to analyze image and audio data as it is captured to identify quality degradation. For example, degradation in the quality of the captured data may indicate possible occlusion of the corresponding sensor. Furthermore, degradation in audio quality may indicate that the audio speaker or source is too far from the microphone 108 of device 100, such that the microphone 108 cannot capture audio with high quality or volume. Thus, in some embodiments, device 100 is configured to detect changes in the quality of the recorded data. For example, device 100 may detect changes in the amplitude of the audio data. Furthermore, such degradation can be determined by comparing features of the captured data with a baseline based on multiple previous data captured by the same sensor over time. In other words, if the sensor is occluded at the start of data capture, occlusion can be detected by comparing the data to a baseline, allowing detection even if there is no significant change in the data from the start of data capture.

[0019] Furthermore, baselines can be used to explain variations in usage by different users. For example, a first user might speak louder than a second user. Therefore, different corresponding baselines can be calculated for each of the first and second users. In some such embodiments, device 100 is configured to identify which user is currently using device 100. For example, device 100 can perform speech recognition analysis, image analysis, fingerprint analysis, and / or a unique password associated with each user, etc., to identify which user is using device 100. Therefore, device 100 can maintain different profiles for multiple users of device 100, such profiles including but not limited to corresponding baselines for detecting quality degradation, historical data associated with users, etc. Furthermore, in some embodiments, in addition to using data collected from device 100 or instead of using data collected from device 100, baselines can also be calculated based on crowdsourced data obtained from other devices.

[0020] Additionally, historical data collected from sensors (such as camera 106 and microphone 108, and other sensors) can be used to help predict when sensor occlusion will occur. For example, based on historical data, device 100 can determine the likelihood that a given user will obstruct the microphone while making a phone call by analyzing patterns in how the user holds the device and the resulting microphone 108 occlusion. In other words, device 100 can predict that when a user is answering a call or making a phone call, the user will typically hold device 100 in a manner that obstructs the microphone. Similar historical patterns can be identified for other sensors and activities, including but not limited to taking photos, participating in video conferences, etc. Furthermore, in some embodiments, device 100 may include additional sensors that help identify patterns in historical data. For example, device 100 may include one or more pressure sensors 130 that provide data on the user's grip pattern, which can be used to predict when a sensor will be obstructed.

[0021] Additionally, in some embodiments, historical data can be further used to help identify potential causes of quality degradation in sensor data. For example, if historical data indicates that the quality of audio received via microphone 108 has gradually degraded over a period of time, this could indicate that lint or dust has accumulated on microphone 108. Conversely, if degradation occurs suddenly, this could indicate that microphone 108 has been covered by an object (such as clothing or a hand). Thus, by analyzing historical data, device 100 is able to determine the relative likelihood of different causes of quality degradation.

[0022] Device 100 is further configured to provide guidance to the user of device 100 to mitigate detected quality degradation. Specifically, device 100 is configured to display one or more visual indicators such as indicators 110, 112, and 114. Visual indicators 110, 112, and 114 are selected to help guide the user in mitigating the causes of data quality degradation or deterioration. For example, many devices have multiple components, such as multiple cameras, microphones, speakers, etc. The user may not know or be able to easily determine where each camera, microphone, etc., is located or which of the multiple sensors is receiving data with degraded quality. Additionally, in some cases, the user may not be aware of the degraded quality. For example, when speaking in a video conference, the user may not be aware that their video is not being captured and transmitted to other participants in the video conference. This may happen, for example, if the user has not removed a sliding privacy shield covering the camera. Similarly, the user may not be aware that the audio being transmitted to other participants in a video conference or phone call has degraded quality (e.g., muted, low volume, distortion, etc.). Therefore, one or more of the visual indicators 110, 112 and 114 can be used to warn users of a decline in detection quality and guide users to mitigate the cause.

[0023] For example, the visual indicator 110 includes multiple concentric circles 122 and an icon 120. In this example, the concentric circles 122 are approximately centered near the microphone 108. Similarly, the icon 120 is located near the physical location of the microphone 108. By positioning the icon 120 and circles 122 near the physical location of the microphone 108, the visual indicator 110 can guide the user to the location of the microphone 108. Furthermore, in this example, the icon 120 depicts a general microphone to indicate to the user that the visual indicator 110 is referencing the location of the microphone 108. As discussed above, the user may not know or be able to easily determine where a sensor is located or which sensor is obstructed. Therefore, by positioning the indicator 110 near the physical location of the microphone 108, the user can easily locate sensors that have been detected as inputting degraded data. Furthermore, although in Figure 1 The visual indicator 110 is shown as a static visual depiction, but in some embodiments, it is not a static visual depiction. That is, the visual indicator 110 may be animated according to a pattern (such as a pulse or wave pattern) (such as changing colors, color intensity, or brightness) to direct the user to the location of the microphone 108. It should be understood that the visual indicator 110 is provided by way of example only. Specifically, in other embodiments, the visual indicator does not include icon 108 and / or concentric circles 122. In other embodiments, the visual indicator may include other components in addition to or in place of icon 108 and / or concentric circles 122.

[0024] For example, visual indicator 112 includes an icon 116 and an arrow 118. Therefore, visual indicator 112 does not include concentric circles 122. Instead, icon 116 and arrow 118 are located near the physical location of the sensor (i.e., camera 106 in this example). Additionally, arrow 118 points to the location of camera 106. Similar to visual indicator 110, visual indicator 112 can also be animated to change color, intensity, brightness, etc., to guide the user to the physical location of the sensor. Furthermore, the implementation of the visual indicator does not need to be located on display 102 near the physical location of the corresponding sensor. For example, visual indicator 114 is a depiction of device 100 that includes labels 126 and 128 indicating the location of the corresponding sensor.

[0025] In some embodiments, visual indicators 110, 112, and / or 114 may be displayed on display 102 by overlaying visual indicators 110, 112, and 114 on other content displayed on display 102. In other embodiments, the content displayed on display 102 is minimized, causing a new screen or window to open to display one or more of visual indicators 110, 112, or 114. Furthermore, more than one visual indicator may be displayed simultaneously. For example, indicator 114 depicting a diagram of device 100 may be displayed simultaneously with visual indicators 110 and / or 112. Additionally, other information may be displayed along with one or more of the visual indicators. For example, a textual list of possible causes and / or possible steps a user may take may be displayed on display 102 along with one or more visual indicators that guide the user to the location of a given sensor or provide additional visual guidance to the user. Such a list of texts can be ordered based on the probability or likelihood of a degradation cause determined by device 100 based on historical data and data from other sensors, such as, for example, sensors 130 and / or 132. In another example, the text information can confirm or guide the user to the location of a sensor. For example, if a given sensor is located on a different side of device 100 than display 102, the text information can convey that information to the user.

[0026] In addition to or replacing the location of a given sensor, other variations of visual indicators can provide additional guidance to the user. For example, an image of a hand instructing the user to move closer could be displayed on display 102, letting the user know that device 100 is too far away and that the user needs to move closer to device 100. Therefore, in Figure 1 The visual indicators 110, 112 and 114 described herein are provided by way of example only, and other visual indicators may be implemented in other embodiments to guide the user to mitigate the possible causes of degradation in the quality of media data captured by one or more sensors.

[0027] Information about the location of sensors on device 100 can be obtained in various ways. For example, in some embodiments, an application running on device 100 can collect identification information about model, manufacturer, etc., from device 100. In some embodiments, the application can collect device information about sensor type, sensor location, display size, display type, sensor characteristics, etc., from the manufacturer or other repositories via a network, based on the collected identification information (e.g., model, serial number, etc.). In other embodiments, a database or repository of device information for multiple devices can be stored on device 100, and the corresponding device information can be located based on identification information collected by an application running on device 100. Device information (such as sensor location and type) can be used to position visual indicators 110, 112, and / or 114 in appropriate locations on display 102. Furthermore, while the functionality of the embodiments described herein can be implemented at the application level, it should be understood that in some embodiments, the functionality of the embodiments described herein can be implemented at the operating system level or a combination of application and operating system levels.

[0028] It should be understood that Figure 1 The components of the device 100 shown are provided by way of example only, and other implementations may include, except for, or replace, the components. Figure 1 Other components of those shown. For example, device 100 may include one or more haptic vibration motors that can provide haptic feedback to the user in addition to visual indicators. For example, if a user is talking on a phone and is not looking at display 102, the haptic vibration motors can be used to notify the user that they should be looking at display 102. Furthermore, device 100 may include one or more speakers, which in some embodiments can provide audio instructions or guidance to the user in addition to the visual guidance provided on display 102.

[0029] Figure 2 This is a block diagram of an embodiment of an example computing device 200 configured to perform the functions of the device 100 discussed above. Figure 2 The components of the computing device 200 shown include one or more processors 202, memory 204, memory interface 216, input / output (“I / O”) device interface 212, and network interface 218, all of which are directly or indirectly communicatively coupled for inter-component communication via memory bus 206, I / O bus 208, bus interface unit (“IF”) 209, and I / O bus interface unit 210.

[0030] exist Figure 2In the illustrated embodiments, computing device 200 includes one or more general-purpose programmable central processing units (CPUs) 202A and 202B, collectively referred to herein as processor 202. In some embodiments, computing device 200 includes multiple processors. However, in other embodiments, computing device 200 is a single CPU system. Each processor 202 executes instructions stored in memory 204. Furthermore, while embodiments are described with respect to central processing unit chips, it should be understood that, in addition to or instead of CPU chips, the embodiments described herein can also be applied to computer systems utilizing digital signal processor (DSP) and / or graphics processing unit (GPU) chips. Therefore, references to processors or processing units herein may refer to CPU chips, GPU chips, and / or DSPs.

[0031] In some embodiments, memory 204 includes random access semiconductor memory, storage device, or storage medium (volatile or non-volatile) for storing or encoding data and programs. For example, memory 204 stores instructions 211, historical data 213, and device information 215. When executed by a processor such as processor 202, instructions 211 cause processor 202 to perform the functions and calculations discussed herein regarding the detection of quality degradation of captured media data (e.g., images, video, audio) and to provide visual indicators to guide the user to mitigate sources of degradation, as discussed herein. Furthermore, although not explicitly stated... Figure 2 As shown, in some embodiments, memory 204 further stores profile data for multiple users as discussed herein. As discussed herein, the profile data can be used for personalized degradation detection and to provide personalized guidance to mitigate the causes of detected degradation.

[0032] In some embodiments, memory 204 represents the entire virtual memory of computing device 200 and may also include virtual memory of other computer devices coupled to computing device 200 via a network. In some embodiments, memory 204 is a single monolithic entity, but in other embodiments, memory 204 includes a hierarchy of caches and other memory devices. For example, memory 204 may reside in multi-level caches, and these caches may be further functionally partitioned such that one cache holds instructions while another holds non-instruction data used by the processor. Memory 204 may be further distributed and associated with different processing units or groups of processing units, as is known in various so-called Non-Uniform Memory Access (NUMA) computer architectures. Therefore, although for illustrative purposes, instructions 211, historical data 213, and device information 215 are stored in Figure 2The example shown uses the same memory 204, but it should be understood that other implementations may be carried out differently. For example, instructions 211, historical data 213, and / or device information 215 may be distributed across multiple physical media.

[0033] Figure 2 The computing device 200 in the illustrated embodiment further includes a bus interface unit 209 for handling communication between the processor 202, memory 204, display system 224, and I / O bus interface unit 210. I / O bus interface unit 210 is coupled to I / O bus 208 for transferring data to and from different I / O units. Specifically, I / O bus interface unit 210 can communicate with multiple I / O interface units 212, 216, and 218 (also referred to as I / O processors (IOPs) or I / O adapters (IOAs)) via I / O bus 208. Display system 224 includes a display controller, display memory, or both. The display controller can provide video, still images, audio, or combinations thereof to display device 226. Display memory can be dedicated memory for buffering video data. Display system 224 is coupled to display device 226. In some embodiments, display device 226 further includes one or more speakers for presenting audio. Alternatively, one or more speakers for presenting audio can be coupled to the I / O interface unit. In alternative embodiments, one or more functions provided by display system 224 are on an integrated circuit that also includes processor 202. Furthermore, in some embodiments, one or more functions provided by bus interface unit 209 are on an integrated circuit that also includes processor 202.

[0034] The I / O interface unit supports communication with various storage and I / O devices. For example, the I / O device interface unit 212 supports the attachment of one or more user I / O devices 220, which may include user output devices and user input devices (such as keyboards, mice, keypads, touchpads, trackballs, buttons, light pens, or other pointing devices). Users can use the user interface to manipulate the user input devices to provide input data and commands to the user I / O device 220. Additionally, users can receive output data via user output devices. For example, a user interface can be presented via the user I / O device 220, such as being displayed on a display device or played through a speaker.

[0035] Storage interface 216 supports the attachment of one or more storage devices 228, such as flash memory. The contents of memory 204, or any portion thereof, can be stored to and retrieved from storage device 228 as needed. Network interface 218 provides one or more communication paths from computing device 200 to other digital devices and computer equipment. For example, in some embodiments, computing device 200 can communicate with a server to request and receive device information via network interface 218.

[0036] although Figure 2 The illustrated computing device 200 shows a specific bus architecture providing a direct communication path between processor 202, memory 204, bus interface unit 209, display system 224, and I / O bus interface unit 210. However, in alternative embodiments, computing device 200 includes different buses or communication paths, which can be arranged in any of a variety of forms, such as point-to-point links in hierarchical, star, or network configurations, multiple hierarchical buses, parallel and redundant paths, or any other suitable type of configuration. Furthermore, although I / O bus interface unit 210 and I / O bus 208 are shown as a single corresponding unit, in other embodiments, computing device 200 may include multiple I / O bus interface units 210 and / or multiple I / O buses 208. While multiple I / O interface units are shown separating I / O bus 208 from different communication paths running to different I / O devices, in other embodiments, some or all I / O devices are directly connected to one or more system I / O buses.

[0037] As discussed above, in some embodiments, Figure 2 One or more of the components and data shown include instructions or statements that execute on processor 202 or are interpreted by instructions or statements that execute on processor 202 to implement the functions as described herein. In other embodiments, instead of or in addition to processor-based systems, implementation is carried out in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices. Figure 2 One or more of the components shown. Furthermore, in other embodiments, [the following may be omitted]. Figure 2 Some of the components shown may be and / or may include other components.

[0038] Figure 3This is a flowchart depicting one implementation of an example method 300 for improving sensor performance. Method 300 may be implemented by a computing device, such as computing device 200 discussed above. It should be understood that, for illustrative purposes, the order of actions in example method 300 is provided, and in other embodiments, the method may be performed in a different order. Similarly, it should be understood that some actions may be omitted or additional actions may be included in other embodiments. Furthermore, it should be understood that a user may opt in to data collection before performing method 300 on a device such as device 100 or 200. For example, a dialog box may be presented to the user requesting their selection whether to authorize data collection and analysis.

[0039] At 302, device information is obtained. For example, as discussed above, in some embodiments, a repository of device information can be searched based on the device's identification information. Device information may include the type of sensor, sensor characteristics, and / or the physical location of one or more sensors on the device. Furthermore, the device information repository may be stored locally on the device or accessed via a network (such as the Internet). Additionally, in some embodiments, the identification information may be obtained by an application running on the device, which obtains the identification information from the device, as discussed above.

[0040] At 304, data is captured by one of the sensors on the device. For example, a camera may capture video or images. Similarly, a microphone may capture audio data. Alternatively, other sensors, such as a light sensor that acquires data about ambient light or a pressure sensor that acquires data about applied pressure, may be used to capture the corresponding data. At 306, it is determined whether the quality of the captured data has degraded. For example, the captured data may be compared to a quality threshold to determine if the captured data has degraded. In some embodiments, sensor profiles may be developed for each sensor to be analyzed based on historical performance data of the sensors. For example, profiles for each camera, each microphone, etc. For example, regarding audio data from a microphone, in some embodiments, Fourier series may be used to develop a mathematical approximation of the sound profile for the microphone, since audio quality tends to follow a sine wave format. In some embodiments, regarding a camera capturing image data, the image data may be evaluated from a pixel-by-pixel perspective. For example, in some such embodiments, a convolutional neural network (CNN) image classifier may be implemented to identify camera obstruction or occlusion that degrades the quality of the acquired image data (e.g., still photos or videos) by identifying what certain pixels look like.

[0041] Convolutional Neural Networks (CNNs) are a class of deep feedforward artificial neural networks that have been successfully applied to the analysis of visual images. A CNN consists of artificial or neuron-like structures with learnable weights and biases. Each neuron receives some input and performs a dot product. A typical CNN architecture comprises a stack of layers that operate to receive input (e.g., a single vector) and transform it through a series of hidden layers. Each hidden layer consists of a set of neurons, each with learnable weights and biases, where each neuron can be fully connected to all neurons in the previous layer, and where neurons in a single layer can function independently without any shared connections. The final layer is a fully connected output layer, and in a classification setting, it represents the class score, which can be any real-valued number or a real-valued objective (e.g., in regression). However, it should be understood that the embodiments described herein are not limited to CNNs, but can be implemented using other types of neural networks, such as recurrent neural networks (RNNs), deep convolutional inverse graph networks (DCIGNs), etc., to analyze images and videos.

[0042] Furthermore, determining the quality threshold for the data acquired by the sensor can be based, at least in part, on the respective user profiles of each of one or more users, as discussed above. For example, each respective user profile can be based on the user's corresponding historical device usage data. Historical device usage data can include, but is not limited to, items such as how the device is held, time of day, actions taken, applications being used, and the volume of the user's voice. Therefore, in some embodiments, the quality threshold can be adapted to interpret the differences in how each user interacts with the device to determine what is considered anomaly or degraded performance of the sensor.

[0043] If it is determined at 306 that the data quality has not degraded, method 300 returns to 304, where subsequent data is captured. If it is determined at 306 that the data quality has degraded, method 300 proceeds to 308, where a visual indicator is displayed on the device's display. This visual indicator includes a non-text component that guides the user to mitigate potential causes of quality degradation in the data captured by the sensor. For example, in some embodiments, the non-text component guides the user to the physical location of the sensor on the device. Additionally, the non-text component may guide the user to take actions such as moving closer to the sensor.

[0044] In some embodiments, the visual indicator is displayed on the display at a location close to the physical location of the sensor. For example, as discussed above, in some embodiments, non-text components may include an icon displayed near the physical location of the sensor, a concentric circle centered near the physical location of the sensor, an arrow displayed near and pointing to the physical location of the sensor, etc. Additionally, in some embodiments, the non-text components of the visual indicator may include an illustration of the device displayed on the device's display. This illustration may show the physical location of the sensor, such as by highlighting the location of the sensor on the device, including labels indicating the location of the sensor on the device, etc. Furthermore, in some embodiments, similar to a pop-up message, the visual indicator may overlay on top of content already displayed on the display. In other embodiments, the content displayed on the device is replaced by the visual indicator. Furthermore, as discussed above, the visual indicator may also include text components (such as labels) and a list of one or more potential causes of degraded data. In some such embodiments, the list may be ordered according to the likelihood that a given potential cause is an actual cause of performance degradation.

[0045] At 310, a success notification may optionally be output or provided to the user. This notification may be a visual notification, an auditory notification, and / or haptic feedback. The success notification is configured to provide the user with feedback that the quality of the degraded data has been improved, such that a comparison with a quality threshold indicates that the data is no longer determined to be degraded. For example, the device may detect and monitor a usage action taken after receiving a notification of degraded data. Furthermore, the device may continue to monitor the quality of data received from sensors. Therefore, in response to detecting a user action and determining that the data captured by the sensors after the user action has improved quality, the device may output a success notification.

[0046] At 312, historical data can be updated based on information about degraded data, any user actions taken to mitigate the degraded data, device usage data when degraded data is detected, etc. This feedback can be used to further improve the device's ability to detect degraded data, such as for updating quality thresholds. Furthermore, in some embodiments, at 314, the device is configured (e.g., a processor on the device executes instructions) to predict when the quality of data captured by the sensor is likely to be degraded. For example, if a given user is detected using the device to make a phone call, the device can use historical data to predict how the user will handle the situation and / or use the device based on historical data to determine whether the user is likely to muffle the microphone, speak too softly, etc. Therefore, in some such embodiments, the device can analyze historical device usage data to identify patterns indicating the likelihood of sensor occlusion and compare current device usage data with the identified patterns. In response to determining that current device usage matches the identified pattern, the device can provide notification and display visual indicators before detecting a degradation in the quality of data captured by the sensor. Thus, the device can provide proactive warnings to reduce the likelihood that the sensor will be blocked or otherwise capture degraded quality data. The notification can be visual, auditory, or tactile feedback.

[0047] In some embodiments, the device can be configured to use machine learning techniques to utilize space-time coordinates (i.e., The device creates paths representing environments or states in historical data, where the quality of data collected by sensors will be degraded. If the currently predicted path is similar to a previous path in historical data where degraded data quality was detected, the device can predict future degradation of the quality of data captured by sensors (e.g., a camera or microphone). For example, let p1, p2, ..., pn represent a set of paths in historical data where degraded data quality was detected. Each path pn is well defined by a point in space-time coordinates. Let ki be a list of paths describing the user's current use of the mobile device. Each path ki is defined by a point in space-time coordinates. If the difference between the current path ki and the historical path pn is less than a predetermined threshold difference ε, the device can predict future degradation of the quality of data captured by sensors (e.g., a camera or microphone). For example, if for a limited number n, |ki-p1| < ε, |ki-p2| < ε, ..., |ki-pn| < ε, then the device predicts future degradation of the quality of the captured data.

[0048] Therefore, the embodiments described herein enable the detection of sensor anomalies or performance degradation, and provide users with visual guidance to locate the sensor on the device and correct potential causes of abnormal performance. Furthermore, the embodiments described herein can proactively predict when degraded performance will occur, along with providing warnings or notifications to users to help them prevent potential causes of quality degradation in the data being collected.

[0049] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0050] Computer-readable storage media can be tangible means for retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0051] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0052] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0053] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0054] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0055] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0056] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0057] While specific embodiments have been shown and described herein, those skilled in the art will understand that any arrangement calculated to achieve the same purpose may replace the specific embodiments shown. Therefore, the invention is clearly intended to be limited only by the claims and their equivalents.

Claims

1. A computer implementation method, comprising: The physical location of the sensors on the device is obtained by the device; The quality threshold is determined at least in part based on historical device usage data associated with users of the device, the historical device usage data indicating patterns in which the sensor is occluded by users of the device; The device determines that the quality of the data captured by the sensor physically located on the device has been degraded based on a comparison of the captured data with the quality threshold. as well as In response to determining that the quality of the data captured by the sensor has been degraded, a visual indicator is displayed on the device's display, wherein the visual indicator includes at least one non-text component that guides the user of the device to the physical location of the sensor on the device.

2. The method of claim 1, wherein the quality threshold is further determined at least in part based on a sensor profile of the sensor developed based on historical performance data of the sensor.

3. The method of claim 1, wherein the historical device usage data of the users of the one or more of the devices is obtained from the respective user profiles of the users of the one or more of the devices, wherein each respective user profile is based on the respective historical device usage data of one of the users of the one or more of the devices.

4. The method of claim 1, further comprising: Compare the current device usage data with the pattern in which the sensor is blocked by the user of the device; as well as In response to determining that the current device usage data matches the pattern in which the sensor is occluded by the user of the device, a notification is provided and the visual indicator is displayed before a degradation in the quality of the data captured by the sensor is detected.

5. The method of claim 1, wherein, The visual indicator is displayed on the device's display at a location close to the physical location of the sensor.

6. The method of claim 5, wherein, The non-text component of the visual indicator includes a plurality of concentric partial circles centered near the physical location of the sensor.

7. The method of claim 1, wherein, The non-text component is animated to guide the user to the physical location of the sensor.

8. The method according to claim 1, wherein, The non-text component of the visual indicator includes a diagram of the device displayed on the display, the diagram showing the physical location of the sensor.

9. The method of claim 1, further comprising providing a success notification to the user in response to detecting a user action and determining that the data captured by the sensor after the user action has improved quality.

10. A computing device, comprising: monitor; The sensor is configured to capture data; as well as A processor communicatively coupled to the sensor and the display, wherein the processor is configured to: Identify the physical location of the sensor on the computing device; The quality of the data captured by the sensor is determined to be degraded based on a comparison of the captured data with a quality threshold, wherein the quality threshold is determined at least in part based on historical device usage data associated with the user of the computing device, the historical device usage data indicating patterns in which the sensor is occluded by the user of the computing device; as well as In response to determining that the quality of the data captured by the sensor has been degraded, a visual indicator is displayed on the display, wherein the visual indicator includes at least one non-text component configured to guide the user of the computing device to mitigate potential causes of the quality degradation of the data captured by the sensor.

11. The computing device according to claim 10, wherein, The processor is configured to: Develop a sensor configuration file for the sensor based on its historical performance data; and The quality threshold is determined in part based on the sensor profile.

12. The computing device of claim 10, wherein historical device usage data of the users of the computing device is obtained from corresponding user profiles of users of one or more of the computing devices, wherein each corresponding user profile is based on corresponding historical device usage data of one of the users of the one or more of the computing devices.

13. The computing device according to claim 10, wherein, The visual indicator is displayed on the display of the computing device at a location close to the physical location of the sensor.

14. The computing device according to claim 10, wherein, The at least one non-text component of the visual indicator includes a diagram of the computing device displayed on the display, the diagram showing the physical location of the sensor.

15. The computing device according to claim 10, wherein, The at least one non-text component of the visual indicator is animated to guide the user of the computing device to the physical location of the sensor.

16. A computer program product having a computer-readable program, wherein, The computer-readable program, when executed by a processor, causes the processor to: Identify the physical locations of sensors on computing devices; The quality threshold is determined at least in part based on historical device usage data associated with users of the computing device, the historical device usage data indicating patterns in which the sensor is occluded by users of the computing device; Based on a comparison of the captured data with the quality threshold, it is determined that the quality of the data captured by the sensor has been degraded; as well as In response to determining that the quality of the data captured by the sensor has been degraded, a visual indicator is displayed on the display, wherein the visual indicator includes at least one non-text component that guides the user of the computing device to the physical location of the sensor on the computing device.

17. The computer program product according to claim 16, wherein, The computer-readable program is further configured to cause the processor to: Develop a sensor configuration file for the sensor based on its historical performance data; and The quality threshold is determined in part based on the sensor profile.

18. The computer program product according to claim 16, wherein, Historical device usage data of users of one or more of the computing devices are obtained from their respective user profiles, wherein each respective user profile is based on the corresponding historical device usage data of one of the users of the one or more of the computing devices.

19. The computer program product according to claim 16, wherein, The visual indicator is displayed on the display of the computing device at a location close to the physical location of the sensor.

20. The computer program product according to claim 16, wherein, The non-text component of the visual indicator includes a diagram of the computing device displayed on the display, the diagram showing the physical location of the sensor.

Images