Deep learning-based real-time detection and correction of compromised sensors in autonomous machines

Abstract

Device (100) for enabling deep learning-based real-time detection and correction of compromised sensors in autonomous machines, the device comprising: one or more processors (112, 114) to enable: Detection and acquisition logic (201) to enable one or more sensors (231) to acquire one or more images of a scene, wherein it is determined that one of the images is unclear, and wherein the one or more sensors comprise one or more cameras; and Classification and prediction logic (207) to enable a deep learning model (421) to identify a sensor associated with the image in real time based on one or more images using a classifier.

Description

AREA

[0001] Embodiments described herein relate generally to data processing and, more particularly, to enabling deep learning-based real-time detection and remediation of compromised sensors in autonomous machines. BACKGROUND

[0002] Autonomous machines are expected to grow exponentially in the coming years, which in turn will likely require sensors, such as cameras, to lead the growth in enabling various tasks, such as autonomous driving.

[0003] Conventional techniques use multiple sensors to attempt to apply data / sensor fusion to provide some redundancy to ensure accuracy; however, these conventional techniques are severely limited in that they are unable to handle or work around sensors that provide low-quality or misleading data.

[0004] REALPE, M. et al.: "Sensor Fault Detection and Diagnosis for Autonomous Vehicles" in: MATEC Web of Conferences, 2015. DOI: 10.1051 / matecconf / 20153004003, describes a method for detecting and diagnosing faulty sensors for autonomous vehicles. Data from different sensors is summarized and compared. Faulty sensors are then identified based on a detected difference.

[0005] DONG, L. et al.: “Camera Anomaly Detection based on Morphological Analysis and Deep Learning”, in: 2016 IEEE International Conference on Digital Signal Processing (DSP), 2016. https: / / ieeexplore.ieee.org / document / 7868559), describes another method for identifying disturbances in sensor data, whereby in particular a morphological analysis in combination with deep learning techniques is used to identify various disturbances. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals refer to similar elements. Fig. 1 illustrates a computing device employing an automatic sensor testing mechanism according to one embodiment. Fig. 2 illustrates the automatic sensor testing mechanism from Fig. 1 according to one embodiment. Fig. 3A illustrates static inputs from multiple sensors according to one embodiment. Fig. Figure 3B illustrates dynamic inputs from a single sensor according to one embodiment. Fig. Figure 3C illustrates dynamic inputs from a single sensor according to one embodiment. Fig. 4A illustrates an architectural layout with a transaction sequence for real-time detection and remediation of compromised sensors using deep learning according to one embodiment. Fig. Figure 4B illustrates a method for real-time detection and remediation of compromised sensors using deep learning according to one embodiment. Fig. 5 illustrates a computing device capable of supporting and implementing one or more embodiments according to an embodiment. Fig. 6 illustrates one embodiment of a computing environment capable of supporting and implementing one or more embodiments according to an embodiment. DETAILED DESCRIPTION

[0007] In the following description, numerous specific details are set forth; however, embodiments as described herein may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques are not shown in detail in order not to obscure the understanding of this description.

[0008] Embodiments provide a novel technique for deep learning-based detection, notification, and remediation of compromised sensors in autonomous machines. In one embodiment, the automated testing of one or more of the following may include detecting compromised sensors, issuing alerts to warn of the compromised sensors, offering the opportunity to remediate any distortions of compromised sensors in real time, and / or the like.

[0009] It is contemplated that embodiments are not limited to any number or type of sensors; however, for brevity, clarity, and understandability, one or more cameras may be used as example sensors in this document, but the embodiments are not so limited.

[0010] It is contemplated that terms such as "request," "query," "job," "work," "work item," and "workload" may be used interchangeably in this document. Similarly, an "application" or "agent" may refer to or comprise a computer program, software application, game, workstation application, etc., offered through an application programming interface (API), such as a free rendering API, such as Open Graphics Library (OpenGL®), DirectX® 11, DirectX® 12, etc., where "dispatch" may be interchangeably referred to as a "work unit" or "draw," and "application" may similarly be interchangeably referred to as a "workflow" or simply an "agent."For example, a workload such as a three-dimensional (3D) game may comprise and output any number and type of "frames," where each frame may represent an image (e.g., sailboat, human face). Further, each frame may comprise and output any number and type of work units, where each work unit may represent a portion (e.g., mast of a sailboat, forehead of a human face) of the image (e.g., sailboat, human face) represented by its corresponding frame. However, for consistency, each element may be referred to by a single term (e.g., "dispatch," "agent," etc.) throughout this document.

[0011] In some embodiments, terms such as "display screen" and "display surface" may be used interchangeably, referring to the visible portion of a display device, while the remainder of the display device may be embedded in a computing device, such as a smartphone, wearable device, etc. It is contemplated and appreciated that embodiments are not limited to any particular computing device, software application, hardware component, display device, display screen or surface, protocol, standard, etc.For example, embodiments may be applied to and used with any number and type of real-time applications on any number and type of computers, such as desktops, laptops, tablet computers, smartphones, head-mounted displays, and other wearable devices, and / or the like. Furthermore, for efficient performance using this novel technique, rendering scenarios may range from simple scenarios such as desktop compositing to complex scenarios such as 3D gaming, augmented reality applications, etc.

[0012] Please note that throughout this document, terms or acronyms such as convolutional neural network (CNN), CNN, neural network (NN), NN, deep neural network (DNN), DNN, recurrent neural network (RNN), RNN, and / or the like may be interchangeably referred to. Furthermore, throughout this document, terms such as "autonomous machine" or simply "machine," "autonomous vehicle" or simply "vehicle," "autonomous agent" or simply "agent," "autonomous device" or "computing device," "robot," and / or the like may be interchangeably referred to.

[0013] Fig. 1 illustrates a computing device 100 employing an automatic sensor testing mechanism ("automatic testing mechanism") 110 according to one embodiment. The computing device 100 represents a communications and computing device that includes or represents any number and type of intelligent devices, such as (without limitation) intelligent command devices or intelligent personal assistants, home / office automation systems, household appliances (e.g., washing machines, televisions, etc.), mobile devices (e.g., smartphones, tablet computers, etc.), gaming devices, handheld devices, wearable devices (e.g., smart watches, smart bracelets, etc.), virtual reality (VR) devices, head-mounted displays (HMDs), Internet of Things (IoT) devices, laptop computers, desktop computers, server computers, set-top boxes (e.g.,Internet-based set-top boxes for cable TV, etc.), Global Positioning System (GPS)-based devices, etc.

[0014] In some embodiments, computing device 100 may include (without limitation) autonomous machines or artificially intelligent agents, such as mechanical agents or machines, electronic agents or machines, virtual agents or machines, electromechanical agents or machines, etc. Examples of autonomous machines or artificially intelligent agents may include (without limitation) robots, autonomous vehicles (e.g., self-driving cars, self-flying aircraft, self-sailing boats, etc.), autonomous devices (self-driving construction vehicles, self-operating medical devices, etc.), and / or the like.Furthermore, “autonomous vehicles” are not limited to automobiles, but may include any number and type of autonomous machines, such as robots, autonomous appliances, autonomous household appliances, and / or the like, and any one or more tasks or operations related to such autonomous machines may be interchangeably referenced with autonomous driving.

[0015] For example, the computing device 100 may include a computing platform that hosts an integrated circuit (“IC”), such as a system on a chip (“SoC” or “SOC”) that integrates various hardware and / or software components of the computing device 100 on a single chip.

[0016] As illustrated, in one embodiment, computing device 100 may include any number and type of hardware and / or software components, such as (without limitation) a graphics processing unit (“GPU,” or simply “graphics processor”) 114, a graphics driver (also referred to as a “GPU driver,” “graphics driver logic,” “driver logic,” user-mode driver (UMD), UMD, user-mode driver framework (UMDF), UMDF, or simply “driver”) 116, a central processing unit (“CPU,” or simply “application processor”) 112, memory 104, network devices, drivers, or the like, and input / output (I / O) sources 108, such as touch screens, touch panels, touch pads, virtual or regular keyboards, virtual or regular mice, ports, connectors, etc.Computing device 100 may include an operating system (OS) 106 that serves as an interface between hardware and / or physical resources of computing device 100 and a user.

[0017] It should be understood that a less or more equipped system than the example described above may be preferred for certain implementations. Therefore, the configuration of computing device 100 may vary from implementation to implementation depending on numerous factors, such as price constraints, performance requirements, technological improvements, or other circumstances.

[0018] Embodiments may be implemented as any one or more microchips or one or more integrated circuits interconnected using a motherboard, hard-wired logic, software stored by a memory device and executed by a microprocessor, firmware, an application-specific integrated circuit (ASIC), and / or a field-programmable gate array (FPGA). The terms "logic," "module," "component," "engine," and "mechanism" may include, by way of example, software or hardware and / or a combination thereof, such as firmware.

[0019] In one embodiment, as illustrated, the automatic testing mechanism 110 may be hosted by the operating system 106 in communication with I / O source(s) 108 of the computing device 100. In another embodiment, the automatic testing mechanism 110 may be hosted or enabled by the graphics driver 116. In yet another embodiment, the automatic testing mechanism 110 may be hosted by the graphics processing unit ("GPU," "Graphics Processing Unit," or simply "graphics processor") 114 or firmware of the graphics processor 114, or a portion thereof. For example, the automatic testing mechanism 110 may be embedded in, or implemented as part of, the processing hardware of the graphics processor 114.Similarly, in yet another embodiment, the automatic checking mechanism 110 may be hosted by the central processing unit ("CPU," "Central Processing Unit," or simply "application processor") 112 or a portion thereof. For example, the automatic checking mechanism 110 may be embedded in or implemented as part of the processing hardware of the application processor 112.

[0020] In yet another embodiment, the automatic testing mechanism 110 may be hosted by any number and type of components of the computing device 100 or a portion thereof. For example, a portion of the automatic testing mechanism 110 may be hosted by an operating system 116 or a portion thereof, another portion may be hosted by a graphics processor 114 or a portion thereof, another portion may be hosted by an application processor 112 or a portion thereof, while one or more portions of the automatic testing mechanism 110 may be hosted by an operating system 116 and / or any number and type of devices of the computing device 100 or a portion thereof.It is contemplated that embodiments are not limited to any particular implementation or hosting of the automatic testing mechanism 110, and that one or more portions or components of the automatic testing mechanism 110 may be used or implemented as hardware, software, or any combination thereof, such as firmware.

[0021] Computing device 100 may host network interfaces to provide access to a network, such as a LAN, a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a Personal Area Network (PAN), Bluetooth, a cloud network, a mobile network (e.g., 3rd Generation (3G), 4th Generation (4G), etc.), an intranet, the Internet, etc. Network interfaces may include, for example, a wireless network interface with an antenna, which may represent one or more antennas. Network interfaces may also include, for example, a wired network interface for communicating with remote devices via a network cable, which may be, for example, an Ethernet cable, a coaxial cable, a fiber optic cable, a serial cable, or a parallel cable.

[0022] For example, embodiments may be provided as a computer program product comprising one or more machine-readable media having machine-executable instructions stored thereon that, when executed by one or more machines, such as a computer, a computer network, or other electronic devices, may cause the one or more machines to perform operations in accordance with embodiments described herein. A machine-readable medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs (Compact Disc-Read Only Memories) and magneto-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable Read Only Memories), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0023] Furthermore, embodiments may be downloaded as a computer program product, where the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by means of one or more data signals embedded and / or modulated in a carrier wave or other propagation medium over a communications connection (e.g., a modem and / or network connection).

[0024] Throughout this document, the term "user" may be interchangeably referred to as "viewer," "observer," "speaker," "person," "individual," "end user," and / or the like. Note that throughout this document, terms such as "graphics domain" may be interchangeably referred to as "graphics processing unit," "graphics processor," or simply "GPU," and similar terms; "CPU domain" or "host domain" may be interchangeably referred to as "computer processing unit," "application processor," or simply "CPU."

[0025] It should be noted that terms such as "node," "compute node," "server," "server device," "cloud computer," "cloud server," "cloud server computer," "machine," "host machine," "device," "computing device," "computer," "computing system," and the like may be used interchangeably in this document. It should also be noted that terms such as "application," "software application," "program," "software program," "package," "software package," and the like may be used interchangeably in this document. Also, terms such as "job," "input," "request," "message," and the like may be used interchangeably in this document.

[0026] Fig. 2 illustrates the automatic sensor testing mechanism 110 of Fig. 1 according to one embodiment. For the sake of brevity, many of the details already described with reference to Fig. 1 are not repeated or discussed below. In one embodiment, the automatic testing mechanism 110 may include any number and type of components, such as (without limitation): detection and acquisition logic 201; chaining logic 203; training and inference logic 205; communication / compatibility logic 209; and classification and prediction logic 207.

[0027] The computing device 100 (interchangeably referred to herein as an “autonomous machine”) is further shown having a user interface 219 (e.g., graphical user interface (GUI)-based user interface, web browser, cloud-based platform user interface, software application-based user interface, other user or application programming interfaces (APIs), etc.). Computing device 100 may further include I / O source(s) 108 with sensing / sensing component(s) 231, such as camera(s) A 242A, B 242B, C 242C, D 242D (e.g., Intel® RealSense™ camera), sensors, microphone(s) 241, etc., and output component(s) 233, such as display device(s) or simply display(s) 244 (e.g., integrated displays, tensor displays, projection screens, display screens, etc.), speaker device(s) or simply speakers 243, etc.

[0028] Computing device 100 is further illustrated as having access to and / or being in communication with one or more databases 225 and / or one or more other computing devices via one or more communication media 230 (e.g., networks such as a cloud network, a proximity network, the Internet, etc.).

[0029] In some embodiments, the database(s) 225 may comprise one or more storage media or devices, repositories, data sources, etc., with any amount and type of information, such as data, metadata, etc., related to any number and type of applications, such as data and / or metadata related to one or more users, physical locations or areas, applicable laws, policies and / or regulations, user preferences and / or profiles, security and / or authentication data, historical and / or preferred details, and / or the like.

[0030] As previously mentioned, computing device 100 may host I / O sources 108 including sensing / sensing component(s) 231 and output component(s) 233. In one embodiment, sensing / sensing component(s) 231 may include a sensor array including, but not limited to, microphone(s) 241 (e.g., ultrasonic microphones), camera(s) 242A-242D (e.g., two-dimensional (2D) cameras, three-dimensional (3D) cameras, infrared (IR) cameras, depth-sensing cameras, etc.), capacitors, radio components, radar components, scanners, and / or accelerometers, etc. Similarly, the output component(s) 233 may include any number and type of speaker(s) 243, display device(s) 244 (e.g., screens, projectors, light-emitting diodes (LEDs)), and / or vibration motors, etc.

[0031] For example, as illustrated, the sensing / sensing component(s) 231 may include any number and type of microphones 241, such as multiple microphones or a microphone array, such as ultrasonic microphones, dynamic microphones, fiber optic microphones, laser microphones, etc. It is contemplated that one or more of the microphones 241 may serve as one or more input devices for accepting or receiving audio input (such as human voice) into the computing device 100 and converting that audio or sound into electrical signals. Similarly, it is contemplated that one or more of the cameras 242A-242D may serve as one or more input devices for detecting and capturing images and / or videos of scenes, objects, etc., and providing the captured data as video inputs to the computing device 100.

[0032] It is contemplated that embodiments are not limited to any number or type of microphone(s) 241, camera(s) 242A-242D, speaker(s) 243, display(s) 244, etc. For example, as enabled by detection and sensing logic 201, one or more of microphones 241 may be used to detect speech or sound simultaneously from multiple users or speakers, such as speaker 250. Similarly, as enabled by the detection and capture logic 201, one or more of the cameras 242A-242D may be used to capture images or videos of a geographic location (e.g., a room) and its contents (e.g., furniture, electronic devices, people, animals, plants, etc.) and form a set of images or a video stream from the captured data for further processing by the automatic inspection mechanism 110 at the computing device 100.

[0033] Similarly, as illustrated, the output component(s) 233 may include any number and type of speaker(s) 243 that serve as output devices for outputting or distributing audio from the computing device 100 for any number or type of reasons, such as human hearing or consumption. For example, the speaker(s) 243 operate in opposition to the microphone(s) 241, with the speaker(s) 243 converting electrical signals into sound.

[0034] As previously mentioned, embodiments are not limited to any number or type of sensors that are part of, embedded within, or coupled to the sensing / sensing component(s) 231, such as microphones 241, cameras 242A-242D, and / or the like. In other words, embodiments are applicable to and compatible with any number and type of sensors; however, cameras 242A-242D are used as examples in this document for the purpose of concise and clear discussion. Similarly, the embodiments are applicable to all types and kinds of cameras, and therefore, cameras 242A-242D need not be of any particular type.

[0035] As mentioned previously, with the growth of autonomous machines, such as self-driving vehicles, drones, household appliances, etc., sensors of all kinds are expected to lead the way in influencing and facilitating certain tasks essential to the viability of autonomous machines, such as sensors that serve as the eyes behind the wheel in self-driving vehicles. Therefore, data quality becomes a critical factor in dealing with autonomous machines for various reasons, such as safety, security, trust, etc., especially in life-and-death situations, business environments, etc.

[0036] It is contemplated that high-quality data can ensure that the artificial intelligence (AI) of an autonomous machine, such as computing device 100, receives high-quality inputs (e.g., images, videos, etc.) to output high-quality performance. It is further contemplated that even if one of the sensors, such as cameras 242A-242D, is defective or not operating at its full potential (e.g., due to mud on the lens or excessive fog, etc.), the overall performance of computing device 100 could be impacted because accuracy is compromised.

[0037] For example, the automatic inspection mechanism 110 provides a novel technique for filtering through the cameras 242A-242D to detect abnormalities in one or more of the cameras 242A-242D that may be responsible for, or have the potential to provide, less than high-quality inputs, where such abnormalities in or with the cameras 242A-242D may include (without limitation) dirt / mud on lenses, obstructions in front of lenses, occlusions (e.g., fog), technical problems due to physical damage, and / or the like.

[0038] Conventional techniques are unable to detect such abnormalities and therefore cannot guarantee the accuracy of the data collected by sensors of autonomous machines, often resulting in low-quality or even misleading data.

[0039] Embodiments provide a novel technique for real-time detection of abnormalities in sensors, such as cameras 242A-242D, issuing alarms or warnings as needed, and correcting or repairing such abnormalities. In one embodiment, automatic checking mechanism 110 provides a novel sensor automatic checking (SAC) technique for detecting and checking the status of each sensor, such as cameras 242A-242D, in a system, such as autonomous machine 100, to ensure that all sensors are functioning properly before performing tasks (e.g., before driving a self-driving car), and to continue checking the sensors to ensure they continue to work, or, if abnormalities exist, that they are corrected in real time during the execution of one of the tasks (e.g., driving).

[0040] In one embodiment, the automatic testing mechanism 110 uses the deep learning of the autonomous machine 100 to ensure in real time that the cameras 242A-242D and all other sensors are in working order, or at least that they are immediately attended to and repaired in the event of any problems. As further described later in this document, the automatic testing mechanism 110 may use deep neural networks (DNNs), such as convolutional neural networks (CNNs) deep learning classifiers, to continuously and accurately check the real-time status of the cameras 242A-242D and other sensors, and then use the training data to detect and predict which of the cameras 242A-242D or other sensors may be defective or out of service.

[0041] One of the biggest weaknesses of conventional techniques is when a camera lens is covered with debris such as dirt, stains, mud, etc., as in this case the camera is unable to detect or capture regardless of the degree or amount of debris.

[0042] Embodiments provide for the use of deep learning on autonomous machines, such as autonomous machine 100, to handle complex issues. For example, in the case of smudges or mud on the lens of a camera, such as camera 242A, this obstruction can be continuously monitored, including taking into account any movements or changes associated with camera 242A, the smudge, and / or the scene. This is detected and monitored in real time so that the defective or obstructed camera 242A can be repaired.

[0043] In one embodiment, the detection and capture logic 201 of the automatic inspection mechanism 110 may be used to trigger one or more cameras 242A-242D located at various positions to capture one or more scenes in front of them. It is contemplated that in some embodiments, as described with reference to Fig. 3A illustrates, multiple cameras 242A-242D may be mounted in their positions, capturing static inputs, such as capturing the scene from different angles at the same time. Similarly, as in Fig. 3B illustrates, in another embodiment, a single camera, such as camera 242A, may be used to capture dynamic inputs, such as capturing the scene from the same angle at different times. For yet another input, as described with reference to Fig. 3C, the spot or deposit itself may be dynamic or moving, so that a camera, such as camera 242B, may be used to capture the scene while capturing the movement of the deposit.

[0044] As discussed above, sensors for sensing / sensing components 231 are not limited to any number or type of cameras 242A-242D or microphones 241, and sensors may further include other sensors, such as light detection and ranging (LiDAR) sensors, ultrasonic sensors, and any number and type of other sensors mentioned or described in this document, and any input from such sensors may be input to a neural network, such as a softmax layer of a CNN, for classification purposes.

[0045] Referring again to the automatic testing mechanism 110, when one or more cameras 242A-242D are capturing a scene, any internal or external problems with any of the cameras 242A-242D may also be detected, where internal problems include any physical defect (e.g., a part of the lens or camera is damaged) or technical problems (e.g., the camera no longer functions), while external problems refer to any form of obstruction, such as snow, trees, dirt, mud, debris, people, animals, etc., that may be on the lens or in the view of the lens and block the view of the scene.

[0046] For example, if mud is found on the lens of camera 242A, detection and capture logic 201 may be triggered to detect this mud, or at least that the view from camera 242A is blocked. In the case of motion associated with one of the cameras 242A, the scene (e.g., moving people, ocean waves, traffic movement, etc.), and / or the mud itself (e.g., flowing downward or downwind, etc.), detection and capture logic 201 may collect data including information related to the blocking of the view from camera 242A as well as one or more of the aforementioned motions.

[0047] Once the data has been collected by the detection and acquisition logic 201, it is passed as inputs to the concatenation logic 203. As mentioned above, embodiments are not limited to camera inputs, and these inputs may be from other sensors and may include LiDAR inputs, radar inputs, microphone inputs, and / or the like, although some overlap may occur when such sensors detect the same object. In one embodiment, in the case of multiple inputs from multiple sensors, such as two or more cameras 242A-242D, at the same time or across different points in time and / or from the same or different angles, and / or the same sensor, such as camera 242A, across multiple points in time and / or from the same or different angles, the concatenation logic 203 may then be triggered to concatenate (or concatenate) these inputs into a single input.

[0048] In one embodiment, each concatenated input of multiple inputs may then be passed to a deep learning neural network model, such as a CNN, for training and inference by the training and inference logic 205. In one embodiment, the concatenation logic 203 performs concatenation outside of or before the deep learning model manipulates the data, allowing for greater flexibility in determining their ordering to further benefit the training process. It is contemplated that embodiments are not limited to a number and type of deep learning models, so a CNN may be any type or kind of CNN commonly used, such as AlexNet, GoogleNet, RESNET, and / or the like.

[0049] It is considered that a deep learning neural network / model, such as a CNN, refers to a combination of an artificial neural network for analyzing, training, and inferring any range of input data. For example, a CNN is much faster and may require relatively less data processing compared to traditional algorithms. It is further considered that once the input data is received at a CNN, the data can then be processed through layers such as a convolutional layer, a pooling layer, a Rectified Linear Unit (ReLU) layer, a fully connected layer, a loss / output layer, etc., with each layer performing specific processing tasks for training and inference purposes.

[0050] For example, a convolutional layer can be considered a core layer with a number of learnable filters or kernels with receptive fields spanning the entire depth of the input volume. In this convolutional layer, processing of data from the inputs can start and proceed to another layer, such as a pooling layer, where some form of nonlinear downsampling is performed. These nonlinear downsampling functions can implement pooling, such as maximum pooling. Similarly, the data is further processed and trained in the ReLU layer, which applies the nonsaturating activation function to enhance the nonlinear properties of the decision function and the network without affecting the receptive fields of the convolutional layer.

[0051] Although embodiments are not limited to a number or type of layers of a neural network, such as a CNN, the training process may continue with the fully connected layer, providing high-level reasoning after multiple convolutional and pooling layers. In other words, a CNN may receive input data and perform feature mapping, sampling, convolutions, subsampling, followed by output results.

[0052] For example, a loss / output layer may specify how training penalizes the deviation between the predicted labels and true labels, where this loss / output layer can be considered the final layer in the CNN. For example, a softmax loss may be used to predict a single class of mutually exclusive classes. Further, in one embodiment, for example, classification and prediction logic 207 may use softmax and classification layers of the loss / output layer for classification and prediction purposes, with the two layers being generated by functions of the softmax layer and the classification layer, respectively.

[0053] In one embodiment, after all input data for training and inference has been processed, classification and prediction logic 207 may then be used to identify which of the sensors, for example, cameras 242A-242D, may have problems. Upon identification, classification and prediction logic 207 may issue a notification regarding the poor camera of cameras 242A-242D, such as displaying the notification on display device(s) 244, outputting it through speaker device(s) 243, etc.In one embodiment, this notification may then be used, such as by a user, to access the defective camera of cameras 242A-242D and correct the problem, such as wiping mud off the lens, manually or automatically correcting any technical issues with the lens, replacing the defective camera of cameras 242A-242D with another one, and / or the like.

[0054] In one embodiment, certain labels may be used for notification purposes, for example, label: 0 may mean all sensors are OK, while label: 1 may mean first sensor is damaged, label: 2 may indicate second sensor is damaged, label: 3 may mean third sensor is damaged, label: 4 may indicate fourth sensor is damaged, and so on. Similarly, label: 1 may indicate first sensor is OK, label: 2 may indicate second sensor is OK, and / or the like. It is contemplated that embodiments are not limited to any one form of notification, and that any or a combination of words, numbers, images, videos, audio, etc. may be used to convey the results of whether sensors are functioning well or not.

[0055] Furthermore, for example, with a single input data layer associated with each of the cameras 242A-242D, a number of channels, such as 12 (3*4=12) channels in the case of four cameras 242A-242D, may provide all the data from four images corresponding to four cameras 242A-242D. This data may be randomly loaded by breaking the order of the channels. Using this data, a deep learning model, such as a CNN, can calculate the loss (during training) and accuracy (during validation), so that labels may not need to be used in prediction.In some embodiments, training data may include a large sample of images, such as thousands or tens of thousands of sample images per camera 242A-242D, while validation data may also include a large sample of images, such as hundreds or thousands of sample images per camera 242A-242D and / or the like, to provide robust training and inference of data, respectively, as enabled by training and inference data, which is then followed by accurate results, including identifications, predictions, etc., as enabled by classification and prediction logic 207.

[0056] The capture / sensing component(s) 231 may further include any number and type of cameras 242A, 242B, 242C, 242D, such as depth-sensing cameras or sensing devices (e.g., Intel® RealSense™ Depth Sensing Camera), which are known for capturing still images and / or video for red-green-blue (RGB) and / or RGB depth (RGB-D, RGB-Depth) images for media, such as personal media. Such images with depth information have been effectively used for various computer vision and computational photography effects, such as (without limitation) scene understanding, refocusing, compositing, cinemagraphs, etc. Similarly, displays may include, for example, any number and type of displays, such as integrated displays, tensor displays, stereoscopic displays, etc., including (but not limited to) embedded or connected display screens, display devices, projectors, etc.

[0057] The sensing / sensing component(s) 231 may further include one or more of vibration components, tactile components, conductivity elements, biometric sensors, chemical detectors, signal detectors, electroencephalography, functional near-infrared spectroscopy, wave detectors, force sensors (e.g., accelerometers), illumination devices, eye tracking or gaze tracking system, head tracking system, etc., used to capture any amount and type of visual data, such as images (e.g., photos, videos, movies, audio / video streams, etc.), and non-visual data, such as audio streams or signals (e.g., sound, noise, vibration, ultrasound, etc.), radio waves (e.g., wireless signals, such as wireless signals containing data, metadata, characters, etc.), chemical changes or properties (e.g., humidity, body temperature, etc.), biometric measurements (e.g.,Fingerprints), brain waves, brain circulation, environmental / weather conditions, maps, etc. It is contemplated that "sensor" and "detector" may be used interchangeably throughout this document. It is further contemplated that one or more sensing / sensing components 231 may further comprise one or more supporting or supplemental devices for sensing and / or sensing data, such as illumination devices (e.g., IR illuminators), lights, generators, sound blockers, etc.

[0058] It is further contemplated that, in one embodiment, the sensing / sensing component(s) 231 may further include any number and type of context sensors (e.g., linear accelerometers) for sensing or sensing any number and type of contexts (e.g., estimating the horizon, linear acceleration, etc., with respect to a mobile computing device, etc.). For example, the sensing / sensing component(s) 231 may include any number and type of sensors, such as (without limitation): accelerometers (e.g., linear accelerometers for measuring linear acceleration, etc.); inertial devices (e.g., inertial accelerometers, inertial gyroscopes, microelectromechanical systems (MEMS) gyroscopes, inertial navigators, etc.).); and gravitational gradiometers for investigating and measuring fluctuations in gravitational acceleration due to gravity, etc.

[0059] Furthermore, the sensing / sensing component(s) 231 may include, for example, (without limitation): audio / visual devices (e.g., cameras, microphones, speakers, etc.); context-aware sensors (e.g., temperature sensors, facial expression and feature measurement sensors that operate with one or more cameras of audio / visual devices); environmental sensors (e.g., for sensing background colors, light, etc.)); biometric sensors (such as for fingerprint detection, calendar maintenance and reading devices, etc.); global positioning system (GPS) sensors; resource requestors; and / or TEE logic. The TEE logic may be used separately or may be part of a resource requestor and / or an I / O subsystem, etc.The detection / sensing component(s) 231 may further include speech recognition devices, photo recognition devices, face and other body recognition components, speech-to-text conversion components, etc.

[0060] Similarly, the output component(s) 233 may include dynamic tactile touch screens with tactile effectors as an example of displaying touch visualization, one embodiment of which may be ultrasonic generators that can send signals into space that, when reaching, for example, human fingers, can evoke a tactile sensation or similar feeling at the fingers.Further, for example, and in one embodiment, the output component(s) 233 may include (without limitation) one or more of light sources, display devices and / or screens, audio speakers, tactile components, conduction elements, bone conduction speakers, olfactory or smell-sensitive visual and / or non-visual presentation devices, haptic or touch-sensitive visual and / or non-visual presentation devices, animation display devices, biometric display devices, x-ray display devices, high-resolution displays, high-dynamic-range displays, multi-view displays, and head-mounted displays (HMDs) for at least one of virtual reality (VR) and augmented reality (AR), etc.

[0061] It is contemplated that the embodiments are not limited to any particular number or type of use case scenarios, architectural placements, or component setups; however, for brevity and clarity, illustrations and descriptions are offered and discussed for exemplary purposes throughout this document, but the embodiments are not limited thereto. Furthermore, "user" throughout this document may refer to someone who has access to one or more computing devices, such as computing device 100, and may be interchangeably referred to as a "person," "individual," "human," "he," "she," "child," "adult," "viewer," "player," "gamer," "developer," "programmer," and / or the like.

[0062] The communication / compatibility logic 209 may be used to enable dynamic communication and compatibility between various components, networks, computing devices, database(s) 225, and / or communication medium(s) 230, etc., and any number and type of other computing devices (such as wearable computing devices, mobile computing devices, desktop computers, server computing devices, etc.), processing devices (e.g., central processing unit (CPU), graphics processing unit (GPU), etc.), sensing / sensing components (e.g., non-visual data sensors / detectors such as audio sensors, olfactory sensors, haptic sensors, signal sensors, vibration sensors, chemical detectors, radio wave detectors, force sensors, weather / temperature sensors, body / biometric sensors, scanners, etc.)and visual data sensors / detectors, such as cameras, etc.), user / context perception components and / or identification / verification sensors / devices (such as biometric sensors / detectors, scanners, etc.), memory or storage devices, data sources and / or database(s) (such as data storage devices, hard drives, solid-state drives, disks, memory cards or devices, memory circuits, etc.), network(s) (e.g., cloud network, internet, internet of things, intranet, cellular network, proximity networks, such as Bluetooth, Bluetooth Low Energy (BLE), Bluetooth Smart, Wi-Fi proximity, Radio Frequency Identification, Near Field Communication, Body Area Network, etc.), wireless or wired communications and relevant protocols (e.g., Wi-Fi®, WiMAX, Ethernet, etc.), connectivity and location management techniques, software applications / websites (e.g.,social and / or business networking websites, business applications, games and other entertainment applications, etc.), programming languages, etc., while ensuring compatibility with changing technologies, parameters, protocols, standards, etc.

[0063] Throughout this document, terms such as "logic," "component," "module," "framework," "engine," "tool," "circuitry," and / or the like may be used interchangeably and include, for example, software, hardware, and / or any combination of software and hardware, such as firmware. In one example, "logic" may refer to or include a software component capable of operating with one or more of an operating system, a graphics driver, etc., of a computing device, such as computing device 100. In another embodiment, "logic" may refer to or include a hardware component capable of being physically installed along with or as part of one or more system hardware elements, such as an application processor, a graphics processor, etc., of a computing device, such as computing device 100.In yet another embodiment, “logic” may refer to or include a firmware component capable of being part of system firmware, such as the firmware of an application processor or graphics processor, etc., of a computing device, such as computing device 100.

[0064] Furthermore, any use of a particular trademark, word, term, phrase, name, and / or acronym, such as “sensors,” “cameras,” “autonomous machines,” “automatic sensor testing,” “deep learning,” “convolutional neural network,” “concatenation,” “training,” “inferencing,” “classifying,” “predicting,” “RealSense™ camera,” “real-time,” “automatic,” “dynamic,” “user interface,” “camera,” “sensor,” “microphone,” “display screen,” “speaker,” “verification,” “authentication,” “privacy,” “user,” “user profile,” “user preference,” “transmitter,” “receiver,” “personal device,” “smart device,” “mobile computer,” “wearable device,” “IoT device,” “proximity network,” “cloud network,” “server computer,” etc., it should not be understood that embodiments are limited to software or devices bearing that designation in products or literature outside this document.

[0065] It is contemplated that any number and type of components may be added to and / or removed from the automatic testing mechanism 110 to enable various embodiments, including adding, removing, and / or enhancing certain features. For brevity, clarity, and understandability of the automatic testing mechanism 110, many of the standard and / or well-known components, such as those of a computing device, are not shown or discussed here. It is contemplated that embodiments as described herein are not limited to any one technology, topology, system, architecture, and / or standard and are dynamic enough to embrace and adapt to future changes.

[0066] Fig. 3A illustrates static inputs from multiple sensors according to one embodiment and as previously described with reference to Fig. 2. For the sake of brevity, many of the details previously described with reference to the Fig. 1-2 may not be discussed or repeated below.

[0067] In the illustrated embodiment, four images A 301, B 303, C 305 and D 307 of a scene are shown as captured by four cameras A 242A, B 242B, C 242C and D 242D, respectively, of Fig. 2, wherein these multiple images 301-307 are based on static data captured by the four cameras 242A-242D over a period of time. For example, sensors such as cameras 242A-242D, radar devices, etc., may be used to capture similar data for the same sensing purpose, such as enabling automated driving vehicles to be aware of objects in their vicinity and surroundings.

[0068] In this embodiment, to simplify design and testing, four cameras 242A-242D are shown capturing four images 301-307 of the same scene at the same time, but from different angles and / or positions. As illustrated, one of the images, for example, image 301, further shows that the corresponding camera 242A has clarity issues, for example, due to some type of smudge 309 (e.g., mud, dirt, debris, etc.) on the lens of camera 242A. It is contemplated that such issues can lead to many problems when dealing with autonomous machines, such as a self-driving vehicle.

[0069] In one embodiment, as described with reference to Fig. 2, enable the collection of a large amount of data, such as thousands of data inputs, and their use as training data, validation data, etc. in deep learning models, such as CNNs, as by the automatic checking mechanism 110 of Fig. 1 enables the real-time detection of spot 309. This real-time detection then enables real-time notification as a real-time correction of spot 309, so that any defects related to camera 242A can be rectified and all cameras 242A-242D can exploit their potential and collect data to facilitate the use of autonomous machines, such as the autonomous machine 100 of Fig. 1, to make it safe, secure and efficient.

[0070] Fig. Figure 3B illustrates dynamic inputs from a single sensor according to one embodiment and as previously described with reference to Fig. 2. For the sake of brevity, many of the details previously described with reference to the Fig. 1-3A may not be discussed or repeated below.

[0071] In this illustrated embodiment, a single sensor, such as the camera D 242D of Fig. 2, can be used to capture four images A 311, B 313, C 315, D 317 of a single scene, but with different timestamps, for example, at different points in time. In this illustrated pattern, capturing the scene at different points in time shows that the scene is moving, for example, from right to left, while spot 319 is shown placed at one location, such as a location on the lens of camera D 242D.

[0072] As clearly shown in the four images 311-317, the object 321 (e.g., a book) as captured by the camera 242D appears to move over time (e.g., from right to left) while the spot 319 is fixed (or, in the true sense, slowly moves in a different pattern, as in the embodiment of Fig. 3C). As explained with reference to the Fig. 2 and Fig. As described in Figure 3A, several thousand images are collected and fed into a deep learning model for training and validation purposes, which then leads to testing of the deep learning model. After testing, the deep learning model can be used to identify and correct sensor problems in real time, such as spot 319 on camera 242D.

[0073] Fig. Figure 3C illustrates dynamic inputs from a single sensor according to one embodiment and as previously described with reference to Fig. 2. For the sake of brevity, many of the details previously described with reference to the Fig. 1-3B may not be discussed or repeated below.

[0074] In one embodiment, as described with reference to Fig. 3B in terms of dynamic inputs from a single sensor, in this illustrated embodiment, a single sensor, such as camera B 242B, captures four images A 331, B 333, C 335, D 337 of a single scene, with the spot 339 on the lens of camera 242B shown to move with the object 341 (e.g., a book) in the background scene. For example, the spot 339 may be a piece of mud on the lens of camera 242B that is pulled downward over time due to gravity or sideways due to wind, movement of camera 242B, and / or the like.

[0075] In one embodiment, as previously described, this data relating to spot 339 and its movements may be captured by one or more sensors, such as camera 242B itself, and input into a trained deep learning neural network / model, such as a CNN, which then predicts and provides in real time the exact location of spot 339, the sensor affected by spot 339, such as camera 242B, and how to resolve this issue, such as removing spot 339 from the lens of camera 242B.

[0076] In one embodiment, this training of deep learning neural networks / models is achieved by inputting a collection of (thousands of) such inputs as examples for training and validating data and testing deep learning models. For example, a deep learning model may first extract features from all sensors, such as camera 242B, using deep learning neural networks such as a CNN, and then fuse the data and use a classifier to identify the problematic sensors, such as camera 242B. For example, camera 242B may be assigned a label, such as Label 2: second sensor is damaged and / or the like.

[0077] Fig. Figure 4A illustrates an architectural design 400 with a transaction sequence for real-time detection and remediation of compromised sensors using deep learning according to one embodiment. For brevity, many of the details previously described with reference to the Fig. 1-3C may not be discussed or repeated below. All processes or transactions may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, etc.), software (such as instructions executing on a processing device), or a combination thereof, as may be performed by the automatic test mechanism 110 of Fig. 1. All processes or transactions associated with this illustration may be illustrated or referred to in linear sequences for the sake of brevity and clarity of presentation; however, it is contemplated that any number of them may be performed in parallel, asynchronously, or in different orders.

[0078] In one embodiment, the transaction sequence in architecture design 400 begins at 401 with the input of collected data from one or more sensors at sequentially multiple points in time for chaining before being input to a deep learning model, such as deep learning model 421. This data may have multiple inputs based on data collected from any number and type of sensors, such as cameras, LiDARs, radars, etc., although there is some degree of overlap in their detections (e.g., when the same object is detected in a scene). At 403, as described with reference to chaining logic 203 of Fig. 2, data from these inputs is sent for concatenation, so that these multiple inputs are then concatenated into a single input and sent to the data learning model 421 for training 405 and inference 407.

[0079] It is contemplated that, in one embodiment, the concatenation may be performed outside of or before sending the data to the deep learning model 421, allowing for greater flexibility in determining orderings to maximize the benefit of training 405. As illustrated, in one embodiment, inferencing 407 may be part of training 405, or inferencing 407 and training 405 may be performed separately in another embodiment.

[0080] In one embodiment, once input to the deep learning model 421, they are input to the CNN 409 and processed there, with the processing of the data passing through several layers, as described with reference to Fig. 2. For example, the classification layer 411 may comprise a common classification layer, such as fully connected layers, softmax layer, and / or the like, as described with reference to Fig. 2 further described.

[0081] In one embodiment, the transaction sequence offered by the architecture 400 may proceed with the results 413 obtained by an output layer, where the results 413 may identify or predict whether one or more sensors are technically defective, obstructed by an object or debris, or malfunctioning for some reason. Once the results 413 are obtained, various labels 415 are compared to determine the loss and the appropriate label to offer to the user regarding the one or more defective sensors. The transaction sequence may proceed with data backpropagation 417, and consequently, more weight updates 419 are performed, all in the deep learning model 421.

[0082] Fig. Figure 4B illustrates a method 450 for real-time detection and remediation of compromised sensors using deep learning, according to one embodiment. For brevity, many of the details previously described with reference to the Fig. 1-4A may not be discussed or repeated below. All processes or transactions may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, etc.), software (such as instructions executing on a processing device), or a combination thereof, as may be performed by the automatic test mechanism 110 of Fig. 1. All processes or transactions associated with this illustration may be illustrated or referred to in linear sequences for the sake of brevity and clarity of presentation; however, it is contemplated that any number of them may be performed in parallel, asynchronously, or in different orders.

[0083] The method 450 begins at block 451 with the detection of data including one or more images of a scene captured by one or more sensors (e.g., cameras) at the same time or over a period of time. If this data is distributed across multiple inputs, these multiple inputs are presented to the multiple inputs for concatenation. At block 453, these multiple inputs are concatenated into a single data input and presented to a deep learning model for further processing, such as training, inferencing, validation, etc. At block 455, this data is received by the deep learning model for training and inferencing. The deep learning model comprises a neural network (such as a CNN) with multiple processing layers.

[0084] It is considered and with reference to Fig. Figure 2 discusses that the data passing through the training and inference phases may be processed and modified at various levels, including the CNN, which may have multiple processing layers of its own. In one embodiment, at block 457, a trained deep learning model classifies the data and predicts the results based on all the processing and classification. For example, predicting the results may indicate and identify in real time whether one of the one or more sensors is defective or disabled, allowing a defective or disabled sensor to be repaired in real time.

[0085] Fig. 5 illustrates a computing device 500 according to one implementation. The illustrated computing device 500 may be used with the computing device 100 of Fig. 1. Computing device 500 includes a system board 502. Board 502 may include a number of components, including, but not limited to, a processor 504 and at least one communications package 506. The communications package is coupled to one or more antennas 516. Processor 504 is physically and electrically coupled to board 502.

[0086] Depending on its applications, the computing device 500 may include other components that may or may not be physically and electrically coupled to the circuit board 502. These other components include a volatile memory (e.g., DRAM) 508, a non-volatile memory (e.g.,ROM) 509, a flash memory (not shown), a graphics processor 512, a digital signal processor (not shown), a crypto processor (not shown), a chipset 514, an antenna 516, a display 518 such as a touch screen display, a touch screen controller 520, a battery 522, an audio codec (not shown), a video codec (not shown), a power amplifier 524, a global positioning system (GPS) device 526, a compass 528, an accelerometer (not shown), a gyroscope (not shown), a speaker 530, cameras 532, a microphone assembly 534, and a mass storage device 510 (such as a hard disk drive, a compact disk (CD), a digital versatile disk (DVD) (not shown), and so on), are not limited thereto.These components can be connected to the system board 502, mounted on the system board, or combined with one of the other components.

[0087] The communications package 506 enables wireless and / or wired communication for transferring data to and from the computing device 500. The term "wireless" and its derivatives can be used to describe circuits, devices, systems, methods, techniques, communication channels, etc., that can communicate data using modulated electromagnetic radiation through a non-solid medium. The term does not imply that the associated devices do not include wires, although in some embodiments they may not. The communications package 506 can implement any of a number of wireless or wired standards or protocols, including Wi-Fi (IEEE 802.11 family), WiMAX (IEEE 802.16 family), IEEE 802.20, Long Term Evolution (LTE), Ev-DO, HSPA+, HSDPA+, HSUPA+, EDGE, GSM, GPRS, CDMA, TDMA, DECT, Bluetooth, Ethernet derivatives thereof, and any other wireless and wired protocols referred to as 3G, 4G, 5G, and beyond. Computing device 500 may include a plurality of communication packets 506. For example, a first communication packet 506 may be for shorter-range wireless communication, such as Wi-Fi and Bluetooth, and a second communication packet 506 may be for longer-range wireless communication, such as GPS, EDGE, GPRS, CDMA, WiMAX, LTE, Ev-DO, and others.

[0088] The cameras 532, including any depth or proximity sensors, are coupled to an optional image processor 536 to perform conversions, analysis, noise reduction, comparisons, depth or distance analysis, image understanding, and other processes described herein. The processor 504 is coupled to the image processor to drive the process with interrupts, set parameters, and control operations of the image processor and cameras. Image processing may instead be performed in the processor 504, the graphics CPU 512, the cameras 532, or another device.

[0089] In various implementations, computing device 500 may be a laptop, a netbook, a notebook, an ultrabook, a smartphone, a tablet, a personal digital assistant (PDA), an ultra-mobile PC, a mobile phone, a desktop computer, a server, a set-top box, an entertainment control unit, a digital camera, a portable music player, or a digital video recorder. The computing device may be fixed, portable, or wearable. In further implementations, computing device 500 may be any other electronic device that processes data or records data for processing elsewhere.

[0090] Embodiments may be implemented using one or more memory chips, one or more controllers, one or more central processing units (CPUs), one or more microchips, or one or more integrated circuits interconnected using a motherboard, an application-specific integrated circuit (ASIC), and / or a field-programmable gate array (FPGA). The term "logic" may include, for example, software or hardware and / or combinations of software and hardware.

[0091] References to "a particular embodiment," "an embodiment," "exemplary embodiment," "various embodiments," etc., indicate that the embodiment(s) so described may include particular features, structures, or characteristics, but not every embodiment necessarily includes the particular features, structures, or characteristics. Furthermore, some embodiments may include some, all, or none of the features described for other embodiments.

[0092] In the following description and claims, the term "coupled" may be used along with its derivatives. "Coupled" is used to indicate that two or more elements cooperate or interact with each other, but they may or may not have intervening physical or electrical components.

[0093] As used in the claims, and unless otherwise indicated, the use of the ordinal adjectives "first," "second," "third," etc., to describe a common element merely indicates that reference is made to different instances of like elements, and is not intended to imply that the elements so described must be in any given order, whether temporally, spatially, in order of precedence, or in any other way.

[0094] The drawings and the foregoing description provide examples of embodiments. Those skilled in the art will recognize that one or more of the described elements may be well combined into a single functional element. Alternatively, certain elements may be separated into multiple functional elements. Elements of one embodiment may be added to another embodiment. For example, the orders of the processes described herein may be changed and are not limited to the manner described herein. Furthermore, the actions of a flowchart need not be implemented in the order shown; nor do all acts necessarily need to be performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of the embodiments is by no means limited to these specific examples.Numerous variations are possible, whether explicitly stated in the description or not, such as differences in structure, dimensions, and use of materials. The scope of the embodiments is at least as broad as indicated in the following claims.

[0095] For example, embodiments may be provided as a computer program product comprising one or more transitory or non-transitory machine-readable storage media having stored thereon machine-executable instructions that, when executed by one or more machines, such as a computer, a computer network, or other electronic device, may cause the one or more machines to perform operations in accordance with embodiments described herein.A machine-readable medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs (Compact Disc-Read Only Memories) and magneto-optical disks, ROMs, RAMs, EPROMs (Erasable Programmable Read Only Memory), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0096] Fig. Figure 6 illustrates one embodiment of a computing environment 600 capable of supporting the operations discussed above. The modules and systems may be implemented in a variety of different hardware architectures and form factors, including those described in Fig. 5 shown can be implemented.

[0097] The instruction execution module 601 includes a central processing unit for caching and executing instructions and distributing tasks among the other modules and systems shown. It may include an instruction stack, a cache memory for storing intermediate and final results, and mass storage for storing applications and operating systems. The instruction execution module may also serve as the central coordination and task allocation unit for the system.

[0098] The screen rendering module 621 draws objects on the one or more screens visible to the user. It can be adapted to receive data from the virtual object behavior module 604, described below, and to render the virtual object and any other objects and forces on the corresponding screen(s). Thus, for example, the data from the virtual object behavior module would determine the position and dynamics of the virtual object and the associated gestures, forces, and objects, and the screen rendering module would accordingly display the virtual object and associated objects and environment on a screen.The screen rendering module could be further adapted to receive data from the adjacent screen perspective module 607, described below, to represent a target landing area for the virtual object if the virtual object could be moved to display the device with which the adjacent screen perspective module is associated. Thus, for example, if the virtual object is moved from a main screen to an auxiliary screen, the adjacent screen perspective module 2 could send data to the screen rendering module to suggest, for example, in shadow form, one or more target landing areas for the virtual object on that track for a user's hand gestures or eye movements.

[0099] The object and gesture recognition module 622 may be adapted to recognize and track a user's hand and arm gestures. Such a module may be used to recognize hands, fingers, finger gestures, hand movements, and a position of hands relative to displays. For example, the object and gesture recognition module could determine that a user made a body part gesture to drop or throw a virtual object onto one or another of the multiple screens, or that the user made a body part gesture to move the virtual object toward a panel of one or another of the multiple screens.The object and gesture recognition system may be coupled to a camera or camera array, a microphone or microphone array, a touch screen or touch surface, or a pointing device, or a combination of these elements to recognize user gestures and commands.

[0100] The touch screen or touch surface of the object and gesture recognition system may include a touch screen sensor. Data from the sensor may be passed to hardware, software, firmware, or a combination thereof to map a user's hand's touch gesture on the screen or surface to a corresponding dynamic behavior of a virtual object. The sensor data may be used for momentum and inertia factors to enable a variety of momentum behaviors for a virtual object based on inputs from the user's hand, such as a swipe rate of a user's finger relative to the screen. Pinch gestures may be interpreted as a command to lift a virtual object from the display screen, to begin creating a virtual binding associated with the virtual object, or to zoom in or out of a display.Similar commands can be generated by the object and gesture recognition system using one or more cameras without the benefit of a touch surface.

[0101] The attention direction module 623 may be equipped with cameras or other sensors to track the position or orientation of a user's face or hands. When a gesture or voice command is issued, the system can determine the appropriate screen for the gesture. In one example, a camera is mounted near each display to detect whether the user is facing that display. If so, the attention direction module's information is provided to the object and gesture recognition module 622 to ensure that the gestures or commands are associated with the appropriate library for the active display. Similarly, if the user looks away from all screens, commands can then be ignored.

[0102] The device proximity detection module 625 may use proximity sensors, compasses, GPS (Global Positioning System) receivers, personal area network radios, and other sensor types, along with triangulation and other techniques, to determine the proximity of other devices. Once a nearby device is detected, it may be registered with the system, and its type may be determined as an input device or display device, or both. For an input device, received data may then be applied to the object and gesture recognition module 622. For a display device, it may be considered by the adjacent screen perspective module 607.

[0103] The virtual object behavior module 604 is adapted to receive input from the object speed and direction module and apply that input to a virtual object shown in the display.For example, the object and gesture recognition system would interpret a user gesture and map the detected movements of a user's hand to detected movements, the virtual object tracking module would associate the position and movements of the virtual object with the movements detected by the object and gesture recognition system, the object speed and direction module would capture the dynamics of the virtual object's movements, and the virtual object behavior module would receive input from the object speed and direction module to generate data that would control the virtual object's movements to match the input from the object speed and direction module.

[0104] The virtual object tracking module 606, on the other hand, may be adapted to track where a virtual object should be located in a three-dimensional space near a display and which body part of the user is holding the virtual object, based on input from the object and gesture recognition module. For example, the virtual object tracking module 606 may track a virtual object as it moves across and between screens and track which body part of the user is holding that virtual object. Tracking the body part holding the virtual object enables continuous perception of the body part's air movements and thus, eventual perception of whether the virtual object has been released on one or more screens.

[0105] The gesture-to-view and screen synchronization module 608 receives the selection of the view and screen, or both, from the attention direction module 623 and, in some cases, voice commands to determine which view is the active view and which screen is the active screen. It then causes the relevant gesture library to be loaded for the object and gesture recognition module 622. Different views of an application on one or more screens can be associated with alternative gesture libraries or a set of gesture templates for a given view. For example, in Fig. 1A a torpedo is launched by a pinch release gesture, in Fig. However, in 1B a water balloon is launched using the same gesture.

[0106] The adjacent screen perspective module 607, which may include or be coupled to the device proximity detection module 625, may be adapted to determine an angle and position of a display relative to another display. A projected display includes, for example, an image projected onto a wall or screen. The ability to detect proximity of a nearby screen and a corresponding angle or orientation of a display projected therefrom may be achieved, for example, either with an infrared emitter and receiver or with an electromagnetic or photodetection sensing capability. For technologies that enable projected displays with touch input, the incoming video may be analyzed to determine the position of a projected display and to correct distortion caused by displaying at an angle.An accelerometer, a magnetometer, a compass, or a camera can be used to determine the angle at which a device is held, while infrared illuminators and cameras could determine the orientation of the display device relative to the sensors on a neighboring device. The neighboring display perspective module 607 can thus determine coordinates of a neighboring display relative to its own display coordinates. Thus, the neighboring display perspective module can determine which devices are close to each other and other possible destinations for moving one or more virtual objects across display screens. The neighboring display perspective module can further enable the position of the display screens to be correlated with a three-dimensional spatial model representing all present objects and virtual objects.

[0107] The object velocity and direction module 603 can be adapted to estimate the dynamics of a virtual object being moved, such as its trajectory, velocity (whether linear or angular), momentum (whether linear or angular), etc., by receiving inputs from the virtual object tracking module. The object velocity and direction module can be further adapted to estimate the dynamics of arbitrary physical forces, for example, by estimating acceleration, deflection, the degree of stretching of a virtual binding, etc., and the dynamic behavior of a virtual object once it is released from a user's body part. The object velocity and direction module can also use image motion, size, and angle changes to estimate the velocity of objects, such as the velocity of hands and fingers.

[0108] The momentum and inertia module 602 can use image motion, image size, and angular changes of objects in the image plane or in a three-dimensional space to estimate the velocity and direction of objects in the space or on a display. The momentum and inertia module is coupled with the object and gesture recognition module 622 to estimate the velocity of gestures performed by hands, fingers, and other body parts and then apply these estimates to determine the momentum and velocities for virtual objects to be affected by the gesture.

[0109] The 3D image interaction and effects module 605 tracks user interaction with 3D images that appear to extend from one or more screens. The influence of objects in the z-axis (toward and away from the screen plane) can be calculated, along with the relative influence of these objects on each other. For example, an object thrown by a user gesture can be affected by 3D objects in the foreground before the virtual object reaches the screen plane. These objects can change the direction or speed of the projectile or destroy it completely. The object can be rendered with the 3D image interaction and effects module in the foreground on one or more of the displays. As illustrated, various components, such as components 601, 602, 603, 604, 605, 606, 607, and 608, are connected via an interconnect or bus, such as bus 609.

[0110] The following clauses and / or examples relate to further embodiments or examples. The specifics in the examples can be used throughout one or more embodiments. The various features of the different embodiments or examples can be combined in various ways, including some features and excluding others, to suit a variety of different applications. Examples can include subject matter such as a method, means for performing acts of the method, at least one machine-readable medium having instructions that, when executed by a machine, cause the machine to perform acts of the method, or an apparatus or system for enabling hybrid communication according to embodiments and examples described herein.

[0111] Some embodiments relate to Example 1, which includes an apparatus for enabling deep learning-based real-time detection and remediation of compromised sensors in autonomous machines, the apparatus comprising: detection and acquisition logic to enable one or more sensors to acquire one or more images of a scene, wherein an image of the one or more images is determined to be unclear, wherein the one or more sensors comprise one or more cameras; and classification and prediction logic to enable a deep learning model to identify a sensor associated with the image in real-time.

[0112] Example 2 includes the subject matter of Example 1, further comprising chaining logic to receive one or more data inputs associated with the one or more images to chain the one or more data inputs into a single data input to be processed by the deep learning model, wherein the apparatus comprises an autonomous machine, including one or more of a self-driving vehicle, a self-flying vehicle, a self-sailing vehicle, and an autonomous household appliance.

[0113] Example 3 includes the subject matter of Examples 1-2, further comprising training and inference logic to enable the deep learning model to receive the single data input to perform one or more deep learning processes, including a training process and an inference process, to obtain a real-time identification of the sensor associated with the ambiguous image, wherein the sensor comprises a camera.

[0114] Example 4 includes the subject matter of Examples 1-3, wherein the training and inference logic is further to enable the deep learning model to receive a plurality of data inputs and run the plurality of data inputs through the training and inference processes such that the real-time identification of the sensor is accurate and timely.

[0115] Example 5 includes the subject matter of Examples 1-4, wherein the deep learning model comprises one or more neural networks, including one or more convolutional neural networks, wherein the image is unclear due to one or more of a technical defect of the sensor or a physical obstruction of the sensors, wherein the physical obstruction is due to a person, plant, animal, or object obstructing the sensor, or dirt, stains, mud, or debris covering a portion of a lens of the sensor.

[0116] Example 6 includes the subject matter of Examples 1-5, wherein the classification and prediction logic is to provide one or more of real-time notification of the unclear image and real-time auto-correction of the sensor.

[0117] Example 7 includes the subject matter of Examples 1-6, wherein the device includes one or more processors including a graphics processor co-located with an application processor on a common semiconductor package.

[0118] Some embodiments relate to Example 8, which includes a method for enabling deep learning-based real-time detection and remediation of compromised sensors in autonomous machines, the method comprising: enabling one or more sensors to capture one or more images of a scene, wherein an image of the one or more images is determined to be unclear, wherein the one or more sensors comprise one or more cameras of a computing device; and enabling a deep learning model to identify a sensor associated with the image in real-time.

[0119] Example 9 includes the subject matter of Example 8, further comprising receiving one or more data inputs associated with the one or more images to concatenate the one or more data inputs into a single data input to be processed by the deep learning model, wherein the device comprises an autonomous machine, including one or more of a self-driving vehicle, a self-flying vehicle, a self-sailing vehicle, and an autonomous household appliance.

[0120] Example 10 includes the subject matter of Examples 8-9, further comprising enabling the deep learning model to receive the single data input to perform one or more deep learning processes, including a training process and an inference process, to obtain a real-time identification of the sensor associated with the ambiguous image, wherein the sensor comprises a camera.

[0121] Example 11 includes the subject matter of Examples 8-10, wherein the deep learning model is further to receive a plurality of data inputs and run the plurality of data inputs through the training and inference processes such that the real-time identification of the sensor is accurate and timely.

[0122] Example 12 includes the subject matter of Examples 8-11, wherein the deep learning model comprises one or more neural networks, including one or more convolutional neural networks, wherein the image is unclear due to one or more of a technical defect of the sensor or a physical obstruction of the sensors, wherein the physical obstruction is due to a person, plant, animal, or object obstructing the sensor, or dirt, stains, mud, or debris covering a portion of a lens of the sensor.

[0123] Example 13 includes the subject matter of Examples 8-12, further comprising providing one or more of a real-time notification of the unclear image and a real-time auto-correction of the sensor.

[0124] Example 14 includes the subject matter of Examples 8-13, wherein the computing device includes one or more processors including a graphics processor co-located with an application processor on a common semiconductor package.

[0125] Some embodiments relate to Example 15, which includes a data processing system with a computing device having memory coupled to a processing device, the processing device to: enable one or more sensors to capture one or more images of a scene, wherein an image of the one or more images is determined to be unclear, the one or more sensors comprising one or more cameras of a computing device; and enable a deep learning model to identify a sensor associated with the image in real time.

[0126] Example 16 includes the subject matter of Example 15, wherein the processing device is further to receive one or more data inputs associated with the one or more images to concatenate the one or more data inputs into a single data input to be processed by the deep learning model, wherein the device comprises an autonomous machine, including one or more of a self-driving vehicle, a self-flying vehicle, a self-gliding vehicle, and an autonomous household appliance.

[0127] Example 17 includes the subject matter of Examples 15-16, wherein the processing device is further to enable the deep learning model to receive the single data input to perform one or more deep learning processes, including a training process and an inference process, to obtain a real-time identification of the sensor associated with the ambiguous image, wherein the sensor comprises a camera.

[0128] Example 18 includes the subject matter of Examples 15-17, wherein the deep learning model is further to receive a plurality of data inputs and run the plurality of data inputs through the training and inference processes such that the real-time identification of the sensor is accurate and timely.

[0129] Example 19 includes the subject matter of Examples 15-18, wherein the deep learning model comprises one or more neural networks, including one or more convolutional neural networks, wherein the image is unclear due to one or more of a technical defect of the sensor or a physical obstruction of the sensors, wherein the physical obstruction is due to a person, plant, animal, or object obstructing the sensor, or dirt, stains, mud, or debris covering a portion of a lens of the sensor.

[0130] Example 20 includes the subject matter of Examples 15-19, wherein the processing device is further to provide one or more of real-time notification of the unclear image and real-time auto-correction of the sensor.

[0131] Example 21 includes the subject matter of Examples 15-20, wherein the computing device includes one or more processors including a graphics processor co-located with an application processor on a common semiconductor package.

[0132] Some embodiments relate to Example 22, which includes an apparatus for enabling simultaneous detection and processing of multiple speech from multiple users, the apparatus comprising: means for enabling one or more sensors to capture one or more images of a scene, wherein it is determined that an image of the one or more images is unclear, wherein the one or more sensors comprise one or more cameras; and means for enabling a deep learning model to identify a sensor associated with the image in real time.

[0133] Example 23 includes the subject matter of Example 22, further comprising means for receiving one or more data inputs associated with the one or more images to concatenate the one or more data inputs into a single data input to be processed by the deep learning model, wherein the device comprises an autonomous machine, including one or more of a self-driving vehicle, a self-flying vehicle, a self-sailing vehicle, and an autonomous household appliance.

[0134] Example 24 includes the subject matter of Examples 22-23, further comprising means for enabling the deep learning model to receive the single data input to perform one or more deep learning processes, including a training process and an inference process, to obtain a real-time identification of the sensor associated with the ambiguous image, wherein the sensor comprises a camera.

[0135] Example 25 includes the subject matter of Examples 22-24, wherein the deep learning model is further to receive a plurality of data inputs and run the plurality of data inputs through the training and inference processes such that the real-time identification of the sensor is accurate and timely.

[0136] Example 26 includes the subject matter of Examples 22-25, wherein the deep learning model comprises one or more neural networks, including one or more convolutional neural networks, wherein the image is unclear due to one or more of a technical defect of the sensor or a physical obstruction of the sensors, wherein the physical obstruction is due to a person, plant, animal, or object obstructing the sensor, or dirt, stains, mud, or debris covering a portion of a lens of the sensor.

[0137] Example 27 includes the subject matter of Examples 22-26, further comprising means for providing one or more of a real-time notification of the unclear image and a real-time auto-correction of the sensor.

[0138] Example 28 includes the subject matter of Examples 22-27, wherein the device includes one or more processors including a graphics processor co-located with an application processor on a common semiconductor package.

[0139] Example 29 includes at least one non-transitory or tangible machine-readable medium comprising a plurality of instructions that, when executed on a computing device, are to implement or perform a method as claimed in any one of claims or examples 8-14.

[0140] Example 30 includes at least one machine-readable medium comprising a plurality of instructions that, when executed on a computing device, are to implement or perform a method as claimed in any one of claims or examples 8-14.

[0141] Example 31 includes a system comprising a mechanism for implementing or performing a method according to any of claims or examples 8-14.

[0142] Example 32 includes an apparatus comprising means for performing a method according to any one of claims or examples 8-14.

[0143] Example 33 includes a computing device arranged to implement or perform a method according to any one of claims or examples 8-14.

[0144] Example 34 includes a communication device arranged to implement or perform a method according to any one of claims or examples 8-14.

[0145] Example 35 includes at least one machine-readable medium comprising a plurality of instructions that, when executed on a computing device, are to implement or perform a method or realize an apparatus as claimed in any preceding claim.

[0146] Example 36 includes at least one non-transitory or tangible machine-readable medium comprising a plurality of instructions that, when executed on a computing device, are to implement or perform a method or realize an apparatus as claimed in any preceding claim.

[0147] Example 37 includes a system comprising a mechanism for implementing or performing a method or for realizing an apparatus according to any one of the preceding claims.

[0148] Example 38 includes an apparatus comprising means for performing a method according to any one of the preceding claims.

[0149] Example 39 includes a computing device arranged to implement or perform a method or to realize an apparatus as claimed in any preceding claim.

[0150] Example 40 includes a communication device arranged to implement or perform a method or to realize an apparatus as claimed in any one of the preceding claims.

[0151] The drawings and the foregoing description provide examples of embodiments. Those skilled in the art will recognize that one or more of the described elements may be well combined into a single functional element. Alternatively, certain elements may be separated into multiple functional elements. Elements of one embodiment may be added to another embodiment. For example, the orders of the processes described herein may be changed and are not limited to the manner described herein. Furthermore, the actions of a flowchart need not be implemented in the order shown; nor do all acts necessarily need to be performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of the embodiments is by no means limited to these specific examples.Numerous variations are possible, whether explicitly stated in the description or not, such as differences in structure, dimensions, and use of materials. The scope of the embodiments is at least as broad as indicated in the following claims.

Claims

[1] Apparatus (100) for enabling deep learning-based real-time detection and correction of compromised sensors in autonomous machines, the apparatus comprising: one or more processors (112, 114) for enabling: Detection and capture logic (201) for enabling one or more sensors (231) to capture one or more images of a scene, wherein it is determined that an image of the one or more images is unclear, wherein the one or more sensors comprise one or more cameras; and Classification and prediction logic (207) to enable a deep learning model (421) to identify, based on the one or more images, a sensor associated with the image in real time using a classifier. [2] The apparatus (100) of claim 1, wherein the one or more processors (112, 114) are further to enable chaining logic to receive one or more data inputs associated with the one or more images to chain the one or more data inputs into a single data input to be processed by the deep learning model (421), the apparatus comprising an autonomous machine, including one or more of a self-driving vehicle, a self-flying vehicle, a self-gliding vehicle, and an autonomous household appliance. [3] The apparatus (100) of claim 1 or 2, wherein the one or more processors are further to enable training and inference logic (205) to enable the deep learning model (421) to receive the single data input to perform one or more deep learning processes, including a training process and an inference process, to obtain a real-time identification of the sensor (231) associated with the ambiguous image, the sensor comprising a camera. [4] The apparatus (100) of claim 3, wherein the training and inference logic (205) is further to enable the deep learning model (421) to receive a plurality of data inputs and run the plurality of data inputs through the training and inference processes such that the real-time identification of the sensor (231) is accurate and timely. [5] The device (100) of any preceding claim, wherein the deep learning model (421) comprises one or more neural networks, including one or more convolutional neural networks, wherein the image is unclear due to one or more of a technical defect of the sensor (231) or a physical obstruction of the sensors (231), wherein the physical obstruction is due to a person, plant, animal or object obstructing the sensor (231), or dirt, stains, mud or debris covering part of a lens of the sensor (231). [6] The apparatus (100) of any preceding claim, wherein the classification and prediction logic (207) is to provide one or more of real-time notification of the unclear image and real-time auto-correction of the sensor. [7] The device (100) of any preceding claim, wherein one or more processors comprise one or more of a graphics processor (114) and an application processor (112), wherein the graphics processor (114) is arranged together with the application processor (112) on a common semiconductor package. [8] A method for enabling deep learning-based real-time detection and correction of compromised sensors in autonomous machines, the method comprising: Enabling one or more sensors (231) to capture one or more images of a scene, wherein it is determined that an image of the one or more images is unclear, wherein the one or more sensors (231) comprise one or more cameras of a computing device; and Enabling a deep learning model (421) based on the one or more images to identify a sensor (231) associated with the image in real time using a classifier. [9] The method of claim 8, further comprising receiving one or more data inputs associated with the one or more images to concatenate the one or more data inputs into a single data input to be processed by the deep learning model (421), wherein the device comprises an autonomous machine, including one or more of a self-driving vehicle, a self-flying vehicle, a self-gliding vehicle, and an autonomous household appliance. [10] The method of any of claims 8-9, further comprising enabling the deep learning model (421) to receive the single data input to perform one or more deep learning processes, including a training process and an inference process, to obtain a real-time identification of the sensor associated with the ambiguous image, wherein the sensor comprises a camera. [11] The method of claim 10, wherein the deep learning model (421) is further to receive a plurality of data inputs and run the plurality of data inputs through the training and inference processes such that the real-time identification of the sensor (231) is accurate and timely. [12] The method of any of claims 8-11, wherein the deep learning model (421) comprises one or more neural networks, including one or more convolutional neural networks, wherein the image is unclear due to one or more of a technical defect of the sensor (231) or a physical obstruction of the sensors (231), wherein the physical obstruction is due to a person, plant, animal or object obstructing the sensor (231), or dirt, stains, mud or debris covering part of a lens of the sensor. [13] The method of any of claims 8-12, further comprising providing one or more of a real-time notification of the unclear image and a real-time auto-correction of the sensor (231). [14] The method of any of claims 8-13, wherein the computing device comprises one or more processors including one or more of a graphics processor and an application processor, wherein the graphics processor is arranged together with the application processor on a common semiconductor package. [15] One or more machine-readable media comprising a plurality of instructions that, when executed on a computing device, are intended to implement or perform a method according to any one of the preceding claims 8 to 14. [16] A system comprising a mechanism for implementing or performing a method according to any one of the preceding claims 8 to 14. [17] Apparatus comprising means for carrying out a method according to any one of the preceding claims 8 to 14. [18] Computing device arranged to implement or perform a method according to any one of the preceding claims 8 to 14. [19] Communication device arranged to implement or perform a method according to any one of the preceding claims 8 to 14.

Images