Neural network for classifying obstruction in optical sensor

Abstract

To provide a neural network (60) configured for classifying whether an image (S) from an optical sensor (30) characterizes obstruction of the optical sensor (30).SOLUTION: A classification is characterized by output (y) of a neural network (60) for input (x) of the neural network (60). The input (x) is based on an image (S). The neural network (60) includes a first convolutional layer (61) that characterizes a 1D-convolution along a vertical axis of convolution output (f) of a preceding convolutional layer (Cn). The neural network (60) includes a second convolutional layer (62) that characterizes a 1D-convolution along a horizontal axis of the convolution output (f). The output (y) of the neural network (60) is based on first convolution output (co1) of the first convolutional layer (61), and the output (y) of the neural network (60) is based on second convolution output (co2) of the second convolutional layer (62).SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a neural network configured to classify whether an optical sensor is obstructed, a method implemented on a computer for determining whether an optical sensor is obstructed, a method implemented on a computer for determining a control signal of a robot, a computer program, and a machine-readable storage medium.

Background Art

[0002] Autonomous devices such as robots typically rely on optical sensors to sense their respective environments. Sensing the environment is often the starting point in a processing pipeline that has the ultimate goal of deriving appropriate actions for the autonomous device given the ambient conditions around it.

[0003] A typical problem that occurs in these scenarios is that the optical sensor may be obstructed and thus may not accurately capture the environment. In this regard, an obstruction may be particularly understood as a situation where the sensor's field of view is obstructed to such an extent that the sensor can still transmit a correct signal, but meaningful information about the environment cannot be inferred from the signal.

[0004] It is desirable to detect whether an optical sensor is obstructed. Advantageously, a neural network as proposed in this invention can achieve this goal with high accuracy. The inventors have found that through a careful design of the neural network, it can be made to execute even on embedded hardware with limited computing power, and its size can be reduced to make the neural network particularly attractive for mobile robots or battery-powered devices.

Summary of the Invention

Means for Solving the Problems

[0005] In a first aspect, the invention relates to a neural network configured to classify whether an image from a light sensor characterizes interference with the light sensor, wherein the classification is characterized by the output of the neural network to an input of the neural network, the input being based on an image, the neural network including a first convolutional layer characterized by a 1D convolution along the height of the convolutional output of a preceding convolutional layer, the neural network including a second convolutional layer characterized by a 1D convolution along the width of the convolutional output, the output being based on the first convolutional output of the first convolutional layer, and the output being based on the second convolutional output of the second convolutional layer.

[0006] The neural networks referred to in this invention may be understood particularly as models from the field of machine learning. Such neural networks are typically implemented in software. However, it is also possible to assume hardware that characterizes the neural network as defined above.

[0007] A neural network may be understood as a convolutional neural network because it includes convolutional layers. A convolutional neural network is characterized by the fact that it includes convolutional layers. A convolutional layer typically accepts a three-dimensional tensor as input, and a typical practice is that the dimensions characterize the height, width, and depth of the tensor in that order. The depth dimension is also sometimes called the channels, i.e., characterizes the number of channels that the tensor contains. In this regard, a three-dimensional tensor may also be understood as the components of a matrix, the matrix being the same size as the tensor and stacked along the depth dimension or channel dimension of the tensor. An image can be understood as a tensor by, for example, encoding the image into the RGB format and designing each color channel as a channel of the tensor.

[0008] A convolutional layer can process tensors and output a tensor, i.e., a convolutional output. The number of channels in the convolutional output is typically given by the number of filters (also known as kernels or filter kernels) used by the convolutional layer to determine the convolutional output.

[0009] When configured for image processing, neural networks may be specifically configured to include two-dimensional convolutional layers. However, neural networks may also include one-dimensional convolutional layers, i.e., layers characterizing 1D convolutions. Such layers may be understood as using a filter kernel that extends either the full width of the tensor used as input to the above layer or the full height of the tensor. Here, with respect to two-dimensional convolutional layers, the filter kernel is typically "slid" along both the width and height of the input tensor, while the filter kernel may be understood as simply "sliding" along one of the two dimensions in a one-dimensional convolutional layer. In a one-dimensional convolutional layer, if its filter kernel is configured to extend the full width of each input tensor in the one-dimensional convolutional layer, the filter kernel slides along the height of the input tensor to determine the convolutional output. Similarly, in a one-dimensional convolutional layer, if its filter kernel is configured to extend the full height of each input tensor in the one-dimensional convolutional layer, the filter kernel slides along the width of the input tensor to determine the convolutional output.

[0010] When a neural network is configured to detect whether a given image characterizes interference from the light sensor that provided the image, the inventors found that a convolutional layer characterizing a 1D convolution, i.e., a one-dimensional convolutional layer, is remarkably well-suited for use in neural networks. Surprisingly, the use of a first convolutional layer and a second convolutional layer improved the accuracy of the neural network. This is due to the fact that the output of the preceding convolutional layer is processed "from top to bottom" by the first convolutional layer and "from left to right" by the second convolutional layer. The inventors found that interference from a light sensor typically reveals the interference itself with very small changes in information within the image of the interfered light sensor, either from top to bottom or from left to right. Since this information is stored in the convolutional output of the preceding convolutional layer (i.e., adjacent elements of the output of the preceding convolutional layer characterize adjacent elements of the image), a one-dimensional convolutional layer is particularly well-suited for neural networks.

[0011] In the context of this invention, the term “interference” may be particularly understood as a situation in which the field of view of the light sensor is obstructed to such an extent that meaningful information about the environment cannot be inferred from the light sensor’s signal, even though the light sensor can still transmit a correct signal. Interference may result particularly from environmental conditions (e.g., bright light, heavy rain, a sensor covered with snow / ice, insufficient light) or intentional actions (e.g., completely covering the sensor with a sticker or metal foil). An obstructed sensor may also be called a “blind sensor” or a “blocked sensor.”

[0012] The optical sensor may be specifically provided by a camera sensor, LiDAR sensor, radar sensor, ultrasonic sensor, or thermal camera. Since the neural network is configured to process image data from the optical sensor, the above neural network may be specifically understood as a neural network configured for image analysis.

[0013] Neural networks may be trained by known methods, particularly using (statistical) gradient descent with backpropagation algorithms or evolutionary algorithms. Labels used in such supervised training methods may be provided specifically by human annotators or known automated labeling methods.

[0014] A neural network can output values ​​that characterize the probability or likelihood of an image being disrupted by a light sensor. Classification can be derived from such values ​​by applying predetermined thresholds. For example, a light sensor might be classified as disrupted if its probability exceeds 50%.

[0015] In a preferred embodiment of the neural network, the first and second convolutional outputs can be used as inputs to the fully connected layer of the neural network, where the output of the neural network is determined based on the output of the fully connected layer.

[0016] A typical architectural choice when designing a convolutional neural network is to place the convolutional layers of the convolutional neural network at the front of the neural network, followed by fully connected layers, where the fully connected layers are configured to determine the output of the convolutional neural network (for example, in the remaining neural network configured for image classification). The convolutional layers may, however, be further followed by multiple fully connected layers, i.e., multilayer perceptrons or MLPs (for example, in VGGnet or AlexNets configured for image classification) that use the output of the last convolutional layer as their input.

[0017] In this invention, there are two parallel convolutional layers, namely a first convolutional layer and a second convolutional layer. They are parallel in that they both take input from the same preceding layer and process each input individually. Using both of its outputs as input to a fully connected layer thus allows the fully connected layer to extract information about both the "top-to-bottom" processing of the image and the "left-to-right" processing of the image.

[0018] The input to a neural network may be the image itself. However, the inventors have found it useful to divide the above image into separate image patches, determine a feature representation (also known as a feature vector) for each of these patches, and use the resulting feature representations as input to a neural network. In other words, the input to a neural network can also be multiple feature vectors, where each feature vector is obtained for a separate patch of the image. The feature vectors may also preferably be organized into a tensor, where the channel dimension of the tensor characterizes the feature vectors, and the feature vectors are arranged along the height and width of the tensor according to the image location of each patch for which the patch is determined. The resulting tensor of feature vectors may then be used as input to a convolutional layer of the neural network.

[0019] The features extracted for each image patch may include feature representations known from computer vision, such as gradients, HOG features, SIFT features, SURF features, or intensity histograms.

[0020] The inventors found it advantageous to use feature representations as input to the neural network because, as is common in the early layers of convolutional neural networks, the neural network does not need to learn about low-level feature representations. This allows for a reduction in the number of convolutional layers within the neural network while still maintaining high accuracy in classifying interference. This is particularly advantageous when the neural network operates on embedded or battery-powered hardware, as smaller neural networks require less computational power and consume less energy.

[0021] In a preferred embodiment of the neural network, it is also possible to characterize a filtered feature vector from multiple feature vectors. This may be understood as performing a filtering operation on feature vectors extracted from an image and then supplying the filtered feature vector to the neural network. Filtering of feature vectors may include, in particular, a smoothing operation. For example, if the image processed by the neural network is part of a stream of images, such as a video signal, then the images in the stream have a certain amount of preceding images, except for the first image in the stream. With respect to filtering, feature vectors may be extracted for patches in the image itself and in a predetermined amount of preceding images. Feature vectors corresponding to patches at the same location in different images may then be processed by a filtering operation. For example, given a patch location, all feature vectors extracted for this patch location from the image and a predetermined amount of preceding images may be combined using a median filter; that is, the element-wise median of the feature vectors for a given patch may be provided as the filtered feature vector for that patch. Alternatively, it is also possible to use mean filtering, that is, to provide the element-wise average of the feature vectors for the patch given as filtered feature vectors for the patch. For the first image in the stream, the feature vector extracted for the patch may be provided as a filtered feature vector.

[0022] Filtering feature vectors is advantageous because it removes high-frequency noise from them. For example, if a light sensor is placed after a wiper used to clear a field of view from raindrops, the filtering operation filters out the feature vectors extracted for the wiper. In other words, unwanted transient interferences that should not be detected as interferences are effectively filtered out. The inventors found that this further improves the accuracy of the neural network.

[0023] In another aspect, the present invention relates to a computer-based method for classifying whether or not an optical sensor is being interfered with, A step of determining an output from a neural network with respect to a provided input to the neural network, wherein the neural network is configured according to any one of the preceding embodiments and the input is based on an image from the optical sensor, The step of determining whether the light sensor is being interfered with or not based on the output of the neural network described above. Includes.

[0024] Essentially, this aspect relates to a method for applying the neural network described above. In the above method, the decision of when to classify a light sensor as disrupted may be selected on a case-by-case basis. For example, a light sensor can be classified as disrupted if its output features that the sensor is disrupted, and otherwise it is classified as not disrupted. In other words, a light sensor can be classified as disrupted based on a classification of a single image (or multiple images if feature vectors are extracted and filtered).

[0025] However, it is also possible to determine whether the optical sensor is disturbed based on the classification of a plurality of images. For example, for each of the plurality of images provided by the optical sensor, it is possible to determine that the output of the neural network results in a plurality of outputs, and further, when the amount of the output characterizing the disturbed optical sensor is greater than or equal to a predetermined threshold, the optical sensor is classified as disturbed.

[0026] For example, when the plurality of images are a stream of images from the optical sensor, the optical sensor may be classified as disturbed when all the images from the plurality are classified as disturbed by the neural network. However, it is also possible for the optical sensor to be classified as disturbed when the amount of the images classified as disturbed is equal to or exceeds a predetermined threshold.

[0027] Preferably, the method for classifying whether the optical sensor is disturbed is part of a method implemented by a computer for determining a control signal of a robot, where the robot senses the environment of the robot using at least the optical sensor, and the control signal is determined based on the classification of whether the optical sensor is disturbed.

[0028] The robot may in particular be at least partially autonomous vehicle, such as a motor vehicle, a drone, a ship, or a mobile robot. The robot may alternatively be, for example, a manufacturing machine configured for welding, soldering, cutting, or installation.

[0029] Such robotic actions, for example, the movement of the entire robot or a part of the robot along a predetermined path, may be triggered according to control signals that may be specifically determined based on the robot's environment. Light sensors may be configured to sense the robot's environment. If the light sensors are classified as being disrupted, appropriate actions may be taken to ensure the robot maintains safe and desirable operation. For example, control of the robot may be handed over to a human operator, thereby ending the robot's autonomous operation at least temporarily. Alternatively, it is also possible to assess the environment without taking into account information from the light sensors. For example, the robot may be equipped with other light sensors, such as different types of light sensors or redundant light sensors, from which the environment may be subsequently sensed, while ignoring information from the light sensors. Methods for classifying whether or not the light sensors are being disrupted can therefore deal with methods for inferring appropriate actions about the current situation.

[0030] Embodiments of the invention will be discussed in more detail with reference to the following figures. [Brief explanation of the drawing]

[0031] [Figure 1] This is a diagram of a tensor. [Figure 2] This is a diagram showing a 1D convolutional layer. [Figure 3] This diagram shows a method for extracting input from an image. [Figure 4] This figure shows a neural network including a 1D convolutional layer. [Figure 5] This figure shows a control system that includes a neural network for controlling actuators within the control system environment. [Figure 6] This figure shows a control system for controlling a robot that is at least partially autonomous. [Figure 7] This figure shows a control system for controlling manufacturing equipment. [Modes for carrying out the invention]

[0032] Figure 1 shows a tensor(t), characterized by elements(e), which are organized along the height(h), width(w), and depth(d) of the tensor. The dimension of tensor(t) corresponding to the height(h) can also be understood as the vertical axis of tensor(t), while the dimension of tensor(t) corresponding to the width(w) can also be understood as the horizontal axis of tensor(t). A matrix slice along the depth(d) dimension of tensor(t) can also be understood as the channels of tensor(t). For example, tensor(t) can characterize an image, such as an RGB image. In this case, each channel characterizes the color channels of the image, and each element(e) characterizes the pixel values ​​of the red, green, or blue channels, respectively. The image may also be given as a grayscale image, in which case tensor(t) contains only a single channel, and each element(e) characterizes the intensity of a pixel. Tensor(t) can also characterize feature vectors. Each feature vector contains elements (e) along the depth dimension of the tensor (t). In addition, each feature vector is characterized by its position along the height (h) and width (w) of the tensor.

[0033] Figure 2 schematically illustrates the operation of a 1D convolutional layer (61). The 1D convolutional layer receives an input (i), preferably given in the form of a tensor (t). The input (i) is provided to a first filter (c1) of the 1D convolutional layer (61). The first filter (c1) contains trainable weights for each element (e) in a slice along the depth (d) dimension of the input (i). The first filter (c1) processes the input (i) by performing a discrete convolution of the first filter (c1) using the input (i). Figure 2 shows an embodiment in which the first filter (c1) operates along the height (h) dimension of the input (i). In other embodiments (not shown), the first filter (c1) also operates along the width (w) of the input (i). The first output (o1) of the first filter (c1) may then be provided as the output (o) of the 1D convolutional layer (61). Alternatively, the 1D convolutional layer (61) may contain multiple filters (c1, c2, c3), each containing trainable weights for each element (e) in the slice along the depth (d) dimension of the input (i). Each of the multiple filters (c1, c2, c3) can then process the input (i) separately and produce its own output (o1, o2, o3). Thus, multiple outputs may be obtained from the filters (c1, c2, c3), which may be sequentially concatenated along a predetermined dimension to determine the output (o) of the 1D convolutional layer (61).

[0034] Figure 3 shows a preferred method for extracting feature vectors from an image (S) and assembling the extracted feature vectors into a tensor (x) which may be used as input to a neural network. The image (S) is divided into separate patches (p). For each image, feature vectors are extracted. The feature vectors may be, for example, a histogram of the directional gradient (g). The histogram may be assigned to a vector used as the feature vector of the tensor (x).

[0035] Figure 4 shows a neural network (60) containing two 1D convolutional layers (61, 62). The neural network (60) has multiple convolutional layers (C1, C2, C n ) includes multiple convolutional layers (C1, C2, C n The ) may be configured to accept an input signal (x) as input and to provide a tensor (t) that characterizes a feature representation (f) of the input signal (x). The feature representation (f) is then provided to a first 1D convolutional layer (61) and a second 1D convolutional layer (62). The first 1D convolutional layer (61) is configured to operate along the height (h) dimension of the feature representation (f), while the second 1D convolutional layer is configured to operate along the width (w) dimension of the feature representation (f). Thanks to what is obtained from the convolutional layers, the feature representation (f) may also be understood as a convolutional output. The first 1D convolutional layer (61) and the second 1D convolutional layer (62) determine a first convolutional output (co1) and a second convolutional output (co2), respectively. The first convolutional output (co1) and the second convolutional output (co2) may each be a matrix in particular. The first convolutional output (co1) and the second convolutional output (co2) may then be provided as inputs to a fully connected layer (F1) of the neural network (60). To be used as inputs, the first convolutional output (co1) and the second convolutional output (co2) may be flattened into vectors, respectively, and the resulting vectors may then be concatenated to form a larger vector to be used as inputs to the fully connected layer (F1). The fully connected layer (F1) may be part of a group of fully connected layers (F1, F2, F3), i.e., part of a multilayer perceptron that is part of the neural network (60). The output of the multilayer perceptron may then be provided as an output signal (y), i.e., as the output of the neural network (60). If the fully connected layer (F1) is used to provide the output of the neural network (60), the output of the fully connected layer (F1) may be provided as an output signal (y).

[0036] In further embodiments, the input signal (x) may also include feature vectors, each of which characterizes a patch of the image. A patch of the image may be understood as multiple pixels of the image, each patch characterizing a rectangular region of the image, and the image may be divided into separate patches. The feature vectors may specifically characterize SIFT, SURF, SWIFT®, or other gradient features of the patch of the image.

[0037] A neural network (60) may be specifically trained to determine, based on the input signal (x), whether a determined light sensor is disturbed or not. The output signal (y) may specifically characterize probability values ​​that characterize the probability of a disturbed light sensor. In addition or alternatively, the output signal (y) may also characterize the probability of an undisturbed light sensor.

[0038] The neural network (60) may be trained in a supervised manner, preferably using a (potentially probabilistic) gradient descent algorithm or an evolutionary algorithm. With respect to training, the neural network (60) may provide an input signal and a desired output signal that characterizes whether or not the training input signal characterizes a disturbed light sensor.

[0039] Figure 5 shows an embodiment of the actuator (10) within its own environment (20). The actuator (10) interacts with a control system (40). The actuator (10) and its environment (20) together are referred to as the actuator system. Preferably at equal intervals, the optical sensor (30) senses the state of the actuator system. The optical sensor (30) may include several sensors. The output signal (S) of the optical sensor (30) (or, in the case where the sensor (30) includes multiple sensors, the output signal (S) for each sensor) encoding the sensed state is transmitted to the control system (40).

[0040] As a result, the control system (40) receives a stream of sensor signals (S). The control system (40) then calculates a series of control signals (A) in response to the stream of sensor signals (S) subsequently transmitted to the actuator (10).

[0041] The control system (40) receives a stream of sensor signals (S) from the sensor (30) in the light receiving unit (50). The light receiving unit (50) converts the sensor signals (S) into input signals (x). Alternatively, in the absence of the light receiving unit (50), each sensor signal (S) may be taken directly as the input signal (x). The input signal (x) may be given, for example, as an excerpt from the sensor signals (S). Alternatively, the sensor signals (S) may be processed to produce the input signal (x) by, for example, extracting feature vectors relating to patches of the sensor signals (S). The input signal (x) may also characterize filtered feature vectors. For example, the light receiving unit (50) can store feature vectors relating to other sensor signals preceding the sensor signal (S), and then determine a median or average feature vector relating to the above sensor signal and other sensor signals (S). In other words, the input signal (x) is provided according to the sensor signals (S).

[0042] The input signal (x) is then transmitted to the neural network (60). The neural network (60) is stored in the parameter memory unit (St1) and is also represented by the provided parameters (Φ).

[0043] The neural network (60) determines the output signal (y) from the input signal (x). The output signal (y) is transmitted to an optical conversion unit (80) that converts the output signal (y) into a control signal (A). The control signal (A) is then transmitted to the actuator (10) to control the actuator (10) accordingly. Alternatively, the output signal (y) may be directly obtained as the control signal (A).

[0044] The actuator (10) receives a control signal (A), is controlled accordingly, and performs an action corresponding to the control signal (A). The actuator (10) may include a control logic unit that converts the control signal (A) into a further control signal, which is then used to control the actuator (10).

[0045] In further embodiments, the control system (40) may include a sensor (30). In further embodiments, the control system (40) may optionally or additionally include an actuator (10).

[0046] In further embodiments, the control system (40) may be envisioned to control the display (10a) instead of or in addition to the actuator (10). Furthermore, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) which stores instructions that, if executed, cause the control system (40) to perform a method according to a certain aspect of the present invention.

[0047] Figure 6 shows an embodiment in which a control system (40) is used to control a robot that is at least partially autonomous, for example, a vehicle (100) that is at least partially autonomous. The optical sensor (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors. Some or all of these sensors are preferably, but not necessarily, incorporated into the vehicle (100).

[0048] The neural network (60) may be configured to detect interference with the vehicle's (100) optical sensor (30). A control signal (A) may then be selected according to an output signal (y) determined by the neural network (60). For example, if the optical sensor (30) is interfered with, the detection of the vehicle's (100) environment (20) may be performed without considering information from the interfered optical sensor (30). For example, the vehicle (100) may be equipped with a camera sensor and a LIDAR sensor, both of which are used to detect objects near the vehicle (100). The neural network (60) may be configured to determine whether the camera sensor is interfered with. If the camera sensor is classified as interfered with, the detection of objects detected based on the camera sensor image or the detection of no objects based on the camera sensor image may be ignored in order to determine the vehicle's (100) route and / or to navigate the vehicle (100).

[0049] Alternatively, for example, when using only camera sensors to determine the environment (20) of the vehicle (100), it is also possible to hand over control of the vehicle (100) to a human driver or operator if it is determined that the vehicle's (100) light sensor (30) is being interfered with.

[0050] The actuator (10), which is preferably incorporated into the vehicle (100), may be provided by the vehicle's (100) brakes, propulsion system, engine, drivetrain, or steering wheel. The control signal (A) may be determined so that the actuator (10) is controlled to avoid collision with an object near the vehicle (100).

[0051] Alternatively or additionally, control signal (A) may also be used to control display (10a) to indicate to the driver or operator of vehicle (100) that the light sensor (30) is being interfered with, for example. It is also conceivable that control signal (A) could control display (10a) to generate a warning signal when the light sensor (30) is classified as being interfered with. The warning signal may be an audible and / or tactile signal, such as vibration of the steering wheel of vehicle (100).

[0052] In further embodiments, the at least partially autonomous robot may be provided by another mobile robot (not shown) which can move, for example, by flying, swimming, diving, or stepping. The mobile robot may, in particular, be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all of the above embodiments, the control signal (A) may be determined to control the propulsion unit and / or handle and / or brakes of the mobile robot so that the mobile robot can avoid collision with the identified object.

[0053] In further embodiments, a robot that is at least partially autonomous may be provided by a horticultural robot (not shown), which uses a light sensor (30) to determine the condition of plants in the environment (20). An actuator (10) can control a nozzle and / or cutting device, such as a blade, for spraying liquid. Depending on the type and / or condition of the plant identified, a control signal (A) may be determined to cause the actuator (10) to spray the plant with the appropriate amount of the appropriate liquid and / or cut the plant.

[0054] In further embodiments, a robot that is at least partially autonomous may be provided by a household appliance (not shown), such as a washing machine, stove, oven, microwave oven, or dishwasher. The optical sensor (30) can detect the state of an object being processed by the household appliance. For example, in the case of a household appliance that is a washing machine, the optical sensor (30) can detect the state of the laundry inside the washing machine. A control signal (A) may then be determined according to the detected material of the laundry.

[0055] Figure 7 shows an embodiment in which a control system (40) is used to control, for example, a manufacturing device (11) of a manufacturing system (200) as part of a production line, such as a punch cutter, cutter, gun drill, or gripper. The manufacturing device may include a conveying device for moving the manufactured products (12), such as a conveyor belt or assembly line. The control system (40) controls an actuator (10), which in turn controls the manufacturing device (11).

[0056] The optical sensor (30) can, for example, capture the characteristics of the manufactured product (12). The actuator (10) may be controlled, for example, according to the position detected by a second neural network. For example, the actuator (10) may be controlled to cut the manufactured product at a specific location on the manufactured product itself. Alternatively, the second neural network may be used to classify whether the manufactured product is broken or defective. The actuator (10) may then be controlled to remove the manufactured product from the conveying device.

[0057] A neural network (60) may be configured to classify whether or not the light sensor (30) is being interfered with. If the light sensor (30) is classified as being interfered with, the manufacturing equipment (11) may be shut down and / or an operator or technician may be warned to perform maintenance on the manufacturing equipment (11).

[0058] The term “computer” may also be understood to cover any device for processing predetermined calculation rules. These calculation rules may be in the form of software, hardware, or a mixture of software and hardware.

[0059] In general, a plural may be understood as being indexed, that is, each element of the plural is assigned its own index, preferably by assigning a consecutive integer to the elements contained in the plural. Preferably, if the plural contains N elements, where N is the number of elements in the plural, then the elements are assigned integers from 1 to N. It may also be understood that elements of the plural are accessed by their own index. [Explanation of symbols]

[0060] 10 Actuators 10a display 11 Manufacturing equipment 12. Manufactured products 30 Light Sensors 40 Control Systems 45 processors 46 Machine-readable storage media 50 Light receiving units 60 Neural Networks 61 1D convolutional layer, first 1D convolutional layer 62 1D convolutional layer, second 1D convolutional layer 80 Light Conversion Unit 100 vehicles 200 manufacturing systems c filter co convolution output d depth e element f Feature display g Directional gradient h height i input o Output p isolation patch w width t tensor x tensor A control signal F fully connected layer S Image S: Sensor output signal, sensor signal Φ parameter

Claims

1. A neural network (60) configured to classify whether an image (S) from an optical sensor (30) characterizes an obstruction of the optical sensor (30), wherein the classification is characterized by an output (y) of the neural network (60) with respect to an input (x) of the neural network (60), wherein the input (x) is based on the image (S), The neural network (60) is preceded by a first convolutional layer (61) that characterizes a 1D convolution along the vertical axis of the convolutional output (f) of the convolutional layer (C ｎ ), and includes a first convolutional layer (61) that characterizes a 1D convolution along the vertical axis of the convolutional output (f) of the convolutional layer (C wherein the neural network (60) includes a second convolutional layer (62) that characterizes a 1D convolution along the horizontal axis of the convolutional output (f), The output (y) of the neural network (60) is based on the first convolution output (co １ ) of the first convolutional layer (61), The output (y) of the neural network (60) is based on the second convolution output (co ２ ) of the second convolutional layer (62). neural network (60).

2. The first convolution output (co １ ), and the second convolution output (co ２ ) are used as inputs to the fully connected layer (F １ ) of the neural network (60), The output (y) of the neural network (60) is determined based on the output of the fully connected layer (F １ ). The neural network (60) according to claim 1.

3. The neural network (60) according to claim 1, wherein the input (x) of the neural network (60) is the image (S).

4. The neural network according to claim 1, wherein the input (x) of the neural network (60) is a plurality of feature vectors, and each feature vector is determined for a separate patch (p) of the image (S).

5. The neural network according to claim 4, wherein the feature vectors from the plurality of feature vectors characterize filtered feature vectors.

6. A computer-implemented method for classifying whether an optical sensor (30) is obstructed, comprising determining an output from the neural network (60) according to claim 1 with respect to a provided input (x) of the neural network (60), wherein the input (x) is based on an image (S) of the optical sensor (30), and determining an output; and determining a classification of whether the optical sensor (30) is obstructed based on the output (y) of the neural network (60). A computer-implemented method comprising the steps above.

7. The method according to claim 6, wherein the optical sensor (30) is classified as obstructed when the output characterizes that the optical sensor (30) is obstructed, and the optical sensor (30) is classified as not obstructed otherwise.

8. For each image (S) of a plurality of images provided by the optical sensor (30), it is determined that the output of the neural network (60) yields a plurality of outputs (y). Further, the optical sensor (30) is classified as being obstructed when the amount of the output (y) characterizing the obstructed optical sensor (30) is equal to or greater than a predetermined threshold, and the optical sensor (30) is classified as not being obstructed otherwise. The method according to claim 6.

9. A method implemented by a computer for determining a control signal (A) of a robot (100, 200), wherein the robot (100, 200) senses the environment (20) of the robot using at least one optical sensor (30), and the control signal (A) is determined based on a classification of whether the optical sensor (30) is obstructed, and the classification is obtained by the method according to claim 6. A method implemented by a computer.

10. The method according to claim 9, wherein the robot (100, 200) is at least partially autonomous vehicle (100) or a manufacturing robot (200).

11. The method according to claim 6, wherein the optical sensor (30) comprises a camera sensor and / or a LIDAR sensor and / or a radar sensor and / or an ultrasonic sensor and / or a thermal camera.

12. A computer program configured to cause a computer to execute the method according to claim 6 in all steps of the method when the computer program is executed by a processor (45).

13. A machine-readable storage medium (46) storing the computer program according to claim 12.

Images