Hazard detection integrated architecture system and method

Abstract

Hazard Detection Integrated Architecture System and Method. A system and method are disclosed that can employ a collection of integrated methods, the collection of integrated methods including multiple machine learning algorithms or statistical algorithms for hazard detection. The system and method can combine image, stereo, and environmental information using multiple deep learning algorithms to accurately detect hazards. The system and method can merge the depth channel into the red, green, and blue (RGB) channels of the image to create a four-channel RGBD image. The system and method can also overlay the RGB image with a color map of the depth channel. The system and method can further connect a region of interest (ROI) to the RGB channels of the image. Finally, the system and method can incorporate an auxiliary semantic segmentation decoder for multi-task learning events on the drivable space.

Description

Technical Field

[0001] The present disclosure relates to an ensemble architecture for detecting hazards within stereo images. Background Art

[0002] Road debris causes many reported crashes each year in the U.S. Robust advanced detection and warning systems and methods for road debris or hazards—whether implemented in an autonomous driving system or in a conventional (ie, human-controlled) driving system—may be desirable. Summary of the invention

[0003] A system and method for detecting hazards within a stereoscopic image having a red channel, a blue channel, and a green channel is disclosed. The red channel, the blue channel, and the green channel produce a 3-channel RGB image. The system and method may include an integrated network having one or more convolutional layers, batch norm layers, YOLO layers, and upsampling layers. It is contemplated that the integrated network may be designed using a YOLOv3 Tiny or Mask R-CNN architecture.

[0004] The integration network may be operable to determine one or more hazards within the stereoscopic image by concatenating the depth channel with the red channel, the blue channel, and the green channel, wherein the concatenation produces a 4-channel RGBD image. The integration network may also be operable to increase the number of bits within each of the red channel, the blue channel, and the green channel to produce a modified 3-channel RGB image. The modified 3-channel RGB image may then be overlaid with the color map of the depth channel.

[0005] The integration network can also determine one or more hazards within the stereo image by extracting a region of interest from the label map and concatenating the region of interest with the red channel, the blue channel, and the green channel, wherein the concatenation produces a 4-channel image. The integration network can also determine one or more hazards within the stereo image by concatenating the output classification from the semantic segmentation algorithm with the red channel, the blue channel, and the green channel, wherein the concatenation produces a 4-channel image.

[0006] The integration network may further include a first RGB encoder network and a second deep encoder network, the first RGB encoder network having a plurality of fusion layers, and output values ​​from the plurality of encoder network layers are added to the plurality of fusion layers. Finally, an auxiliary semantic segmentation decoder may be included to force the integration network to learn one or more features of the drivable space to determine one or more hazards within the stereo image. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An exemplary computing system is illustrated.

[0008] Figure 2 An exemplary convolutional neural network is illustrated.

[0009] Figure 3A and Figure 3B An exemplary integrated machine learning architecture for detecting hazards is illustrated.

[0010] Figure 4 A computing system for controlling an at least partially autonomous robot is illustrated. DETAILED DESCRIPTION

[0011] Embodiments of the present disclosure are described herein. However, it will be understood that the disclosed embodiments are merely examples, and other embodiments may take various forms and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but only as a representative basis for teaching those skilled in the art to adopt the embodiments in different ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the figures may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of illustrated features provides representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable for specific applications or implementations.

[0012] In the United States, road debris causes many police-reported crashes each year. Therefore, it is desirable for both autonomous and conventional driving systems to be able to perform advanced detection, alert, and respond to road hazards / debris. While it is contemplated that existing systems can operate to detect objects (e.g., pedestrian detection or traffic light / sign detection), hazard detection systems are not as easy to use, although vehicle controllers are not typically designed to detect them.

[0013] To solve the problem of hazard detection, one or more computer vision algorithms (e.g., stereo vision and optical flow) can be used. Alternatively, an occlusion edge map can also be used to help detect small hazards. On a pixel-level metric, one or more deep learning algorithms can also be used after a training process using a lost and found dataset.

[0014] For example, a fast direct plane hypothesis test (FPHT) that combines a lost and found dataset with a stereo vision-based algorithm can be used for hazard detection. It is understood that the lost and found dataset can be a known benchmark for training a hazard detection machine learning algorithm. The FPHT algorithm can be based on a statistical analysis of disparity maps from a stereo camera for drivable space and obstacles. If there are objects in a computationally robust and efficient manner, the FPHT algorithm can operate by performing hypothesis testing. Deep learning algorithms may also be operable for appearance-based small road hazard detection. It is envisioned that the predicted FPHT stixels can then be fused with stixels extracted from a semantic segmentation network using one or more known machine learning architectures (e.g., GoogLeNet).

[0015] Another hazard detection algorithm can use semantic segmentation methods (e.g., MergeNet), which has the same potential performance as that provided by FPHT fusion. It is envisioned that semantic segmentation can use RGB images and conventionally derived disparity Figure 2 The MergeNet process can also employ two or three semantic segmentation networks. An additional and relatively shallow network can be used to fuse the different semantic maps. Typically, one semantic segmentation network operating on a full-resolution RGB image can be combined with another network operating on vertical stripes of the image.

[0016] Anomaly detection algorithms can also be used to detect hazards by reconstructing roads using compression networks such as restricted Boltzmann machines (RBMs) or autoencoders. It is envisioned that generative adversarial networks (GANs) can also be used to detect unexpected hazards. The semantic segmentation can then be used as input to the GAN to resynthesize the original map. The generated image can then be compared to the input image to detect differences.

[0017] A stereo vision algorithm that computes depth (i.e., disparity map) that requires less computation time (i.e., is faster) but may be more sensitive to occlusion and sensor noise. It is therefore contemplated that utilizing disparity information in combination with RGB images purely as a fourth channel in deep learning may be challenging because early fusion methods may be sensitive to noise. The computing system 102 may be operable to overcome potential challenges in utilizing this approach because depth may be represented using alternative methods, or alternative late fusion architectures and novel depth data augmentation techniques may be employed. It is also contemplated that a mid-term fusion strategy may be additionally used to employ a FuseNet approach. The computing system 102 may therefore be operable to employ a mid-term or late fusion algorithm.

[0018] Multi-task learning algorithms have also been envisioned for hazard detection, where multiple computer vision tasks can be solved with a joint neural network. This approach can employ both object detection and semantic segmentation, but these techniques may not exclude two-dimensional tasks. Due to shared computation, the combined network may be cheaper than the separate networks, and multi-task learning algorithms may also provide improved generalization capabilities.

[0019] While one or more of these algorithms can be used for hazard detection, there are still potential challenges. For example, one challenge that may exist is related to "class imbalance", where - unlike traffic sign, vehicle, or pedestrian detection tasks - the hazard detection problem may be more difficult because road hazards may be less common. Although less common, road hazards are still the cause of a large number of accidents each year (e.g., about 50,000 accidents in the United States alone). But what is less common is that natural driving datasets with hazards that can be used to train machine learning algorithms are generally less available, and the training process may be more expensive than for the balanced detection task.

[0020] Another challenge that may exist is related to the “environment” where an object may be considered a hazard depending on whether it is in the potential path of the vehicle. In terms of machine learning algorithms, the challenge is not only detecting objects, but also developing an algorithm that can determine whether those objects may be problematic given their current location. For example, a given object of reasonable size (e.g., a pothole) may be considered a hazard if and only if the object is within the potential path of the vehicle.

[0021] Additional challenges that may exist are related to the “diversity” of objects. Since there may be countless objects that can be classified as hazards, machine learning algorithms dealing with hazard detection should be able to predict things that may not be available in the training set. For example, debris from car collisions, fallen cargo, cracks in the road, and toys may be diverse potential hazards that do not have any consistency in shape, color, or composition. In addition, driving scenarios featuring hazards can be extremely different - both in reality and in publicly available datasets. The diversity (i.e., variability) of hazards makes it challenging to obtain sufficient data and may affect the efficiency of machine learning algorithms in real-time operation in detecting hazards with high recall and precision.

[0022] The last challenge that may exist is related to the “metrics” of objects. For example, conventional object detection or semantic segmentation metrics may not fully capture the quality of algorithms used for hazard detection. Recent work in the metric space also does not seem to be related to instance-level metrics. Therefore, there is a need to come up with a consistent set of metrics for hazard detection.

[0023] Regarding these challenges, detecting road hazards using pixel-by-pixel prediction algorithms can provide accurate results, while semantic segmentation algorithms can be computationally expensive and have difficulty tracking objects over time. In addition, hazard detection may not require predicting pixel-accurate contours. Instead, hazard detection can be used as a subset of object detection or instance segmentation. It is therefore envisioned that when starting to detect hazards, either object detection or instance segmentation architectures can be used.

[0024] By using object / instance detection algorithms for hazard detection, potential recall-precision tradeoffs can be reduced and objects can be tracked over time. Confidence thresholds can also be employed and can be tunable, allowing for greater flexibility across applications and improved performance in meeting the needs of a given application.

[0025] While it may be possible to achieve similar performance from a semantic segmentation algorithm, such a solution may be more complex to implement and may require greater adjustments to the loss during the training process. Furthermore, in order to distinguish between different instances, there is a need to track objects with different trajectories. It is envisioned that a Kalman filter algorithm can be used to make these distinctions.

[0026] It is also contemplated that for a given application (e.g., an autonomous vehicle), hazards may need to be reliably detected at increased distances because the hazards will be approaching at high speeds. Furthermore, the machine learning algorithm may need to avoid false positives when employed with or as part of an operational control strategy for an autonomous vehicle.

[0027] It is also contemplated that machine learning algorithms may need to operate against a diverse set of hazards that may only be hazards if they appear in the "right" environment - as explained above with respect to the challenges associated with hazard detection. Furthermore, machine learning algorithms may need to be trained and operable to detect "potential" hazards (e.g., "there is something potentially dangerous lying on the road") despite the potential for high-level imbalance.

[0028] It is thus contemplated that one or more machine learning algorithms and statistical algorithms may be employed to provide detection of unexpected, undefined hazards in roadways (e.g., highways, side streets, parking lots), which may be of various shapes (e.g., cars, trucks, and pedestrians) and sizes (e.g., objects having a size between 5 centimeters and 1 meter). The machine learning and statistical algorithms may also employ multi-task learning to improve generalization capabilities in detecting hazards. Due to potential limitations on computing resources (e.g., embedded applications), the machine learning and statistical algorithms may also be optimized to provide functionality necessary for a given application. It is further contemplated that the machine learning and statistical algorithms may be designed with reduced computational effort.

[0029] Figure 1 An exemplary system 100 is illustrated that can be used to employ one or more machine learning algorithms and statistical algorithms to detect unexpected, undefined hazards. System 100 may include at least one computing device 102. Computing system 102 may include at least one processor 104 operably connected to a storage unit 108. Processor 104 may be one or more integrated circuits that implement the functionality of a processing unit (PU) 106. PU 106 may be a commercially available central processing unit (CPU) that implements instructions such as one of the x86, ARM, Power, or MIPS instruction set families. Alternatively, processing unit 106 may be a commercially available graphics processing unit (GPU) that consists of hundreds of cores that are operable to simultaneously process a large number of parallel tasks (i.e., parallel computing).

[0030] During operation, the PU 106 may execute stored program instructions retrieved from the memory unit 108. The stored program instructions may include software that controls the operation of the PU 106 to perform the operations described herein. In some examples, the processor 104 may be a system on a chip (SoC) that integrates the functionality of the PU 106, the memory unit 108, a network interface, and an input / output interface into a single integrated device. The computing system 102 may implement an operating system for managing various aspects of operation.

[0031] The memory unit 108 may include volatile memory and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 102 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, the memory unit 108 may store a machine learning model 110 or algorithm, a training data set 112 for the machine learning model 110, and / or raw source data 115.

[0032] The computing system 102 may include a network interface device 122 that is configured to provide communications with external systems and devices. For example, the network interface device 122 may include a wired and / or wireless Ethernet interface as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 122 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 122 may further be configured to provide a communication interface to an external network 124 or a cloud.

[0033] The external network 124 may be referred to as the World Wide Web or the Internet. The external network 124 may establish a standard communication protocol between computing devices. The external network 124 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 130 may communicate with the external network 124.

[0034] Computing system 102 may include input / output (I / O) interface 120, which may be configured to provide digital and / or analog input and output. I / O interface 120 may include an additional serial interface (e.g., a universal serial bus (USB) interface) for communicating with external devices.

[0035] The computing system 102 may include a human-machine interface (HMI) device 118, which may include any device that enables the system 100 to receive control inputs. Examples of input devices may include human-machine interface inputs such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 102 may include a display device 132. The computing system 102 may include hardware and software for outputting graphics and text information to the display device 132. The display device 132 may include an electronic display screen, a projector, a printer, or other suitable device for displaying information to a user or operator. The computing system 102 may be further configured to allow interaction with a remote HMI and a remote display device via a network interface device 122.

[0036] System 100 may be implemented using one or more computing systems. Although this example depicts a single computing system 102 implementing the described features, it is intended that the various features and functions may be separated and implemented by multiple computing units in communication with each other. The architecture selected may depend on a variety of factors.

[0037] The system 100 may implement one or more machine learning algorithms 110 configured to analyze raw source data 115 (or data sets). The raw source data 115 may include raw or unprocessed sensor data, which may represent an input data set for a machine learning system. The raw source data 115 may include videos, video clips, images, and raw or partially processed sensor data (e.g., data from a digital camera or LiDAR sensor). In some examples, the machine learning algorithm 110 may be a neural network algorithm (e.g., a Transformer, CNN, RNN, or DNN) that may be designed to perform a predetermined function.

[0038] For example, Figure 2A non-limiting example of a CNN 200 is illustrated, comprising: an input data set 210; one or more convolutional layers 220-240; one or more pooling layers 250-270; a fully connected layer 280; and a softmax layer 290.

[0039] The input data set 210 may include raw images, speech data, or text data. The input data set 210 may also include measurements received from one or more sensors. Alternatively, the input data set 210 may be slightly processed before being provided to the CNN 200. The convolutional layers 220-240 may also be operable to extract features from the input data set 210. It is generally understood that the convolutional layers 220-240 may be operable to apply filtering operations (e.g., kernels) before passing the results to another layer of the CNN 200. For example, for a given data set (e.g., a color image), the convolutional layer may perform filtering routines to perform operations such as image recognition, image edge detection, and image sharpening.

[0040] It is also contemplated that the CNN may include one or more pooling layers 250-270 that receive convolution data from corresponding convolutional layers 220-240. The pooling layers 250-270 may include one or more pooling layer units that apply a pooling function to one or more convolutional layer outputs computed at different frequency bands using a pooling function. For example, the pooling layer 250 may apply a pooling function to the kernel output received from the convolutional layer 220. The pooling function implemented by the pooling layers 250-270 may be an average or maximum function or any other function that aggregates multiple values ​​into a single value.

[0041] The fully connected layer 280 may also be operable to learn nonlinear combinations of high-level features in the output data received from the convolutional layers 220-240 and the pooling layers 250-270. Finally, the CNN 200 may include a softmax layer 290 that combines the outputs of the fully connected layer 280 using a softmax function. Imagine, Figure 2 The neural network algorithm shown may be configured to operate within an automotive application to recognize objects (eg, pedestrians) based on images provided from a digital camera and / or depth maps from a LiDAR sensor.

[0042] The machine learning algorithm 200 may be operable to combine image, stereo, and environmental information to provide improved detection accuracy and computational speed. For example, the machine learning algorithm 200 may employ an object detection architecture that supplements RGB with additional information. The machine learning algorithm may also be designed using known or custom architectures (e.g., Mask R-CNN, DeepLabV3+, YOLOv3Tiny) that may provide cost-effective, deployable, real-time applications. The first set of strategies may include parallax, and the second set of strategies may be designed to achieve a better understanding of the drivable space.

[0043] The machine learning algorithm 200 may also undergo a training process using one or more datasets (e.g., Lost and Found dataset, KITTI, MS Coco, ImageNet, Cityscape). It is understood that the Lost and Found dataset may be a stereo vision dataset that contains one or more pre-computed disparity maps obtained by semi-global block matching. It is also contemplated that the Lost and Found dataset may provide a label map for segmentation that includes drivable space and hazards. The format of the Lost and Found dataset may also generally follow the structure of the Cityscape dataset.

[0044] The machine learning algorithm 200 may also be operable to detect hazards by concatenating the depth channels of one or more images received as data 115. It is contemplated that concatenation may be employed because it may be considered a naive approach for incorporating depth.

[0045] For example, Figure 3A An embodiment of a machine learning architecture 300 that can be employed by a computing system 102 is illustrated. It is contemplated that the machine learning architecture 300 can be operably designed like a YOLOv3 Tiny architecture. As illustrated, the machine learning architecture 300 can receive an RGB image 302 and a depth image 304 as data 115. The machine learning architecture 300 can then include one or more layers 306-326 that operate to join the depth channel to the red, green, and blue channels of the image, thereby producing a 4-channel RGBD image.

[0046] It is contemplated that layers 306-316 may include one or more convolutional layers, batch norm layers, leaky ReLU activation functions, and max pooling layers. Layers 318-320 may also include one or more convolutional layers, batch norm layers, and leaky ReLU activation functions. However, layers 318-320 may not include one or more max pooling layers. Layers 322 and 326 may include one or more convolutional layers, batch norm layers, leaky ReLU activation functions, and YOLO (You Only Look Once) layers. It is contemplated that a YOLO layer may include a convolutional layer, a YOLO output process, and a cost function. Finally, layer 324 may include one or more convolutional layers, a batch norm layer, a leaky ReLU activation function, and one or more upsampling layers. It is also contemplated that one or more skip architectures may be employed. For example, the output of layer 314 may be provided as an input to layer 326. Similarly, the output of layer 320 may be provided as an input to layer 324. Figure 3A Also illustrated is the number of connections (eg, 1, 1 / 2 / 14, etc.) and the number of summations after activation (eg, 32, 64, 128, etc.).

[0047] It is also contemplated that the machine learning 200 may incorporate depth in alternative approaches. For example, by increasing the number of bits in each RGB channel, the RGB image 302 may be overlaid with a color map for the depth channel. Alternatively, the machine learning architecture 300 may detect hazards by joining regions of interest (RoIs) as a fourth channel. For example, the machine learning algorithm 200 may extract the RoIs from the label map. It is also contemplated that during runtime processing, the machine learning algorithm may be operable to join the output of a semantic segmentation algorithm. For example, a semantic segmentation algorithm may already be employed within a vehicle controller. The machine learning architecture 300 may receive the output of a semantic segmentation algorithm already employed by a vehicle controller to provide hazard detection at a reduced cost. However, it is also contemplated that the resulting bit-level joins may produce similar results as channel joins.

[0048] Alternatively, if Figure 3B As shown, a deep mid-term fusion architecture can be used to perform hazard detection. For example, a first encoder 340 (ie, an RGB encoder) and a second encoder 342. It is contemplated that the second encoder 342 can be additionally added to the YOLOv3Tiny architecture (ie, the first encoder 340) to provide one or more single-channel disparities as input to the first encoder 340.

[0049] As shown, the first encoder 340 may include layers 344-364. It is envisioned that layers 344-354 may include one or more convolutional layers, batch standard layers, leaky ReLU activation functions, maximum pooling layers, and fusion layers. Layers 356-358 may include one or more convolutional layers, batch standard layers, and leaky ReLU activation functions. However, layers 356-358 may not include one or more maximum pooling layers or fusion layers. Layers 360 and 364 may include one or more convolutional layers, batch standard layers, leaky ReLU activation functions, and YOLO layers. Finally, layer 362 may include one or more convolutional layers, batch standard layers, ReLU activation functions, and one or more upsampling layers. It is also envisioned that one or more skip architectures may be used. For example, the output of layer 352 may be provided as an input to layer 364. Similarly, the output of layer 358 may be provided as an input to layer 362. Figure 3B Also illustrated is the number of connections (eg, 1, 1 / 2 / 14, etc.) and the number of summations after activation (eg, 32, 64, 128, etc.).

[0050] The second encoder 342 may include layers 366-376. It is contemplated that layers 366-374 may include one or more convolutional layers, batch normalization layers, leaky ReLU activation functions, and maximum pooling layers. Layer 376 may include one or more convolutional layers, batch normalization layers, and leaky ReLU activation functions. However, layer 376 may not include one or more maximum pooling layers, nor a fusion layer. It is also contemplated that the outputs of layers 366-376 may be provided as inputs to layers 344-354, respectively. In other words, after each activation layer of the second encoder 342, the value of the second encoder 342 (i.e., the depth encoder) is added to the value of the first encoder 340 (i.e., the RGB encoder). It is also contemplated that the second encoder 342 may be randomly initialized. However, it is also contemplated that pre-trained (i.e., established during the training process) weights may be used to initialize all other layers 344-364.

[0051] It is contemplated that the YOLOv3 architecture may not be highly optimized for applications with high levels of imbalance. It is also contemplated that the imbalance of objects relative to no objects may increase quadratically with the width / height of the input image (i.e., image 302). For example, when nine anchors are applied on an input image having 2048×1024 pixels (92,160 potential predictions) instead of 416×416 pixels (7,605 possible predictions), the YOLOv3 network may systematically predict lower confidence scores compared to its probability of being the correct prediction. To compensate, Figure 3B The illustrated machine learning algorithm 200 (ie, the deep mid-term fusion architecture) may employ the loss function shown in equation (1) below:

[0052] (1)

[0053] Where L xy can be the original YOLOv3 bounding box center loss, and L GIoU It can be a generalized Intersection over Union (GIoU) ​​loss. Suppose that the term L xy This may not be necessary, since the center of the bounding box prediction is also determined by L GIoU However, item L xy can be included to reduce the number of epochs until the training process converges. obj It is also possible to express the binary cross entropy loss as the inverse of the class frequency Power weighting. This can include items , because when using the inverse class frequency (k=1), training may become unstable. To help with tuning, the term ,in Nc can be the number of object / no-object labels in the mini-batch. It is also assumed that for Equation (1), the loss term can be represented by the corresponding “ ” hyperparameters to balance. It is also envisioned that a focused loss algorithm can be used to adopt Figure 3B However, for some classification applications (e.g., binary classification), the L obj The use of a focus loss algorithm may not provide improved performance compared to the previous one.

[0054] Finally, the machine learning algorithm 200 may employ a multi-task learning algorithm on the drivable space to perform hazard detection. The auxiliary semantic segmentation decoder may be operable to force the machine learning network 200 to learn the characteristics of the drivable space. It is contemplated that a SegNet decoder (e.g., MergeNet) may be employed by a multi-task learning algorithm. However, it is also contemplated that the auxiliary semantic segmentation decoder may not require the deployment of the machine learning algorithm 200 for object detection. In addition, when a multi-task learning algorithm is employed, item L sem can be included in the overall cost of Equation (1). Since the features of roads can be considered more relevant, the multi-task algorithm can use the unweighted cross entropy as the auxiliary semantic segmentation output. This method acts as a mid-term fusion with the auxiliary semantic segmentation.

[0055] It is therefore envisioned that for improved hazard detection, an ensemble of ensemble methods (i.e., multiple learning algorithms that can achieve better predictive performance than can be achieved from any one of the constituent learning algorithms alone) can be employed. Various hazard detectors combined into a single prediction have been disclosed. The integration of instance segmentation and object detection algorithms can be used to overcome many potential challenges of hazard detection. In addition, the integration of computationally expensive machine learning algorithms (e.g., Mask R-CNN) with less computationally expensive machine learning algorithms (e.g., YOLOv3 Tiny) can be used for hazard detection. An integration that combines similar machine learning algorithms with different “backbones” (e.g., different encoder architectures and pre-training datasets) can also be used for hazard detection. Finally, an integration that fuses similar machine learning algorithms that are trained to classify roads (i.e., a good understanding of what a road is) can also be used for hazard detection.

[0056] Figure 4 An exemplary computing system 440 is illustrated that can be used to detect hazards and control a partially autonomous robot, such as partially autonomous vehicle 400. Computing system 440 can be used as Figure 1 The system 100 described in . The sensors 430 may include one or more video / camera sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors and / or one or more location sensors (such as, for example, GPS). Some or all of these sensors are preferably, but not necessarily, integrated in the vehicle 400.

[0057] Alternatively, the sensor 430 may include an information system for determining the state of the actuator system. The sensor 430 may collect sensor data or other information to be used by the computing system 440. Using the input signal x, the system may, for example, detect a hazard in the vicinity of the at least partially autonomous robot. The output signal y may include information characterizing where an object is in the vicinity of the at least partially autonomous robot. A control command A may then be determined based on this information, for example to avoid a collision with the detected object.

[0058] The actuator 410, which may be integrated into the vehicle 400, may be provided by the brakes, propulsion system, engine, transmission system, or steering system of the vehicle 400. The actuator control command A may be determined such that the actuator (or actuators) 410 are controlled such that the vehicle 400 avoids colliding with the detected object. The detected hazards may also be classified according to what the machine learning algorithm 200 considers them to be most likely to be, such as pedestrians or trees, and the actuator control command A may be determined depending on the classification.

[0059] The process, method or algorithm disclosed herein may be delivered to / implemented by a processing device, a controller or a computer, which may include any existing programmable electronic control unit or a dedicated electronic control unit. Similarly, the process, method or algorithm may be stored in a variety of forms as data and instructions that can be executed by a controller or a computer, including but not limited to information permanently stored on a non-writable storage medium such as a ROM device and information that can be replaced and stored on a writable storage medium such as a floppy disk, a tape, a CD, a RAM device, and other magnetic and optical media. The process, method or algorithm may also be implemented in a software executable object. Alternatively, the process, method or algorithm may be embodied in whole or in part using suitable hardware components, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a state machine, a controller, or other hardware components or devices, or a combination of hardware, software and firmware components.

[0060] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms included in the claims. The words used in the specification are words of description, not words of limitation, and it is understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of various embodiments can be combined to form additional embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being preferred over other embodiments or prior art implementations in terms of one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics may be compromised to achieve the desired overall system properties depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. As such, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations in terms of one or more features, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method for detecting danger, comprising: Receiving a stereoscopic image having a red channel, a blue channel, and a green channel, wherein the red channel, the blue channel, and the green channel produce a 3-channel RGB image; and receiving a depth image; The depth image is concatenated to the red, green, and blue channels of the 3-channel RGB image to produce a 4-channel RGBD image, and the 4-channel RGBD image is provided as a single input to the same layer of a neural network, wherein the neural network comprises: a first plurality of layers, wherein each layer of the first plurality of layers comprises: Convolutional layers; Batch standard layer; leaky ReLU activation function; and Max pooling layer; a second plurality of layers, wherein each layer of the second plurality of layers comprises: Convolutional layers; Batch Standard Layer; and a leaky ReLU activation function, wherein the second plurality of layers does not include a max pooling layer; a third plurality of layers, wherein each layer of the third plurality of layers comprises: Convolutional layers; Batch standard layer; leaky ReLU activation function; and You Only Look Once ("YOLO") layer; and one or more skip architectures; determining one or more hazards in the stereoscopic image using the neural network; concatenating an output of the neural network with an output of an auxiliary semantic segmentation decoder to force the neural network to learn one or more features of the drivable space to determine the one or more hazards in the stereo image; The vehicle is driven through the drivable space while avoiding the one or more hazards.

2. The method of claim 1, wherein the one or more skip architectures comprise an output from a layer in the first plurality of layers, the output providing an input to a layer comprising an upsampling layer.

3. The method of claim 1, wherein the one or more skip architectures include an output from a layer in the second plurality of layers, the output providing an input to a layer including an upsampling layer.

4. A system for detecting danger, comprising: one or more processors; as well as One or more non-transitory computer-readable media having computer-executable instructions stored therein, which when executed cause the system to perform the following operations: a receiving stereo camera operable to receive a stereo image having a red channel, a blue channel, and a green channel, wherein the red channel, the blue channel, and the green channel produce a 3-channel RGB image; and receiving a depth image; The depth image is concatenated to the red, green, and blue channels of the 3-channel RGB image to produce a 4-channel RGBD image, and the 4-channel RGBD image is provided as a single input to the same layer of a neural network, wherein the neural network comprises: a first plurality of layers, wherein each layer of the first plurality of layers comprises: Convolutional layers; Batch standard layer; leaky ReLU activation function; and Max pooling layer; a second plurality of layers, wherein each layer of the second plurality of layers comprises: Convolutional layers; Batch standard layer; and a leaky ReLU activation function, wherein the second plurality of layers does not include a max pooling layer; a third plurality of layers, wherein each layer of the third plurality of layers comprises: Convolutional layers; Batch standard layer; leaky ReLU activation function; and You Only Look Once ("YOLO") layer; and one or more skip architectures; determining one or more hazards in the stereoscopic image using the neural network; concatenating an output of the neural network with an output of an auxiliary semantic segmentation decoder to force the neural network to learn one or more features of the drivable space to determine the one or more hazards in the stereo image; The vehicle is driven through the drivable space while avoiding the one or more hazards.

5. The system of claim 4, wherein the one or more skip architectures include an output from a layer in the first plurality of layers, the output providing an input to a layer including an upsampling layer.

6. The system of claim 4, wherein the one or more skip architectures include an output from a layer in the second plurality of layers, the output providing an input to a layer including an upsampling layer.

7. A non-transitory computer readable medium operable to employ a neural network, the non-transitory computer readable medium having computer readable instructions stored thereon, the computer readable instructions operable to be performed to perform the following functions: receiving a stereoscopic image having a red channel, a blue channel, and a green channel, wherein the red channel, the blue channel, and the green channel produce a 3-channel RGB image; and receiving a depth image; The depth image is concatenated to the red, green, and blue channels of the 3-channel RGB image to produce a 4-channel RGBD image, and the 4-channel RGBD image is provided as a single input to the same layer of a neural network, wherein the neural network comprises: a first plurality of layers, wherein each layer of the first plurality of layers comprises: Convolutional layers; Batch standard layer; leaky ReLU activation function; and Max pooling layer; a second plurality of layers, wherein each layer of the second plurality of layers comprises: Convolutional layers; Batch standard layer; and a leaky ReLU activation function, wherein the second plurality of layers does not include a max pooling layer; a third plurality of layers, wherein each layer of the third plurality of layers comprises: Convolutional layers; Batch standard layer; leaky ReLU activation function; and You Only Look Once ("YOLO") layer; and one or more skip architectures; determining one or more hazards in the stereoscopic image using the neural network; concatenating an output of the neural network with an output of an auxiliary semantic segmentation decoder to force the neural network to learn one or more features of the drivable space to determine the one or more hazards in the stereo image; The vehicle is driven through the drivable space while avoiding the one or more hazards.

8. The non-transitory computer-readable medium of claim 7, wherein the one or more skip architectures include an output from a layer in the first plurality of layers, the output providing an input to a layer including an upsampling layer.

9. The non-transitory computer-readable medium of claim 7, wherein the one or more skip architectures include an output from a layer in the second plurality of layers, the output providing an input to a layer including an upsampling layer.

10. The non-transitory computer-readable medium of claim 7, wherein the neural network comprises a first RGB encoder network and a second deep encoder network, the first RGB encoder network having a plurality of fusion layers, and output values ​​from a plurality of encoder network layers are added to the plurality of fusion layers.

11. A computer program product comprising computer instructions, characterized in that The computer instructions implement the steps of the method according to any one of claims 1 to 3 when executed by a processor.

Images