Adaptive lane, drivable region, and terrain detection and recognition for vehicular control

Abstract

This disclosure relates to automatic lane and and / or terrain detection and recognition under various ambient and driving conditions for robotic / vehicular control purposes / applications. In one aspect, a rule-based detection system is configured to perform lane recognition using optical images. Various edge detection thresholds are adaptively tuned from frame to frame using a feedback signal derived from a fuzzy logic inference mechanism. Such detection techniques may also be adapted for drivable region detection, which may be fused together with drivable region detection based on LiDAR cloud point. Such detection techniques may be further adapted for a preprocessing of optical images of road to generate preprocessed images that may then be processed by learning-based lane detection models. In another aspect, an intelligent terrain classification system may be configured to include a fusion of both a vision-based terrain classification convolutional neural network (CNN) and a parallel proprioception-based terrain classification CNN. These CNNs may be configured to process a small number of selected set of proprioceptive signals and to adaptively detect / classify the terrain as platform-independent as possible.

Description

ADAPTIVE LANE, DRIVABLE REGION, AND TERRAIN DETECTION AND RECOGNITION FOR VEHICULAR CONTROLINCORPORATION BY REFERENCE

[0001] This application is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 530,008 filed on July 31 , 2023 and entitled "Adaptive Lane and Terrain Detection and Recognition for Vehicular Control," which is herein incorporated by reference in its entirety.BACKGROUNDTechnical Field

[0002] This disclosure relates to automatic lane and and / or terrain detection and recognition under various ambient and driving conditions for robotic / vehicular control purposes / applications.Background Technologies

[0003] Accurate machine recognition of lanes and / or terrains helps improve robotic / vehicular safety and provide automatic driver assistance, and is particularly critical for autonomous robotic / vehicular control and / or navigation in various operational contexts, including on-road and off-road environments. Accuracy of lane and / or terrain recognition based on machine vision and / or sensing may be greatly affected by ambient conditions that vary depending on lighting conditions, weather conditions, and visibility conditions. It may further be desirable for a system for lane and / or terrain recognition to be modularized and configured as vehicle / robot independent and agnostic as possible.BRIEF SUMMARY

[0004] This disclosure relates to an automatic lane and / or terrain detection and recognition under various ambient and driving conditions for robotic / vehicular control purposes / applications. In one aspect, a rule-based detection system is configured to perform lane recognition by edge detection in a sequence of optical image frames.The various edge detection thresholds are adaptively tuned from frame to frame using a feedback signal derived from a fuzzy logic inference mechanism.

[0005] In another aspect, LiDAR (Light Detection And Ranging) data may be processed for detection of road edge and drivable regions and for detection of obstacles. Such detection may provide recognition of drivable regions particular in conditions whether roads are covered by snow and in other conditions. In some example implementations, an image / vision-based road and drivable area recognition and the LiDAR-based detection approaches may be fused using, for example, a pixel-wise fuzzy inference system to provide a more accurate road and drivable region detection.

[0006] In another aspect, the components in the image-based detection approach may be used to form a preprocessing system that generate multi-channel preprocessed images. Such preprocessed images may then be further processed by pre-trained learning-based modes, such as convolutional neural network (CNN) models for lane detection. Such hybrid lane detection system may be more accurate in detection lane in various weather conditions.

[0007] In yet another aspect, an intelligent terrain classification system is configured to include a fusion of both a vision-based terrain classification convolutional neural network (CNN) and a parallel proprioception-based terrain classification CNN. These CNNs are separately trained and are configured to focus on extracting distinct features that are more prominent in different ambient conditions for more accurate terrain detection and classification. These CNNs may be configured to process a small number of selected sets of proprioceptive signals and to adaptively detect / classify the terrain as platform-independent as possible.

[0008] In an example implementation, a method for automatic lane detection for robotic or vehicular control is disclosed. The method may include receiving a current-frame RGB image to generate a pro-processed current-fame image; automatically pre-processing the current-frame RGB image to generate a preprocessed current-fame image for lane contrast enhancement; automatically detecting a plurality of edges in the preprocessed current frame image; automatically determining a region of interest (ROI) within the preprocessed current-frame image;automatically detecting a set of candidate lines within the ROI of the preprocessed current-frame image based on the plurality of edges; automatically determining at least two lane-boundary lines from the set of candidate lines; generating a fuzzy logic inference signal from at least one property of the set of candidate lines using a fuzzy logic rule-set; and feeding forward the fuzzy logic inference signal for an automatic edge detection of a next-frame imageBRIEF DESCRIPTION OF THE DRAWINGS

[0009] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0010] Fig. 1 illustrates a general example lane detection algorithm.

[0011] Fig. 2 illustrates an example lane detection algorithm for a sequence of image frames incorporating feedback to edge detection based on fuzzy logic inference.

[0012] Fig. 3 illustrates an example RGB image taken from a vehicle with dashboard reflections.

[0013] Fig. 4 illustrates an example manner for selecting a region of interest (ROI).

[0014] Fig. 5 illustrates a grayscale image showing lane lines.

[0015] Fig. 6 illustrates an example effect of color space transformation to enhance contrast for the yellow lane line.

[0016] Fig. 7 illustrates an example effect of noise reduction filtering.

[0017] Fig. 8 shows input membership functions for an example fuzzy logic inference rule-base for generating frame-to-frame feedback to adjust edge detection parameters of Fig. 2.

[0018] Fig. 9 shows an output membership function for the example fuzzy logic inference rule-base for generating frame-to-frame feedback to adjust edge detection parameters of Fig. 2.

[0019] Fig. 10 illustrates an example of general triangular ROI.

[0020] Fig. 11 illustrates an example manner for determining the apex of a triangular ROL

[0021] Fig. 12 illustrates an example of the Hough Transform where many possible straight line in an image is translated into the Hough space with the parameters p and 9, representing a distance from the origin to the line and a corresponding angle, respectively.

[0022] Fig. 13 illustrates an example of the detected lane lines.

[0023] Figs. 14-17 illustrates lane line detection under a variety of distinct time and weather conditions.

[0024] Fig. 18 shows an example of a general terrain classifier by fusing an exteroceptive module and a proprioceptive module.

[0025] Fig. 19 illustrates a particular terrain classifier by fusing an exteroceptive module and a proprioceptive module according to Fig. 18.

[0026] Fig. 20 shows a schematic diagram of an example system setup for terrain data collection.

[0027] Fig. 21 shows example sample images of the eight outdoor terrain classes.

[0028] Fig. 22 shows gravel sample images in the visible spectral range under several lighting conditions in grayscale or full color.

[0029] Fig. 23 shows gravel sample images in Near InfraRed (NIR) spectral range under several lighting conditions.

[0030] Fig. 24 shows confusion matrices from a stereo camera in the visible spectral range for characterizing the effectiveness of the terrain detection algorithm of Fig. 2 and its various components.

[0031] Fig. 25 shows confusion matrices from a stereo camera in the NIR spectral range for characterizing the effectiveness of the terrain detection algorithm of Fig. 2 and its various components.

[0032] Fig. 26 shows an example system for fusing drivable region detection based on optical images and LiDAR cloud point using a pixel-wise fuzzy inference system.

[0033] Fig. 27 illustrates an example preprocessing of images for a learningbased model to perform lane detection.

[0034] Fig. 28 illustrates example input member functions for a fuzzy adaptive parameter tuner of FIG. 27.

[0035] Fig. 29 illustrates example output member functions for a fuzzy adaptive parameter tuner of FIG. 27.

[0036] Fig. 30 illustrates example results of a channel allocation process.DETAILED DESCRIPTION

[0037] Various aspects for intelligent visual image or proprioception data processing for lane and / or terrain detection / recognition / classification in robotic / vehicular applications will now be described in detail hereinafter with reference to the accompanied drawings, which form a part of the present disclosure, and which show, by way of illustration, various example implementations and embodiments. The optical / photonic devices and systems may, however, be embodied in a variety of different forms and, therefore, the disclosure herein is intended to be construed as not being limited to the embodiments set forth. Further, the disclosure may be embodied as methods, components, and / or platforms in addition to the disclosed devices and systems. Accordingly, embodiments of the disclosure may, for example, take the form of hardware, software, firmware or any combination thereof.

[0038] In general, terminology may be understood at least in part from usage in its context. For example, terms, such as "and", "or", or "and / or," as used herein may include a variety of meanings that may depend at least in part upon the context in which such terms are used. Typically, the term "or", if used to associate a list, such as A, B or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B or C, here used in the exclusive sense. In addition, the term "one ormore" or "at least one" as used herein, depending at least in part upon context, may be used to describe any feature, structure, or characteristic in a singular sense or may be used to describe combinations of features, structures or characteristics in a plural sense. Similarly, terms, such as "a", "an", or "the", again, may be understood to convey a singular usage or to convey a plural usage, depending at least in part upon context. In addition, the term "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of factors and may, instead, allow for the existence of additional factors not necessarily expressly described, again, depending at least in part on context.

[0039] Many other modifications of the implementations above may be made to adapt a particular situation or material to the teachings without departing from the scope of the current disclosure. Therefore, it is intended that the present methods and systems not be limited to the particular embodiments disclosed, but that the disclosed methods and systems include all embodiments falling within the scope of the appended claims.General Lane and / or Terrain Detection / Recognition / Classification

[0040] Accurate machine detection / recognition / classification of lanes and / or terrains helps improve vehicular safety and provide automatic driver assistance, and is particularly critical for autonomous vehicular control and / or navigation in various operational contexts, including on-road and off-road environments.

[0041] For on-road navigational applications, the detection of lanes may be based on the recognition of on-road lane marks and / or lines. Accurate lane detection may help facilitate the generation of driver warning signals, autonomous control of the vehicle to stay in or change lanes, and the like. For off-road vehicular applications, automatic detection / recognition / classification of terrain may likewise be used to generate appropriate control signals for vehicular speed, direction, and the like. In some situations, the detection / recognition / classification of terrain may be used to adaptively control other vehicular properties. For example, vehicular operation in different terrains (e.g., a sandy terrain vs. a solid dirt terrain) may benefit from distinct different levels of tire pressure, which may be controlled depending on automatic terrain detection.

[0042] Such lane and / or terrain detection / recognition / classification may be obtained automatically by a computing machine based on input data collected from various sensors configured to record the surroundings of the vehicle and / or detect the operational states of the vehicle. These input data may be visual (e.g., images of surroundings at various spectral ranges), proximity, or may be proprioceptive (e.g., speed, acceleration, vibrational data, and the like).

[0043] Collection of at least some of such input data, including but not limited to the visual data and proximity data in particular, may be affected by various ambient conditions, such as lighting conditions, time of the day, season of the year, weather-related conditions (e.g., snow, ice, or rain on the ground), visibility, and the like. In addition, some input data, the proprioceptive data in particular, may be vehicle dependent. As such, it may be difficult but desirable to generate a model for lane and / or terrain detection / recognition / classification that can provide accurate predictions under all conditions and is independent / agnostic of vehicles as much as possible.Lane Detection - Overview

[0044] Accurate lane detection for on-the-road applications may be used for driver assistance or may be used to facilitate full autonomous driving. For example, lane detection is a particularly critical underlying component of driver assistance functions including but not limited to Lane Keep Assistance (LKA), Autonomous Emergency Braking (AEB), and Autonomous Cruise Control (ACC).

[0045] Several different example approaches / implementations may be employed for automatic lane detection. For example, in some example implementations, a lane map may be generated by collecting positions of the lane along a large number of roads and may be used in conjunction of GPS signals to determine instant lane information of the vehicle. Though the lane map data can be downsized / compressed by approximating the lanes with curves, a considerable amount of data collection is required before applying them to a practical system. Such an approach lacks robustness when the lanes of the roads are redesigned, or reconfigured, sometimes frequently.

[0046] In some other example implementations, apart from lane maps, proximity sensors may be used to detect lanes. For example, LIDAR sensors could be leveraged to construct a 3D map of a scene. Specifically, by differentiating reflective (or scattering) intensity from the road and lane lines, the position of the lane lines could be identified. For another example, camera sensors at various wavebands may be used in vision-based approaches. Because road lane lines are usually designed for human eyes, camera sensors in the visible light spectral range may be adopted for generating visual information as input to lane detection algorithms. In some other example implementations, a fusion of vision and proximity sensing (e.g., LIDAR) may be employed to further improve the detection accuracy. However, complex weather conditions can greatly influence the lane detection accuracy based on input data from LIDARs and cameras. For example, intensity of light during day and night conditions widely varies, usually requiring different detection parameters or even different detection algorithms. For another example, during rainy weather with raindrops on the windshield, reduced visibility and fastmoving wipers may pose additional challenges for LIDAR and camera-based algorithms. Likewise, water accumulation on the ground changes the reflected intensity of LIDAR rays and thus affects the accuracy of lane detection based on LIDAR proximity sensing. For another example, in winter, lane detection is also greatly affected by snow covering the road and lines / marks on the road.

[0047] In at least one of the following example implementations, a detection algorithm that is adaptable to various weather conditions is disclosed. Such an example algorithm may contain module / circuitry configured to perform contrast adjustment, noise reduction, ROI selection, and line detection from a sequence of input image frames. The line detection information generated from such an algorithm may then be applied to either an artificial-intelligence (Al) or traditional lane detection frameworks for the recognition of road lanes.Lane Detection Using Images - Approaches and Components

[0048] Lane detection algorithms may be generally categorized into geometrical and learning-based approaches. Geometrical approaches may leverage intrinsic characteristics of the road lane lines, such as parallelism and color. On the other hand, learning-based algorithms rely on data collected under different roadconditions and weather. For example, geometrical approaches may achieve a better detection rate than learning-based approaches in some situations (with good weather, clear road marks, and the like). However, geometrical approaches lack consistency under different weather conditions. By improving the adaptability of geometrical approaches under challenging weather, a higher detection rate and detection consistency can be achieved simultaneously.

[0049] Example underlying principles for geometrical approaches using image data are first described below in relation to Fig. 1 . As shown in Fig. 1 , geometrical approaches generally include the following example components: a contrast adjustment module / circuitry, a Region of Interest (ROI) selection module / circuitry, and a line detection module / circuitry.

[0050] The contrast adjustment module / circuitry may be configured to process typical input images taken from an RGB camera. Since lane lines are usually bright in color, compared to, for example, the background asphalt road, the RGB images may be translated into grey-level (brightness) representations. A typical translation formula considering the luminance may be based on the relative luminosity of RGB components of the image to human eyes:G ylevel = 0.3 x J -j- 0.59 x G 4- 0.11 x (1) where R, G and B stand for the intensity (or luminosity) in red, green and blue channels of the image, respectively.

[0051] It may be more difficult to detect yellow lane lines than the typical white lines in normal RGB color space. As such, in some example implementations, the YUV tri-color-space representation may be leveraged to increase the detection rate of yellow and blue lines. An example transformation from RGB color space to YUV color space may be written as

[0052] The YUV color space above may be capable of enhancing the detection rate of the yellow and blue lane lines in typical daylight but may provide poorer performance than RGB space in nighttime. As such, the YUV color space may needto be switched off under certain conditions, such as during nighttime operation. In some implementations, a more robust algorithm may automatically enable / disable the YUV channel (or switch between the YUV and RGB color spaces) under the proper situation for increasing the overall lane detection rate.

[0053] The ROI selection module / circuitry may be configured to deploy an ROI selection strategy to increase the performance of lane detection. An ROI selection would help reduce the amount of imaging data to be processed and reduce overall noise in the processed data. For example, the top (sky) and the bottom (hood) of the image may be cropped to form rectangular ROIs.

[0054] Typically, lane lines are parallel to one another. According to normal optical property of cameras, they coincide at a vanishing point near the center of an optical image. As such, triangular rather than rectangular ROIs may be used in some example lane detection algorithms. The advantage of triangular ROIs is that the information from adjacent lanes can be filtered out, thereby increasing the accuracy of the detection of the center lane.

[0055] However, the vanishing point location may change with the angle of the camera and when a curved road is encountered, a part of the lanes might not fit within such triangular ROI. In some example implementations, a unique A-shaped ROI may be adopted. For example, with one vanishing point at the center and four adaptively calculated points at the bottom of an image, an A-shaped ROI may be fully defined with two triangles with a shared apex at the vanishing point. In adopting this example A-shaped ROI, the system may be able to further avoid the distraction of lane marks such as arrows and speed limits painted on the ground between lane lines. However, more calculations may be required for all anchor points in such ROI. Additionally, the smaller ROI selection may limit the tolerance of detection errors. When such an ROI did not include the lane lines, it may significantly increase the possibility of a detection failure.

[0056] As such, with a variety of ROI selection strategies, it is important to design one that can adapt to the road and the field of view of the cameras.

[0057] The line detection module / circuitry may be the most critical part of the lane detection algorithms of Fig. 1 . The line detection process may be categorizedinto two parts - feature detection and feature connection. The lane lines are usually composed of elements with a similar color or gradient value at edges. However, lane lines can be painted in various colors and their appearances may also vary with illumination and shadows. Even with a gradient-calculating edge detection method, the edge detection threshold value that differentiates edges / non-edges may need to be adaptively tuned and determined. In some example implementations, such tuning may be rule-based. For example, one or more equations / formula may be used to adjust the edge gradient threshold value with input parameters depending on weather, lighting, and general visibility conditions. However, such equations / formula may generally have difficulty functioning properly at nighttime.

[0058] With properly detected edges in an image, multiple example feature connection approaches may be used to connect the edges to define the lane lines. For example, the EDLines algorithm may be used to connect the edges directly with the closest points to form the lines. For another example, a Hough Transform and its similar variations may be used to transform the edges into Hough space and to identify edges in the Hough space that could be connected into straight lines.Similar to edge detection algorithms, threshold(s) for differentiating lines / non-lines remains parameters that may need to be adaptively adjusted based on varying weather and illumination conditions in such connection algorithms.Adaptive Lane Detection Using Images and Based on Fuzzy-Logic - Examule Implementations

[0059] The example implementations below are configured to enable adaptive precise lane detection with a vision-only approach. It processes input images in a sequence of, for example, image frames. The lane detection result is calculated by a series of modules / circuitry, including a contrast adjustment modules / circuitry, a noise reduction modules / circuitry, an ROI selection modules / circuitry, and a line detection modules / circuitry. A tracking function is further included to enhance the stability of the algorithm when lane detection in a particular image frame fails. The data and logic flow of this example algorithm is shown in Fig. 2.

[0060] As shown in Fig. 2, an input image frame is first processed by the contrast adjustment module / circuitry 202 of Fig. 2. After the contrast adjustment, theimage undergoes a noise reduction processing by the noise reduction module / circuitry 204. Following a Canny edge detection function of the edge detection module / circuitry 206, a proper ROI may then be selected from the image by the ROI selection module / circuitry 208. Line detection is then performed within the ROI by the line detection model / circuitry 210. The detected number of lines 212 may then be used in adapting the edge detection and contrast adjustment function for a next frame of image, respectively, using for example, fuzzy logic inference. The line detection module / circuitry 210 may further determine whether a yellow line exists (based on color detection) and use such determination to decide whether to switch to YUV color space in the contrast adjustment module / circuitry 202 for the next image frame. Generally, in the case where no line is detected, a tracking function 214 may be enabled. When a valid line result is present, the positions of the lines may be used to adaptively modify the ROI for a better result for the next image frame.

[0061] In further detail, the images in an example DSDLDE dataset may be processed by the example algorithm of Fig. 2 in sequence. The example dataset contains 50 video clips collected under different weather conditions. The distribution of the scenes is shown in Table I. The resolution of the videos is 1080 x 1920 with 3 color channels (Red, Green, and Blue). The videos were collected in the United States and South Korea. The length of each of the clips is around 1 minute.TABLE I: The distribution of the weather conditions in the DSDLDE dataset with six different scenarios.

[0062] The dataset is challenging for proper lane detection because of the huge variance in the weather and road conditions. For example, raining conditions that range from drizzling to pouring are all available in the dataset. Various scenes such as tunnels, highways, and cities are also included.

[0063] Although the DSDLDE dataset provides an abundance of data, one drawback is the inconsistency of its camera setup. For example, for this dataset, the camera is installed inside the car without a fixed view angle among different videos. The hood of the car takes up to 1 / 3 of the image in some of the clips, while it takes less than 1 / 10 in the others. Additionally, in some of the videos, the camera was installed far from the windshield. In these clips, the screen of the navigation system, as well as its reflection on the windshield, were visible in the frames, as shown in Fig. 3. To maintain a consistency of the input images, a clip from the DSDLDE data set was discarded if the hood of the car in the images takes 1 / 3 or more of the images. In addition, videos with prominent unnecessary information, such as the internal reflection of the car, were also excluded.

[0064] The experiments using the various modules / circuitry of Fig. 2 are performed independently between different videos. The frames in the videos are processed in sequence by the algorithm, simulating the scenario of a computer processing the images captured by an onboard camera. The processing of the image sequence following the algorithm of Fig. 2 is performed with a PC having an Intel core i7, 3.0 GHz CPU with 16 GB RAM.

[0065] Contrast adjustment

[0066] The contrast adjustment module / circuitry 202 may be responsible for transforming the image from RGB to YUV or Grayscale color space. The decision may be based on comparing the current image frame and the detection results of one or more previous image frames. With a default setting, the contrast adjustment module / circuitry may transform the images from RGB to grayscale using Equation (1). When yellow lines are detected in the images (e.g., in a previous frame, the transformation may be switched from RGB to YUV color space in the next image frame.

[0067] The difference between the U channel and the V channel in the YUV color space being below a threshold value may be indicative of yellow element. In some example implementations, a threshold of such U V difference may be predefined. For example, the threshold of a yellow pixel for U-V may be defined asU - V < -15 (3)

[0068] For indicating yellow element. To identify a yellow line, a horizontal line may be plotted at a predefined portion (e.g., %) of the image height, centering at a validly detected lane in the previous image frame in the horizontal direction. A predefined number (e.g., 11 ) of points may be sampled along the horizontal line. An example is shown in Fig. 4. If any of the points in a point group had met the requirement of Equation 3, this point group is then regarded as a portion of a yellow line. If a yellow line is detected on the left / right-hand side, the corresponding side of the current image frame may be transformed into the YUV color space. This approach may successfully increase the contrast between the yellow lane lines and an asphalt ground, as shown in Figs. 5 and 6, where Fig. 5 and the right side of Fig. 6 show grayscale color-space and the left side of Fig. 6 shows YUV color space. However, the performance of the YUV color space may deteriorate under illumination of yellow light sources because of the potential large yellow background in the image. As such, in some example implementations, the YUV transforming module within the contract adjustment module / circuitry 202 may be switched off when more than, for example, 15%, of the whole image is determined as yellow pixels under Equation (3).

[0069] The approach above thus may facilitate successfully detection of yellow lines, switching on the YUV color space representation, and identifying the lane lines better than directly using grayscale images. By processing the lane lines on the left and right-hand sides independently, the image can be processed correctly even when the color of the lines differed from each side. An example of a single sided color space transformation is shown in Fig. 6. In some example implementations, the colors of the lane lines may be checked every 30 frames in order to maintain the robustness of the contrast adjustment module / circuitry 204.

[0070] Noise reduction in Image-Based Lane Detection

[0071] As an example, the noise reduction technique used in the algorithm of Fig. 2 may include a Bilateral filter. Such a filter may be capable of eliminating various noises in the image while preserving the edges.

[0072] The Bilateral filter may be based on a convolution of a kernel with the image. The example kernel may be represented aswhere W represents a normalization factor of the kernel, the first exponential represents a Gaussian smoothing function of space, and the second exponential represents a Gaussian smoothing function of intensity. In Equation (4) above, x / represents elements in the kernel while x represents an element in the image.

[0073] In some example implementations, a kernel size of 15 x 15 may be used, with an example variance of 50 in both space and intensity being adopted. Example images before and after applying the filtering process are shown in Fig. 7. The top panel shows the original image. The bottom left and bottom right panels show unfiltered and filtered portions of the image highlighted in the top panel, respectively. The bottom right panel of Fig. 7 shows that, via the convolution-based filtering process above, the noise in the image are suppressed while the edges of the lane lines remain clear.

[0074] Fuzzy-adaptive edge detection in Images

[0075] In the example algorithm of Fig. 2, a Canny edge detection algorithm may be used to identify the points in the image with larger gradient values. Canny edge detection may be provided with two independent threshold values. With such two threshold values, Canny edge detection may outperform other algorithms with its higher degree of flexibility. For example, when the gradient value detected in a pixel of an image is larger than the higher threshold value, this pixel may be regarded as an edge, whereas when it is smaller than the lower threshold, the pixel is regarded as a non-edge point. When the gradient value of the pixel is between the two thresholds, it may be identified as an edge only if its neighbors include one or more other edge points.

[0076] The flexibility provided by the two thresholds above may contribute to the algorithm’s adaptability over varying weather conditions, particularly when combined with a fuzzy inference system configured to perform fuzzy logic-based tuning for the threshold values. Fuzzy inference systems are also referred to as expert systems because they integrate the decision-making processes of expertswith linguistic explainability. Their ability to handle uncertainty in the algorithm of Fig.2 may provide higher flexibility and better tolerance against noises in the images.

[0077] In some example implementations, the adjustment of the Canny thresholds may be performed recursively. The thresholds to be adjusted may depend on the number of detected lines in a previous frame (as determined in a previous image frame and fed back from the line detection module / circuitry 210). For example, when the detected line number is higher than expected in the previous frame, the Canny thresholds would be tuned higher, discarding edge pixels with lower gradient values. On the other hand, when the edges of the lines are less obvious, only a smaller number of pixels could exceed the threshold value for edge detection, leaving fewer lines to be detected. In this case, the threshold values would be tuned lower to allow more pixels to be considered as edges.

[0078] The tuning process of the “high threshold” among the two threshold values of the Canny edge detection may thus be performed using a fuzzy inference system (FIS). The detected line number may be categorized into five example different categories of conditions. Example linguistic variable membership functions that map the input into the five example categories are shown in Fig. 8. According to the detected line number, an example FIS rule base may be adopted as defined in Table II. Finally, the decision made according to the rule base may be defuzzified according to example output membership function shown in Fig. 9.

[0079] The fuzzy rule set of Table II may be based on differential adjustment of the previous threshold value. “Zero” in Table II, for example, may represent no adjustment to the threshold being tuned. After tuning the “high threshold”, the “low threshold” may be correspondingly calculated by dividing the high threshold by a predetermined factor, e.g., a factor of 3.TABLE II: The rule base of the fuzzy edge detection module

[0080] Fuzzy logic inference may also be used to adaptively adjust the parameters in the contrast adjustment module / circuitry 202 for the next image frame. For example, the U-V differential threshold of Equation (3) for yellow line detection may be adjusted using fussy logic rules constructed in a similar manner of Table II.

[0081] ROI selection in Images

[0082] As described above, ROI (Region of Interest) selection may be used as an effective approach to exclude non-lane objects from the images. Though the A- shaped ROI described above could exclude many non-lane objects, it includes limited area and could lead to failure in detecting lane lines. As such, a triangular rather than A-shaped ROI may be adopted for the example algorithm of Fig. 2. An example ROI is shown in Fig. 10.

[0083] The triangular ROI of Fig. 10 may be based on a bottom of the image, with its tip located near the extended intercept of the two detected lines in the previous frame. Ideally, the camera may be horizontally installed relative to the road. The vanishing point of a road is likely to be located at the center of the image. However, in practice, the tip of the ROI may need to be adjusted adaptively according to the changing road directions. The location of the vanishing point in the image may be represented by (XROI, YROI), as shown in Fig. 10.

[0084] The ROI selection may first involve a calculation of the XROI. The default position of XROI may be set at the center of the image. When the vehicle is going through a turn in the road, one of the lines of the road might be cropped from the ROI. As a result, the XROI may be adjusted by an example amount of 5% towards the left (or right) for a current image frame when the lane line on the lefthand (or right-hand) side disappears from the previous image frame, and vice versa. This example setting for the adjustment may be effective in reducing failure frames when the vehicle is turning. This adjustment percentage amount may depend on theparameters of the camera. It may be tuned again when a different camera is installed on the vehicle.

[0085] The ROI selection may then involve a calculation of the YROI. Though a typical vanishing point is at the center of the image, setting the YROI at 1 / 2 of the image height might crop part of the lane when either the road or the camera is not horizontal. Therefore, an example default position of the y-axis intercept may be set at 2 / 3 of the image height rather than the ! > height to ensure or increase a probability that that complete lane lines are included in the ROI.

[0086] However, larger or more error tolerant ROIs may include more unwanted backgrounds or objects. In some example implementations, an additional mechanism may be used to further cut down on the size of ROIs. For example, when a number of frames with successfully detected lines are present, e.g. both the lane lines on the left and right-hand side are successfully detected in a current frame, the y-axis intercept YROI may be lowered to a predetermined percentage amount (e.g., 10%) higher than the average vanishing point height of a number of previous frames, e.g., a total of 1 .1 times the average vanishing point height, as shown in Fig. 1 1 .

[0087] The example ROI selection methodology above thus facilitates a derivation of (XROI, YROI) while effectively cutting out unwanted objects, leaving only the lane in view for the next step in line detection in Fig. 2.

[0088] Hough Transform

[0089] As described above, Hough Transform may be used for lane line detection in the example algorithm of Fig. 2. Hough Transform translates a straight line from the image space into Hough space using the equationwhere p represents the shortest distance from the origin to the straight line and 0 is the corresponding angle of the straight line, as shown in Fig. 12.

[0090] In some example implementations, a set of all straight lines that go through one edge pixel in the image space may be transformed into a sine wave in the Hough space. When multiple edge pixels are colinear, their corresponding sinewaves would coincide with one another at the same point in the Hough Transformed space (same line, thus same p and 0). By identifying the most accumulated points, the longest lines could be identified. The Hough Transform may thus return a series of 0 and p values in the order of accumulation values, representing detected lines. The longest lines (most highly repeated p and 0) may be considered as the lane lines.

[0091] Lane Line detection in Images

[0092] Since the ROI selection process limits the line detection region, the line detection could simply be done by the Hough Transform with consideration of some geometrical properties. The distance resolution of the Hough transform may be predetermined at, for example, 1 pixel, while the angular resolution may also be predetermined and set at, for example, 1 degree. The accumulator threshold of Hough Transform may be predetermined and set as, for example, 5, which means only line candidates that go through at least 5 edge points are considered. This value could be fixed since the threshold of Canny edge detection may have been made adaptive. For example, if only very few lines go through over 5 edge points, the threshold of Canny edge detection may decrease for the next image frame, allowing more edge points to be considered.

[0093] In some example implementations, various geometrical properties of the typical lane lines may also be used to filter out the lines that are not compatible with normal road conditions. For example, only the lines with their angles between, e.g., 25 and 65 degrees, or 110 and 155 degrees may be considered. In addition, only the lines that intercepted the bottom of the image on a reasonable side are considered. For example, when the detected line angle is between 25 and 65 degrees, it may be regarded as the line located on the left-hand side of the lane. In other words, for this case, only the lines that intercept with the bottom of the image at the left half of the image are considered valid candidates.

[0094] Using the geometrical criteria, the lane line candidates could be effectively identified. According to the Hough accumulator, the two highest accumulated candidates on each side may be chosen. Their Hough Transform parameters, p and 0, may be averaged and recorded.

[0095] In some example implementations, the averaged psand 0Smay be used to visualize the detection result. The detected lane lines may be accordingly drawn on the images. The intercept of both lane lines may be used to determine the ROI in the next frame, as described above. An example of the detection result visualization is shown in Fig. 13.

[0096] Tracking

[0097] A tracking function as shown in Fig. 2 may be essential for maintaining a stability of the example algorithm. When driving in the lane, the lane lines tend to stay in similar positions. As such, in some example implementation, when the algorithm fails to detect the left and / or right lane line in a current frame, the p and 0 from the previous frame may be adopted.

[0098] Results for Image-Based Lane Detection

[0099] A series of experiments are conducted on the DSDLDE dataset using the example algorithm of Fig. 2 with the specifics described above. Since the chosen dataset does not have labeled ground truth available, multiple different manners of validation of the resulting lane detection are used. For example, when both left / right detected results overlap with more than 2 / 3 of the lines on the image in a frame, that frame is considered as a successfully detected frame. The detection rate may be calculated for every video independently aswhere TP stands for true positive and FN stands for false negative.

[0100] Only frames with visible lines on both sides are considered effective detection cases. No false positive or true negative cases are included in the calculation. For instance, the cases where the vehicle is switching between the lanes are excluded from the validation experiment. The validation result shows that the algorithm specified above with respect to Fig. 2 successfully detect most of the frames in the dataset. Figs. 14 and 15 show several successfully detected frames.

[0101] However, some scenes still pose challenge to the algorithm. For example, in the frames where some objects are parallel and close to the lines, thealgorithm may tend to take the average of the lane line and the object as the detection result. Some example failed detection results are shown in Figs. 16 and 17. The detection rate is compared with methods described elsewhere. The detection rate results are also shown below.TABLE III: Detection rate comparison between the algorithm of Fig. 2 and several previous approaches (the unit of all numbers is percentage)time applications,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 12, pp. 4043-4048, 2018;Ahmed Method: S. Sultana and B. Ahmed, “Robust nighttime road lane line detection using bilateral filter and sage under challenging conditions,” in 2021 IEEE 13th International Conference on Computer Research and Development (ICCRD). IEEE, 2021 , pp. 137-143; “Lane detection and tracking under rainy weather challenges,” in 2021 IEEE Region 10 Symposium (TENSYMP). IEEE, 2021 , pp. 1-6.

[0102] In the Lee method, for example, different numbers of videos may be chosen from each weather condition to calculate the average. An identical approach is performed using the algorithm above to achieve a more effective comparison shown in Table III. Although a simpler triangular ROI design is adopted, the algorithm showed higher detection accuracy compared to the Lee method.

[0103] Table III also shows that the Ahmed method attempts to tackle the challenging weather but the parameters so developed only work well for rainy and night time clips, respectively.

[0104] From Table III, it is shown that the algorithm of Fig. 2 is effective on all challenging weather conditions with the same initial parameters. The adaptive algorithm reached the highest detection rate among all the compared previous studies done on the same dataset.

[0105] An analysis is also performed on the processing time of the algorithm above. A comparison is made between the current algorithm and the referenced methods / approaches. Both reference approaches reduce the resolution of the videos from the dataset. Their reduced resolution and processing time are shown in the comparison to the current algorithm in Table. IV.TABLE IV: The processing time of a single frame compared between the current method and several previous studies.

[0106] To compare the processing time under the same condition, the current algorithm may be applied to images that are down-scaled to the corresponding resolutions. When down-scaled to 350 x 640, the processing time is about 5.8 ms / frame. Therefore, comparing to the previous methods and approaches, the current algorithm leads not only in the detection rate, but also the processing efficiency.

[0107] For better comparison between the current algorithm and prior studies in Table III and Table IV, down-sampling of the images into lower resolutions (e.g., 540 x 690) is performed.

[0108] To summarize the example lane detection disclosed above, an algorithm with a fuzzy logic-based adaptive module is disclosed. The example algorithm has shown its ability in overcoming all challenging weather conditions withidentical initial parameters. Moreover, the algorithm is shown to exceed the previous studies in both detection rate and processing speed.

[0109] In some other implementations, the detected information above from the vehicle may be integrated with the visual detection system. By incorporating the read-out from the turn signal and, f o r exam p l e , wiper signal, false detections caused by lane switching and wipers can be further reduced. In addition, the disclosure above has proved that the fuzzy logic-based adaptive module could corroborate with geometrical approaches. In some example implementations, integration with learning-based algorithms may be further used. For example, the fuzzy logic membership functions may be learned and tuned as described in U.S. Provisional Patent Application No. 63 / 529,967 entitled “ HIERARCHICAL FUZZY CONTROLLER WITH MULTIPLE CONTROLOUTPUT filed by the same applicant on July 31 , 2024. The entirety of that patent application is herein incorporated by reference.Road-Edge Detection and Drivable Region Identification

[0110] The above example implementations focus on image-based lane detections. In some situations, image-based approaches may be impaired by weather conditions. For example, the road may be covered by snow. In such conditions, lane information in the image may be lacking or inadequate for recognition, but for safety purposes, it may nevertheless be desired to at least identify obstacles resent on the road and to map out the drivable or navigable areas.

[0111] In some example implementations, the image-based lane detection approaches above may be supplemented by some other detection mechanism configured for detection obstacles and for mapping the drivable areas, particular under the example weather conditions above. As a supplement, such detection mechanism may be based on input data other than the images. For example, the input data for such detection mechanisms may be taken from one or more LiDAR (Light Detection And Ranging) sensors. The LiDAR data may be processed to detect, for example, road edges, in order to determine drivable regions.

[0112] The data generated by the LiDAR may be provided in the form of a dense point cloud, each point in the point cloud may include a direction and adistance of a light reflection point in the detected 3D scene. The point cloud data may be provided to a drivable region detection module which output the drivable region in pixel-wise manner.

[0113] In addition, a parallel drivable area identification module adapting the image or vision-based detection mechanism above may also be employed. Such a module, like the lane detection system described above, may include the various components of Fig. 2 and its variations (particularly the noise reduction module, the Canny edge detectors, the fuzzy logic member function threshold tuning mechanisms, the Hough line detecting mechanism, and the like) in order to detect lane and / or the edge of the road (as opposed to only lane detection) and in order to derive a drivable area under various weather conditions and illumination conditions while maintaining reasonable processing time. To adapt the detection mechanism of Fig. 2 to detect road edge and drivable areas, adaptation may be provided with respect to, for example, the ROI determination mechanism. Given the prevalence of confounding features like tire tracks and watermarks in, for example, snowy or rainy conditions, ROI with some other shapes (such as A-shape or trapezoidal shape, other than triangular) may be deemed more suitable despite a slight increase in processing time

[0114] A multi-level fuzzy-logic-based fusion method may be further deployed to fuse the detection results of the vision / image-based system and the LiDAR-based system. Such fusion may be performed by a pixel-wise Fuzzy logic Inference System (FIS). At a first level, the LiDAR-based detection system and the image / vision-based detection system may both include parameter tuners that tune the threshold values of image / point cloud processing functions. At a second level, the FIS may be configured to controls the sensor fusion process.

[0115] An example is illustrated in Fig. 26. Fig. 26 illustrates two processing branches, including an upper branch for the image / vision-based processing and a lower branch for the point cloud-based processing. The image / vision-based processing module 2602, for example and as described above, may be based on the underlying principles described in relation to Fig. 2 with some of the processing components adapted for road edge detection and drivable region determination (as described above). The output of the image / vision-based processing module may bea first representation or mapping of detected drivable regions 2604 according to the input images.

[0116] A further shown in Fig. 26, the LiDAR processing module 2606 may include, for example, a normalization component for normalizing the dense point cloud generated from the LiDAR sensors. The normalization process may help mitigate variations in point density resulting from differences in distance. The road edge detection may be based on identifying the left / right most points in the dense point cloud. For example, by tallying the density of point clouds intersecting each grid within a grid map, the leftmost and rightmost points on the road can be pinpointed. The generation of the road edges may be further based on a set of interpolation processes. The output from the LiDAR processing module 2606 may be a second representation or mapping of detected drivable regions 2608 according to the input LiDAR point cloud data.

[0117] As further shown in Fig. 26, the first representation or mapping of detected drivable regions 2604 according to the input images and the separately generated second representation or mapping of detected drivable regions 2608 according to the input LiDAR point cloud may be fused by the FIS module 2610 to generate a fused output for the drivable regions. Rather than applying probabilitybased methods (such as Bayesian methods or conditional random field methods), an example approach based on pixel-wise FIS may be deployed to generate the fusion result, given the probability nature of FIS. Based on the detection result of each pixel, the designed rule base for the fuzzy logic in the FIS will determine the probability of the pixel being road / non-road.Preprocessing for CNN-Based Lane Detection

[0118] In order to better handle the challenging weather conditions above, in some other example implementations, lane detection above may be performed using a learning approach rather than the geometric or rule-based approaches above.

[0119] Specifically, ensuring the robustness of lane detection systems is essential for the reliability of autonomous vehicles, particularly in the presence of diverse weather conditions. Addressing challenges posed by varying weather remains an ongoing issue. Geometric-based lane detection methods rooted in theinherent properties of road geometry, provide enhanced generalizability. However, these methods often require manual parameter tuning to accommodate it fluctuating illumination and weather conditions. Conversely, learning-based approaches, trained on pre-labeled datasets, excel in localizing intricate and curved lane configurations but grapple with the absence of diverse weather datasets

[0120] The further example implementations below provide a hybrid approach that merges the strengths of both geometry-based and learning-based methodologies. An adaptive preprocessing method described, which utilizes a fuzzy inference system (FIS), to dynamically adjust parameters in geometric-based image processing functions and enhances adaptability to diverse weather conditions. This preprocessing algorithm may be designed to seamlessly integrate with various types of learning-based lane detection models including but not limited to CNN models. When implemented in conjunction with CNN-based models, the hybrid approach provides generalizability across weather conditions and adaptability to complex lane configurations.

[0121] The example implementations below provide preprocessing framework in order to integrate with most learning-based lane detection models. Such framework process may process input color images and output images with the same sizes and number of channels.

[0122] An example overall structure of such preprocessing framework or algorithm is illustrated as 2700 in Fig. 27. The input images may be processed by a noise reduction module 2702, an edge detection module 2704, an ROI selection module 2706, and a Line detection module 2708 sequentially. The threshold parameter in the edge detection module 2704 may be adjusted by a fuzzy parameter tuner in each iteration. That is, the measurement acquired in frame A / may be processed by the parameter tuner which determined the parameter used in frame N + 1 . The functionality and the details of each module are described in further detail below. At least some of the underlying principles above for the lane detection implementations applies.

[0123] Noise Reduction

[0124] To simplify the preprocessing, the input images may be first transformed to grayscale when imported into the preprocessing module 2700. For example, a bilateral filter may be used to suppress the noise in the images, enhancing the quality of the edge detectors. The bilateral filter may calculate weighted average of neighboring pixels with the weights determined by assigned spatial and color variance values. The mathematical expression of the bilateral filter may be expressed as:

[0125]

[0126] where x represents the center pixel and x / represents the neighboring pixels in the kernel, osand oi represent the pre-assigned variance value in space and intensity respectively, and W is the normalization parameter. A kernel size of 7 x 7, spatial variance of 50, and intensity variance of 25 may be used as an example.

[0127] Edge Detection

[0128] After reducing the noise in the input images, a canny edge detection function may be used in the edge detection module 2704 for edge detection. For example, the Canny edge detection function may leverage a 3 x 3 filter to detect the gradients in the images in the x and y directions. An L2 summation of both gradient values may then subject to a non- maximum suppression before a thresholding process.

[0129] For eample, two threshold parameters may be employed in the Canny edge detection function, Thhigh and Thiow. The pixels with gradient values higher than Thhigh may be reserved while the ones lower than Thiowmay be deleted. On the other hand, pixels with gradient values between Thhigh and Thiowmay be reserved only when neighboring any candidates that are greater than Thhigh. In some example implementations, Thhigh may be adjusted by the fuzzy parameter tuner, and Thiowmay be set at 1 / 3 of Thhigh

[0130] ROI Selection

[0131] As generally described above, the number of candidate edge pixels to process greatly influence the processing time of the preprocessing framework. Inorder to select better candidates among the edge pixels derived by the Canny edge detector, the ROI selection module 2706 may be deployed in the preprocessing module 2700, as indicated in Fig. 27.

[0132] As described above, typical roads consist of parallel lines that converge to a vanishing point in images. Therefore, with a triangular ROI, the possibility of capturing at least one lane in the image is high. When the slope in the road is limited, the vanishing point resides around the center of the images. Similar to the lane detection approach above, an ROI region that covers the vanishing point with a margin was adopted. The apex of the ROI, for example, may be set at the center of the image with 3 / 4 of the image height.

[0133] Line Detection

[0134] A line detection function 2708 may be incorporated into the preprocessing module 2700 as indicated in Fig. 27 to tune the threshold of the Canny edge detection module. The line detection result may not be directly used to assist the lane detection process. Instead, it may be used as a measure to evaluate the performance of the current threshold value.

[0135] Considering the design of the triangular ROI and the typical structure of roads, it is expected that the ROI would contain 1 -2 lanes, which may include from 1 - 3 line marks. As a result, when applying line detection module 2708, the detected line number should fall into a fixed range if the edge detection module 2704 is correctly tuned. To be more specific, when the edge detection module 2704 outputs more edge pixels than actually there are, an overly higher detected line number may be detected. As such, an increase in the threshold value may be needed in this case.

[0136] The commonly used Hough Transform (as described above) may be adopted to identify and quantify the line numbers in the images. The Hough Transform method may generate straight lines in all directions, with each line centered on every edge pixel present in the images. A “vote” value may be assigned to each line by tallying the number of edge pixels that align with the generated line. Lines with higher vote counts may be considered as exhibiting a stronger connection with edge pixels, thus making them preferred candidates for representing lane lines.

[0137] In some example implementations, an angular resolution of 1 degree and a minimum intersection of 3 may be adopted in this work. The detected line count may then be used to tune the threshold value in the Canny edge detector, moderating the number of candidate pixels. Such a closed-loop feedback process may help maintain a proper number of candidates, thereby reducing excessive processing time of Hough Transform.

[0138] Fuzzy Parameter Tuner

[0139] Similar to the implantations described above, a parameter-tuning procedure may be employed for fuzzy logic parameters, particular the ones associated with the member functions. To be more specific, the parameter tuner shown as 2709 may decide a direction (increase / decrease) and a level (more / less) of the applied change instead of outputting an absolute threshold value.

[0140] As mentioned previously, given the road structure and the inclusion of the ROI, the number of detected lines in the triangular ROI should fall into a fixed range. Therefore, the detected line number was leveraged as the “observable parameter” that could be used to tune the threshold value of Canny edge detector.

[0141] In some example implementations, a single input single output (SISO) FIS system may be leveraged in the parameter tuning process. As an example, five input membership functions may be defined according to the detected line count, as shown in FIG. 28. Based on the membership functions, the probability of the input belonging to each linguistic category may be calculated. The rule-base used as an example is listed in Table V. When the line number detected was fewer than the expected value, the threshold of the Canny edge detector is reduced in order to introduce more information to the system and vice versa. The decisions made by the FIS may then be defuzzified with the output membership functions, as shown in FIG. 29Table V: The rule base of the fuzzy edge detection module

[0142] All membership functions may be designed experimentally. The initial Canny threshold value may be set to, as an example, 1 , because it allows the most information from the images to be processed. However, to raise the threshold to the appropriate value effectively and thereby enhance the detection rate, an asymmetric design may be adopted for the “Add2” function (step up by 2). This off-center function may help raise the threshold value quickly when the experiment is initiated, helping the system to stabilize within fewer frames.

[0143] Channel Allocation

[0144] At the end of the preprocessing algorithm, a channel allocation process may be applied, as shown by 2705 of Fig. 27. Due to the adaptive parameter tuning capability of the algorithm, the edge detection result may be able to outline the lane lines effectively. Therefore, the edge detection results may be included in the final output to the CNN-based lane detection models.

[0145] In some example implementations, based on the experiments conducted, the green channel in the images may be determined as carrying more information on the white and yellow lines. Therefore, in the channel allocation process, the detected edge images may be normalized to 8-bit and used to replace the red and blue channels of the original images. An example of the allocated result is shown in FIG. 30. Fig. 30 shows that the green channel preserves most image features of the lines while the edge channels well-outlined the marks. This approach may help maintaining the dimensions and the number of channels of the input images, enabling seamless integration with most CNN-based lane detection models.

[0146] Integration with CNN Models

[0147] Finally, as shown in FIG. 27, the preprocessed output images may be feed to any types of learning-based modes for further lane detection, including but not limited to CNN based lane detection system 2710.Terrain Detection / Recocinition / Classification - Overview

[0148] For outdoor robotic applications, including delivery, construction, environment inspection, mining, and agricultural applications, it is critical for a robot to recognize its operation environment, such as its terrain, in real-time, in order to control the robot to improve its traversability, power consumption, and path-planning decision. Different terrains, for example, may significantly affect the operation of such outdoor robots. For example, lower tire pressure and smoother steering should be applied to vehicles in sandy terrain to prevent them from being stuck in loose sand. Hazardous areas, such as muddy terrains, may cause wheeled robotic vehicles to sink and therefore harm the electronics. These situations could result in property loss to stakeholders and even environmental damages. In order to reduce the risk of vehicle operations and improve the efficiency of the system, robust automatic terrain classification may be critical.

[0149] In some example implementations, robotic terrain classification (RTC) may rely on exteroceptive sensors, proprioceptive sensors, or a combination thereof. Example exteroceptive sensors may include stereo cameras, near-infrared cameras, laser scanners, and LIDAR for sensing the environment of the robot in two or three dimensions. The data output from the exteroceptive sensors may be processed via rule-based terrain detection / classification schemes or via more advanced trained neural network classifiers.

[0150] However, similar to the issues described with respect to the on-the-road lane detection context above, illumination / lighting variation and visibility issues all pose great challenges for terrain classifiers solely based on exteroceptive sensors. In some example implementations, other types of exteroceptive sensors such as acoustic sensors (microphones) and ultrasonic sensors may be used to provide supplemental datasets. However, effectiveness of these sensors for automatic terrain classification may still be significantly limited.

[0151] In some example implementations, proprioception may be adopted to address the issues associated with external illumination variation, since the proprioceptive signals are based on the robot’s physics-based dynamic feedback and are thus insensitive to variations such as illumination. Proprioception-based classifiers may assess the terrain by dimensional parameters and direct ground feedback such as vibration, linear acceleration, motor current, or angular velocity. Inaddition, with a sampling rate of, for example, at least 100 Hz, proprioceptive sensors may be configured to provide more continuous information than standard exteroceptive devices such as RGB cameras (below 30 Hz).Terrain Detection / Recocinition / Classification - Architecture

[0152] In the additional example implementations below, a general, compact, modularized and illumination-robust terrain classifier for robotic applications is disclosed. These example implementations combine the processing of real-time data from both exteroceptive and proprioceptive sensors to determine terrain classification. As shown in Fig. 18, an example terrain classifier may include two classifier modules 1802 and 1804, for exteroceptive (vision) and proprioceptive processing, respectively. The vision-based exteroceptive classifier 1802 processes data from one or more exteroceptive sensors 1810 and includes an exteroceptive network 1812 that may be based on a pre-trained neural network model, such as a MobileNet v2 model. The exteroceptive sensors 1810, for example, may include a stereo Near-Infrared (NIR) camera. The proprioception classifier 1804 processes data from one or more proprioceptive sensors 1820 and includes a proprioceptive network 1822 which again may be based on a pre-trained neural network model. The neural network, for example, may be based on a 1 D convolutional neural network (CNN) for processing the raw signal from the proprioceptive sensors.

[0153] The classifying output from the exteroceptive classifier 1802 and the proprioceptive classifier 1804 may then be fused by a fusion classifier 1830 which combines features extracted from modules 1802 and 1804 and outputs the final classification result. In some example implementations, all data may be acquired from a particular remote-control robot platform. All the data training / testing may be performed offline.

[0154] The exteroceptive network 1802 may be referred to as Vision Net, whereas the proprioceptive network 1822 may be referred to a Proprioceptive Net, and the fusion classifier 1830 may be referred to as Fusion Net.

[0155] Fig. 19 shows further detailed example architecture of the Vision Net, the Proprioceptive Net, and the Fusion Net.

[0156] In some example implementations, as shown in Fig. 19, the Vision Net may include a pre-trained 1 D Mobile Net v2 module embedded in a vision CNN 1902 to acquire the visual terrain properties, features, and textures 1904 more effectively under limited computation resources. The vision CNN 1902 may be pre-trained using an example ImageNet image database. In addition, various image augmentation methods may be applied to extend the complexity of training data.These image augmentation approaches may include image flipping, random rotation, random brightness, and random contrast. These data augmentation processes help increase the complexity of the training dataset in the iterative train / validate process in order to avoid the over-fitting effects. Of all adopted processes, random brightness and random contrast augmentation may be especially helpful in improving the classifying results in brightness-varying environments. An example architecture for the vision Net is shown below:TABLE V: Vision Net model structure

[0157] In some example implementations, terrain images 1906 of Fig. 19 for the vision CNN 1902 may be collected by a highly reliable stereo camera. The stereo camera may provide either or both RGB or grayscale images. The stereo camera may operate in either visible or IR spectral range. In some implementations, dual spectral output may be generated by the stereo camera (in both visible and IR spectral range).In some example implementations, the Proprioception Net of Fig. 19 may be deployed as the example 1 D proprioception CNN 1606 of Fig. 19, which may process one or more signals from various proprioceptive sensors to extract proprioceptive features 1908. For example, five different proprioceptive signals may be processed by the proprioception CNN. The proprioceptive sensors may include,for example, one or more accelerometer, one or more gyroscopes, one or more speedometers, and the like. Architecture of an example proprioception CNN is shown below.TABLE VI: Proprioception Net model structure

[0158] In an effort to make the terrain classifier framework platform (vehicle / robot) independent, use of vehicle / robot related parameters may be avoided. For example, the dimension of the vehicle and its coordinate- may not be considered as part of the proprioceptive data. In some example implementations, as few general types of proprioceptive signals from as few sensors as possible may be used. In some specific examples, as few as 5 different proprioceptive signals may be used and processed by the Proprioception Net. For example, three linear accelerations of the vehicle / robot in three different directions may be used. Two signals representing the angular speed of the wheels on both sides of the robot / vehicle may be used. These example 5 signals may be measured by as few as three proprioceptive sensors. The acceleration signals, for example, may be provided by a built-in accelerometer as part of the stereo IR camera. These signals are chosen to be independent of robot / vehicle dimensions and other robot / vehicle properties so that the algorithm disclosed herein may be as platform agnostic as possible. These signals are also processed in raw format by the Proprioceptive Net and thus further render the Proprioceptive Net platform independent.

[0159] In some example implementations, the Fusion Net 1830 of Fig. 8 may be deployed as a multilayer neural network 1910 of Fig. 19. The Fusion net 1910may be configured to merge the features 1904 and features 1908 extracted from the vision CNN 1902 and proprioception CNN 1906, respectively. The fusion neural network 1910 may adopt a feature map-level fusion scheme. For example, a flattened output from the Proprioception Net (1 x 1024) and that from the Vision Net (1 x 1280) are stacked into a hybrid feature (e.g., 1 x 2304) and may be processed by the Fusion Net 1910.Terrain Detection / Recognition / Classification - Operation Configuration

[0160] An example Jackal vehicle is used to implement the terrain classifier above. The example Jackal vehicle is a skid-steer four-wheel-drive vehicle produced by Clearpath robotics, and is applied to collect the intrinsic proprioceptive signals in the experiments below. Being of skid-steering type, the wheels of this vehicle on each (left / right) side share the same spin rate. The Jackal vehicle further integrates an onboard Inertial Measurement Unit (IMU), DC motors with encoders, and motor current sensors for wheels on both sides. The onboard IMU provides the platform attitude with linear acceleration and angular rate in the Euclidean coordinate system.

[0161] All the outdoor terrain data collection experimental tasks are carried out by the Jackal vehicle with normal velocity mode (0.2-0.5 m / s) to prevent the vehicle from accidentally tipping over. A human operator controlled the Jackal via Bluetooth at a visible distance during all data collection and experimental processes.

[0162] The images for the vision-based classifier are collected by a highly reliable stereo camera, Multisense S7, from Carnegie Robotics. Multisense S7 stereo camera has dual types of image output, RGB and grayscale type. Both are experimented with. In addition, experiments with two sensing versions of the stereo camera, visible and NIR bands, are also conducted to compare the capabilities of the example terrain classifier under both vision spectral bands.

[0163] The stereo camera is mounted on the top platform of the Jackal by an aluminum extrusion structure with a down-tilt angle in order to capture a broad horizontal view ahead. The terrain patch covered by the stereo camera field of view is around 1000 mm x 900 mm in front of the vehicle. This example robotic vehicle setup is shown in Fig. 20. The versatile MultiSense S7 camera is integrated with an additional onboard IMU and a magnetometer. To maintain the consistency inimaging quality under varying illumination environments, the camera's exposure time is fixed at 10 ms. Furthermore, the frame rate of the camera is reduced to 10 Hz in order to reduce the data storage size. The camera parameter setting procedures mentioned above are all completed on an original interface of MultiSense S7 via the Robot Operating System (ROS).Terrain Detection / Recognition / Classification - Training and Testing Data Collection

[0164] Eight outdoor terrain classes are investigated in the experiment above to build a robust terrain classification method for outdoor environments. The terrain classes are divided depending on the diverse size of soil particles and the surface properties, including concrete pavement, sand, dirt, gravel, soil, grass, pine needles, and tree bark. Fig. 21 shows the sample images from the eight outdoor terrain classes. During data collection, the vehicle traversed those terrain classes multiple times to collect the images and corresponding proprioceptive feedback.

[0165] In the example terrain classifier above, as few proprioceptive signals as possible are used. For example, as described above, as few as five (5) proprioceptive signals or features coming from the stereo camera and the wheel encoder of the vehicle are used in order to use the reduced-order neural network model. Specifically, the IMU in the stereo camera provides the three-axis linear acceleration, and the wheel encoders on the vehicle provide the angular velocity. That is a total of 5 signals of 2 different types.

[0166] All collected sample data from each terrain class may be stacked into two primary datasets, as training dataset and testing dataset. For each terrain type, data of 1600 seconds in length are collected as the training dataset, and another 500 seconds-long data are collected as testing dataset.

[0167] Since the goals of those two datasets are distinct, the vehicle is operated in two driving modes for data collection: (1) straight-driving mode and (2) remote-control mode. In the straight-driving mode, the vehicle only moved forward straightly without turning, stopping, or reversing. A 400-second-long data collection in the straight-driving mode for each terrain provides essential terrain features and vehicle dynamic properties in a stable driving pattern. However, the simple straight path would not reflect the off-road vehicle operating conditions in the real world.Therefore, the datasets collected in the straight-driving mode are excluded from the testing datasets. The other 1200 second of data are collected in the remote-control mode as part of the training dataset and all of the 500 seconds of testing dataset. To simulate an actual field task, the vehicle is driven in the investigated areas in the remote-control mode without a particular path. Similar to straight-driving mode, no stopping and reverse driving is involved in the remote-control mode. In total, the training datasets contain a total of 8 terrain types x (400s + 1200s) =12800s (3.56 hours) of images and proprioception data, while the testing dataset contained 8 x 500s =4000s (1.11 hours) of data. The detailed properties of the training dataset are shown below.TABLE VII: Properties of Datasets

[0168] Though the training dataset only contains the data collected in the daytime, the testing dataset for evaluating the example terrain classifier with data from both daytime and nighttime. Daytime testing dataset (Day Set) and night-time testing dataset (Night Set) are collected from identical terrain areas within normallight and low-light conditions. Fig. 22 shows the visible-band sample images in both Day Set and Night Set. The NIR band camera received broad electromagnetic spectrum energy through a designed optical filter and therefore has a color difference in the output image. Fig. 23 shows grass sample images captured by the NIR band camera. However, the raw images in Night Set are almost all-black. Therefore, the contrast and brightness of the sample images in Fig. 22 and Fig. 23 are adjusted by an image editing tool in order to present the terrain features. The RGB and grayscale images are saved simultaneously in separate files to generate the dataset with different input image types.

[0169] Overall, both sensing versions (visible and NIR) of cameras are operated to collect their own Day Set and Night Set for testing the trained model separately. The processed data are saved in the HDF5 format and sorted by the timestamp, as shown by 1940 of Fig. 19. The saved images are all resized to 224 x 224 pixels to fit the input format of the trained module Mobile Net v2 of the Vision Net (or vision CNN) 1902 of Fig. 19.

[0170] Further, the training dataset and Day Set are all collected on non-rainy days during normal light conditions. The data are acquired after sunrise (11 am - 5 pm), while the Night Set is collected on non-rainy days under low-illumination conditions after sunset (7:30 pm - 8:30 pm). Weather.com is referred to for weather forecast updates. The terrain data collection tasks are completed in 2022 from July 29 to Aug. 23 in Champaign, Illinois. During the data acquisition days, the temperature range is 18-29 degrees Celsius.Terrain Detection / Recognition / Classification - Data Processing

[0171] The data rates of proprioceptive sensors and visual sensors are different from one another. Therefore, an interpolation is performed as part of the data preprocessing. The input proprioceptive signals and images are chopped into one-second data segments. Every segment of data contained 1 image array and 100 sets of proprioceptive measurements. In the case where the data rates from various sensors are different, a Zero-Order-Hold (ZOH) interpolation is adopted to adjust all sampling frequencies to 100 Hz for proprioceptive data.

[0172] As described above, five proprioceptive parameters are measured from the proprioceptive sensors. Three of them represent the linear acceleration of the x, y, and z axes. The other two parameters represent the angular speed of the wheels on both sides. Therefore, a 100 x 5 vector is generated in one data segment. Furthermore, the example 1 D proprioception CNN processes the proprioceptive signals with partial temporal information.

[0173] In addition, the data corresponding to where the vehicle is stopped are excluded from the dataset in order to prevent meaningless data from influencing the training quality. When a data segment contains stopping or linear speed below 0.2 m / s, it is excluded from the data processing step. That is because the low-speedphysical feedback from the vehicle contains less valuable information for neural network training.

[0174] With the Multisense S7 stereo camera set to 10Hz FPS, only the first image captured within every second is stacked with the proprioceptive data. That is, as an example, all single-segment data contains one image and five proprioceptive parameters (100Hz) extracted within a second.Terrain Detection / Recognition / Classification - Training and Experimental Outcome

[0175] As described below in further detail, confusion matrices are adopted to comprehensively present the classifying results in each terrain type. The number of misclassification cases and the confusing relatives in each terrain type are all presented in the confusion matrices.

[0176] Visible Spectral Range (Visible Band) Camera

[0177] According to all tested results from the two testing datasets, Fig. 24 shows the confusion matrices under the three trained models applied with the dataset collected with the visible band camera. The overall classification accuracy is displayed in the top column of confusion matrices. Three main observations are identified in the confusion matrices.

[0178] First, an extremely high accuracy, above 99% (matrix A of Fig. 24), is achieved by the Vision Net model under normal light. The terrain texture and appearance are precisely characterized by the trained model when the testing data are collected under good lighting circumstances. However, the accuracy dropped dramatically in matrix (B) of Fig. 24 when processing the Night Set sample data. As a result, lacking visual information from dim environments poses a significant challenge to the visual classifier based on a single sensor.

[0179] Second, the confusion matrices of Proprioception Net in Fig. 24 (C)&(D) provides little difference in accuracy under the two testing light conditions. This observation could easily be explained by the fact that proprioceptive signals are independent of illumination conditions. Compared to Vision Net, it is clear that Proprioception Net could provide robust characteristics for the terrain classifier.

[0180] Eight complex outdoor terrain types are used here in this experiment. The proposed reduced-order Proprioception Net maintains an average accuracy above 82%. However, certain terrain types are difficult to classify using proprioceptive feedback alone because they have similar surface properties. For example, rocky granules are found in sand, gravel, soil, and dirt terrain types. Specifically, the size of rocks from gravel and dirt are similar in some investigating regions. This results in similar proprioception data and therefore causes a correlative result in the classifier. Aside from the granule, some terrain types are covered by natural mulch, such as grass and pine terrain types. The thick layer on top results in comparable dynamic feedback and therefore causes confusion in classification. On the other hand, the tree bark layer could be pushed by the vehicle easily. The ground beneath the tree bark might cause the classifier to mistake this terrain type for dirt or soil. In short, the terrain types that are very distinct visually might be proprioceptively similar and therefore may impede the overall accuracy of the Proprioception Net.

[0181] The confusion matrices (E)&(F) in Fig. 24 demonstrate the high accuracy achieved by Fusion Net. In the confusion matrix (E), the Fusion Net preserves extremely high accuracy, 99.7%, under sufficient light. Furthermore, the confusion matrix (F) of Fig. 24 shows that the accuracy of the fusion classifier under Night Set is raised to 92%, which significantly outperforms benchmark results (matrices (B) and (D) of Fig. 24).

[0182] The complete result of the experiment is listed Table VIII below.TABLE VIII: Overall Classification Accuracy from Visible Spectral Range Camera

[0183] Near-Infrared Band Camera

[0184] The confusion matrix in Fig. 25 demonstrates the testing results collected by the NIR band camera. Under the same data collecting, model training, and model testing procedures, results with better accuracy are obtained with the NIR sensing band MultiSense S7 camera. The results of the experiment are listed in Table IX.TABLE IX: Overall Classification Accuracy from NIR Spectral Range Camera

[0185] The NIR visual data provides a more abundant visual appearance since the imager receives a broader electromagnetic spectrum from the environment than visible band cameras. In Fig. 25 (A)&(B), the Vision Net model reached a good classification accuracy (89.2%) when tested with Night Set (B) and 99.6% in the daytime testing dataset. Compared to the visible band dataset, the result in the NIR experiment maintains high accuracy in the Day Set, and significantly improves the result in the Night Set. In addition, Fig. 25 (C) and (D) show that the average accuracy of Proprioception Net from the two testing datasets is 81 .2%, which is similar to the result collected from the visible band stereo camera (82.45%). Therefore, it may be concluded that the Proprioception Net is robust over the illumination conditions as well. Finally, the matrix (F) of Fig. 25 shows that the terrain classification accuracy is greatly improved (97.8%) when the NIR camera isadopted. The fusion of visual and proprioceptive terrain features enables the precise classification of the eight terrain classes under low-light conditions.

[0186] The accuracy of different input image types

[0187] Experiments are also done to assess the influence of RGB color images and grayscale images. In these trials, grayscale images are used to replace the RGB color images. Though the RGB images with three color channels are expected to contain more abundant features than grayscale images, Tables above show that the result is maintained at a satisfactory level even when the color information is removed from the dataset. The difference in average Fusion Net accuracy is only 0.2% and 1 .2% for visible and NIR cameras. By using the grayscale images, the data size can be reduced to 1 / 3, and thereby providing a feasible and computationally efficient way to improve the framework.Terrain Detection / Recocmition / Classification - Summary

[0188] The example terrain classifier implementation above provides higher accuracy while requiring fewer data and fewer proprioceptive sensor signals. More terrain categories and higher complexity are included in the experiment to show the capabilities of the model. In order to demonstrate the performance of the example terrain classifier in a low illumination environment, the experiment is done with both a visible band camera and a NIR camera. By combining the proprioception and NIR images, the example framework is able to perform the terrain classification task with an accuracy higher than 97% in both daytime and nighttime, demonstrating the robustness of the model under varying illumination. In addition, by reducing the vehicle-related data needed, the flexibility of applying the model to different platforms is promising, thereby rendering the example terrain classifier as platform independent as possible.

[0189] Additional details are included in Appendix A.

[0190] It is to be understood that the various implementations above are not limited in its application to the details of construction and the arrangement of components set forth above and in the accompanying drawings. The disclosure is intended to cover other embodiments that may be practiced or carried out in various ways following the underlying principles disclosed herein.

[0191] It should also be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components may be used to implement the various embodiments of the disclosure. In addition, it should be understood that embodiments of this disclosure may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components are implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this disclosure, would recognize that, in at least one embodiment, the electronic based aspects of the invention may be implemented in software (e.g., stored on non- transitory computer-readable medium) executable by one or more processors. As such, it should be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components may be utilized to implement the invention. Furthermore, and as described in subsequent paragraphs, the specific mechanical configurations illustrated in the drawings are intended to exemplify embodiments of the invention and that other alternative mechanical configurations are possible. For example, “controllers” described in the specification can include standard processing components, such as one or more processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the components. These controllers may be implemented as dedicated processing circuitry or in general- purpose processors, in combination of various software and / or firmware, and in combination of other wired or wireless communication interfaces.

Claims

CLAIMSWe claim:1 . A method for automatic lane detection for robotic or vehicular control, comprising: receiving a current-frame RGB image to generate a preprocessed currentfame image; automatically pre-processing the current-frame RGB image to generate a preprocessed current-fame image for lane contrast enhancement; automatically detecting a plurality of edges in the preprocessed current-frame image; automatically determining a region of interest (ROI) within the preprocessed current-frame image; automatically detecting a set of candidate lines within the ROI of the preprocessed current-frame image based on the plurality of edges; automatically determining at least two lane-boundary lines from the set of candidate lines; generating a fuzzy logic inference signal from at least one property of the set of candidate lines using a fuzzy logic rule-set; and feeding forward the fuzzy logic inference signal for an automatic edge detection of a next-frame image.

2. The method of claim 1 , wherein preprocessing the current-frame RGB image comprises converting the current-frame RGB image to a grayscale image to generate the preprocessed current-frame image.

3. The method of claim 1 , further comprising automatically detecting yellow lines in the preprocessed current-frame image.

4. The method of claim 3, wherein automatically detecting the yellow lines are performed in an YUV color space using difference between the U and V channels as a yellow indicator.

5. The method of claim 3, further comprising: upon detection of yellow lines in the preprocessed current-frame image, switching a preprocessing mode for the nextframe image to a YUV mode in which the next-frame image is preprocessed in a YUV color space rather than an RGB color space.

6. The method of claim 1 , wherein automatically detecting the plurality of edges comprises edge detection based on pixel gradients in the preprocessed currentframe image in comparison to at least one gradient threshold.

7. The method of claim 6, wherein the ROI comprises a triangular ROI.

8. The method of claim 7, wherein a bottom of the preprocessed current-frame image forms a base of the triangular ROI and an apex of the triangular ROI is determined based on a lane-line converge point determined from at least one previous-frame image.

9. The method of claim 8, wherein the ROI apex is determined from averaging line-line converging points of the at least one previous-frame image.

10. The method of claim 8, wherein the apex of ROI is increased in a height direction by a predetermined percentage from the lane-line converge point determined from the at least one previous-frame image.1 1 . The method of claim 6, wherein the fuzzy logic inference signal is configured to adjust the at least one gradient threshold for the automatic edge detection of the next-frame image.

12. The method of claim 11 , wherein input membership functions to the fuzzy logic rule-set involves linguistic variables describing the at least one property of the set of candidate lines and the fuzzy logic inference signal indicates an amount of adjustment to the at least one gradient threshold for the automatic edge detection of the next-frame image.

13. The method of claim 12, wherein the at least one property of the set of candidate lines comprises a quantity lines in the set of candidate lines.

14. The method of claim 13, wherein the fuzzy logic inference signal is configured to increase the at least one gradient threshold for a detection of a large number of lines and to decrease the at least one gradient threshold for a detection of a small number of lines.

15. The method of claim 14, wherein the at least one gradient threshold comprises a high gradient threshold and a low gradient threshold, and wherein the high gradient threshold and low gradient threshold are adjust jointly adjusted based on the fuzzy logic inference signal.

16. The method of claim 6, wherein detecting the set of candidate lines based on the plurality of edges comprises detecting each of the set of candidate lines by identifying colinear edges having a length larger than a predetermined length threshold.

17. The method of claim 16, wherein identifying colinear edges is performed in a Hough space where each point in the Hough space represents a line having a unique combination of direction distance to an origin.

18. The method of claim 17, wherein identifying colinear edges comprises identifying points in the Hough space having accumulated count higher than a predetermined count threshold.

19. The method of claim 1 , further comprising determining the at least two laneboundary lines from lane lines of a previous-frame image when failing to detect the at least two lane-boundary lines in the preprocessed current-frame image.

20. A robotic / vehicular controller, comprising a circuitry configured to executed a set of instructions to:receive a current-frame RGB image to generate a preprocessed current-fame image; automatically pre-process the current-frame RGB image to generate a preprocessed current-fame image for lane contrast enhancement; automatically detect a plurality of edges in the preprocessed current-frame image; automatically determine a region of interest (ROI) within the preprocessed current-frame image; automatically detect a set of candidate lines within the ROI of the preprocessed current-frame image based on the plurality of edges; automatically determine at least two lane-boundary lines from the set of candidate lines; generate a fuzzy logic inference signal from at least one property of the set of candidate lines using a fuzzy logic rule-set; and feed forward the fuzzy logic inference signal for an automatic edge detection of a next-frame image.

Images