Free path detection using a hierarchical approach

Abstract

Method for detecting a free travel path (50) for a vehicle (100) using at least one image (10) generated by a camera (110), wherein the method comprises that the image (10) is being monitored; a first piece-based analysis (504, 604) of the image (10) is used, which includes that the image (10) is divided into a large number of pieces (60) sufficient to adequately analyze the image (10), using several overlapping pieces (60); a feature identification algorithm is applied to the identified pieces (60); and each piece (60) is analyzed to determine for each of the pieces (60) of the first piece-based analysis (504, 604) a first confidence probability of a free path (50); Each of the pieces (60) where the first confidence probability of a free path (50) is greater than a first threshold confidence probability is defined as a piece (60) of a free path (50); A second analysis on a subset of the pieces (60) where the first confidence of a free path (50) is not greater than the first threshold confidence probability is used to determine a second confidence probability of a free path (50) for each of the pieces (60) of the second analysis; Each of the pieces (60) of the second analysis where the second confidence probability of a free path (50) is greater than a second threshold confidence probability is defined as a piece (60) of a free path (50); and the free path (50) for the vehicle (100) is identified on the basis of the segments (60) of a free path (50).

Description

[0001] This disclosure relates to automated or semi-automated control of a motor vehicle.

[0002] Autonomous and semi-autonomous driving systems use inputs regarding the road and other driving conditions to automatically control throttle and steering mechanisms. Accurate estimation and identification of a clear path for a vehicle to follow is critical for successfully replacing the human brain as the control mechanism for vehicle operation.

[0003] Road conditions can be complex. During normal vehicle operation, the human operator makes hundreds of observations per minute and adjusts the vehicle's operation based on perceived road conditions. One aspect of perceiving road conditions is perceiving the road in the context of objects on and around the roadway and navigating a clear path through any such objects. Replacing human perception with technology preferably involves a means to accurately perceive objects, including stationary objects such as curbs and moving objects such as other vehicles, as well as road conditions such as lane markings, potholes, and icy patches on the roadway, and to proceed with effectively navigating around such navigational problems.

[0004] A technological means of perceiving an object or road conditions includes data from visual cameras, radar image acquisition, and LiDAR. Cameras translate visual images in the form of radiation, such as light patterns or infrared signatures, into a readable data format. This data format comprises pixel images, where a perceived scene is broken down into a series of pixels. Radar image acquisition uses radio waves generated by a transmitter to estimate shapes and objects in front of the transmitter. Patterns in the waves reflected by these shapes and objects can be analyzed, and the locations of objects can be estimated. Similarly, LiDAR uses a light source and the reflected light from the vehicle's surroundings to generate information about the environment.

[0005] Once data regarding the ground in front of the vehicle has been generated, this data must be analyzed to estimate the presence of objects or road conditions. Using cameras, radar image acquisition systems, and LiDAR, the ground or roadway ahead can be analyzed for the presence of objects or road conditions that should be avoided. However, simply identifying potential navigational problems to be avoided does not constitute the analysis. A crucial component of any autonomous system involves how potential navigational problems identified in the perceived ground data are processed and modified to identify a clear path for the vehicle to follow.

[0006] A well-known method for identifying a clear path for the vehicle to follow is to catalog and preliminaryly identify all perceived navigational problems and then identify a clear path based on the locations and behavior of these problems. Images can be processed to identify and classify navigational problems according to their shape and relationship to the roadway. While this method can be effective in identifying a clear path, it requires significant processing power, such as recognizing and separating different objects in the visual image, for example, distinguishing between a tree at the side of the road and a pedestrian walking toward the curb.Such methods may be slow or ineffective when dealing with complex situations, or may require cumbersome and expensive equipment to provide the necessary processing capacity.

[0007] From WO 2008 / 071473 A1, it is known to determine, from an image of a vehicle's environment divided into fields, the probability of an obstacle in each field. This probability is compared with two threshold values, and if either threshold is exceeded, the field is classified as obstructed. The result of the comparison with the lower threshold is passed to a comfort system, and the result of the comparison with the higher threshold is passed to a safety system of the vehicle.

[0008] DE 10 2006 046 843 A1 describes a method for detecting the environment of a vehicle, in which data for detecting objects as well as data for detecting free areas are determined, which are combined according to a combination rule to give a probability for the presence of a free area.

[0009] From the publication TT Son et al. “Road detection using segmentation by weighted aggregation based on visual information and a posteriori probability of road regions”, 2008 IEEE International Conference on Systems, Man and Cybernetics, Singapore, 2008, pp. 3018-3025, DOI: 10.1109 / ICSMC.2008.4811758, a method for road detection based on an image segmentation algorithm is known.

[0010] It is an object of the invention to provide a method for detecting a free driving path for a vehicle using an image generated by a camera, which is more reliable and efficient than methods known from the prior art.

[0011] The problem is solved by a method having the features of claim 1. The problem is also solved by a method having the features of claim 9 and by a method having the features of claim 10.

[0012] A method for detecting a free path for a vehicle using at least one image generated by a camera comprises monitoring the image and performing an initial piece-based analysis of the image. This initial analysis includes dividing the image into a sufficient number of pieces for adequate analysis, using multiple overlapping pieces. A feature identification algorithm is then applied to the identified pieces to determine a first confidence probability of a free path for each piece. Each piece for which the first confidence probability of a free path is greater than a first threshold confidence probability is defined as a piece of a free path.A second analysis is performed on a subset of the pieces where the first confidence level of a free path is not greater than the first threshold confidence level, in order to determine a second confidence level of a free path for each of the pieces in the second analysis. Each of the pieces in the second analysis where the second confidence level of a free path is greater than a second threshold confidence level is defined as a piece of a free path. The free path for the vehicle is then identified based on these pieces of a free path.

[0013] Below, one or more embodiments are described by way of example with reference to the accompanying drawings, wherein: Fig. 1 shows an exemplary arrangement of a vehicle equipped with a camera and a radar image acquisition system according to the disclosure; Fig. 2 shows a known method for determining a free path to autonomous driving according to the revelation; Fig. 3 shows an exemplary procedure for determining a free path using a probability analysis of an image according to the revelation; Fig. 4 shows an exemplary procedure for analyzing an image according to the revelation; Fig. 5 shows an exemplary procedure for defining a classification error by tuning a single threshold according to the disclosure; Fig. 6A, Fig. 6B and Fig. 6C demonstrates an exemplary determination of an image difference by calculating an absolute image intensity difference according to the disclosure; Fig. 7 shows an exemplary procedure for classifying a feature as a section of a free path and simultaneously as a detected object as a method for image analysis according to the disclosure; Fig. 8 further shows an exemplary method for classifying a feature as a section of a free path and simultaneously as a detected object as a method for image analysis according to the disclosure; Fig. 9 is a flowchart for an exemplary piece-based method for detecting a free path according to the revelation; Fig. 10 exemplary several perspective pieces that are applied to an exemplary image, graphically representing according to the revelation; Fig. 11 a flowchart for a structure-rich pixel-based method for detecting a free path according to the disclosure; Fig. 12 a flowchart for a structureless pixel-based method for detecting a free path according to the disclosure; Fig. 13 a flowchart for detecting a free path for a vehicle using a first hierarchical embodiment of a free path detection method according to the disclosure; Fig. 14 a flowchart for detecting a free path for a vehicle using a second hierarchical embodiment of a free path detection method according to the disclosure; and Fig. Figure 15 schematically shows an operation of an exemplary example-based process according to the present disclosure.

[0014] Referring now to the drawings, in which the illustrations serve only to explain certain exemplary embodiments and not to restrict them, it shows Fig. 1 An exemplary arrangement of a camera 110, which is arranged at the front of a vehicle 100 and directed towards the ground in front of the vehicle 100, according to the disclosure. The camera 110 communicates with the processing module 120, which contains logic for processing inputs from the camera 110. The vehicle 100 may also be equipped with a radar image acquisition system 130, which, if present, also communicates with the processing module 120.Experts will recognize that, in addition to or alternatively using the camera 110 and the radar image acquisition system 130, the vehicle 100 could use a number of methods for identifying road conditions, including LIDAR, GPS information combined with a digital image, information from other vehicles communicating with the vehicle 100, information from an infrastructure facility connected to the vehicle 100, historical data relating to the specific lane, biometric information such as systems that read the driver's gaze, or other similar systems. The disclosure herein can be applied to various installation arrangements and is therefore not limited thereto.

[0015] The camera 110 is a device well known in the art that can translate visual inputs in the form of light, infrared, or other electromagnetic radiation (EM radiation) into a data format that can be easily analyzed, e.g., a digital pixel image. In one embodiment, the camera 110 uses a sensor of a charge-coupled device (CCD sensor) to generate images that define a field of view. Preferably, the camera 110 is configured for continuous image generation, e.g., 30 generated images per second. Images generated by the camera 110 can be stored in a memory within the camera 110 or transferred to the processing module 120 for storage and / or analysis. Preferably, each image generated by the camera 110 is a two-dimensional image with known pixel dimensions, comprising several identifiable pixels.The multiple identifiable pixels can be stored and analyzed using an array. Each pixel can be represented in the array as a set of bits or multiple sets of bits, where the bits correspond to a color on a predetermined palette or color chart. Each pixel can be expressed as a function of multiple color intensity values, such as in a red-green-blue (RGB) color model or a cyan-magenta-yellow-key (CMYK) color model. Preferably, each pixel comprises multiple sets of bits, where each set of bits corresponds to a color intensity and a color intensity value. For example, a first set of bits corresponds to a red color intensity value, a second set of bits corresponds to a green color intensity value, and a third set of bits corresponds to a blue color intensity value in the RGB color model.

[0016] The radar image acquisition device 130 is a device well known in the art, comprising a transmitter capable of emitting radio waves or other electromagnetic radiation, a receiver capable of detecting the emitted waves reflected back to the receiver by objects in front of the transmitter, and a means of converting the detected waves into a data format capable of analysis, indicating, for example, the distance and angle from the objects from which the waves were reflected. Alternatively, the radar image acquisition device 130 can be replaced or supplemented by a Light Detection and Ranging (LIDAR) system configured to transmit and receive optical energy. The received optical energy can be used to determine the geometric dimensions of an object and / or its geometric proximity to the vehicle 100.It should be noted that the radar image acquisition device 130 is optional and not necessary to perform many of the methods disclosed herein, whereby the processing of visual images can perform free-path detection. The term "free path," as used herein, is intended to have an ordinary and customary meaning for a person skilled in the art (and is not limited to a special or specifically adapted meaning) and refers, without limitation, to a path that is free of objects exceeding a threshold, e.g., a path free of objects classified in the second and third categories mentioned above.

[0017] The processing module 120 is in Fig. Figure 1 is shown and described herein as a discrete element. This representation serves to simplify the description, and it should be noted that the functions performed by this element may be combined in one or more devices, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. The processing module 120 can be a general-purpose digital computer comprising a microprocessor or central processing unit, storage media with non-volatile memory including read-only memory and electrically programmable read-only memory, random-access memory, a high-speed clock, analog-to-digital and digital-to-analog circuitry, input / output circuitry and devices, and suitable signal conditioning and buffering circuitry.Alternatively, the processing module 120 can be a digital signal processing unit (DSP), such as a custom integrated circuit like a field-programmable gate array. The processing module 120 includes a set of processing algorithms, comprising resident program instructions and calibrations, which are stored in non-volatile memory and executed to provide desired functions. The algorithms are preferably executed during predefined loop cycles. Executed by the central processing unit, the algorithms monitor inputs from the aforementioned sensing devices and perform control and diagnostic routines to manage the operation of the actuators using preset calibrations.The loop cycles can be executed at regular intervals, for example every 3, 125, 6, 25, 12.5, 25, and 100 milliseconds during ongoing engine and vehicle operation. Alternatively, the algorithms can be executed in response to the occurrence of an event.

[0018] The processing module 120 executes stored algorithmic code to monitor related devices, such as the camera 110 and the radar image acquisition system 130, and to execute commands or data transmissions as specified by the analysis performed in the processing module. The processing module 120 may include algorithms and mechanisms for actuating an autonomous driving control system by means known in the art and not described herein, or the processing module 120 may simply provide information to a separate autonomous driving system. Depending on the specific embodiment used in conjunction with the control module, the processing module 120 is capable of receiving input signals from other systems and the operator as required.

[0019] Fig. Figure 2 shows a known method for determining a free path for autonomous driving according to the disclosure. An image 10 is generated corresponding to the roadway in front of the vehicle 100. In one of several procedures, the objects 40A, 40B, and 40C in the image 10 are identified, and each object is categorized and classified according to a filter and trained object behavior. Separate processing of each object can be computationally intensive and requires expensive and cumbersome equipment to handle the computational load. An algorithm processes all available information regarding the roadway and the objects 40 to estimate a free path available to the vehicle 100. The determination of the free path depends on the specific classifications and behavior of the identified objects 40.

[0020] Fig. Figure 3 shows an exemplary method for determining a free path for autonomous or semi-autonomous driving according to the disclosure. Figure 10 is shown comprising a ground 20, a horizon 30, and objects 40. Figure 10 is captured by camera 110 and represents the road environment in front of vehicle 100. The ground 20 represents the zone of all available paths open for driving, without reference to any potential objects. The method of Fig. Method 3, which identifies a free path on floor 20, begins by assuming that the entire floor 20 is free and then uses available data to exclude sections of floor 20 as not being free. In contrast to the method of Fig. 2, which classifies each object 40, analyzes the procedure of Fig. 3. Instead, the system focuses on the ground 20 and attempts to define, from available data, a confidence probability of a free path that a detectable deviation, which object 40 may represent, restricts or prevents this section of the ground 20 from being free. This focus on the ground 20 rather than the objects 40 avoids the complex computational tasks associated with managing object detection. Individual classification and tracking of each object is unnecessary, as the individual objects 40 are simply grouped together as part of the overall uniform constraint on the ground 20. The ground 20, described above as all paths open to driving without distinction, minus the constraints arranged on the ground 20 by areas found to be free, defines a free path 50, which is located in Fig. 3 as the area shown within the dashed lines, or an area open with a threshold confidence probability of 100 for a journey of the vehicle.

[0021] The object 40, which creates non-free constraints on the ground 20, can take many forms. For example, an object 40 can represent a discrete object, such as a parked car, a pedestrian, or a road obstruction. Alternatively, an object 40 can represent a less discrete change in surface patterns that indicates an edge to a road, such as a curb, a boundary with grass, or water covering the roadway. The object 40 can also encompass the absence of a level road connected to the ground 20, as might be detected, for example, by a large pothole in the road. Furthermore, the object 40 can encompass an indicator without any definable elevation change from the road, but with clear effects on a free path for that segment of the road, such as a color pattern on the roadway indicating a lane marking.The method disclosed herein, which does not attempt to identify an object 40 but takes visual evidence from the ground 20 and everything near the ground in the image 10, evaluates a confidence probability of a free path of free versus not free and adjusts the control of the vehicle 100 for the presence of each object 40.

[0022] Numerous methods exist for the automated analysis of two-dimensional images (2D images). An algorithm in processing module 120 performs an analysis of image 10. Fig. Figure 4 shows an exemplary procedure that can be applied to analyze image 10 according to the disclosure. This procedure subdivides image 10 and identifies a sub-image or piece 60 of the ground 20 for analysis, extracts features or analyzes the available visual information from piece 60 to identify any features of interest or distinguishing characteristics within the piece, and classifies the piece according to the confidence probability that it is a free path based on the analysis of the features. The pieces that are classified as free with a probability greater than a certain threshold, and an assembly of the pieces, can be used to reconstruct a free path in the image.

[0023] Piece 60, as a sub-image of image 10, can be identified by any known means, such as a random search or a search of image 10. Alternatively, information regarding the presence of an object 40, available from another information source, such as the radar image acquisition system 130, can be used to identify a piece in order to analyze the section of image 10 that should depict the object 40. Analyzing the entire image 10 may require many pieces 60. Furthermore, several overlapping pieces or pieces of different sizes could be used to fully analyze an area of ​​image 10 containing information of interest.For example, a small piece 60 might be used to analyze a small point on the road; however, a large piece 60 might be required to analyze a series of points that, on their own, might appear uninteresting but, in the context of the entire series, might indicate an object 40 of interest. Furthermore, the resolution of pieces applied to a particular area can be modulated based on available information, for example, applying several pieces to an area of ​​the image 10 in which an object 40 is believed to exist. Many schemes or strategies can be used to define the pieces 60 for analysis, and the disclosure is not intended to be limited to the specific embodiments described herein.

[0024] Once a piece 60 has been identified for analysis, the processing module 120 processes the piece by applying a known feature identification algorithm. Furthermore, the processing module 120 can perform an analysis of the piece's location within the context of the vehicle's location. The feature identification algorithms search the available visual information for characteristic patterns in the image associated with an object, including features defined by track orientation, track location, color, corner properties, other visual attributes, and learned attributes. The feature identification algorithms can be applied to sequential images to identify changes corresponding to vehicle movement, although changes not associated with ground movement cannot be identified as free paths.The learned attributes can be learned in the vehicle using machine learning algorithms, but are mostly programmed offline and can be developed experimentally, empirically, predictively, through modeling or other techniques suitable for accurately training distinguishing attributes.

[0025] Once features have been extracted from the piece 60, the piece is classified based on these features to determine the confidence level that the path is a free path. Probability analysis is a process known in engineering by which a probability value, or confidence level, is developed that a certain condition exists. Applied to the present disclosure, classification includes probability analysis to determine whether the piece represents a free path or whether the ground 20 in that piece is restricted by an object 40. In one exemplary embodiment, the classification is performed by applying classifiers or algorithms trained on a database of exemplary road conditions and interactions with detected objects.These classifiers enable the processing module 120 to develop a fractional probability value of a free path for the piece 60, quantitatively determining a confidence level between zero and one that the features identified in the piece do not indicate a restrictive object 40 that would prevent the vehicle 100 from traveling freely. A threshold confidence level can be set, defining the probability of a free path required to classify the piece as a free path, for example, using the following logic. Confidence = ProbabilityFreePath(i)If_Confidence>0.5, then_Piece = FreePath

[0026] In this particular exemplary embodiment, a confidence level of 50% or 0.5 is selected as the threshold confidence. This number can be developed experimentally, empirically, predictively, through modeling, or other techniques suitable for accurately evaluating pieces with respect to free-path properties.

[0027] As mentioned above, in an exemplary embodiment, probability analysis can be performed by applying trained classifiers to features extracted from a single image. One method analyzes the features a priori using a training image set. In this training stage, distinguishing features are selected from a raw feature set, with the distinguishing features being defined by techniques known in engineering, such as the Haar wavelet, Gabor wavelet, and Leung-Malik filter bank. Furthermore, 2D image location information based on the minimum classification errors of each feature, calculated as the sum of a false acceptance rate (FAR) and a false rejection rate (FRR), can be used by tuning a single threshold, as described in Fig. 5 is shown. This classification error can Classification error(i)=FARi+FRRi

[0028] The information from the trained classifiers is used to classify or weight the feature as either a free path or a non-free path, with the specific classification depending on the strength of comparisons with the trained data. If the feature is the only feature in the piece, the classification can be applied directly to the piece. Classifying a piece with multiple identified features can take many forms, including defining the piece by the feature that most strongly indicates the piece is not free, or defining the piece by a weighted sum of all the features it contains.

[0029] The above procedure can be used to examine a single image 10 and, based on the visual information contained in the image 10, estimate a free path 50. This procedure can be repeated at intervals as the vehicle travels along the road to incorporate new information and extend the formulated free path to an area ahead of the vehicle's new position. The interval must update the image 10 with sufficient frequency to accurately provide the vehicle 100 with a free path on which to travel. However, the interval can also be chosen as a minimum value to adequately control the vehicle without excessively increasing the computational load placed on the processing module 120.

[0030] The detection of a free path can be achieved with a single image, as described above. However, adding a second image, captured close in time to the original, such as sequential frames from a streaming video clip, can improve processing speed and accuracy. A second image allows for a direct comparison with the first, providing updated information regarding the vehicle's trajectory and the movement of detected objects.Changing the perspective of camera 110 also allows for a different analysis of features from the first image: a feature that may not have been clearly visible or was indistinct in the first image may be shown from a different camera angle, may be more noticeable, or may have moved since the first image, giving the classification algorithm an additional way to define the feature.

[0031] Processing a second image relative to the original image 10 can be accomplished by calculating an image difference. If the image difference of a point of interest, such as a feature identified by radar, is not zero, the point can be comprehensively identified as new information. Points with zero image differences can be omitted from the analysis, thus conserving computational resources. Methods for determining an image difference include absolute image intensity difference and vehicle motion-compensated image difference.

[0032] Determining an image difference by calculating an absolute image intensity difference can be used to gather information between two images. An absolute image intensity difference procedure involves determining equivalent image properties between the original and second images to compensate for vehicle movement between the images, superimposing the images, and recording any significant change in intensity between them. A comparison between the images that indicates a change in image intensity in a specific area provides new information. Areas or segments that show no change in intensity can be given less attention during analysis, while the focus can be on areas that show significant changes in intensity, using the aforementioned procedures to analyze segments of one or both captured images.

[0033] Fig. 6A, Fig. 6B and Fig. Figure 6C shows an exemplary determination of an image difference by calculating an absolute image intensity difference according to the disclosure. Fig. 6A shows an original image. Fig. Figure 6B shows a second image with changes compared to the original image. Specifically, the circular shape shown has shifted to the left. A comparison of the two images as shown in Fig. Figure 6C shows an output that, based on a comparison of absolute image intensity differences, identifies an area that has become darker from the first to the second image and another area that has become brighter. Such a procedure can be described as differentiation. An analysis of the comparison provides the information that a change likely occurred in this area of ​​the images as a result of motion or a change in perspective. In this way, an absolute image intensity difference can be used to analyze a pair of sequential images to identify a potentially non-free path.

[0034] Similarly, determining an image difference by calculating a vehicle motion-compensated image difference can be used to capture information between two images. Many methods for calculating a vehicle motion-compensated image difference are known. One exemplary method involves simultaneously analyzing a potential object as both a stationary segment of a free path and a detected object. A probability analysis is performed on features identified simultaneously according to the potential object from both classifications, and the classifications can be compared, for example, using the following logic. Confidence(i) = ProbabilityFreePath(i) − ProbabilityDetectedObject(i) If Confidence > 0, then Piece = FreePath

[0035] In this example comparison, if confidence(i) is greater than zero, the piece containing the feature is classified as a free path. If confidence(i) is less than or equal to zero, the piece containing the feature is classified as a restricted or non-free path. However, different confidence levels can be chosen to classify the piece as a free path. For example, testing might show that false positives are more likely than false negatives, and thus a factor or offset can be introduced.

[0036] Fig. Figure 7 shows a method for simultaneously classifying a feature as a section of a free path and as a detected object, as described above according to the disclosure. Figure 10 includes the object 40, a trapezoidal projection 70, and a rectangular projection 80. This method uses an assumption that projects the object 40 onto the ground within projection 70 as a flat object to test the classification of the feature as a section of a free path. The method also uses an assumption that projects the object 40 onto the ground within rectangular projection 80 as a vertical object to test the classification of the feature as a detected object. Fig. Figure 8 shows comparisons drawn between the two images using the collected data to evaluate the nature of object 40 according to the revelation. Camera 110 observes and captures data of object 40 in the form of a first image at time t1. If object 40 is an actual detected object, the profile of object 40 observed by camera 110 at time t1 corresponds to point 90A. If object 40 is a flat object in the same plane as the ground 20, the profile of object 40 observed by camera 110 at time t1 corresponds to point 90B. Between times t1 and t2, camera 110 moves a certain distance. At time t2, a second image is captured, and information regarding object 40 can be tested by applying an algorithm that considers visible attributes of the object in the second image compared to the first image.If object 40 is an actual detected object extending upwards from the ground 20, its profile will be observed at point 90C at time t2. If object 40 is a flat object in the same plane as the ground 20, its profile will be observed at point 90B at time t2. The comparison derived from the vehicle motion-compensated image difference can directly assign a confidence level by applying classifiers based on the observations of point 90, or the comparison can simply indicate the area where the change is a point of interest.Testing the object with respect to both classifications, as a flat object and as an actual detected object, allows either the area containing object 40 to be identified for further analysis by analyzing a piece as described above, or a direct development of a free path probability and a detected object probability for comparison, as in the logical expression (3) above.

[0037] The information available from the analysis of the second image can be further enhanced by integrating information regarding vehicle motion, such as rotational speed and yaw rate. Vehicle motion information is available from a number of sources, including the vehicle speedometer, vehicle dynamics sensors or wheel speed sensors, anti-lock braking systems, and GPS positioning systems. The algorithms can, for example, combine this vehicle motion information with the data in Fig. 7 and Fig. Use the projections described in section 8 to project angles that should be present for a feature lying flat on the ground into the second image, based on data from the first image and the measured movement of the vehicle between the images.

[0038] The number of images used for comparison need not be limited to two. A multi-image analysis with several iterations can be performed, tracking and comparing an object over a number of cycles. As mentioned above, computational efficiency can be increased by using image difference analysis to identify points of interest and discarding areas with a zero difference in subsequent analyses. Such efficiencies can be leveraged across multiple iterations, for example, by analyzing points of interest identified between the first and second images in the third and fourth acquired images.At one point, a fresh set of images must be compared to ensure that no change has occurred in any of the areas showing a zero difference, such as a moving object appearing on a previously identified free path. The use of image difference analysis and focused analysis, omitting areas identified with a zero change, varies from application to application and can differ between various operating conditions, such as vehicle speed or perceived operating environment. The specific use of image difference analysis and focused analysis can encompass many different embodiments, and the disclosure is not intended to be limited to the specific embodiments described herein.

[0039] This document discloses several methods for identifying a free path, including piece-based and pixel-based methods. These methods are preferably implemented in the processing module 120, but can be combined in one or more units, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. The piece-based methods are described above and herein with respect to Fig. 9 and Fig. 10. The pixel-based methods are described therein in relation to Fig. 11 and Fig. 12. Exemplary pixel-based methods include structureless and structure-rich methods. Structure-rich methods analyze pixel features in an image with respect to certain points of interest based on a context-dependent view of the image. The points of interest are mapped according to a field of view of vehicle 100, and a free path is determined based on the topography of the points of interest in the image. Structureless methods filter out non-conforming areas of an image as not belonging to a planar, consistent roadway; the remaining pixels correspond to the free path. As described above, computation is relatively fast for piece-based methods, while computation is relatively slow for pixel-based methods.

[0040] Fig. Figure 9 shows an exemplary piece-based method 200, in which an input from a camera is analyzed to determine a probability of a free path, according to the disclosure. The piece-based method 200 is described in Fig. Figure 9 shows and is comprehensively described herein as discrete elements. Such a representation serves to simplify the description, and it can be seen that the functions performed by these elements can be combined in one or more devices, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. For example, the piece-based method 200 can be executed in the processing module 120 as one or more algorithms.

[0041] During operation, the camera 110 generates an image for analysis in the processing module 120 (202). The processing module 120 identifies pieces in the image and selects one piece for analysis (204). This disclosure considers several methods for defining a plurality of pieces sufficient to adequately analyze the image. As described above, pieces can be defined according to random search or swarm search methods. Alternatively, information from another information source, such as a radar image acquisition system or LiDAR, can be used to define a piece for analyzing the section of the image. Furthermore, multiple superimposed pieces can be used based on the perceived area of ​​interest in the image to be analyzed.Furthermore, methods can be used to define pieces according to an expected roadway ahead of the vehicle and resulting free-path patterns. These methods include, for example, a fixed-grid piece method that divides part or all of the image according to a regular piece pattern, and a perspective piece method similar to the fixed-grid method, except that the piece sizes and / or shapes are matched based on a perspective view of the road and consideration of a perimeter of the roadway contained within each piece. Such an expected roadway ahead of the vehicle can be assessed as an initial reference for defining a piece pattern by an initial review of the input image, for example, by using clearly visible lines that could define lane boundaries.In another example, a piece pattern from an immediately preceding iteration of the free path procedure could be used and slowly tuned over the iterations, based on the iteratively defined free path and other clues that can be identified in the analyzed images.

[0042] The fixed-grid method identifies and divides the image into multiple pieces based on an area of ​​interest and applies a fixed-grid pattern to the area of ​​interest. The fixed-grid pattern essentially divides the entire area of ​​interest into pieces. The area of ​​interest preferably comprises the field of view immediately in front of the vehicle, although the area may be defined to encompass a narrower field of view. In one embodiment, the area of ​​interest comprises the field of view immediately in front of the vehicle, delimited by a horizon line or vanishing point perceived by the roadway. The fixed-grid method may include piece tracking, identification, and analysis via a matrix, wherein each piece can be identified by a number from a number sequence.

[0043] The fixed grid piece can be applied to the image using a number of methods. A first method involves applying the fixed grid piece to sequential images using the same image coordinates. A second method involves applying the fixed grid piece to the image using an identified point of interest on the image, such as the horizon line. A third method involves applying the fixed grid piece to the image with compensation for vehicle motion, such as vehicle yaw.

[0044] The perspective piece method identifies and divides the image into multiple pieces based on field-of-view coordinates rather than image coordinates. An area of ​​interest is identified as described above, applying a perspective piece pattern based on estimated field-of-view coordinates to the area of ​​interest and allocating piece sizes relative to an approximate ground area covered by each piece. Such perspective piece definitions allow for a more detailed review of pieces describing the ground closer to the vehicle and a less wasteful review of pieces describing the ground farther away. The pieces can be aligned with perceived lane markings on the roadway, defined, for example, by lane markings and / or curb edges.Such piece definitions are often trapezoidal, with the parallel sides of the trapezoids being parallel to the horizon or the vehicle's horizontal axis, and the remaining sides of the trapezoids depending on the defined lane in the image view. Such a piece orientation is efficient for defining the lane. However, calculating the mesh of the pieces and analyzing the pieces is complicated by the many different shapes. The pieces can instead be normalized to rectangular (or square) shapes, still parallel to the horizon or the vehicle's horizontal axis.Such rectangular pieces are in some ways less efficient when covering the roadway, since, for example, sections of the curb edge overlap pieces that actually comprise a clear roadway; however, sufficient resolution of the pieces and improved computational efficiency can make such normalized perspective pieces advantageous.

[0045] Fig. Figure 10 graphically illustrates several normalized perspective pieces applied to a sample image. The size of the perspective pieces varies based on a geographical area assigned to each perspective piece. Fig. Figure 10 shows that the size of pieces assigned to a geographic area near the vehicle is larger than that of pieces assigned to the geographic area further away from the vehicle. Defining pieces on an image according to a geographic area allocates processing resources to the pieces based on that geographic area, thereby increasing object detection capabilities for objects near the vehicle when time for driving decisions is more limited.

[0046] Back on Fig. 9 By reference, a filter or set of filters can be applied to the selected piece (206), e.g., an illumination normalization filter. An exemplary normalization filter changes the range of pixel intensity values ​​in the piece, thereby bringing the piece into a range more suitable for machine processes. For example, normalizing each pixel value to a mean of zero and unit variance improves image contrast, especially in dimly lit environments or when the contrast is poor due to glare. A number of exemplary filters and filtering methods useful for image analysis are considered, and the disclosure is not intended to be limited to the specific exemplary embodiments described herein.

[0047] After filtering, feature extraction is performed on the selected piece using feature extraction algorithms (208). Feature extraction algorithms can analyze the selected piece with respect to predetermined features, e.g., edges, corners, and spheres, and / or shapes, e.g., circles, ellipses, and lines. It should be noted that some features are meaningful and others are not, and that a feature selection process can be used to identify a group of best features for analysis. A classifier training algorithm analyzes each feature and assigns a probability value (210). As mentioned above, classifiers or logic used in generating probability values ​​are initially trained offline. Optionally, training can continue in the vehicle based on fuzzy logic, neural networks, or other learning mechanisms known in engineering.These trained classifiers perform a probability analysis on the extracted features, and a probability value is determined for the item. This probability value expresses a confidence that the selected item is free.

[0048] Probability analysis can be extended using spatially and / or temporally related pieces to evaluate identified features during vehicle operation. Once the feature extraction algorithm has identified a feature, the processing module 120 can spatially analyze the identified feature for consistency among adjacent or nearby pieces. A feature identified in one piece can be compared with surrounding pieces to determine whether it is an anomaly or consistent with them. A feature consistent with surrounding pieces can be assigned a similar probability value, whereas a feature inconsistent with surrounding pieces can be assigned a different probability value.Similar to spatial analysis, once the feature extraction algorithm has identified a feature, processing module 120 can temporarily analyze that feature for consistency across temporarily related frames, compensating for vehicle movement. For example, a feature analyzed across multiple frames that is found to have a high probability value across the frames can be temporarily compared to the same feature that receives a low probability value in a later frame. If the temporary analysis of the feature reveals new information, such as movement of the feature relative to the roadway or an elevated perspective indicating that the feature represents an object on the roadway, the lower probability value for the feature can be confirmed.If no new information is available, the lower probability value for the feature in the present frame can be dismissed as not necessarily indicating a non-free path. Subsequent frames can be analyzed similarly to determine whether the feature is significant or not. Similarly, the feature in question can be analyzed with increased computational effort, either in the present frame or in subsequent frames, using the procedures described herein.

[0049] The processing module 120 compares the probability value with a threshold probability value (214). If the probability value is greater than the threshold, the piece is identified as a free path (218). If the probability value is not greater than the threshold, the piece is identified as a non-free path.

[0050] As described above, the piece-based method 200 can be repeated or iterated in a number of ways, whereby the same image is repeatedly analyzed with the selection and analysis of different pieces, and an identified piece can be tracked and analyzed for change over a number of sequential images.

[0051] Fig. Figure 11 shows an exemplary feature-rich pixel-based method 300, in which an input from a camera is analyzed to determine a probability of a free path, according to the disclosure. The feature-rich pixel-based method 300 is described in Fig. Figure 11 shows and is comprehensively described herein as discrete elements. Such a representation serves to simplify the description, and it can be seen that the functions performed by these elements can be combined in one or more devices, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. For example, the pixel-based method 300 can be executed in the processing module 120 as one or more algorithms.

[0052] During operation, the camera 110 generates an image for analysis in the processing module 120 (302). The processing module 120 analyzes the image with respect to points of interest, for example, by checking a pixel color intensity as described above and comparing the pixel or group of pixels with surrounding pixels. A point of interest is an identifiable pixel in an image and is associated with a set of visual information, i.e., rich features, and is associated with objects located in the field of view. Points of interest can be identified and used to perform the procedures described herein (306) by means of methods known in the art, e.g., scale-invariant feature transformation (SIFT), methods employing corner detection or the detection of other shapes, or a Sobel filter. In one embodiment, redundant points of interest, e.g.,Several points of interest corresponding to the same feature were removed for reasons of computational efficiency.

[0053] Once the points of interest are identified, the processing module compares 120 sequential images as the vehicle moves to match points of interest from each image with corresponding points in the sequential images that correspond to the same points in the field of view, if possible (310). Matching involves using correspondence matching programming, e.g., feature matching, scale-invariant feature transformation (SIFT) programming, and optical flow programming, and may further include locating corresponding points by template matching, taking into account the motion of the host vehicle, and generating a best estimate of whether two points represent the same object or feature visible in the field of view.Matching points to a template can be determined using one of several methods, including one of various known techniques for programming a template match to find the corresponding points of interest, e.g., Lucas-Kanade or Horn-Schunck. The resulting pairs of matched points correspond to the same feature present in both images, with the same feature associated with the same object in the field of view. While points of interest can be matched, not all pairs of matched corresponding points represent high-quality pairs of corresponding points that allow for the identification of their three-dimensional positions in the field of view for classification as a free path for vehicle passage.

[0054] The processing module 120 filters the pairs of matched corresponding points to identify high-quality pairs of corresponding points that can be used for three-dimensional position identification with high confidence (314). The preferred pairs of matched points can be selected based on quality control criteria, such as distance between points, distance from an image boundary, and color similarities between respective adjacent pixels.Furthermore, a selection of criteria for assessing matched pairs may be made on the basis of conditions such as the level of illumination outside the vehicle, the weather, the speed of the vehicle, and any other factors that affect the ability to assess matched pairs or an urgency to define a free path quickly and accurately.

[0055] The pairs of high-quality corresponding points are analyzed to determine the three-dimensional positions of objects in the field of view represented by these pairs (318). It should be noted that corresponding points at different heights relative to ground level move differently between sequential images. Analyzing the movement of the points of interest between sequential images can provide estimated three-dimensional coordinates of these points. The object position can be determined based on the pairs of high-quality corresponding points, the sampling time between sequential images, and vehicle data such as vehicle speed and yaw rate. These triangulation methods can provide the object's position in a horizontal plane and its height relative to ground level.

[0056] The determined object positions are used to map object positions in front of the host vehicle, including an estimated topography of the field of view (322). The topography can be estimated by assigning predetermined spatial dimensions around the object. Preferably, the predetermined spatial dimensions decrease with respect to height at a predetermined rate with respect to the distance from the object. Using the mapping and the estimated topography, the processing module 120 can determine a clear path in front of the host vehicle (326).

[0057] The above procedure uses sequential images to create a mapping of object positions and vertical heights in front of the vehicle, allowing a free path to be defined. It should be noted that in this particular analysis, any two given images might not classify a specific object as comprising two high-quality points of interest sufficient for mapping. However, the above analysis is performed several times per second while the vehicle is moving. As the vehicle progresses along the free path, different perspectives of an object are acquired, and a large number of images are analyzed. Driving along a path and analyzing the multiple iterative images along that path builds confidence that no object contradicting the free path exists on the identified free path.

[0058] Fig. Figure 12 shows an exemplary structureless pixel-based method 400, in which an input from a camera is analyzed to determine a probability of a free path, according to the disclosure. The structureless pixel-based method 400 is described in Fig. Figure 12 shows and is comprehensively described herein as discrete elements. Such a representation serves to simplify the description, and it can be seen that the functions performed by these elements can be combined in one or more devices, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. For example, the structureless pixel-based method 400 can be executed in the processing module 120 as one or more algorithms.

[0059] During operation, camera 110 generates an image for analysis in processing module 120 (453). Processing module 120 analyzes the image using several filtering methods to identify and remove non-conforming pixels. The remaining pixels indicate a potential clear path for the vehicle to pass through. Fig. Figure 12 shows the structureless pixel-based method 400 with four exemplary filtering methods for identifying and removing non-conforming pixels from the image. A structureless method could be used with some of the four exemplary methods and / or could include unnamed but similar methods to process the image.

[0060] A first exemplary filtering procedure removes pixels above a horizon or vanishing point, including the sky and other vertical features that cannot be part of a roadway (456). The term 'vanishing point' as used here is a broad term and represents its common and usual meaning to experts, referring to a point on the horizon at infinity that is intersected by a multitude of parallel lines on the ground in the view. A roadway, which creates a clear path to be driven on, must necessarily be identified below the vanishing point or horizon line. Filtering images only to analyze an area below the horizon line helps to distinguish the pixels to be analyzed for identifying a roadway from irrelevant pixels.An expert will recognize that there are many known methods for determining a vanishing point and a corresponding horizon line.

[0061] A second exemplary filtering method applies a filter based on the variation of pixel color intensity, based on the premise that a roadway covers a large area with a visual intensity that is uniform across the area (459). Pixels are removed from the image when an associated pixel color intensity changes more than a predetermined threshold. For pixels to which multiple colors have been assigned, the pixel color intensity of any specific color that changes more than the predetermined threshold can be removed from the image. The predetermined threshold can be updated based on the color intensity of pixels in the past that have been identified as free.

[0062] A third exemplary filtering method applies a filter based on differentiating sequential images, enabling analysis of changes between the images (462). Pixels assigned a pixel color intensity that differs from the sequential images by more than a predetermined threshold can be removed from the image. In one embodiment, the adjustments to the images are made based on the motion of the vehicle such that pixels are differentiated and compared as if all pixels corresponded to points at ground level. Known triangulation methods can be used to determine motion adjustments to the images. By using the assumption that all objects in the image are at ground level, points that are not at ground level can be identified by changes that are inconsistent with ground-level pixels.For example, pixels above ground level may move faster than expected between sequential images, and this movement can be identified by checking the difference between pixel color intensities between sequential images.

[0063] A fourth exemplary filtering method applies a filter based on identifying pixels that represent edges or transitions in the visual data (465). To generate the fourth filtered image, the processing module 120 extracts pixels from the image based on color intensity values ​​corresponding to the edges, using one of several known edge detection filters, e.g., a Sobel filter. The edge detection filter is preferably implemented in the processing module 120, but can be combined in one or more arrangements, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. In one embodiment, each pixel is analyzed using a Sobel operator.The Sobel operator calculates a gradient vector of color intensity at each pixel, resulting in a direction of greatest possible increase from light to dark and a rate of change in that direction. Points corresponding to a rate of change exceeding a threshold, and corresponding gradient vectors at nearby pixels, define edges and are included in the fourth filtered image. These pixels are included in the image, while the others are removed.

[0064] When the different methods are applied in parallel, the results can be combined into a single image (468). A combination includes pixels identified in each of the different filtering methods. Pixels on the combined free path image correspond to desired driving locations in the field of view. Locations on the combined free path image without pixels correspond to undesired driving locations in the field of view. The processing module 120 can analyze the image for visual data indicating a free driving path (471).

[0065] The in Fig. The 12 described structureless pixel-based methods 400 apply various methods in parallel to images to identify features in a view that are relevant for defining a free path. It should be noted, however, that these methods do not have to be performed in parallel; instead, they can be used to process images sequentially step by step or to selectively identify features in a view that are relevant for defining a free path.

[0066] An addition to a pixel-based method for detecting a free path involves applying a combined rich-structured and unstructured method.

[0067] The rich-structured and structureless methods can be combined in a number of ways. An image with identified points and determined elevations, identified using rich-structured methods, can be overlaid with a filtered image generated by structureless methods, and an agreement of the two methods can be used to define a free path through the overlaid image. In an alternative method for combining the two schemes, the data from both schemes can be used to project information onto a programmed top-down map of an area in front of the vehicle, and this top-down map, incorporating data obtained from an analysis of the two schemes, can include the construction of confidence indicators for areas of the map.In an alternative method for unifying the two schemes, one scheme can be used as the primary or dominant scheme, and the second scheme can be used or activated to analyze areas in the view identified as ambiguous or unclear. In each unifying method, the strengths of one processing scheme can be used to mitigate the weaknesses of the other. If both schemes agree that the path is clear, the processing module employing the schemes can determine with increased confidence that the path is desirable for the vehicle to travel. A number of methods for unifying the identified schemes are considered, and the disclosure is not intended to be limited to the specific embodiments described herein.Furthermore, one or both schemes can be combined with the above-mentioned method, which uses piece analysis.

[0068] An expert will recognize that illumination normalization can be applied to the captured image data. Normalization is a process that modifies the range of pixel intensity values. The purpose of normalization is to bring the image into a range more suitable for a machine process, thereby improving reliability. For example, each pixel value is normalized to a mean of zero and unit variance to improve image contrast, particularly in low-light environments or when contrast is poor due to glare.

[0069] As described above, an example-based method can alternatively or additionally be used to define a free path based on an input image. An example-based method can be described as capturing a number of sample images of views, defining a free path for each of the sample images, matching a current image with one or more of the sample images, and determining a free path based on the match.Matching the current image with one or more of the sample images can be achieved by, for example, extracting features from each of the sample images using the methods described herein, extracting features from the current image, comparing the extracted features from the current image with a database of extracted features from the sample images, and selecting sample images that match the current image. A free path can be selected from the best-matching sample image or can be determined based on a combination of the closest matches with the current image.

[0070] Fig. Figure 15 schematically shows an operation of an exemplary process according to the present disclosure. The example-based process 700 is in Fig. Figure 15 shows and is comprehensively described herein as discrete elements. Such a representation serves to simplify the description, and it can be seen that the functions performed by these elements can be combined in one or more devices, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. For example, the pixel-based method 700 can be executed in the processing module 120 as one or more algorithms.

[0071] Method 700 describes a comparison of a current image with a number of sample or sample images to determine a free path. As described above, a number of sample images are analyzed by feature extraction (704), and the determined features are stored, for example, in an offline training process in a refined database. Method 700 inputs a current image and performs feature extraction on the image (702). Current features of the current image are generated. The current features are used to select one or more matches in the refined database with the current features (706). In this exemplary embodiment, three matching feature sets are identified.The corresponding images of the matches are then compared with the instantaneous image, and a free path for the instantaneous image is determined or derived on the basis of the free path for the matching sample image or images (708).

[0072] A person skilled in the art will recognize that an example-based method such as the one described above can be useful for assessing situations where visual differences between surfaces, such as similar colors or isotropic structures lacking a dominant direction, or less pronounced features in an image, might not yield accurate results from the other exemplary methods described herein. In such cases, the human-defined free paths of the scanned images can provide a priority input for the location of the free path. It should also be noted that ambiguities or weaknesses in camera detection, such as large radial distortion that may be present in some camera images or densely packed buildings in the background, can distort or introduce inaccuracies into the interpretation of images according to other methods described herein.However, such ambiguities or weaknesses in images exist in both the sampled images and the instantaneous image used in the disclosed example-based method, and therefore tend to cancel each other out. Specifically, the example-based method, as described in relation to the hierarchical method disclosed herein, provides an alternative method for processing an image and identifies a free path in cases where a piece-based and / or pixel-based method cannot clearly identify the free path.

[0073] As described above, the example piece-based and pixel-based methods are useful for analyzing a piece or pixel details within a piece to determine whether the piece represents a free path or not. The example-based method, on the other hand, identifies or infers the location of a free path in a snapshot image. When applied to the hierarchical method, the identified free path from the example-based method can be compared to the location of the piece being further analyzed, and the piece can be flagged as free or not based on this comparison.

[0074] This document discloses methods for detecting a free path for a vehicle using hierarchical configurations of free path detection algorithms, arranged based on computational intensity. In this hierarchical configuration, less computationally intensive detection methods identify free paths in the image and leave remaining image segments not identified as free paths for more computationally intensive analysis. This increases the computational efficiency for free path identification. As described above, the specific methods used and the hierarchical structure of the specific hierarchical methods used may differ or change.A first exemplary method for detecting a free path using a hierarchical design comprises the use of a piece-based free path detection method and a pixel-based free path detection method. The piece-based free path detection method first identifies free paths in the image, and the pixel-based free path detection method is used to analyze the remaining portions of the image that were not identified as free paths by the piece-based method. A second exemplary method disclosed herein comprises an additional hierarchical layer. In the second method, a sample-based method further analyzes portions that were not identified as free paths by the pixel-based method.

[0075] Further exemplary embodiments of hierarchical structures are considered. For example, a hierarchical structure can be defined as comprising a piece-based method, then a pixel-based method, and then another pixel-based method. In another example, the hierarchical structure can be defined as comprising a pixel-based method, then a combination of pixel-based methods, and then an example-based method. It should be noted that such a structure, no longer bound to pieces of a piece-based method, could define the image areas to be analyzed by the different layers of the hierarchy in a different way.For example, a pixel-based method could identify a circular area or a cluster of pixels, and then only that area or cluster could be analyzed by a subsequent free-path method. A number of different hierarchical configurations are considered, and the disclosure is not intended to be limited to the specific embodiments described herein.

[0076] Fig. Figure 13 shows a flowchart for a control scheme 500 for detecting a free path for a vehicle, using an exemplary hierarchical configuration of free path detection methods, including a piece-based method and a pixel-based method. The control scheme 500 is shown as a plurality of discrete elements. This representation simplifies the description and illustrates that the functions performed by these elements can be combined in one or more devices, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. For example, the control scheme 500 can be executed as one or more algorithms in the processing module 120.

[0077] During operation, the camera 110 generates an image for analysis in the processing module 120 (502). The images are analyzed using one of the piece-based free-path detection methods described above (504). The processing module 120 identifies several component pieces in the images using the piece-based method. Each component piece is analyzed for features using the piece-based method to determine a confidence probability, as described above (506). As described above, the confidence probability expresses a probability that the component piece is part of a free path for the vehicle.

[0078] After determining the confidence probability value for the multiple component pieces, the processing module 120 compares the confidence probability value with a first and second predetermined threshold to determine whether the component piece is free, not free, or selected for further analysis. Component pieces with a confidence probability greater than a first predetermined threshold are designated as a free path (508). Component pieces that are not greater than the first predetermined threshold but are greater than a second predetermined threshold are selected for further analysis (510).

[0079] Component pieces that are smaller than the second predetermined threshold are designated as not free (520).

[0080] After selecting a subset of the component pieces for further analysis, the processing module 120 analyzes the component pieces using the pixel-based free path detection method (512). The processing module 120 applies a pixel-based free path detection method described above to determine a confidence level, as described above, to determine, based on features identified by the pixel-based method, whether the selected component piece corresponds to a free path or a non-free path (514). It should be noted that pixel-based methods make decisions based on a single pixel or a group of pixels.This decision regarding the piece to be analyzed can be based on pixels contained within the piece, with the piece being determined, for example, to be non-free if any or a minimal number of the pixels in the piece are determined to be non-free. Component pieces assigned a confidence probability value greater than a predetermined threshold correspond to a free path, while component pieces assigned a confidence probability value less than the predetermined threshold correspond to a non-free path (515).

[0081] After analyzing the selected component pieces, a free path can be identified using the component pieces that correspond to a free path (516) and the component pieces that correspond to a non-free path (520).

[0082] Fig. Section 13 describes the use of a first predetermined threshold and a second predetermined threshold to assess whether a given piece is free, not free, or requires further analysis. It should be noted that such a pair of thresholds can instead be described as a range of values ​​that define when the additional analysis should be performed.

[0083] Fig. Paragraph 13 further describes the use of the second, lower predetermined threshold to define pieces that are immediately considered non-free. It should be noted, however, that a similar procedure could be used without the second threshold, in which each piece not identified as free in the first analysis would be further analyzed in the second analysis.

[0084] Fig. Figure 14 shows a flowchart for a control scheme 600 for detecting a free path for a vehicle using a hierarchical design of free path detection methods, including the piece-based method, the pixel-based method, and the example-based method. The control scheme 600 is shown as several discrete elements. This representation simplifies the description and illustrates that the functions performed by these elements can be combined in one or more devices, e.g., implemented in software, hardware, and / or an application-specific integrated circuit. For example, the control scheme 600 can be executed as one or more algorithms in the processing module 120.

[0085] During operation, the camera 110 generates an image for analysis in the processing module 120 (602). The images are analyzed using one of the piece-based free-path detection methods described above (604). The piece-based method identifies multiple component pieces and features within the component pieces in the images. After identifying features within the component pieces, the processing module 120 determines a confidence probability for each component piece, as described above, using probability analysis. Component pieces assigned a confidence probability greater than a first predetermined threshold are referred to as a free path (606).Component pieces that are assigned a confidence probability value that is not greater than the first predetermined threshold, but is greater than a second predetermined threshold, are selected for further analysis (608). Component pieces that are assigned a confidence probability value that is less than the second predetermined threshold are designated as not free (622).

[0086] After selecting a subset of the component pieces for further analysis, the processing module 120 applies a pixel-based free-path detection procedure, as described above, to determine a confidence probability, as described above, to ascertain whether the selected component piece corresponds to a free path or a non-free path (610). The pixel-based procedure identifies features for use in the probability analysis used above to determine confidence values ​​for the component piece. The component pieces that are assigned a confidence probability value greater than a third predetermined threshold correspond to a free path (612).Component pieces assigned to a confidence probability value no greater than the third predetermined threshold but greater than a fourth predetermined threshold are selected for further analysis (614). Component pieces assigned to a confidence probability value less than the second predetermined threshold are designated as non-free (622).

[0087] After selecting the pixels, the processing module 120 analyzes the pixels using an exemplary example-based free-path detection procedure and determines a confidence probability (616). Component pieces that are assigned a confidence probability value greater than a predetermined threshold correspond to a free path (618). Component pieces that are assigned a confidence probability value less than the second predetermined threshold are designated as not free (622).

[0088] After analyzing the selected component pieces, a free path can be identified using the component pieces and pixels that correspond to a free path (620) and / or the component pieces and pixels that correspond to a non-free path (622).

[0089] Fig. Section 14 describes a series of thresholds, and a series of numerical designations (first, second, third, etc.) are assigned to the thresholds. It should be noted that these designations are assigned solely for the purpose of indicating that the thresholds are potentially different from one another. The upper and lower thresholds may be different or the same for any given comparison. The assigned designations in Fig. However, 14 are for illustrative purposes only, and no meaning should be implied by the numerical designations outside of the potential distinction.

[0090] The hierarchical configurations or procedures described above define the different levels of the hierarchy in the various embodiments considered, based on computational or processing complexity, with a preference for less intensive methods over more intensive ones. It should be noted, however, that computational intensity is not the only consideration that can be used to define the specific method employed at each level of the hierarchy. For example, the hierarchy of Fig. 14. Under normal conditions, the procedure is used, first employing a piece-based method, then a pixel-based method, and finally an example-based method. However, the order of the methods used, or the exclusion or substitution of any of the methods, can be implemented based on perceived factors. For example, driving on a dirty or snow-covered road may produce particularly poor results for one of the methods. Similarly, an urban environment with rapidly changing traffic conditions or pedestrian traffic may produce particularly poor results for one of the methods. Contamination of the camera image caused by a smudge on the camera lens may similarly lead to poor results from one of the methods. Vehicle speed may also affect the quality of the results from one of the methods.Such factors can be determined experimentally or computationally, or assessed based on modeling or other methods sufficient to accurately estimate the effectiveness of different free-path methods for determining a free path based on the input image. Furthermore, methods such as machine learning algorithms or fuzzy logic can be implemented to select the appropriate free-path methods for use in the hierarchical process, based on the vehicle's internal use of these methods and the resulting vehicle operation.Such operation of the vehicle can be assessed in many ways known in the art, such as monitoring for the occurrence of operator intervention in the vehicle's control, or by monitoring the location of lane markings in images, which describes the subsequent control of the vehicle on the roadway. A number of methods for selecting between free-path procedures and for adapting the selection of the various free-path procedures are considered, and the disclosure is not intended to be limited to the specific exemplary embodiments disclosed herein. As in . Fig. 13 and Fig.Figure 14 shows a number of threshold confidence probabilities for comparison with confidence probabilities developed at each layer of the hierarchical procedures. It should be noted that these thresholds can each be distinct and unique, and can be described as the first threshold, second threshold, third threshold, etc. Some of the thresholds may be the same, with, for example, identical thresholds defining when the particular piece is a free path. The exact levels of each threshold depend on the free path procedures used to analyze the piece and can be developed experimentally or by any method sufficient to estimate or evaluate the operation of the image processing techniques employed.

[0091] As mentioned above, the processing module 120 can include algorithms and mechanisms for actuating an autonomous driving control system by means known in the art and not described herein, or the processing module 120 can simply provide information to a separate autonomous driving system. The responses to perceived objects can vary and include, without limitation, steering inputs, throttle changes, braking responses, and warning the vehicle operator and transferring control of the vehicle to the operator.

Claims

[1] Method for detecting a free travel path (50) for a vehicle (100) using at least one image (10) produced by a camera (110), the method comprising that the image (10) is being monitored; a first piece-based analysis (504, 604) of the image (10) is used, which includes that the image (10) is divided into a large number of pieces (60) sufficient to adequately analyze the image (10), using several overlapping pieces (60); a feature identification algorithm is applied to the identified pieces (60); and each piece (60) is analyzed to determine for each of the pieces (60) of the first piece-based analysis (504, 604) a first confidence probability of a free path (50); Each of the pieces (60) where the first confidence probability of a free path (50) is greater than a first threshold confidence probability is defined as a piece (60) of a free path (50); A second analysis on a subset of the pieces (60) where the first confidence of a free path (50) is not greater than the first threshold confidence probability is used to determine a second confidence probability of a free path (50) for each of the pieces (60) of the second analysis; Each of the pieces (60) of the second analysis where the second confidence probability of a free path (50) is greater than a second threshold confidence probability is defined as a piece (60) of a free path (50); and the free path (50) for the vehicle (100) is identified on the basis of the segments (60) of a free path (50). [2] The method of claim 1, further comprising comparing the first confidence probability of a free path (50) for each of the pieces (60) of the first piece-based analysis (504, 604) with a third threshold confidence probability; and where the use of the second analysis further includes applying the second analysis to a subset of the pieces (60) of the first piece-based analysis (504, 604) where the first confidence probability of a free path (50) is greater than the third threshold confidence probability, preferably further comprising that each of the pieces (60) of the first piece-based analysis (504, 604) where the first confidence probability of a free path (50) is not greater than the third threshold confidence probability is defined as a piece (60) of a non-free path. [3] The method of claim 1, further comprising that each of the pieces (60) of the second analysis, in which the second confidence probability of a free path (50) is not greater than the second threshold confidence probability, is defined as a piece (60) of a non-free path. [4] Method according to claim 1, wherein the second analysis an example-based free path method (50) and / or a pixel-based method (512, 610) of a free path (50) includes, wherein the pixel-based method (512, 610) of a free path (50) preferably a structure-rich free path procedure (50) and / or a structureless free path procedure (50) includes. [5] The method of claim 1, further comprising that a third analysis on the pieces (60) of the second analysis where the second confidence probability of a free path (50) is not greater than the second threshold confidence probability, is used to determine a third confidence probability of a free path (50) for each of the pieces (60) of the third analysis; and Each of the pieces (60) of the third analysis where the third confidence probability of a free path (50) is greater than a third threshold confidence probability is defined as a piece (60) of a free path (50). [6] The method of claim 5, further comprising comparing the second confidence probability of a free path (50) for each of the pieces (60) of the second piece-based analysis (504, 604) with a fourth threshold confidence probability; and where the use of the third analysis further includes applying the third analysis to a subset of the pieces (60) of the second piece-based analysis (504, 604) where the second confidence probability of a free path (50) is greater than the fourth threshold confidence probability, preferably further comprising that each of the pieces (60) of the second piece-based analysis (504, 604) where the second confidence probability of a free path (50) is not greater than the fourth threshold confidence probability is defined as a piece (60) of a non-free path. [7] The method of claim 5, further comprising that each of the pieces (60) of the third analysis, in which the third confidence probability of a free path (50) is not greater than the third threshold confidence probability, is defined as a piece (60) of a non-free path. [8] Method according to claim 5, wherein the third analysis is a pixel-based method (512, 610) of a free path (50) on the part of the pieces (60) and / or an example-based procedure (616) of a free path (50) on the part of the pieces (60) comprises. [9] Method for detecting a free travel path (50) for a vehicle (100) using at least one image (10) produced by a camera (110), the method comprising that the image (10) is being monitored; a first piece-based analysis (504, 604) of the image (10) is used, which includes that the image (10) is divided into a large number of pieces (60) sufficient to adequately analyze the image (10), using several overlapping pieces (60); a feature identification algorithm is applied to the identified pieces (60); and each piece (60) is analyzed to determine for each of the pieces (60) of the first piece-based analysis (504, 604) a first confidence probability of a free path (50); Each of the pieces (60) where the first confidence probability of a free path (50) is greater than a first threshold confidence probability is defined as a piece (60) of a free path (50); a pixel-based analysis (512, 610) on a subset of the pieces (60) where the first confidence of a free path (50) is not greater than the first threshold confidence probability is used to determine a second confidence probability of a free path (50) for each of the pieces (60) of the second analysis; Each of the pieces (60) of the pixel-based analysis (512, 610) where the second confidence probability of a free path (50) is greater than a second threshold confidence probability is defined as a piece (60) of a free path (50); the free path (50) for the vehicle (100) is identified on the basis of the segments (60) of a free path (50); and The free route (50) is used to navigate the vehicle (100). [10] Method for detecting a free travel path (50) for a vehicle (100) using at least one image (10) produced by a camera (110), wherein the method comprises that the image (10) is being monitored; a first piece-based analysis (504, 604) of the image (10) is used, which includes that the image (10) is divided into a large number of pieces (60) sufficient to adequately analyze the image (10), using several overlapping pieces (60); a feature identification algorithm is applied to the identified pieces (60); and each piece (60) is analyzed to determine for each of the pieces (60) of the first piece-based analysis (504, 604) a first confidence probability of a free path (50); Each of the pieces (60) where the first confidence probability of a free path (50) is greater than a first threshold confidence probability is defined as a piece (60) of a free path (50); an example-based analysis (616) on a subset of the pieces (60) where the first confidence of a free path (50) is not greater than the first threshold confidence probability is used to determine a second confidence probability of a free path (50) for each of the pieces (60) of the second analysis; Each of the pieces (60) of the example-based analysis (616) where the second confidence probability of a free path (50) is greater than a second threshold confidence probability is defined as a piece (60) of a free path (50); the free path (50) for the vehicle (100) is identified on the basis of the segments (60) of a free path (50); and The free route (50) is used to navigate the vehicle (100).

Images