Image processing method and apparatus implementing the same

Abstract

An image processing method and a device configured to implement the image processing method are disclosed. The method comprises: obtaining image data comprising temporally consecutive image frames from an imaging device; performing feature extraction on each of the obtained image frames; dynamically maintaining the extracted feature data of the obtained image frames in a feature accumulation database by adjusting the data maintained in the feature accumulation database to a selective subset of the extracted feature data from the obtained image frames; performing a random sample consensus (RANSAC) operation on the selective subset of the extracted feature data from the feature accumulation database; and generating an estimated model from an output of the RANSAC operation based on at least one of extracted feature data of a current image frame or extracted feature data of one or more temporally preceding image frames of the obtained image frames.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to image processing methods and apparatuses implementing image processing methods, and more particularly to an image processing implementation that performs selective estimation processing on a set of obtained image data, thereby conserving computational resources while enhancing estimation stability. BACKGROUND

[0002] Random Sample Consensus (RANSAC) is currently applied in computer vision and automotive related applications. RANSAC is an iterative method that is suitable for estimating parameters of a mathematical model from a set of observations that contain outliers. In computer vision, this set of observations is usually a set of features extracted from an image. In automotive related applications, RANSAC can be used to estimate lane vanishing points, driving direction vanishing points, camera rotation angles, ground planes, stereo camera calibration parameters, etc. In such applications, the stability of RANSAC estimation is largely affected by the content of the road scene. In some scenarios, we can only extract a small number of features from a single self-captured frame, and the feature content can not be sufficient for a stable RANSAC estimation. SUMMARY BRIEF DESCRIPTION OF DRAWINGS

[0003] In order that the manner in which the above-recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, can be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this disclosure and are therefore not to be considered limiting of its scope, for the disclosure can admit to other equally effective embodiments.

[0004] Figure 1 is a schematic diagram of an exemplary image processing framework according to some embodiments of video stream data estimation.

[0005] Figure 2 is another schematic diagram of an exemplary image processing framework for video stream data estimation according to some embodiments of the present disclosure.

[0006] Figure 3 shows an exemplary dynamic feature accumulation process for an image processing method according to some embodiments of the present disclosure.

[0007] Figure 4 shows an exemplary implementation of a dynamic feature accumulation process according to some embodiments of the present disclosure.

[0008] Figure 5 shows an exemplary key frame selection process for an image processing method according to some embodiments of the present disclosure.

[0009] Figure 6A photograph image of exemplary road scene data captured by an image processing system according to some embodiments of the disclosure is shown.

[0010] Figure 7 A photograph image of exemplary road scene data captured by an image processing system according to some embodiments of the disclosure is shown.

[0011] Figure 8 A photograph image of exemplary road scene data captured by an image processing system according to some embodiments of the disclosure is shown.

[0012] Figure 9 A photograph image of exemplary road scene data captured by an image processing system according to some embodiments of the disclosure is shown.

[0013] Figure 10 A flowchart of an exemplary image processing method according to some embodiments of the disclosure is shown. DETAILED DESCRIPTION

[0014] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the disclosure are shown. The disclosure may, however, be embodied in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like reference numerals refer to like elements throughout.

[0015] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," or "includes" and / or "including" when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0016] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0017] In the pursuit of further autonomy for automotive applications, various advanced driver assistance systems (ADAS) have been developed with the aim of achieving vehicle automation and enhancing driver safety and driving experience. Most driver assistance systems act based on information collected around the vehicle's position, orientation, and motion state to assist the driver in various ways. For example, in recent years, camera (image sensor) based driver assistance systems (DAS) have proliferated in the market.

[0018] Images obtained by on-board sensor systems can be used to provide driving support and vision support. For example, based on images captured by on-board cameras, the position and orientation of the vehicle can be detected and used for guidance information. Also, oncoming vehicles, vehicles ahead, pedestrians, obstacles, etc. can be detected to avoid accidents. Additionally, in low observable driving conditions, images captured by on-board cameras can be processed to sense the headlamps of oncoming vehicles (or rear vehicles) and the taillights of vehicles ahead, generating useful information about other surrounding vehicles and helping the driver to increase awareness. The functions of this driver assistance system can include lane departure warning (LDW), automatic high beam control (AHC), traffic sign recognition (TSR), forward collision warning (FCW), and pedestrian detection.

[0019] In some applications, such as visual odometry, image processing techniques can be used to determine the position of a vehicle. In a typical application, various types and configurations of image sensors are used to capture image input. Image rectification can then be applied, and then predefined features can be identified / extracted from the captured image data. The extracted features can then be matched across a sequence of captured image frames, and an optical flow field can be constructed, e.g., by using correlation. Correspondence between time-sequential images can then be established, e.g., by feature extraction and correlation, or by using the Lucas-Kanade method. Tracking errors can be detected and corresponding outliers can be compensated for / dropped, while camera motion can be estimated from the optical flow, e.g., by using a Kalman filter or by minimizing a cost function based on the geometric properties of the features. In some applications, camera calibration for orientation parameters of the image sensor system, e.g., its tilt angle and horizontal angle, can be performed using detected parallel lines and corresponding vanishing points. In some applications, the detection of vanishing points can be achieved by using projection on a normalized unit sphere.

[0020] Among other algorithms suitable for ADAS applications, random sample consensus (RANSAC) is an iterative algorithm designed to estimate parameters of a mathematical model from a set of observations that contain outliers. In computer vision applications, such observations can include a set of features (e.g., feature points, line segments) extracted from a captured image frame (e.g., a single or a series of image frames). In automotive-related applications, RANSAC processing can be used to estimate lane vanishing points, driving direction vanishing points, camera rotation angles, ground planes, stereo camera calibration parameters, etc.

[0021] By way of example, reference is made to Figure 7 which shows a photo image of exemplary road scene data from an image processing system in accordance with some embodiments of the present disclosure. In the illustrated photo image, an image sensor system mounted on the dashboard captures an image frame showing the host vehicle moving along traffic in a suburban residential community.

[0022] In the exemplary road scene image, RANSAC estimation of the driving direction vanishing point from the optical flow is visualized as an arrow DD in the center region of the image. By way of example, the arrow DD indicating the optical flow substantially follows the backward relative motion of the tree top leaves. In addition, RANSAC estimates the lane vanishing point (LVP) from extracted lane line segment data at a lower center location of the image (e.g., by estimating line segment intercepts).

[0023] By dynamically applying RANSAC processing to feature data extracted from a spontaneously captured sequence of image frames, valuable navigation information can be obtained that can be further used to implement valuable driving assistance functions.

[0024] Figure 1 is a schematic diagram of an exemplary image processing framework in accordance with some embodiments of the present disclosure. By way of example, Figure 1 An exemplary image processing framework 10 utilizing random sample consensus (RANSAC) algorithms and Kalman filters for road scene video stream data estimation in accordance with some embodiments of the present disclosure is schematically illustrated.

[0025] Figure 1 The exemplary image processing framework of employs RANSAC processing on features extracted from each individual frame from a road scene video stream. By way of example, the exemplary electronic image processing framework begins with an image acquisition module P11 that operates to obtain image data including temporally consecutive image frames from an imaging device. In some embodiments, the image acquisition operation can be implemented by utilizing an on-board image sensing device (e.g., a forward looking camera).

[0026] A feature extraction module P12 is then provided for performing a feature extraction operation on each of the obtained image frames in the obtained data sequence. In some embodiments, the feature extraction module P12 is implemented by a processing device, which can incorporate necessary hardware circuitry, software or firmware components configured to perform a feature extraction operation on each of the captured time-sequential image frame sequences.

[0027] An estimation module P13 is then incorporated for applying a RANSAC algorithm to the features extracted from each of the obtained image frames. In some embodiments, the estimation module P13 is implemented by a processing device, which can incorporate necessary hardware circuitry, software or firmware components configured to perform a RANSAC estimation operation on the extracted feature data corresponding to each of the captured image frames.

[0028] The exemplary framework 10 is then incorporated with a model generation module P14, in which an estimated model corresponding to each of the obtained image frames can be generated. In some embodiments, the model generation module P14 is implemented by a processing device, which can incorporate necessary hardware circuitry, software or firmware components configured to generate an estimated model based on the output from the RANSAC estimation operation.

[0029] In the context of the present disclosure, "feature" can refer to a 2D image point, a 3D image point (e.g., obtained by a time-of-flight / TOF camera or some scanner), or a line segment, etc. observed / identified in the obtained image frames. The term "feature extraction" can refer to a line detection, an optical flow estimation, or a 3D scanning of a scene, etc. process performed on the obtained image frame data. For example, in some embodiments, a parallel line of one vanishing point can correspond to a lateral boundary of a street. In some embodiments, a parallel line can correspond to a demarcation on a street. The term "weight" can refer to an importance value assigned to each of the extracted features, which can represent a length of a line segment or an optical flow matching measure. In some embodiments, the weight is used for RANSAC voting. In addition, the term "estimated model" can refer to a predefined model estimation of a vanishing point of a lane, some extrinsic / intrinsic camera parameters, or a 3D plane equation of a ground, etc.

[0030] In Figure 1In the illustrated embodiment, the exemplary image processing framework 10 performs a RANSAC estimation process on feature data extracted from each of the captured image frames. In some situations, this single frame-by-frame estimation scheme can be less than ideal. On one hand, experiments have shown that the actual number of iterations required to obtain a solution during the RANSAC estimation operation is typically an order of magnitude greater than the estimated number of iterations. Thus, for dynamic high frame rate applications, the computational resources required for this single frame processing scheme will be quite substantial. On the other hand, the stability of the RANSAC estimation for each obtained frame is largely dependent on the content of the scene. In situations where the scene from a single captured frame contains only a few extractable features, the limited number of useful features can not be sufficient to provide adequate stability in the single frame RANSAC estimation scheme. Thus, the reliability of the generated estimation model can be compromised due to the lack of useful scene content.

[0031] Figure 2 is another schematic diagram of an exemplary image processing framework for video stream data estimation in accordance with some embodiments of the present disclosure. For example, Figure 2 An image processing framework 20 is schematically illustrated that utilizes a random sample consensus (RANSAC) algorithm along with additional process modules designed to improve processing efficiency and increase estimation model stability.

[0032] The illustrated embodiments introduce a framework for dynamic feature accumulation referred to as the time to cross a RANSAC model estimation (tRANSAC) that is designed to dynamically accumulate features across time so that RANSAC can be stably performed with a large number of features accumulated over a longer time period. To implement the newly proposed framework, functional modules are integrated into the RANSAC framework.

[0033] The exemplary electronic image processing framework 20 begins with an image acquisition module P21 that operates to obtain image data from an imaging device that includes temporally consecutive image frames. A feature extraction module P22 can be implemented by a processing device that includes a feature extraction component (a) that can include necessary hardware circuitry, software or firmware components configured to perform a feature extraction operation on each of the obtained image frames in the obtained data sequence. An estimation module P23 is then provided to perform a RANSAC process on a selected subset of the feature data extracted from the obtained image frames. The framework then proceeds to a model generation module P24 in which an estimation model corresponding to the subset of the obtained image frames can be generated.

[0034] Embodiments of the present disclosure do not perform RANSAC processing directly on features extracted from each individual frame, but rather seek to dynamically collect features extracted from captured frames in a dynamic cumulative feature pool, which is then selectively forwarded for RANSAC processing. In the present embodiments, a dynamic cumulative feature pool (module C) is interposed between the feature extraction module P22 and the estimation module P23, which operates to dynamically maintain feature data for obtained image frames. In some embodiments, the dynamic cumulative feature pool C is implemented by a feature accumulation database (c), which can include a data storage component configured to selectively maintain extracted feature data from obtained image frames. For example, in some embodiments, the feature pool module C can be adjusted to maintain a selective subset of extracted feature data from a sequence of obtained image frames.

[0035] In the illustrated embodiments, a keyframe selection module F is arranged downstream between the dynamic cumulative feature pool C and the RANSAC estimation module P23. In some embodiments, the keyframe selection module F is implemented by a processing device, which can include a keyframe selection component (f) that, in conjunction with necessary hardware circuitry, software or firmware components, and operates to compare and determine whether newly extracted features from a currently obtained image frame are consistent with an existing RANSAC estimation model generated based on previously obtained feature data. In some embodiments, the keyframe selection module F is configured to selectively forward new feature data from a current image frame to the RANSAC estimation module P23. In some embodiments, the keyframe selection module F is configured to forward new feature data for further RANSAC processing when a newly acquired frame contains feature data that is no longer consistent with a previously computed RANSAC model. In some cases, when there is a significant change in the scene from a previous image frame to a subsequent frame, the newly obtained current feature data is no longer suitable for a previously generated RANSAC model. For example, when a vehicle-mounted camera system captures a significant scene change when the host vehicle is driving down a hill and there is otherwise smooth highway cruising.

[0036] For example, a sequence of image data captured during highway cruising can be monotonous and predictable, especially when driving on a straight road with open scenery that remains largely unchanged. Thus, a corresponding estimation model generated from such a steady sequence of image data will largely remain consistent. Therefore, from a resource management perspective, it makes little sense to spend valuable computing and storage resources on such consistent (repetitive / redundant) feature data. In contrast, when the host vehicle enters a ramp and exits the highway, the significant change in scenery indicates that the vehicle is about to enter a new, unknown environment. Thus, a corresponding estimation model generated from image data with significant scenery changes (e.g., a noticeable change in extracted features) can no longer be consistent with a previous estimation model generated based on a previous sequence of images whose extracted features are significantly different from the extracted features of the current frame.

[0037] This implication of consistency between current feature data and a previously generated estimation model would be meaningful because it can indicate that the model needs to be updated. In some embodiments, RANSAC processing of feature data associated with more steady frames can be skipped, thereby saving computing resources. In some embodiments, only when a newly acquired frame contains feature data that is no longer consistent with a previously computed RANSAC model (e.g., an unsteady frame) will the features of the new frame be forwarded for RANSAC processing. In some embodiments, the keyframe selection module F operates to identify such inconsistent frames in the time-continuous image data and designate them as keyframes.

[0038] The integration of the feature pool module C and the keyframe selection module F into the estimation module P23 provides a selective buffer and enables it to perform RANSAC operations on a subset of the extracted feature data from the feature accumulation database (c). Furthermore, with the improved processing scheme of the exemplary framework 20, the model generation module P24 can generate an estimation model from the output of the estimation module P23 based on more than a single image frame’s feature data, thereby facilitating higher stability of model generation. In some embodiments, an estimation model can be generated based on at least one of the extracted feature data of a current image frame (e.g., a keyframe) or the extracted feature data of one or more temporally preceding image frames (e.g., one or more selectively accumulated frames preceding the keyframe).

[0039] In the illustrated embodiment, a model estimate quality measurement module E is further introduced downstream of the model generation module P24. In some embodiments, the model estimate quality measurement module E is implemented by processing means which can include a model quality measurement component (e) which incorporates necessary hardware circuitry, software or firmware components and operates to measure the quality (e.g., consistency) of the estimate model currently employed. In some embodiments, the model estimate quality measurement module E determines the quality of the estimate model by evaluating a quality variable (q) which can be a function of parameters such as RANSAC support votes, total number of features used for RANSAC processing, and current model residual error, among others. In some embodiments, the results of the model quality measurement module E are used as a determining parameter for the scheduling of the dynamic data retention policy / dynamic accumulated feature pool C.

[0040] The exemplary framework 20 further incorporates a dynamic feature accumulation control module B interposed between the dynamic accumulated feature pool C and the model estimate quality measurement module E. The dynamic feature accumulation control module B can be provided to regulate data accumulation in the feature accumulation database (c) based on output from the model estimate quality measurement module E. In some embodiments, the dynamic feature accumulation control module B is implemented by processing means which can include a dynamic feature accumulation controller (b) which incorporates necessary hardware circuitry, software or firmware components and operates to control data access and retention scheduling in the feature accumulation database (c). In some embodiments, the control module B can operate to dynamically control retention of extracted feature data from each obtained image frame, adjusting the data retention in the feature accumulation database (c) to a selective subset of extracted feature data from a continuous sequence of obtained image frames.

[0041] Accordingly, based on the control of the control module B, the extracted feature data from the newly obtained frame is initially added to the feature accumulation database (c). The updated feature data obtained subsequently in the pool is used to check the consistency with the previously generated RANSAC model. In some embodiments, only when the consistency is found to be below a certain threshold, the corresponding new frame is considered as a key frame, and the current accumulation in the feature accumulation database (c) is forwarded to the estimation module P23 for RANSAC estimation. Accordingly, when the RANSAC estimation tends to perform stably (e.g., when the newly extracted feature data from the currently obtained image frame is consistent with the previously generated RANSAC estimation model), the accumulated feature data in the feature accumulation database (c) can be discarded at a higher rate so as to keep less redundant features and use them for RANSAC estimation. Conversely, when the RANSAC estimation is found to perform poorly, the accumulated features in the feature accumulation database (c) are adjusted to be discarded at a slower rate so that more features can be accumulated and used to enhance the stability of the RANSAC estimation.

[0042] Reference is made in advance Figure 8 which shows a photo image of exemplary road scene data processing according to some embodiments of the present disclosure. For example, Figure 8 The image in (A) illustrates the result of the travel direction vanishing point estimation using the line segment features.

[0043] In particular, Figure 8 The photo image (A) illustrates the result of the RANSAC estimation using the feature data extracted from a single captured image frame. It can be observed that with less available features extracted from a single captured frame, the result accuracy of the RANSAC estimation will be less stable. For example, the estimated travel direction vanishing point Dl in the photo image (A) inaccurately falls on the left side across the opposite lane of the road. In contrast, Figure 8 The photo image (B) illustrates the result of the RANSAC estimation using the feature data extracted from a set of accumulated image frames. It can be observed that with more available features accumulated from a set of more than one captured frames, the result accuracy of the RANSAC estimation can be improved. In this case, the estimated travel direction vanishing point D2 in the photo image (B) more accurately reflects the travel direction along the road.

[0044] Reference is made in advance Figure 9 which shows a photo image of exemplary road scene data processing according to some embodiments of the present disclosure. For example, Figure 9 The image in (A) illustrates the result of the travel direction vanishing point estimation using the line segment features.

[0045] In particular, Figure 9Photo image (A) illustrates the result of RANSAC estimation using line segment features extracted from a single captured image frame. Due to the lack of sufficient feature volume in a single image frame, a correct roll angle estimation cannot be obtained. In contrast, Figure 9 Photo image (B) illustrates the result of RANSAC estimation using line segment features extracted from a cumulative series of image frames. It is observed that as more available features are accumulated from a series of captured frames, a proper roll angle estimation can be achieved, enabling the generation of a rectified image (as shown in the lower right photo).

[0046] Referring back to Figure 3 which demonstrates an exemplary keyframe selection process for an image processing method according to some embodiments of the present disclosure. For instance, Figure 3 demonstrates a schematic flow of keyframe selection operation from a dynamic cumulative feature pool (c) through a keyframe selection component (f) to a tRANSAC processing component (d).

[0047] In some embodiments, performing RANSAC operation on a selective subset of extracted feature data can include determining whether a newly obtained image frame constitutes a keyframe, e.g. by a keyframe selection component (f), in the obtained image frames; and subsequently forwarding the extracted feature data of the keyframe from the feature pool (c) to the tRANSAC processing component (d) for selective RANSAC processing.

[0048] Depending on the stability of the tRANSAC estimation result, in some embodiments, cumulative feature data preceding the feature data of a keyframe can be further forwarded from the feature pool (c) to the tRANSAC processing component (d) for selective RANSAC estimation. In some embodiments, the keyframe determination process includes performing a consistency check on the extracted feature data from a current image frame against an estimated model that is temporally preceding. In some embodiments, a current image frame is designated as a keyframe when a consistency level of the extracted feature data of the current image frame is below a consistency threshold. In some embodiments, upon designating a currently captured image frame as a keyframe, the extracted feature data of the keyframe is kept in the feature pool (c) for a longer time duration. In contrast, when a newly captured image frame is not designated as a keyframe, its extracted feature data can be kept in the feature pool (c) for a shorter time period.

[0049] Please also refer to Figure 4 and Figure 5 . Figure 4 demonstrates an exemplary implementation of a dynamic feature accumulation process according to some embodiments of the present disclosure. Figure 5 demonstrates an exemplary dynamic feature accumulation process for an image processing method according to some embodiments of the present disclosure.

[0050] For example, Figure 4 An exemplary image processing flow of captured image frame data from feature extraction component (a) of the feature extraction module to the feature accumulation database (c) of the feature pool module, with selective retention of feature data for further RANSAC processing. The selectively accumulated feature data is then forwarded to the processing component (d) of the estimation module for tRANSAC computation, which outputs for generating an updated estimate model. The generated estimate model is then fed to the model quality measurement component (e) of the model estimate quality measurement module for consistency evaluation. The results of the model quality evaluation are then fed to the dynamic feature accumulation controller (b) of the dynamic feature accumulation control module as the basis for determining the dynamic data retention policy / schedule of the feature accumulation database (c).

[0051] In some embodiments, the model quality measurement component (e) of the model estimate quality measurement module is configured to measure the current model estimate quality. The determination of the current model quality can be based on a quality measurement variable q, which can be a function of the quality variable q, the RANSAC support votes, the total number of features used for RANSAC, and the current model residual error, among others.

[0052] For example, the dynamic feature accumulation control module (b) can operate to regulate access into / out of the feature accumulation database (c) of, for example, the faucet and sink. For example, the control module (b) can be configured to allow new features from new frame data to flow into the feature pool, and let old feature data to be gradually phased out from the feature pool. In some embodiments, the control of the faucet and sink relies on a dynamic accumulation control variable d, which can be a function of the model estimate quality variable q.

[0053] In some embodiments, the dynamic feature accumulation control module (b) can be configured to take the quality measurement variable q from the model quality measurement component (e) as input, and adjust the dynamic accumulation control variable d. In some embodiments, the dynamic accumulation control variable d is set as

[0054] d = f c (q),

[0055] where f c is a monotonically decreasing function.

[0056] In some embodiments, the function is arranged such that the better the model estimate quality (e.g., higher q value), the smaller the control variable d. In some embodiments, it is arranged such that the smaller d is, the less feature data tends to be accumulated over time to reduce the RANSAC computation cost. Conversely, the larger d is, the more feature data tends to be accumulated over time to improve the RANSAC model estimate stability.

[0057] In some example image processing methods of the present disclosure, for each generated estimate model, a model quality measurement variable q is generated for evaluating the estimate model quality. In some embodiments, the model quality measurement variable q comprises a function of RANSAC support votes, total number of features used for RANSAC, and current model residual error, among others. In some embodiments, dynamic retention of extracted feature data in the feature accumulation database (c) can be performed by adjusting a discard rate of retained extracted feature data in the feature accumulation database based at least in part on the model quality measurement variable q. In some embodiments, adjusting the discard rate of the feature accumulation database can comprise generating a control variable d with a value not greater than 1 by inputting the model quality measurement variable q into a dynamic accumulation control function (e.g., d = fc(q)), wherein the dynamic accumulation control function comprises a monotonically decreasing function.

[0058] In some embodiments, example image processing methods of the present disclosure can be further arranged such that the control variable d is set to a value less than 1 when the dynamic accumulation control function satisfies a predetermined first quality threshold (e.g., T1). Thus, in some embodiments, a smaller value of the control variable d affects a reduction in the retention volume of extracted feature data in the feature accumulation database (c). In some embodiments, reducing the retention volume in the feature accumulation database (c) can be performed by reducing the current data accumulation weight factor by a factor of d, and removing retained feature data in the feature accumulation database (c) with a weight factor less than a predetermined second quality threshold (e.g., T2). In some embodiments, the dynamic accumulation control function is specified to affect an inverse relationship between the estimate model quality q and the control variable d.

[0059] In one example embodiment, the dynamic accumulation control variable d is set to

[0060]

[0061] where σ is proportional to the model quality measurement variable q.

[0062] With this arrangement, the dynamic accumulation control variable d, which is a continuous monotonically decreasing function of q, serves as a feature weight temporal decay factor. For example, the greater the value of q, the smaller d, so that older feature data is discarded faster. It should be noted that while the limited length of the present disclosure allows for positively introducing only a few feasible examples, other arrangements of the above functions and variables can also be applicable in some embodiments.

[0063] Figure 6An exemplary image processing system configured to perform an image processing method according to some embodiments of the present disclosure is shown. For example, the exemplary image processing system is shown configured to receive a stream of image data from an imaging system mounted on a dashboard (e.g., system 61) and output a computed result to an on-board image display (e.g., system 62).

[0064] In the illustrated embodiment, the exemplary image processing system includes a processing device 60 configured to obtain image data including temporally successive image frames from an imaging device. The processing device 60 can include a feature extraction component (6a) configured to receive image data from the imaging device and extract feature data from each of the obtained image frames, a feature accumulation database (6c) in data communication with the feature extraction component (6a) configured to selectively maintain the extracted feature data, a random sample consensus (RANSAC) processing component (6d) configured to apply a RANSAC process to the extracted features from the feature accumulation database (6c), a model generator (6g) disposed downstream of the RANSAC processing component configured to generate an estimated model from an output of the RANSAC processing circuit, and a key frame selection component (6f) disposed in data communication between the feature accumulation database (6c) and the RANSAC processing component (6d) configured to perform a consistency check on feature data extracted from a newly obtained image frame relative to a temporally preceding estimated model from the model generator.

[0065] In some embodiments, the key frame selection component (6f) is further configured to designate the newly obtained image frame as a key frame when a consistency level of the extracted feature data of the newly obtained image frame is below a predetermined consistency threshold, and to forward accumulated extracted feature data including the extracted feature data of the key frame from the feature accumulation database (6c) to the RANSAC processing component (6d) for selective RANSAC processing.

[0066] In some embodiments, the exemplary system 60 further includes a model quality measurement component (6e) disposed in data communication downstream of the model generator (6g) configured to evaluate an estimated model quality of each generated estimated model and generate a model quality measurement variable q, such as the model quality measurement variable depicted in the foregoing discussion.

[0067] In some embodiments, the exemplary system 60 further includes a dynamic feature accumulation controller (6b) configured to dynamically adjust data accumulation in the feature accumulation database according to the model quality measurement variable q.

[0068] In this embodiment, each of the system components can be implemented by a processing device in conjunction with necessary hardware circuitry, software or firmware components and operating to perform the respective functions discussed. It is further noted that each of the various system components of the processing device 60 can be formed of common or different circuitry within the processing unit and configured to execute program instructions read from memory coupled thereto. For example, one or more of the illustrated circuits of the processing device 60 can use memory for temporary storage during computation and for storage of computed data. The memory can also store program instructions read and executed by the processing device 60 to perform its operations.

[0069] Depending on the application, the processing device 60 can be implemented as a multi-purpose computing hardware or a special-purpose hardware. For example, suitable types of processing devices can include central processing units (CPUs), digital signal processors (DSPs), image signal processors (ISPs), etc. In some embodiments, the processor can include a multi-core processor containing multiple processing cores in a computing device. In some embodiments, various elements associated with the processing device 60 can be shared by other devices.

[0070] Figure 10 A flowchart of an exemplary image processing method according to some embodiments of the disclosure is shown.

[0071] The exemplary image processing method includes:

[0072] Operation P101 : obtaining image data including temporally consecutive image frames from an imaging device;

[0073] Operation P102: performing feature extraction on each of the obtained image frames;

[0074] Operation P103: dynamically accumulating the extracted feature data of the obtained image frames in a feature accumulation database; and

[0075] Operation P104: updating the estimation model by performing RANSAC operations on the extracted feature data of a selective subset of the temporally consecutive image frames.

[0076] In some embodiments, performing RANSAC operations on the extracted feature data of the selective subset of operation P104 includes determining whether a newly obtained image frame constitutes a key frame among the obtained image frames, and forwarding the extracted feature data of the key frame from the feature accumulation database for selective RANSAC operations.

[0077] In some embodiments, the keyframe determination process includes performing a consistency check on the feature data extracted from the current image frame relative to an estimated model that precedes in time. In some embodiments, the current image frame is designated as a keyframe when a consistency level of the extracted feature data of the current image frame is below a consistency threshold.

[0078] In some embodiments, when the current image frame is designated as a keyframe, the extracted feature data of the keyframe in the feature accumulation database is adjusted to maintain a first time duration. In some embodiments, when the current image frame is not designated as a keyframe, the extracted feature data of the current image frame in the feature accumulation database is set to maintain a second time duration that is shorter than the first time duration.

[0079] In some embodiments, for each generated estimated model, the estimated model quality is evaluated by generating a model quality measurement variable (e.g., quality variable q). In some embodiments, the model quality measurement variable q includes a function of RANSAC support votes, a total number of features used for RANSAC, and a current model residual error.

[0080] In some embodiments, dynamically accumulating the extracted feature data in the feature accumulation database in operation P103 includes adjusting a discard rate of the extracted feature data maintained in the feature accumulation database based at least in part on the model quality measurement variable (e.g., variable q).

[0081] In some embodiments, adjusting the discard rate of the feature accumulation database includes generating a control variable (e.g., dynamic accumulation control variable d) having a value not greater than 1 by inputting the model quality measurement variable (e.g., variable q) into a dynamic accumulation control function. In some embodiments, the dynamic accumulation control function includes a monotonically decreasing function.

[0082] In some embodiments, when the dynamic accumulation control function does not satisfy a predetermined first quality threshold (e.g., first threshold T1), the control variable d is set to a value of 1, and a maintained volume of the extracted feature data in the feature accumulation database is expanded by accumulating the extracted feature data from the current image frame into the feature accumulation database. In some embodiments, the dynamic accumulation control function is designated to affect an inverse relationship between the estimated model quality and the control variable d.

[0083] While the standard RANSAC method can be used to solve general computer vision problems, the tRANSAC framework presented immediately can better solve problems in automotive applications. The tRANSAC framework can be suitable for ADAS (Advanced Driver Assistance System) related applications with better stability and less time / computational cost. In one aspect, the presently disclosed tRANSAC framework leverages the dynamic nature of road scene video streams by accumulating features over time in a dynamic manner as compared to the baseline RANSAC method. In this way, the vulnerability of RANSAC estimation to road scene content can be overcome. In another aspect, by adding a dynamic feature accumulation component, features are accumulated only when a RANSAC model cannot be stably estimated. This reduces unnecessary time cost as compared to naively accumulating features over time. Furthermore, by adding a key frame selection component, RANSAC can be performed only in selective frames where the accumulated feature data no longer agrees with the previous estimated model. This arrangement can significantly save computational time.

[0084] Accordingly, one aspect of the present disclosure provides a computer-implemented image processing method. The method comprises performing feature extraction on each obtained image frame; dynamically maintaining extracted feature data of the obtained image frame in a feature accumulation database by adjusting data in the feature accumulation database to a selective subset of the extracted feature data from the obtained image frame; performing a random sample consensus (RANSAC) operation on the selective subset of the extracted feature data from the feature accumulation database; and generating an estimated model from an output of the RANSAC operation based on at least one of the extracted feature data of a current image frame or the extracted feature data of one or more temporally preceding image frames of the obtained image frames.

[0085] In some embodiments, performing the RANSAC operation on the selective subset of the extracted feature data comprises determining, among the obtained image frames, whether a newly obtained image frame constitutes a key frame, and forwarding the extracted feature data of the key frame from the feature accumulation database for the selective RANSAC operation.

[0086] In some embodiments, the method further comprises forwarding further accumulated extracted feature data preceding the feature data of the key frame from the feature accumulation database for the selective RANSAC operation.

[0087] In some embodiments, the key frame determination process comprises performing a consistency check on the extracted feature data from the current image frame with respect to a temporally preceding estimated model, and designating the current image frame as a key frame when a consistency level of the extracted feature data of the current image frame is below a consistency threshold.

[0088] In some embodiments, the extracted feature data of the keyframes in the feature accumulation database is adjusted to be maintained at a first time duration when the current image frame is designated as a keyframe; the extracted feature data of the current image frame in the feature accumulation database is set to be maintained at a second time duration shorter than the first time duration when the current image frame is not designated as a keyframe.

[0089] In some embodiments, the method further comprises, for each generated estimation model, evaluating the estimation model quality by generating a model quality measurement variable, wherein the model quality measurement variable comprises a RANSAC support vote, a total number of features used for RANSAC, and a function of current model residual error.

[0090] In some embodiments, dynamically maintaining the extracted feature data in the feature accumulation database comprises adjusting a discard rate of the extracted feature data maintained in the feature accumulation database based at least in part on the model quality measurement variable.

[0091] In some embodiments, adjusting the discard rate of the feature accumulation database comprises generating a control variable having a value no greater than 1 by inputting the model quality measurement variable into a dynamic accumulation control function.

[0092] In some embodiments, the dynamic accumulation control function comprises a monotonically decreasing function.

[0093] In some embodiments, the method further comprises, when the dynamic accumulation control function satisfies a predetermined first quality threshold, setting the control variable to a value less than 1; and reducing a maintained volume of the extracted feature data in the feature accumulation database.

[0094] In some embodiments, reducing the maintained volume of the extracted feature data in the feature accumulation database is performed by reducing a current data accumulation weight factor by a factor of the control variable, and removing the maintained extracted feature data in the feature accumulation database having a weight factor less than a predetermined second quality threshold.

[0095] In some embodiments, the dynamic accumulation control function is specified to affect an inverse relationship between the estimation model quality and the control variable.

[0096] Accordingly, another aspect of the present disclosure provides a computer-implemented image processing method. Comprising: obtaining image data comprising temporally consecutive image frames from an imaging device; performing feature extraction on each of the obtained image frames; dynamically accumulating the extracted feature data of the obtained image frames in a feature accumulation database; and updating an estimation model by performing a RANSAC operation on the extracted feature data of a selective subset of the temporally consecutive image frames.

[0097] In some embodiments, performing RANSAC operations on the extracted feature data in the selective subset includes determining, among the obtained image frames, whether a newly obtained image frame constitutes a key frame, and forwarding the extracted feature data of the key frame from the feature accumulation database for selective RANSAC operations.

[0098] In some embodiments, the key frame determination process includes performing a consistency check on the extracted feature data from a current image frame relative to an estimated model that precedes in time, and designating the current image frame as a key frame when a consistency level of the extracted feature data of the current image frame is below a consistency threshold.

[0099] In some embodiments, when the current image frame is designated as a key frame, the extracted feature data of the key frame in the feature accumulation database is maintained for a first time duration; when the current image frame is not designated as a key frame, the extracted feature data of the current image frame in the feature accumulation database is maintained for a second time duration that is shorter than the first time duration.

[0100] In some embodiments, the method further includes, for each generated estimated model, evaluating the estimated model quality by generating a model quality measurement variable, wherein the model quality measurement variable includes a RANSAC support vote, a total number of features used for RANSAC, and a function of a current model residual error.

[0101] In some embodiments, dynamically accumulating the extracted feature data in the feature accumulation database includes adjusting a discard rate of the extracted feature data maintained in the feature accumulation database based at least in part on the model quality measurement variable.

[0102] In some embodiments, adjusting the discard rate of the feature accumulation database includes generating a control variable with a value no greater than 1 by inputting the model quality measurement variable into a dynamic accumulation control function, wherein the dynamic accumulation control function includes a monotonically decreasing function.

[0103] In some embodiments, the method further includes, when the dynamic accumulation control function does not satisfy a predetermined first quality threshold, setting the control variable to a value of 1, and expanding a maintained volume of the extracted feature data in the feature accumulation database by accumulating the extracted feature data from a current image frame into the feature accumulation database.

[0104] In some embodiments, the dynamic accumulation control function is designated to affect an inverse relationship between the estimated model quality and the control variable.

[0105] Therefore, another aspect of this disclosure provides an image processing system. The system includes: a processing unit configured to acquire image data comprising temporally consecutive image frames from an imaging device. The processing unit includes: a feature extraction component configured to receive image data from the imaging device and extract feature data from each of the acquired image frames; a feature accumulation database configured to selectively retain the extracted feature data in data communication with the feature extraction circuitry; a Random Sample Consensus (RANSAC) processing component configured to apply the RANSAC process to the extracted features from the feature accumulation database; a model generator disposed downstream of the RANSAC processing component configured to generate an estimated model from the output of the RANSAC processing circuitry; and a keyframe selection component disposed in data communication between the feature accumulation database and the RANSAC processing component, configured to perform a consistency check on the feature data extracted from newly acquired image frames relative to a temporally earlier estimated model from the model generator.

[0106] In some embodiments, the keyframe selection component is further configured to: designate the newly acquired image frame as a keyframe when the consistency level of the extracted feature data of the newly acquired image frame is lower than a predetermined consistency threshold; and forward the accumulated extracted feature data containing the extracted feature data of the keyframe from the feature accumulation database to the RANSAC processing circuit for selective RANSAC processing.

[0107] In some embodiments, the system further includes: a model quality measurement component, which is arranged downstream of the model generator in a data communication manner, and is configured to evaluate the estimated model quality of each generated estimated model and generate model quality measurement variables.

[0108] In some embodiments, the system further includes a dynamic feature accumulation controller configured to dynamically adjust the data accumulation in the feature accumulation database based on model quality measurement variables.

[0109] Those skilled in the art will readily discover that numerous modifications and alterations can be made to the apparatus and method while maintaining the teachings of this disclosure. Therefore, the above disclosure should be understood as being limited only by the boundaries and limits of the appended claims.

[0110] Figure 1

[0111] Figure 2

[0112] Road scene video stream

[0113] Frame

[0114] Feature extraction

[0115] RANSAC random sample consensus

[0116] Kalman filter

[0117] Estimated model+votes

[0118] Updated estimated model+votes

[0119] Figure 3

[0120] Road scene video stream

[0121] Frame

[0122] Feature extraction(a)

[0123] B dynamic feature accumulation control

[0124] Kalman filter

[0125] C feature accumulation database

[0126] Key frame selection(f)

[0127] Features consistent?

[0128] Y is

[0129] N no

[0130] Skip

[0131] Estimated model+votes

[0132] Updated estimated model+votes

[0133] E model estimation quality measure

[0134] Figure 4

[0135] Dynamic accumulated feature pool (c) Dynamic accumulated feature pool (c) Key frame selection (f) Key frame selection (f)

[0136] Feature consistent with previous model? Feature consistent with previous model?

[0137] 1 round voting 1 round voting

[0138] Y is Y is

[0139] N is N is

[0140] Skip Skip

[0141] Transac (d) (n round hypothesis generation + voting) Transac (d) (n round hypothesis generation + voting)

[0142] Figure 5

[0143] Feature extracted from current frame (a) Feature extracted from current frame (a)

[0144] Dynamic feature accumulation control (b) Dynamic feature accumulation control (b)

[0145] Dynamic accumulated feature pool (c) Dynamic accumulated feature pool (c)

[0146] Estimated model quality measure (e) Estimated model quality measure (e)

[0147] q = f q (RANSAC votes, model residual error, # of features) q = f q (RANSAC votes, model residual error, # of features)

[0148] Figure 6

[0149] Estimated model quality measure (e) Estimated model quality measure (e)

[0150] q = previous model support votes

[0151] Dynamic feature accumulation control

[0152] previous model support votes q > T1?

[0153] N (poor RANSAC estimation)

[0154] Y (good RANSAC estimation)

[0155] Feature from current frame

[0156] Tap water faucet

[0157] Dynamic accumulated feature pool

[0158] Open sink

[0159] Close sink

[0160] Set control variable d to some value <1.0 (e.g. d = 0.99)

[0161] Set control variable d to 1.0

[0162] Reduce current accumulated features in feature pool weights by x d & remove all features with weights < T2

[0163] Feature weight decay

[0164] Updated accumulated features

[0165] Figure 7

[0166] 6a Feature extraction

[0167] 6b Dynamic feature accumulation controller

[0168] 6c Feature accumulation database

[0169] 6d Random sample consensus processing

[0170] 6e Model quality measurement

[0171] 6f Keyframe selection

[0172] 6g Model generator

[0173] Figure 9

[0174] LVP lane vanishing point

[0175] Figure 10

[0176] View angle corrected output

[0177] ​

[0178] P101 Obtaining image data comprising temporally consecutive image frames

[0179] P102 Performing feature extraction on each of the obtained image frames

[0180] P103 Dynamically accumulating extracted feature data of the obtained image frames in a feature accumulation database

[0181] P104 Updating an estimation model by performing a random sample consensus operation on extracted feature data of a selective subset of the temporally consecutive image frames.

Claims

1. A computer-implemented image processing method, comprising: Obtain image data, including temporally continuous image frames, from the imaging device; Feature extraction is performed on each of the obtained image frames; The extracted feature data of the obtained image frame is dynamically maintained in the feature accumulation database by adjusting the data in the feature accumulation database to a selective subset of the extracted feature data from the obtained image frame; Perform a Random Sampling Consensus (RANSAC) operation on a selective subset of the extracted feature data from the feature accumulation database; and An estimated model is generated from the output of the RANSAC operation based on the extracted feature data of the current image frame or at least one of the extracted feature data of one or more temporally earlier image frames.

2. The method according to claim 1, The RANSAC operation performed on the selective subset of the extracted feature data includes: Among the acquired image frames, determine whether a newly acquired image frame constitutes a keyframe, and The extracted feature data of the keyframe is forwarded from the feature accumulation database for selective RANSAC operation.

3. The method according to claim 2, further comprising: The extracted feature data accumulated prior to the feature data of the keyframe is further forwarded from the feature accumulation database for selective RANSAC operation.

4. The method according to claim 2, The keyframe determination process includes: The extracted feature data from the current image frame is subjected to a consistency check relative to the time-preceding estimation model. When the consistency level of the extracted feature data of the current image frame is lower than the consistency threshold, the current image frame is designated as a keyframe.

5. The method according to claim 4, When the current image frame is designated as the key frame, the extracted feature data of the key frame in the feature accumulation database will be maintained at a first time interval. When the current image frame is not designated as the key frame, the extracted feature data of the current image frame in the feature accumulation database is kept at a second time interval shorter than the first time interval.

6. The method of claim 4, further comprising: For each of the generated estimated models, the quality of the estimated model is evaluated by generating model quality measurement variables. The model quality measurement variables mentioned above include a function of RANSAC support votes, the total number of features used for RANSAC, and the current model residual error.

7. The method according to claim 6, The process of dynamically maintaining the extracted feature data of the obtained image frames in the feature accumulation database includes: The discard rate of the retained extracted feature data of the obtained image frames in the feature accumulation database is adjusted at least in part based on the model quality measurement variables.

8. The method according to claim 7, Adjusting the discard rate of the feature accumulation database includes: Control variables with values ​​no greater than 1 are generated by inputting the model quality measurement variables into a dynamic cumulative control function. The dynamic cumulative control function mentioned therein includes a monotonically decreasing function.

9. The method of claim 8, further comprising: When the dynamic cumulative control function satisfies a predetermined first quality threshold, the control variable is set to a value less than 1; and The following operations reduce the hold-up volume of the extracted feature data in the feature accumulation database: Reduce the current data cumulative weight factor by the factor of the control variable, and Remove extracted feature data from the feature accumulation database that has a weight factor less than a predetermined second quality threshold.

10. The method according to claim 9, The dynamic cumulative control function is specified to influence the inverse relationship between the quality of the estimated model and the control variables.

11. A computer-implemented image processing method, comprising: Obtain image data, including temporally continuous image frames, from the imaging device; Feature extraction is performed on each of the obtained image frames; The extracted feature data of the obtained image frames are accumulated in the feature accumulation database; and An estimation model is generated by performing a RANSAC operation on the extracted feature data of a selective subset of the temporally continuous image frames. The RANSAC operation performed on the extracted feature data of the selective subset includes: Among the acquired image frames, determine whether a newly acquired image frame constitutes a keyframe, and The extracted feature data of the keyframe is forwarded from the feature accumulation database for selective RANSAC operation; The keyframe determination process includes: The extracted feature data from the current image frame is subjected to a consistency check relative to the time-preceding estimation model. When the consistency level of the extracted feature data of the current image frame is lower than the consistency threshold, the current image frame is designated as a keyframe; When the current image frame is designated as a key frame, the extracted feature data of the key frame in the feature accumulation database will be maintained at a first time interval. When the current image frame is not designated as a keyframe, the extracted feature data of the current image frame in the feature accumulation database is kept at a second time interval shorter than the first time interval.

12. A computer-implemented image processing method, comprising: Obtain image data, including temporally continuous image frames, from the imaging device; Feature extraction is performed on each of the obtained image frames; The extracted feature data of the obtained image frames are accumulated in the feature accumulation database; An estimation model is generated by performing a RANSAC operation on the extracted feature data of a selective subset of the temporally continuous image frames. For each of the generated estimated models, the quality of the estimated model is evaluated by generating model quality measurement variables. The model quality measurement variables mentioned above include a function of RANSAC support votes, the total number of features used for RANSAC, and the current model residual error. The RANSAC operation performed on the extracted feature data of the selective subset includes: Among the acquired image frames, determine whether a newly acquired image frame constitutes a keyframe, and The extracted feature data of the keyframe is forwarded from the feature accumulation database for selective RANSAC operation; The keyframe determination process includes: The extracted feature data from the current image frame is subjected to a consistency check relative to the time-preceding estimation model. When the consistency level of the extracted feature data of the current image frame is lower than the consistency threshold, the current image frame is designated as a keyframe.

13. The method according to claim 12, The process of accumulating the extracted feature data of the obtained image frames in the feature accumulation database includes: The discard rate of the accumulated extracted feature data in the feature accumulation database is adjusted at least in part based on the model quality measurement variable.

14. The method according to claim 13, Adjusting the discard rate of the feature accumulation database includes: Control variables with values ​​no greater than 1 are generated by inputting the model quality measurement variables into a dynamic cumulative control function. The dynamic cumulative control function mentioned therein includes a monotonically decreasing function.

15. The method of claim 14, further comprising: When the dynamic cumulative control function does not meet the predetermined first quality threshold, the control variable is set to the value 1; and The volume of extracted feature data in the feature accumulation database is increased by accumulating the extracted feature data from the current image frame into the feature accumulation database.

16. The method according to claim 15, The dynamic cumulative control function is specified to influence the inverse relationship between the quality of the estimated model and the control variables.

17. An image processing system comprising: A processing apparatus configured to acquire image data comprising temporally consecutive image frames from an imaging apparatus, the processing apparatus comprising: A feature extraction component configured to receive image data from the imaging device and extract feature data from each of the acquired image frames; A feature accumulation database, which communicates with the feature extraction component, is configured to selectively retain the extracted feature data; A random sampling consistency (RANSAC) processing component configured to apply the RANSAC process to the extracted feature data from the feature accumulation database; A model generator, positioned downstream of the RANSAC processing component, is configured to generate an estimated model from the output of the RANSAC processing component; and A keyframe selection component, communicatively arranged between the feature accumulation database and the RANSAC processing component, is configured to perform a consistency check on the extracted feature data from newly acquired image frames relative to a temporally earlier estimation model from the model generator.

18. The system according to claim 17, The keyframe selection component is further configured to: When the consistency level of the extracted feature data of the newly acquired image frame is lower than a predetermined consistency threshold, the newly acquired image frame is designated as a keyframe. The accumulated extracted feature data containing the keyframe is forwarded from the feature accumulation database to the RANSAC processing component for selective RANSAC processing.

19. The system of claim 18, further comprising a model quality measurement component disposed downstream of the model generator in a data communication manner, configured to evaluate the estimated model quality of each of the generated estimated models and generate model quality measurement variables.

20. The system of claim 19, further comprising a dynamic feature accumulation controller configured to dynamically adjust the data accumulation in the feature accumulation database according to the model quality measurement variables.

Images