A vision-based obstacle recognition and ranging method

Abstract

This invention relates to the field of visual perception technology and discloses a vision-based obstacle recognition and ranging method. The method includes simultaneously acquiring visible light and infrared image pairs and fusing their features, identifying potential obstacles and extracting descriptors, and determining the dynamic scene state of the obstacle by matching it with a pre-built scene feature library. Based on this state, a motion trajectory is constructed for the dynamic obstacle, trajectory intersection and collision time analysis is performed, and the fusion weights of the dual-modal data are dynamically optimized based on the analysis results to generate an optimized fused image. Subsequently, 3D reconstruction is performed on the optimized image to obtain point cloud data, and the real-time distance and velocity of the obstacle are estimated by combining the motion trajectory. The scene feature library is then updated to complete the final recognition and ranging. This method guides the processing flow through scene semantic state and adjusts the front-end fusion strategy according to dynamic threats, improving the system's perception accuracy, adaptability, and decision reliability in complex environments.

Description

Technical Field

[0001] This invention relates to the field of visual perception technology, specifically to a vision-based obstacle recognition and ranging method. Background Technology

[0002] In perception systems used in fields such as autonomous driving and robot navigation, fusing visible light and infrared images to improve obstacle detection and ranging capabilities in complex environments has become a common technique. Existing solutions typically employ fixed weights or simple rules based on image quality for pixel-level or feature-level fusion at the front end to generate higher-quality fused images for subsequent recognition and ranging modules. However, this fusion strategy, which is decoupled from scene semantics and dynamic threats, has limitations. In dynamic traffic scenarios, the motion states and threat levels of different obstacles vary greatly. A uniform fusion process cannot achieve optimal allocation of perception resources, potentially leading to insufficient feature extraction for highly dynamic, high-threat targets, or unnecessary redundant calculations for static, low-threat targets.

[0003] Conventional technical workflows typically treat obstacle identification, tracking, trajectory prediction, and distance estimation as relatively independent sequential modules. After the identification module outputs the obstacle category and location, the tracking module is responsible for associating them, while the trajectory prediction and distance estimation modules perform independent calculations based on the associative results. This architecture lacks a global understanding and utilization of the "scene state." The system cannot clearly distinguish whether an obstacle is in a stable cruising state, interacting with other traffic participants, or performing a potentially risky intrusion behavior. This results in subsequent trajectory analysis and risk estimation lacking high-level semantic guidance, having a simplistic processing strategy, and insufficient adaptability and decision-making rationality in complex interaction scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a vision-based obstacle recognition and ranging method to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a vision-based obstacle recognition and ranging method, the method comprising:

[0006] By simultaneously acquiring image pairs of the same scene using a binocular camera and a visible light imaging device and an infrared imaging device, an original set of image pairs is formed.

[0007] Pixel-level feature fusion is performed on each image pair in the original image pair set to generate a fused feature image;

[0008] Based on the fused feature image, potential obstacle regions in the scene are identified, and multimodal descriptors of the potential obstacle regions are extracted;

[0009] The multimodal descriptors are matched with a pre-built scene feature library, and the scene state of the potential obstacle region is determined based on the matching results.

[0010] Based on the scene state, construct motion trajectories for obstacles in the dynamic scene, and perform trajectory intersection analysis and collision time estimation;

[0011] Based on the results of trajectory cross-analysis and collision time estimation, the fusion weights of visible light and infrared data are dynamically adjusted to generate an optimized fused image.

[0012] Dense 3D reconstruction is performed on the optimized fused image to calculate the 3D point cloud data of the obstacle surface;

[0013] Based on the three-dimensional point cloud data and the motion trajectory, the real-time distance and speed of the obstacle are estimated;

[0014] Based on the real-time distance and movement speed, update the feature records of the corresponding obstacles in the pre-built scene feature library;

[0015] Based on the updated pre-built scene feature library, the final identification and ranging of obstacles in the current frame are completed.

[0016] Preferably, image pairs of the same scene are simultaneously acquired by a binocular camera and obtained by a visible light imaging device and an infrared imaging device to form an original image pair set, specifically including:

[0017] Control the visible light imaging device and the infrared imaging device to capture images at the same frame rate and timestamp;

[0018] The captured visible light and infrared images are time-stamp aligned and verified, and images with time-stamp deviations exceeding a threshold range are discarded.

[0019] The visible light image that has passed the timestamp alignment verification is paired with the infrared image to form the original image pair set.

[0020] Preferably, pixel-level feature fusion is performed on each image pair in the original image pair set to generate a fused feature image, specifically including:

[0021] Edge enhancement processing is performed on the visible light images in the original image set, and temperature region segmentation processing is performed on the infrared images.

[0022] The visible light image after edge enhancement and the infrared image after temperature region segmentation are mapped to the same scale space;

[0023] Within the scale space, a fusion coefficient is assigned to each pixel location, and the corresponding pixel values ​​from the visible light image and the infrared image are weighted and calculated based on the fusion coefficient to generate the fused feature image.

[0024] Preferably, based on the fused feature image, potential obstacle regions in the scene are identified, and multimodal descriptors of the potential obstacle regions are extracted, specifically including:

[0025] A sliding window is applied to the fused feature image to perform region scanning, and the texture complexity and gradient statistical features of the image within the window are calculated.

[0026] Windows whose texture complexity and gradient statistical features exceed a preset threshold are marked as potential obstacle regions.

[0027] Within each potential obstacle region, its color histogram, local gradient direction histogram, and temperature distribution histogram are calculated respectively, and these histograms are connected to form the multimodal descriptor.

[0028] Preferably, the multimodal descriptor is matched with a pre-built scene feature library, and the scene state of the potential obstacle region is determined based on the matching result, specifically including:

[0029] Calculate the similarity distance between the multimodal descriptor and each feature template in the pre-built scene feature library;

[0030] Select the feature template with the smallest similarity distance, and determine whether the smallest similarity distance is less than the recognition threshold;

[0031] If the value is less than the recognition threshold, the potential obstacle area is determined to be a known obstacle, and its scene state is either known static or known dynamic; otherwise, the potential obstacle area is determined to be a newly appearing obstacle, and its scene state is unknown.

[0032] Preferably, based on the scene state, motion trajectories are constructed for obstacles in a dynamic scene, and trajectory intersection analysis and collision time estimation are performed, specifically including:

[0033] For obstacles whose scene state is known dynamic or unknown, track their position changes in multiple consecutive frames;

[0034] The positional change is fitted into a smooth curve to construct the motion trajectory;

[0035] Predict the extended path of the motion trajectory in the future, and calculate the intersection point and intersection time between the machine's motion path and the extended path. The intersection time is the estimated collision time.

[0036] Preferably, based on the results of the trajectory intersection analysis and collision time estimation, the fusion weights of visible light and infrared data are dynamically adjusted to generate an optimized fused image, specifically including:

[0037] Set an attention factor that is inversely proportional to the collision time;

[0038] When the collision time is short, the attention factor is increased, and the weight of infrared data in the fusion is increased accordingly to enhance the temperature characteristics;

[0039] When the collision time is long or there is no intersection, the attention factor is reduced and the weight of visible light data in the fusion is increased accordingly to enhance texture details;

[0040] The pixel-level feature fusion step is re-executed using the adjusted weights to generate the optimized fused image.

[0041] Preferably, dense 3D reconstruction is performed on the optimized fused image to calculate the 3D point cloud data of the obstacle surface, specifically including:

[0042] A binocular vision system is constructed by utilizing the fixed positional relationship and imaging parameters between the visible light imaging device and the infrared imaging device.

[0043] Highly robust feature points are extracted from the optimized fused image, and the parallax of the feature points between the views of the visible light imaging device and the infrared imaging device is calculated.

[0044] Based on the parallax and the geometric model of the binocular vision system, the three-dimensional spatial coordinates corresponding to the feature points are calculated using the triangulation principle to form the three-dimensional point cloud data.

[0045] Preferably, estimating the real-time distance and speed of the obstacle based on the three-dimensional point cloud data and the motion trajectory specifically includes:

[0046] From the three-dimensional point cloud data, select the set of three-dimensional spatial coordinate points belonging to the same obstacle;

[0047] Calculate the Euclidean distance from the geometric center of the three-dimensional spatial coordinate point set to the optical center of the binocular vision system, and use it as the real-time distance;

[0048] Based on the geometric center position calculated in the previous frame, the displacement of the geometric center between the current frame and the previous frame is calculated, and the motion velocity is calculated based on the frame interval time.

[0049] Preferably, based on the updated pre-built scene feature library, the final identification and ranging of obstacles in the current frame are completed, specifically including:

[0050] The real-time distance, the motion speed, and the multimodal descriptor calculated in the current frame are associated with and stored in the historical records of obstacles in the pre-built scene feature library.

[0051] The real-time distance, the movement speed, and the final obstacle marker are used as the output of the current frame.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] The fusion weights of visible light and infrared data are dynamically adjusted based on trajectory analysis and collision time estimation. This transforms front-end data fusion from a passive, fixed process into a proactive decision-making process guided by real-time security assessment. When the system estimates a short collision time and determines that the target poses a high threat, the algorithm will strengthen the contribution weight of the imaging modality that better represents the target in real time, thereby highlighting and retaining the key features of high-threat targets in the generated fused image. This ensures that the input quality of subsequent recognition and ranging modules is optimized at critical moments, improving the reliability and accuracy of high-risk target perception. Meanwhile, for low-threat targets, a more balanced or computationally efficient fusion strategy is adopted, achieving adaptive optimal allocation of the system's perception computing resources.

[0054] By matching a pre-built feature library to determine the "scene state," this semantic result serves as the direct input and process control basis for subsequent trajectory construction and analysis. This upgrades the entire system from a serial link where each module operates independently to a collaborative closed loop under unified scheduling by a higher-level scene understanding. Based on different determined states, the system can dynamically invoke or configure different trajectory prediction models and analysis logics. This not only reduces unnecessary calculations for static or irrelevant targets but also allows for the incorporation of richer semantic context into the motion analysis and behavior prediction of dynamic obstacles. Consequently, the trajectory prediction results better align with the interaction logic in real-world scenarios, enhancing the system's overall decision-making rationality and scene adaptability in complex dynamic environments. Attached Figure Description

[0055] Figure 1 This is a schematic diagram illustrating the working principle of the vision-based obstacle recognition and ranging method described in this invention.

[0056] Figure 2 A flowchart for pixel-level feature fusion;

[0057] Figure 3 A flowchart for extracting multimodal descriptions;

[0058] Figure 4 A graph showing the correlation between collision time and fusion weight in visual obstacle recognition;

[0059] Figure 5This is a grouped bar chart for the analysis of obstacle movement speed. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Please see Figure 1 This invention provides a vision-based obstacle recognition and ranging method, comprising: simultaneously acquiring image pairs of the same scene using a binocular camera and a visible light imaging device and an infrared imaging device, forming an original image pair set; performing pixel-level feature fusion on each image pair in the original image pair set to generate a fused feature image; identifying potential obstacle regions in the scene based on the fused feature image and extracting multimodal descriptors of the potential obstacle regions; matching the multimodal descriptors with a pre-constructed scene feature library, and determining the scene state of the potential obstacle regions based on the matching results; and determining the scene state of the potential obstacle regions based on the scene state. In dynamic scenes, obstacles are constructed with motion trajectories, and trajectory cross-analysis and collision time estimation are performed. Based on the results of trajectory cross-analysis and collision time estimation, the fusion weights of visible light and infrared data are dynamically adjusted to generate an optimized fused image. Dense 3D reconstruction is performed on the optimized fused image to calculate the 3D point cloud data of the obstacle surface. Based on the 3D point cloud data and motion trajectory, the real-time distance and motion speed of the obstacle are estimated. Based on the real-time distance and motion speed, the feature records of the corresponding obstacles in the pre-built scene feature library are updated. Based on the updated pre-built scene feature library, the final identification and ranging of the obstacle in the current frame are completed.

[0062] Example 1: See Figure 2The system simultaneously acquires image pairs of the same scene using a binocular camera, obtained from both a visible light imaging device and an infrared imaging device, forming an initial image pair set. Specifically, the visible light and infrared imaging devices capture images at the same frame rate and timestamp. The captured visible light and infrared images undergo timestamp alignment verification, discarding images with timestamp deviations exceeding a threshold. The visible light and infrared images that pass the timestamp alignment verification are then paired to form the initial image pair set. Pixel-level feature fusion is performed on each image pair in the initial image pair set to generate a fused feature image. Specifically, edge enhancement processing is applied to the visible light image, and temperature region segmentation processing is applied to the infrared image. The edge-enhanced visible light image and the temperature-segmented infrared image are mapped to the same scale space. A fusion coefficient is assigned to each pixel location within this scale space. The corresponding pixel values ​​from the visible light and infrared images are weighted according to the fusion coefficient to generate the fused feature image.

[0063] In practical implementation, the following example scenario illustrates this: a vehicle is traveling on a road at night, with a visible light imaging device and an infrared imaging device fixed at the front of the vehicle, synchronously acquiring information about the environment ahead. Both devices capture images at the same frame rate of 30 frames per second, attaching a timestamp accurate to the millisecond level to each captured image. In practice, the timestamps are generated by a unified hardware clock source to ensure the synchronization of the visible light and infrared images at the moment of capture. When performing timestamp alignment verification on the captured visible light and infrared images, a threshold range of five milliseconds for timestamp deviation is set. If the timestamp difference between the visible light and infrared images exceeds five milliseconds, the image pair is discarded. Only image pairs with timestamp differences within five milliseconds are retained. The visible light and infrared images that pass the timestamp alignment verification are then paired to form the original image pair set.

[0064] In the specific implementation, pixel-level feature fusion is performed on each image pair in the original image pair set to generate a fused feature image. Specifically, edge enhancement processing is performed on the visible light image in the original image pair set, and temperature region segmentation processing is performed on the infrared image. Edge enhancement processing of the visible light image uses the Sobel operator to calculate gradient magnitude, enhancing the contour information in the image. The specific implementation of the Sobel operator for calculating gradient magnitude includes using predefined horizontal and vertical convolution kernels to perform convolution operations on the visible light image, extracting the spatial gradient change of each pixel through the convolution operation. The horizontal convolution kernel is used to detect vertical edges in the image, while the vertical convolution kernel is used to detect horizontal edges. The convolution results in the two directions are combined to calculate the gradient magnitude of each pixel, thereby highlighting the contour and texture information in the image. Temperature region segmentation processing of the infrared image is based on a preset temperature threshold, dividing pixels into different temperature ranges to highlight areas of thermal radiation difference. The edge-enhanced visible light image and the temperature-segmented infrared image are mapped to the same scale space. The mapping process involves image resampling, ensuring that the visible light image and the infrared image have the same pixel size and spatial alignment. The mapping of this scale space is based on the calibration parameters of the binocular vision system. Relying on the binocular vision structure composed of a visible light imaging device and an infrared imaging device, it ensures precise spatial registration of the two images, enabling a one-to-one correspondence between pixels in the visible light and infrared images within the unified coordinate system of binocular vision. This provides a spatial alignment basis for subsequent pixel-level fusion. Within the scale space, a fusion coefficient is assigned to each pixel location. Based on the fusion coefficient, the corresponding pixel values ​​from the visible light and infrared images are weighted to generate a fused feature image. In some embodiments, the allocation of the fusion coefficient is based on the local features of the pixel location; for example, visible light data is given higher weight in areas with rich texture, and infrared data is given higher weight in areas with significant temperature contrast. Optionally, the weighting calculation uses a linear combination method, expressed by the formula:

[0065]

[0066] in: It is the fusion of feature images at location pixel values, It is a visible light image at position pixel values, It is an infrared image at the location pixel values, It is a visible light image at position fusion coefficient, It is an infrared image at the location The fusion coefficient, and satisfying It is understandable that the fusion coefficient can be determined through predefined rules, such as dynamic adjustment based on local image variance. In specific implementation, data comparison is reflected in the timestamp alignment verification stage. When the timestamp deviation is 3 milliseconds, the image pair is retained, while when the timestamp deviation is 6 milliseconds, the image pair is discarded. This ensures the spatiotemporal consistency of visible light images and infrared images in subsequent processing. Optionally, the scale space mapping uses a bilinear interpolation method to ensure the smoothness of the transformed image.

[0067] Example 2: See Figure 3 Based on fused feature images, potential obstacle regions in a scene are identified, and multimodal descriptors of these potential obstacle regions are extracted. Specifically, a sliding window is applied to the fused feature image to scan the region, and the texture complexity and gradient statistical features of the image within the window are calculated. Windows whose texture complexity and gradient statistical features exceed a preset threshold are marked as potential obstacle regions. Within each potential obstacle region, its color histogram, local gradient direction histogram, and temperature distribution histogram are calculated, and these histograms are connected to form a multimodal descriptor.

[0068] In a specific implementation, the following example scenario illustrates this: the fused feature image presents a nighttime road scene containing vehicles and pedestrians, with a resolution of 1280 pixels by 720 pixels. In this implementation, potential obstacle regions in the scene are identified based on the fused feature image, and multimodal descriptors for these regions are extracted. Specifically, a sliding window is applied to the fused feature image for region scanning. The sliding window size is set to 64 pixels by 64 pixels, and the scanning step size is set to 32 pixels. The texture complexity and gradient statistics of the image within each sliding window are calculated. In some embodiments, texture complexity is obtained by calculating the local entropy of the pixels within the window, and gradient statistics are obtained by calculating the average and standard deviation of the gradient magnitudes of all pixels within the window. Optionally, the formula for calculating texture complexity is expressed as:

[0069]

[0070] in: Represents a sliding window The texture complexity value, This indicates the number of gray levels in the pixels within the sliding window. Represents grayscale level The probability of a pixel appearing within the window. It's understandable that the size and step size of the sliding window can be adjusted according to the actual application scenario; for example, a smaller window and step size can be used in scenarios requiring higher spatial resolution.

[0071] In specific implementations, windows whose texture complexity and gradient statistical features exceed preset thresholds are marked as potential obstacle regions. These preset thresholds include a texture complexity threshold and a gradient statistical feature threshold; the texture complexity threshold is set to 4.5, and the average gradient magnitude threshold in the gradient statistical features is set to 15.0. Data comparison is reflected in the feature value calculation results of different windows. For example, a window containing a vehicle outline has a texture complexity value of 5.2 and an average gradient magnitude of 18.3, both exceeding the preset thresholds, and this window is marked as a potential obstacle region. Conversely, a window containing only a flat road surface has a texture complexity value of 2.1 and an average gradient magnitude of 5.7, neither exceeding the preset thresholds, and this window is not marked. In some embodiments, the marking process assigns a unique region identifier to each window identified as a potential obstacle region and records its bounding box coordinates in the fused feature image.

[0072] In practice, a multimodal descriptor is extracted from each potential obstacle region. Specifically, this involves calculating the color histogram, local gradient direction histogram, and temperature distribution histogram for each region. The color histogram is calculated based on the RGB color space of the corresponding region in the fused feature image, quantizing each color channel into 16 intervals. The local gradient direction histogram calculates the gradient direction of each pixel within the corresponding region, evenly dividing the direction range from 0 degrees to 180 degrees into 9 directional intervals for statistical analysis. The temperature distribution histogram is calculated based on the pixel values ​​of the corresponding region in the infrared image used to generate the fused feature image, quantizing the temperature values ​​into 8 intervals. These histograms are then concatenated to form the multimodal descriptor. The concatenation method involves sequentially stitching the statistical values ​​of each interval of the color histogram, local gradient direction histogram, and temperature distribution histogram into a one-dimensional feature vector. In essence, the multimodal descriptor integrates appearance, shape, and thermal radiation information, resulting in a 65-dimensional feature vector with a length of 16 x 3 + 9 + 8.

[0073] Example 3: The multimodal descriptor is matched with a pre-built scene feature library. Based on the matching results, the scene state of potential obstacle regions is determined. Specifically, the similarity distance between the multimodal descriptor and each feature template in the pre-built scene feature library is calculated. The feature template with the smallest similarity distance is selected, and it is determined whether the smallest similarity distance is less than the recognition threshold. If it is less than the recognition threshold, the potential obstacle region is determined to be a known obstacle, and its scene state is either known static or known dynamic. Otherwise, the potential obstacle region is determined to be a newly appeared obstacle, and its scene state is unknown. Based on the scene state, motion trajectories are constructed for obstacles in dynamic scenes, and trajectory cross-analysis and collision time estimation are performed. Specifically, for obstacles with a scene state of known dynamic or unknown, their position changes are tracked in multiple consecutive frames of images. The position changes are fitted into a smooth curve to construct a motion trajectory. The extension path of the motion trajectory in the future is predicted, and the intersection point and intersection time between the local motion path and the extension path are calculated. The intersection time is the estimated collision time. Before fitting the motion trajectory, the position coordinates of the obstacle in the image coordinate system are converted into preliminary position information in the three-dimensional global coordinate system of the binocular vision system based on the imaging parameters of the binocular vision system. Then, the trajectory is fitted based on the preliminary three-dimensional position information of the continuous frames, so that the constructed motion trajectory is more in line with the motion state of the obstacle in the actual space.

[0074] In specific implementation, the following description uses a continuing example scenario where a multimodal descriptor representing a vehicle ahead has been extracted from the fused feature image. In this implementation, the multimodal descriptor is matched against a pre-built scene feature library. Based on the matching result, the scene state of potential obstacle regions is determined. Specifically, this involves calculating the similarity distance between the multimodal descriptor extracted in the current frame and each feature template in the pre-built scene feature library. In some embodiments, the pre-built scene feature library is stored in a database format, containing multimodal descriptor templates of known obstacles and their corresponding identifiers and status labels. Optionally, the similarity distance is calculated using Euclidean distance, expressed by the formula:

[0075]

[0076] in: Indicates similarity distance. The dimension of the multimodal descriptor is represented. This indicates that the current multimodal descriptor is in the th... eigenvalues ​​of dimension This indicates that a certain feature template in the pre-built scene feature library is in the [number]th [position]. Eigenvalues ​​of dimension.

[0077] In practice, the feature template with the smallest similarity distance is selected, and it is determined whether the minimum similarity distance is less than the recognition threshold, which is set to 0.85. Data comparison is reflected in different distance calculation results. For example, if the calculated similarity distance between the current multimodal descriptor and the "Car A" template in the feature library is 0.52, and the similarity distance with the "Pedestrian B" template is 1.30, then the minimum similarity distance is 0.52, which is less than the recognition threshold of 0.85. In this case, the potential obstacle area is determined to be a known obstacle, and its scene state is determined as "known dynamic" based on the label of the "Car A" template. If the minimum similarity distance between the current multimodal descriptor and all templates in the feature library is 1.20, which is greater than the recognition threshold of 0.85, then the potential obstacle area is determined to be a newly appeared obstacle, and its scene state is marked as "unknown".

[0078] In practical implementation, based on the determined scene state, motion trajectories are constructed for obstacles in dynamic scenes, and trajectory intersection analysis and collision time estimation are performed. Specifically, for obstacles with known or unknown scene states, their position changes are tracked across multiple consecutive frames. Position information is represented by the center coordinates of the bounding box of the potential obstacle region in the image. In five consecutive frames, the sequence of bounding box center coordinates for an obstacle determined to be "known dynamic" is [(320,480),(325,478),(330,475),(336,472),(342,470)]. In practical implementation, the position changes are fitted into a smooth curve to construct the motion trajectory. The fitting method uses quadratic polynomial fitting to obtain parametric equations describing the horizontal and vertical position of the obstacle in the image coordinate system over time. The extended path of the motion trajectory is predicted over a future period of one second (i.e., the next thirty frames), and the intersection point and intersection time between the local motion path and the extended path are calculated.

[0079] In some embodiments, the vehicle's own motion path is provided by the vehicle's localization and heading system, represented as a predicted straight path over a future time period. Optionally, the intersection point and intersection time are calculated based on kinematic equations in a unified global coordinate system, and the calculated intersection time is the estimated collision time. It is understood that if the calculation results show that the two paths have no intersection within the predicted time period, the collision time is either infinite or marked as invalid. Data comparison is reflected in a specific calculation case: for the trajectory fitted by the above coordinate sequence, the intersection time between its extended path and the vehicle's path is calculated to be 4.3 seconds, while the trajectory calculation result for another obstacle shows no intersection.

[0080] See Figure 4This is a graph illustrating the correlation between collision time and fusion weights in visual obstacle recognition. The dynamic adjustment logic for the weights is as follows: shorter collision times result in higher infrared fusion weights (e.g., car A has a short collision time, resulting in an infrared weight of 0.7); longer collision times result in higher visible light weights (e.g., guardrail D has a long collision time, resulting in a visible light weight of 0.8), reflecting the optimization strategy of "strengthening infrared temperature features as danger approaches." Obstacle risk differences are also considered: car A and the new obstacle E have shorter collision times (higher risk), corresponding to a higher infrared weight; guardrail D is a static obstacle (longer collision time), corresponding to a higher visible light weight. This type of graph is used in the fusion weight adjustment stage of visual obstacle recognition, visually demonstrating the correlation between collision risk and multimodal data fusion strategies, providing a basis for dynamically adjusting the fusion weights.

[0081] Example 4: Based on the results of trajectory intersection analysis and collision time estimation, the fusion weights of visible light and infrared data are dynamically adjusted to generate an optimized fused image. Specifically, an attention factor inversely proportional to the collision time is set. When the collision time is short, the attention factor is increased, and the weight of infrared data in the fusion is correspondingly increased to enhance temperature features. When the collision time is long or there is no intersection, the attention factor is decreased, and the weight of visible light data in the fusion is correspondingly increased to enhance texture details. The pixel-level feature fusion step is then re-executed using the adjusted weights to generate the optimized fused image. Dense 3D reconstruction is performed on the optimized fused image to calculate the 3D point cloud data of the obstacle surface. Specifically, a binocular vision system is constructed using the fixed positional relationship and imaging parameters between the visible light imaging device and the infrared imaging device. Highly robust feature points are extracted from the optimized fused image, and the disparity between the viewpoints of the feature points in the visible light imaging device and the infrared imaging device is calculated. Based on the disparity and the geometric model of the binocular vision system, the 3D spatial coordinates corresponding to the feature points are calculated using the triangulation principle to form 3D point cloud data.

[0082] In practical implementation, the following description uses a continued example scenario where a trajectory intersection analysis of a dynamic obstacle has been completed and its collision time estimated to be 3.5 seconds. In practice, the fusion weights of visible light and infrared data are dynamically adjusted based on the results of the trajectory intersection analysis and collision time estimation to generate an optimized fused image. Specifically, an attention factor inversely proportional to the collision time is set, and the formula for calculating the attention factor is as follows:

[0083]

[0084] in: Indicates the attention factor. This represents the estimated collision time. This represents a predefined reference time constant, set to five seconds, which can be understood as a focus factor. It is a dimensionless ratio, relative to the collision time. The correlation is inverse; the shorter the collision time, the higher the attention factor value. In some embodiments, the collision time... The value of this factor is greater than zero. When there is no intersection between trajectories, the collision time is assigned a maximum value, such as 1000 seconds, to make the attention factor approach zero. Data comparison is reflected in the calculation of the attention factor under different collision times. At 3.5 seconds, the attention factor The calculated value is approximately 1.43; when the collision time... Attention factor when it is 10 seconds The calculation is 0.5; when the collision time... At a time of 2 seconds, the attention factor The value is calculated to be 2.5.

[0085] In practice, the weight of infrared data in the fusion is dynamically adjusted based on the attention factor. Weighting of visible light data in fusion The weight adjustment follows the principle that the weight of infrared data increases with the increase of the attention factor, while the weight of visible light data increases with the decrease of the attention factor. The adjusted weights satisfy... Optionally, the mapping relationship between weights and attention factors is determined by a pre-defined lookup table, see Table 1 for a portion of the lookup table.

[0086] Table 1: Correspondence between Collision Time, Attention Factor, and Fusion Weight

[0087]

[0088] In practice, when the collision time is short, such as 0.5 seconds, the attention factor is increased to 10, and the weight of infrared data in the fusion is correspondingly increased to 0.85 to enhance temperature features. When the collision time is long or there is no intersection, such as 10 seconds or no intersection, the attention factor is decreased to 0.5 or 0, and the weight of visible light data in the fusion is correspondingly increased to 0.8 or 0.9 to enhance texture details. The pixel-level feature fusion step is re-executed using the adjusted weights to generate an optimized fused image, and the fusion calculation is re-executed but with dynamically adjusted weights. and Instead of the original fixed fusion coefficients. In some embodiments, weight adjustments are performed on a pixel-by-pixel or region-by-region basis, depending on the spatial correlation at collision time.

[0089] In the specific implementation, dense 3D reconstruction is performed on the optimized fused image, and 3D point cloud data of the obstacle surface is calculated. Specifically, a binocular vision system is constructed using the fixed positional relationship and imaging parameters between a visible light imaging device and an infrared imaging device. The optical centers of the visible light and infrared imaging devices are 0.2 meters apart, and their optical axes are parallel. Highly robust feature points are extracted from the optimized fused image using the SIFT algorithm, detecting thousands of key points and calculating their descriptors. The disparity between the feature points in the views of the visible light and infrared imaging devices is calculated. Disparity calculation is performed by matching corresponding feature points in the left and right views of the binocular vision system and calculating their horizontal pixel coordinate differences. Both feature point extraction and matching are based on the imaging constraints of the binocular vision system. The visible light and infrared imaging devices are used as fixed imaging references for the left and right views of the binocular vision system, limiting the search range for feature point matching, improving the accuracy and efficiency of feature point matching between the two views, and reducing invalid matches. Optionally, feature point matching uses descriptor-based nearest neighbor search and ratio testing. Based on the geometric model of parallax and binocular vision systems, the three-dimensional spatial coordinates of feature points are calculated using the triangulation principle to form three-dimensional point cloud data. The triangulation formula is:

[0090] in: This represents the depth distance of a feature point relative to the binocular vision system. Indicates the focal length of the imaging device. This represents the baseline distance between the optical centers of the visible light imaging device and the infrared imaging device. This represents the calculated disparity. It can be understood as focal length. Distance from baseline These are fixed parameters obtained from the calibration of a binocular vision system. In some embodiments, the dense 3D reconstruction process is performed on all extracted feature points on the optimized fused image, generating a set of 3D spatial coordinate points, i.e., 3D point cloud data.

[0091] Example 5: Based on 3D point cloud data and motion trajectory, the real-time distance and motion speed of obstacles are estimated. Specifically, a set of 3D spatial coordinate points belonging to the same obstacle is selected from the 3D point cloud data. The Euclidean distance from the geometric center of the 3D spatial coordinate point set to the optical center of the binocular vision system is calculated as the real-time distance. The displacement of the geometric center between the current frame and the previous frame is calculated by combining the geometric center position calculated in the previous frame. The motion speed is calculated by combining the frame interval time. Based on the updated pre-built scene feature library, the final identification and ranging of obstacles in the current frame are completed. Specifically, the real-time distance, motion speed, and multimodal descriptors calculated in the current frame are associated and stored with the historical records of obstacles in the pre-built scene feature library. The real-time distance, motion speed, and final obstacle identification are used as the output results of the current frame.

[0092] In practical implementation, the following description uses a continuation example scenario where dense 3D reconstruction has yielded 3D point cloud data containing a car and a pedestrian in front. The implementation estimates the real-time distance and speed of obstacles based on the 3D point cloud data and their movement trajectories. Specifically, this involves filtering the 3D point cloud data to identify sets of 3D spatial coordinate points belonging to the same obstacle. This filtering process associates the projection positions of the 3D spatial coordinate points onto the image pixel plane with the bounding boxes of previously identified potential obstacle regions. All 3D spatial coordinate points falling within the bounding boxes of the same potential obstacle region are classified as belonging to the same obstacle. Data comparison is reflected in the division of the 3D point cloud data. For example, out of a total of 10,000 3D spatial coordinate points, approximately 3,000 points have projection positions falling within the image region labeled "Obstacle 1," and these points are filtered into the set of 3D spatial coordinate points belonging to "Obstacle 1." Another approximately 500 points have projection positions falling within the image region labeled "Obstacle 2," and these points are filtered into the set of 3D spatial coordinate points belonging to "Obstacle 2." The remaining points are considered background points.

[0093] In practical implementation, the Euclidean distance from the geometric center of the three-dimensional spatial coordinate point set to the optical center of the binocular vision system is calculated as the real-time distance. The geometric center is obtained by calculating the arithmetic mean of the coordinates of all points in the three-dimensional spatial coordinate point set. This Euclidean distance is calculated using the optical center of the binocular vision system as a unified reference origin, which is determined during the calibration of the binocular vision system. Combined with the baseline distance and focal length of the visible light and infrared imaging devices, and other core binocular vision parameters, the calculated real-time distance ensures that it conforms to the three-dimensional spatial measurement logic of the binocular vision system. For the three-dimensional spatial coordinate point set of "obstacle one," its geometric center coordinates are calculated as (10.2, -0.3, 25.5), in meters. The coordinate system has its origin (0,0,0) at the optical center of the binocular vision system, with the Z-axis pointing directly forward. The Euclidean distance from the geometric center to the optical center is calculated to be 28.1 meters, which is the real-time distance of "obstacle one." It can be understood that the Euclidean distance calculation formula is the same as the formula for the straight-line distance between two points in three-dimensional space. In some embodiments, the position of the optical center of the binocular vision system is defined as the origin of the coordinate system during system calibration. Based on the geometric center position calculated in the previous frame, the displacement of the geometric center between the current frame and the previous frame is calculated, and the motion velocity is calculated based on the frame interval. The geometric center positions in both the current and previous frames are coordinate information in the three-dimensional global coordinate system of the binocular vision system. The displacement calculation is based on the three-dimensional spatial measurement rules of binocular vision, ensuring that the displacement calculation results accurately reflect the actual spatial movement distance of the obstacle within the binocular vision perception range. For the same "obstacle 1," its geometric center coordinates calculated in the previous frame are (10.0, -0.3, 26.8). The time interval between the current frame and the previous frame... Displacement is 0.1 seconds. The difference between the geometric center coordinates of the current frame and the previous frame is calculated, and the formula is expressed as:

[0094]

[0095] in: This represents the geometric center coordinates of the current frame (10.2, -0.3, 25.5). The coordinates of the geometric center of the previous frame are (10.0, -0.3, 26.8), and the displacement is calculated. Approximately 1.3 meters, speed of movement The calculated speed is approximately 13.0 m / s. Alternatively, the speed can be decomposed into components along the coordinate axes, for example, the speed in the X direction is approximately 2.0 m / s and the speed in the Z direction is approximately -13.0 m / s, indicating that the obstacle is approaching.

[0096] In specific implementation, the final identification and ranging of obstacles in the current frame are completed based on the updated pre-built scene feature library. Specifically, the real-time distance, speed, and multimodal descriptor calculated for the current frame are associated with and stored in the historical records of obstacles in the pre-built scene feature library. For "Obstacle 1" identified as having "known dynamics," a feature template record already exists in the pre-built scene feature library. The update operation includes appending the obstacle's currently calculated real-time distance (28.1 meters), speed (13.0 meters / second), and latest multimodal descriptor vector to the historical data list of this feature template. Simultaneously, the multimodal descriptor of the template can be updated to a moving average of historical data to reflect possible slow changes in its appearance. For new obstacles whose scene state is determined to be "unknown," a new feature template record is created in the pre-built scene feature library, storing its first extracted multimodal descriptor, calculated real-time distance, and speed, and assigning it a unique new identifier. In some embodiments, the pre-built scene feature library is organized using relational database tables or data structures. Understandably, the update process allows the scene feature library to accumulate historical information and adapt to environmental changes. Finally, the real-time distance, movement speed, and the final obstacle identifier are used as the output of the current frame. The output can be a structured data object; for example, for "Obstacle 1," the output is {Identifier: "Car_A_001", Real-time distance: 28.1 meters, Movement speed: 13.0 meters / second}, and for "Obstacle 2," the output is {Identifier: "Pedestrian_New_002", Real-time distance: 15.5 meters, Movement speed: 1.2 meters / second}.

[0097] See Figure 5 This is a grouped bar chart analyzing obstacle movement speeds. It displays the movement speeds of two types of obstacles: cars and pedestrians. This type of chart is commonly used for motion state analysis in fields such as obstacle recognition and autonomous driving, visually illustrating the speed differences and directions of movement of different obstacles. The total speed of the car is much higher than that of the pedestrian, and the Z-axis component is negative, indicating that the car is rapidly approaching the monitoring equipment, requiring close attention to collision risk. The pedestrian's speed is lower, and while the trend is also towards the pedestrian, the risk level is lower than that of the car.

[0098] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0099] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A vision-based obstacle recognition and ranging method, characterized in that, Includes the following steps: By simultaneously acquiring image pairs of the same scene using a binocular camera and a visible light imaging device and an infrared imaging device, an original set of image pairs is formed. Pixel-level feature fusion is performed on each image pair in the original image pair set to generate a fused feature image; Based on the fused feature image, potential obstacle regions in the scene are identified, and multimodal descriptors of the potential obstacle regions are extracted; The multimodal descriptors are matched with a pre-built scene feature library, and the scene state of the potential obstacle region is determined based on the matching results. Based on the scene state, construct motion trajectories for obstacles in the dynamic scene, and perform trajectory intersection analysis and collision time estimation; Based on the results of trajectory cross-analysis and collision time estimation, the fusion weights of visible light and infrared data are dynamically adjusted to generate an optimized fused image. Dense 3D reconstruction is performed on the optimized fused image to calculate the 3D point cloud data of the obstacle surface; Based on the three-dimensional point cloud data and the motion trajectory, the real-time distance and speed of the obstacle are estimated; Based on the real-time distance and movement speed, update the feature records of the corresponding obstacles in the pre-built scene feature library; Based on the updated pre-built scene feature library, the final identification and ranging of obstacles in the current frame are completed.

2. The vision-based obstacle recognition and ranging method according to claim 1, characterized in that, By simultaneously acquiring image pairs of the same scene using a binocular camera and a visible light imaging device and an infrared imaging device, an original image pair set is formed, specifically including: Control the visible light imaging device and the infrared imaging device to capture images at the same frame rate and timestamp; The captured visible light and infrared images are time-stamp aligned and verified, and images with time-stamp deviations exceeding a threshold range are discarded. The visible light image that has passed the timestamp alignment verification is paired with the infrared image to form the original image pair set.

3. The vision-based obstacle recognition and ranging method according to claim 2, characterized in that, Pixel-level feature fusion is performed on each image pair in the original image pair set to generate a fused feature image, specifically including: Edge enhancement processing is performed on the visible light images in the original image set, and temperature region segmentation processing is performed on the infrared images. The visible light image after edge enhancement and the infrared image after temperature region segmentation are mapped to the same scale space; Within the scale space, a fusion coefficient is assigned to each pixel location, and the corresponding pixel values ​​from the visible light image and the infrared image are weighted and calculated based on the fusion coefficient to generate the fused feature image.

4. The vision-based obstacle recognition and ranging method according to claim 3, characterized in that, Based on the fused feature image, potential obstacle regions in the scene are identified, and multimodal descriptors of the potential obstacle regions are extracted, specifically including: A sliding window is applied to the fused feature image to perform region scanning, and the texture complexity and gradient statistical features of the image within the window are calculated. Windows whose texture complexity and gradient statistical features exceed a preset threshold are marked as potential obstacle regions. Within each potential obstacle region, its color histogram, local gradient direction histogram, and temperature distribution histogram are calculated respectively, and these histograms are connected to form the multimodal descriptor.

5. The vision-based obstacle recognition and ranging method according to claim 4, characterized in that, The multimodal descriptors are matched with a pre-built scene feature library, and the scene state of the potential obstacle region is determined based on the matching results, specifically including: Calculate the similarity distance between the multimodal descriptor and each feature template in the pre-built scene feature library; Select the feature template with the smallest similarity distance, and determine whether the smallest similarity distance is less than the recognition threshold; If the value is less than the recognition threshold, the potential obstacle area is determined to be a known obstacle, and its scene state is either known static or known dynamic; otherwise, the potential obstacle area is determined to be a newly appearing obstacle, and its scene state is unknown.

6. The vision-based obstacle recognition and ranging method according to claim 5, characterized in that, Based on the scene state, motion trajectories are constructed for obstacles in the dynamic scene, and trajectory intersection analysis and collision time estimation are performed, specifically including: For obstacles whose scene state is known dynamic or unknown, track their position changes in multiple consecutive frames; The positional change is fitted into a smooth curve to construct the motion trajectory; Predict the extended path of the motion trajectory in the future, and calculate the intersection point and intersection time between the machine's motion path and the extended path. The intersection time is the estimated collision time.

7. The vision-based obstacle recognition and ranging method according to claim 6, characterized in that, Based on the results of trajectory cross-analysis and collision time estimation, the fusion weights of visible light and infrared data are dynamically adjusted to generate an optimized fused image, specifically including: Set an attention factor that is inversely proportional to the collision time; When the collision time is short, the attention factor is increased, and the weight of infrared data in the fusion is increased accordingly to enhance the temperature characteristics; When the collision time is long or there is no intersection, the attention factor is reduced and the weight of visible light data in the fusion is increased accordingly to enhance texture details; The pixel-level feature fusion step is re-executed using the adjusted weights to generate the optimized fused image.

8. The vision-based obstacle recognition and ranging method according to claim 7, characterized in that, Dense 3D reconstruction is performed on the optimized fused image to calculate the 3D point cloud data of the obstacle surface, specifically including: A binocular vision system is constructed by utilizing the fixed positional relationship and imaging parameters between the visible light imaging device and the infrared imaging device. Highly robust feature points are extracted from the optimized fused image, and the parallax of the feature points between the views of the visible light imaging device and the infrared imaging device is calculated. Based on the parallax and the geometric model of the binocular vision system, the three-dimensional spatial coordinates corresponding to the feature points are calculated using the triangulation principle to form the three-dimensional point cloud data.

9. The vision-based obstacle recognition and ranging method according to claim 8, characterized in that, Based on the three-dimensional point cloud data and the motion trajectory, the real-time distance and speed of the obstacle are estimated, specifically including: From the three-dimensional point cloud data, select the set of three-dimensional spatial coordinate points belonging to the same obstacle; Calculate the Euclidean distance from the geometric center of the three-dimensional spatial coordinate point set to the optical center of the binocular vision system, and use it as the real-time distance; Based on the geometric center position calculated in the previous frame, the displacement of the geometric center between the current frame and the previous frame is calculated, and the motion velocity is calculated based on the frame interval time.

10. The vision-based obstacle recognition and ranging method according to claim 9, characterized in that, Based on the updated pre-built scene feature library, the final identification and ranging of obstacles in the current frame are completed, specifically including: The real-time distance, the motion speed, and the multimodal descriptor calculated in the current frame are associated with and stored in the historical records of obstacles in the pre-built scene feature library. The real-time distance, the movement speed, and the final obstacle marker are used as the output of the current frame.

Images