Depth estimation system and method based on image segmentation

Abstract

A depth estimation system based on image segmentation is provided. The system includes a first photographic device for capturing an observation image; a second photographic device for capturing a target image; and a processing unit that executes instructions stored in a storage unit. The processing unit generates a feature map via feature extraction, identifies multiple segmentation blocks in multiple images via an image segmentation process, calculates a pair of epipolar constraints for each observation point, performs segmentation block-level matching and pixel-level matching based on the epipolar constraints to obtain a target point corresponding to that observation point, and estimates a depth value based on the disparity between the observation point and the corresponding target point.

Description

Technical Field

[0001] This invention relates to image analysis and depth estimation techniques, and more particularly to a depth estimation system based on image segmentation. Background Technology

[0002] Depth estimation is a fundamental task in computer vision and is widely used in autonomous driving, robotics, and environmental perception. When using multiple sensors or cameras for depth estimation, the system typically identifies the same object captured from different viewpoints and calculates the relative distance between the sensor and the object based on the disparity of the captured images. Therefore, the accuracy of feature matching between corresponding regions or objects in different images has a crucial impact on the reliability of the overall depth estimation.

[0003] However, feature matching for images taken from different perspectives is a challenging task. In practice, even the same entity can appear significantly different due to variations in lighting, perspective, image scale, or lens distortion, leading traditional algorithms to misidentify corresponding points or objects.

[0004] To mitigate these issues, traditional stereo vision systems often employ cameras of identical specifications and restrict their mounting to a parallel orientation and minimal baseline displacement to maintain image similarity and facilitate feature matching. This configuration limits the design flexibility and deployment adaptability of the sensing system.

[0005] Therefore, it is necessary to propose a depth estimation system and method that can overcome the above limitations and provide accurate and reliable depth estimation. Summary of the Invention

[0006] One embodiment of this disclosure provides a depth estimation system based on image segmentation. The system includes: a first imaging device configured to capture an observation image; and a second imaging device configured to capture a target image. The system further includes a processing unit and a storage unit coupled to the processing unit. The storage unit stores a plurality of instructions, which, when executed by the processing unit, cause the processing unit to: generate a first feature map and a second feature map based on the observation image and the target image, respectively, through feature extraction. The plurality of instructions further cause the processing unit to identify a plurality of segmented blocks in the observation image and the target image through an image segmentation process. For each observation point in the observed image, multiple instructions further instruct the processing unit to: calculate a pair of epipolar constraints corresponding to the observation point based on multiple external parameters of the first and second imaging devices; identify an observation segmentation block where the observation point is located among multiple segmentation blocks of the observed image; determine a target segmentation block that matches the observation segmentation block among multiple segmentation blocks of the target image based on the epipolar constraints; search for a target point that matches the observation point within the target segmentation block based on the first feature map, the second feature map, and the epipolar constraints; and estimate a depth value based on a disparity between the observation point and the target point.

[0007] In one embodiment, the processing unit determines the target segmentation block by performing the following steps: determining one or more candidate segmentation blocks among a plurality of segmentation blocks of the target image based on the epipolar constraint; and filtering the one or more candidate segmentation blocks by selecting only those among the one or more candidate segmentation blocks that have the same semantic category as the observed segmentation block.

[0008] In one embodiment, the processing unit determines the target segmentation block by performing the following steps: determining one or more candidate segmentation blocks among a plurality of segmentation blocks of the target image based on the epipolar constraint; and filtering the one or more candidate segmentation blocks by selecting only those among the one or more candidate segmentation blocks that have an appearance similarity greater than a similarity threshold with the observed segmentation block.

[0009] In one embodiment, the processing unit calculates the appearance similarity between the observed segmentation block and each of the one or more candidate segmentation blocks based on the first feature map and the second feature map.

[0010] In one embodiment, the processing unit determines the target segmentation block by performing the following steps: determining one or more candidate segmentation blocks among a plurality of segmentation blocks of the target image based on the epipolar constraint; calculating an observation distance between the first imaging device and a ground point of the observed segmentation block in a bird's-eye view coordinate system; calculating a candidate distance between the first imaging device and the ground point of each of the one or more candidate segmentation blocks in the bird's-eye view coordinate system; and selecting the candidate segmentation block having the smallest difference between the candidate distance and the observation distance as the target segmentation block.

[0011] In one embodiment, a plurality of instructions further cause the processing unit to: evaluate a discriminability score for each of a plurality of segmented blocks in the observed image and the target image, and assign a textureless label to the plurality of segmented blocks whose discriminability scores are lower than a discriminability threshold; generate reliable depth information from the depth values ​​estimated by the plurality of segmented blocks without the textureless label, and refine the depth values ​​of the plurality of segmented blocks with the textureless label to obtain refined depth information; and integrate the reliable depth information and the refined depth information to generate a depth map.

[0012] In one embodiment, the processing unit calculates the discriminability score of each of the plurality of segmented blocks based on a variance of a plurality of feature values ​​within the plurality of segmented blocks; wherein the variance of the plurality of feature values ​​is calculated based on at least one of the first feature map and the second feature map.

[0013] In one embodiment, the multiple instructions further instruct the processing unit to: determine whether the number of matching points of each of the multiple segmented blocks having the textureless label is sufficient; and mark the depth values ​​corresponding to the multiple segmented blocks with insufficient matching points as invalid in the refined depth information.

[0014] In one embodiment, the multiple instructions further instruct the processing unit to: for each of the multiple segmented blocks having a sufficient number of matching points and the textureless label, determine whether the multiple depth values ​​of the matching points in the segmented block are continuous; and mark the depth values ​​of the multiple segmented blocks with discontinuous depth values ​​as invalid in the refined depth information.

[0015] In one embodiment, the multiple instructions further cause the processing unit to assign multiple interpolated depth values ​​to multiple segmented blocks having the textureless label, a sufficient number of matching points, and multiple depth values ​​of the multiple matching points being consecutive, and to incorporate the multiple interpolated depth values ​​into the refined depth information.

[0016] In one embodiment, the first photographic device is a pinhole camera, and the second photographic device is a fisheye camera.

[0017] In one embodiment, multiple instructions further cause the processing unit to apply the depth value estimated from the observation point and the target point to perform obstacle detection in at least one autonomous navigation system or an advanced driver assistance system.

[0018] In one embodiment, multiple instructions further cause the processing unit to apply the depth value estimated from the observation point and the target point to perform scene reconstruction in an augmented reality system or a virtual reality system.

[0019] One embodiment of this disclosure provides a depth estimation method based on image segmentation. The method is executed by a processing unit and includes: generating a first feature map and a second feature map based on a viewing image and a target image respectively via feature extraction; and identifying multiple segmentation blocks in the viewing image and the target image via an image segmentation process. The viewing image is captured by a first photographic device, and the target image is captured by a second photographic device. For each viewing point in the viewing image, the method further includes: calculating a pair of epipolar constraints corresponding to the viewing point based on multiple extrinsic parameters of the first and second photographic devices; identifying a viewing segmentation block containing the viewing point among the multiple segmentation blocks of the viewing image; determining a target segmentation block matching the viewing segmentation block among the multiple segmentation blocks of the target image based on the epipolar constraints; searching for a target point matching the viewing point within the target segmentation block based on the first feature map, the second feature map, and the epipolar constraints; and estimating a depth value based on a disparity between the viewing point and the target point.

[0020] In one embodiment, determining the target segmentation block includes: determining one or more candidate segmentation blocks among multiple segmentation blocks of the target image based on the epipolar constraint; and filtering the one or more candidate segmentation blocks by selecting only the one or more candidate segmentation blocks that have the same semantic category as the observed segmentation block.

[0021] In one embodiment, determining the target segmentation block includes: determining one or more candidate segmentation blocks among multiple segmentation blocks of the target image based on the epipolar constraint; and filtering the one or more candidate segmentation blocks by selecting only those among the one or more candidate segmentation blocks that have an appearance similarity greater than a similarity threshold with the observed segmentation block.

[0022] In one embodiment, based on the first feature map and the second feature map, the appearance similarity between the observed segmentation block and each of the one or more candidate segmentation blocks is calculated.

[0023] In one embodiment, determining the target segmentation block includes: determining one or more candidate segmentation blocks among multiple segmentation blocks of the target image based on the epipolar constraint; calculating an observation distance between the first imaging device and a ground point of the observed segmentation block in a bird's-eye view coordinate system; calculating a candidate distance between the first imaging device and the ground point of each of the one or more candidate segmentation blocks in the bird's-eye view coordinate system; and selecting the candidate segmentation block that has the smallest difference between the candidate distance and the observation distance as the target segmentation block.

[0024] In one embodiment, the method further includes: evaluating a discriminability score for each of a plurality of segmented blocks in the observed image and the target image, and assigning a textureless label to the plurality of segmented blocks whose discriminability scores are lower than a discriminability threshold; generating reliable depth information from the depth values ​​estimated by the plurality of segmented blocks without the textureless label, and refining the depth values ​​of the plurality of segmented blocks with the textureless label to obtain refined depth information; and integrating the reliable depth information and the refined depth information to generate a depth map.

[0025] In one embodiment, the discriminability score of each of the plurality of segmented blocks is calculated based on a variance of a plurality of feature values ​​within the plurality of segmented blocks; wherein the variance of the plurality of feature values ​​is calculated based on at least one of the first feature map and the second feature map.

[0026] In one embodiment, the method further includes: for each of the plurality of segmented blocks having the textureless label, determining whether the number of matching points of the segmented block is sufficient; and in the refined depth information, marking the depth values ​​corresponding to the plurality of segmented blocks with insufficient matching points as invalid.

[0027] In one embodiment, the method further includes: for each of the plurality of matching points having a sufficient number and having the plurality of segmented blocks without texture labels, determining whether the plurality of depth values ​​of the matching points in the segmented block are continuous; and marking the plurality of depth values ​​of the plurality of segmented blocks with discontinuous depth values ​​as invalid in the refined depth information.

[0028] In one embodiment, the method further includes: assigning multiple interpolated depth values ​​to multiple segmentation blocks having the textureless label, a sufficient number of matching points, and multiple depth values ​​of the multiple matching points being consecutive, and incorporating the multiple interpolated depth values ​​into the refined depth information.

[0029] In one embodiment, the method further includes, in at least one of an autonomous navigation system or an advanced driver assistance system, applying the depth value estimated from the observation point and the target point to perform obstacle detection.

[0030] In one embodiment, the method further includes applying the depth value estimated from the observation point and the target point to perform scene reconstruction in an augmented reality system or a virtual reality system.

[0031] The depth estimation system and method disclosed herein integrate image segmentation, epipolar geometry, and hierarchical refinement techniques to achieve robust and reliable depth estimation in multi-view or cross-sensor configurations. By limiting the corresponding search space through segmentation block-level and pixel-level matching, the system of this disclosure effectively reduces computational complexity and minimizes false matches. Furthermore, the discriminative refinement mechanism improves the integrity and consistency of depth information, ensuring that both textured and textureless regions are accurately represented in the generated depth map. Through the combined use of semantic and geometric cues, the system of this disclosure maintains stable performance even in challenging scenarios, such as differences in viewpoint, varying lighting conditions, or differences in sensor modes, making it particularly suitable for applications such as autonomous navigation, obstacle detection, and scene reconstruction. Therefore, the system and method of this disclosure provide a practically deployable and scalable high-precision, highly robust depth estimation framework for real-world environments. Attached Figure Description

[0032] This disclosure will be more fully understood by reading the following detailed description and embodiments, and by referring to the accompanying drawings, in which: Figure 1 According to an embodiment of the present disclosure, an example of an original scene image and its corresponding image segmentation map is provided; Figure 2 This is a system architecture diagram of a depth estimation system according to an embodiment of the present disclosure; Figure 3A The data flow of a depth estimation method according to an embodiment of the present disclosure is described; Figure 3B The data flow of a depth calculation process according to an embodiment of the present disclosure is described; Figure 4 This provides an implementation example of the segmentation block-level matching steps according to an embodiment of the present disclosure; Figure 5 This section describes an implementation example of a pixel-level matching step according to an embodiment of the present disclosure; Figure 6 The detailed implementation of the segmentation block-level matching step according to an embodiment of the present disclosure is described; Figure 7 An implementation example of a semantic category filter according to an embodiment of the present disclosure is described; Figure 8An implementation example of an appearance similarity filter according to an embodiment of the present disclosure is illustrated; Figure 9 A flowchart illustrating an embodiment of the present disclosure describes a detailed implementation of a bird's-eye view distance filter. Figure 10 This describes an exemplary scenario of applying a bird's-eye view distance filter according to an embodiment of the present disclosure; Figure 11 This illustrates the advantages of a hierarchical matching strategy that first performs segmented block-level matching and then pixel-level matching, according to an embodiment of the present disclosure. Figure 12 This describes a data flow for generating a depth map according to an embodiment of the present disclosure; and Figure 13 The detailed implementation of the deep refinement steps according to an embodiment of the present disclosure is described.

[0033] The attached figures are labeled as follows: 101: Original scene image 102: Image segmentation diagram 20: Depth Estimation System 21: First Photographic Device 22: Second photographic device 23: Processing Unit 24: Storage Unit 204: Instruction 205: Depth value OI: Observation Image TI: Target Image 30: Depth Estimation Methods 301: First Feature Map 302: Second Feature Map 303, 304: Block splitting S31: Feature Extraction Steps S32: Image Segmentation Steps DCP: Depth Computation Process 305: External Parameters 306: Depth value 307: Epipolar Constraint S33: Steps for epipolar calculation S34: Block-level matching steps S35: Pixel-level matching steps S36: Depth Estimation Steps OP: Observation point OS: Observation Segmentation Block TS: Target Segment Block TP: Target Point 401: Elliptical segmentation block 402: Star-shaped partition block 403: Square Divided Block 411, 412, 413: Divide the blocks EL: Polar Line C1, C2, C3: Candidate points 61: Semantic Category Filter 62: Appearance Similarity Filter 63: Bird's-eye view distance filter 711, 712, 713: Divide the blocks 801, 802: Block partitioning S901, S902, S903: Steps 1001, 1003, 1005: Students 1002, 1004: Ground points Observation distance Bird's-eye view distance 1101, 1102: Divide the blocks 1201: No segmented blocks without texture labels 1202: Segmentation blocks with or without texture tags 1203: Reliable Depth Information 1204: Refining Deep Information 1205: Depth Map S121: Discrimination Assessment Steps S122: Deep Refinement Steps S123: Steps for Deep Information Integration 1301, 1302: Conditions 1303: Invalid depth value 1304: Interpolation depth value Detailed Implementation

[0034] The following description is intended to illustrate the general principles of this disclosure and should not be construed as limiting. The scope of this disclosure is defined by the appended claims.

[0035] In the following embodiments, the same reference numerals denote the same or similar elements or components.

[0036] The serial numbers used in the claims, such as "first", "second", "third", etc., are for ease of explanation only and do not indicate any sequential relationship between them.

[0037] The descriptions provided below for embodiments of apparatus or systems are equally applicable to embodiments of methods, and vice versa.

[0038] This disclosure provides a depth estimation system that utilizes semantic priors to improve the accuracy and robustness of feature matching. Unlike searching for pixel-level correspondences across the entire image, this system first performs region-level or segment-level correspondence searches based on image segmentation. By determining corresponding object segments among multiple images, the system restricts the pixel-level matching process to a finite search space defined by matched segmented segments and an epipolar constraint. This hierarchical matching strategy reduces the probability of mismatches, improves computational efficiency, and generates more stable and accurate depth estimation results even with significant viewpoint differences.

[0039] Figure 1 An example of an original scene image 101 and its corresponding image segmentation map according to an embodiment of the present disclosure is provided. For example... Figure 1 As shown, the original scene image 101 includes multiple objects, such as vehicles and riders, located at different depths within the driving environment. Image segmentation image 102 is generated from the original scene image 101 through an image segmentation process, where pixels are classified into different object or region categories, such as roads, vehicles, pedestrians, and buildings. In image segmentation image 102, each object category is presented as a visually distinguishable region, for example, by assigning different colors or labels to pixel groups to clearly mark the extent of each object category. Image segmentation image 102 provides this category hierarchy information, allowing subsequent depth estimation to be performed at the segmentation block level, rather than on the entire image, thereby limiting the search space corresponding to the pixel level and improving the overall matching stability.

[0040] Figure 2 This is a system architecture diagram of a depth estimation system 20 according to an embodiment of the present disclosure. Figure 2 As shown, the depth estimation system 20 includes two imaging devices (hereinafter referred to as the first imaging device 21 and the second imaging device 22, respectively), a processing unit 23, and a storage unit 24.

[0041] The first camera device 21 and the second camera device 22 are image capture devices used to acquire image data of a scene from different perspectives. The first camera device 21 captures an observation image OI, and the second camera device 22 captures a target image TI. The first camera device 21 and the second camera device 22 can be implemented using a pinhole camera, a fisheye camera, or any other type of optical or digital image sensor, respectively. The first camera device 21 and the second camera device 22 can be fixed at different positions or orientations so that the same object in the scene can be photographed from different perspectives, thereby allowing depth information to be derived.

[0042] The processing unit 23 can be implemented using any computing device or hardware circuit capable of performing arithmetic or logical calculations, such as a central processing unit (CPU), graphics processing unit (GPU), neural processing unit (NPU), application-specific integrated circuit (ASIC), programmable logic array (FPGA), system-on-a-chip (SoC), or a combination thereof. The processing unit 23 may further include multiple computing cores or parallel accelerators configured for parallel computing.

[0043] Storage unit 24 is coupled to processing unit 23 and may include one or more types of memory, such as read-only memory (ROM), random access memory (RAM), or one or more non-volatile storage media. Storage unit 24 stores instructions 204, which may be implemented as software programs, executable code, or firmware. These programs, code, or firmware may be written in any programming language, such as C, C++, or Python, but are not limited thereto. When instructions 204 are executed by processing unit 23, processing unit 23 executes the depth estimation method described in this disclosure to obtain one or more depth values ​​205 based on the observed image OI and the target image TI.

[0044] Processing unit 23 can acquire the observation image OI and target image TI from the first imaging device 21 and the second imaging device 22 via any wired or wireless communication interface. Such interfaces include, but are not limited to, Universal Serial Bus (USB), PCIe, Gigabit Ethernet, or serial interfaces such as the Mobile Industry Processor Interface Camera Serial Interface (MIPI-CSI). In other embodiments, wireless transmission interfaces include Wi-Fi, Bluetooth, or dedicated automotive communication buses. The choice of a specific communication interface may depend on system bandwidth, latency, and installation limitations, but this disclosure is not limited thereto.

[0045] In one embodiment, the first imaging device 21 is a pinhole camera, and the second imaging device 22 is a fisheye camera. The pinhole camera provides a relatively narrow field of view with minimal geometric distortion, enabling it to capture scenes at a perspective close to human visual perception while maintaining accurate spatial proportions of objects. In contrast, the fisheye camera has an ultra-wide field of view (typically exceeding 180 degrees), capable of capturing peripheral areas not covered by the pinhole image, despite significant geometric distortion. Due to their complementary characteristics, the observation image (OI) obtained by the pinhole camera can accurately identify objects in the forward direction, while the target image (TI) obtained by the fisheye camera covers a wider scene, including lateral areas or objects appearing near the edge of the vehicle's field of view. This configuration is particularly advantageous in automotive applications, such as in autonomous driving systems, where combining detailed perspective views with wide-angle view maps allows for more comprehensive depth estimation from different perspectives and improves robustness.

[0046] In one embodiment, the processing unit 23 may further apply the estimated depth value 205 to at least one of an autonomous navigation system or an advanced driver assistance system to perform obstacle detection. By determining the relative distance between the vehicle and surrounding objects based on the depth value 205, the system can identify obstacles falling within a preset safety range. Once an obstacle is detected, the autonomous navigation system can generate a collision risk map and determine an avoidance trajectory based on the spatial distribution of the depth value 205. Alternatively, the depth value 205 may also be applied to an advanced driver assistance system to support functions including but not limited to lane keeping, adaptive cruise control, and emergency braking, wherein the system assesses the proximity of the obstacle and the feasibility of the trajectory to provide corrective steering or speed adjustments, thereby improving driving safety.

[0047] The depth value 205 can also be used to construct a map of the drivable area to distinguish between ground areas and obstacles above ground. Based on this information, the navigation system can dynamically adjust the vehicle's steering angle, acceleration, and braking parameters to ensure safe driving along a feasible path. By continuously acquiring and updating the depth value 205, the system can perform obstacle detection and path planning in real time with greater accuracy and responsiveness.

[0048] In another embodiment, processing unit 23 may further apply the estimated depth value 205 to perform scene reconstruction in an augmented reality (AR) or virtual reality (VR) system. In this embodiment, the estimated depth value 205 provides a three-dimensional representation of the surrounding environment, enabling virtual objects to be correctly placed and rendered relative to real-world objects, with appropriate occlusion and scaling. This scene reconstruction can enhance spatial realism and user immersion in AR or VR applications.

[0049] Figure 3AThis describes an embodiment of the present disclosure, by Figure 2 The data flow of the depth estimation method 30 executed by the processing unit 23 in the middle. For example Figure 3A As shown, this depth estimation method includes a feature extraction step S31, an image segmentation step S32, and a depth calculation process DCP.

[0050] In the feature extraction step S31, a first feature map 301 and a second feature map 302 are generated by feature extraction based on the observed image OI and the target image TI, respectively.

[0051] The observation image (OI) and the target image (TI) represent images captured from different viewpoints, for example... Figure 2 The images shown are captured by the first camera device 21 and the second camera device 22. The two images present the same scene, but due to the geometric relationship between the two cameras and the differences in their specifications, the two images differ in proportion, angle and appearance.

[0052] Feature maps 301 and 302 are both high-dimensional representations obtained from the corresponding images, where the original pixel data is converted into compact descriptors that encode local textures, edges, shapes, and other distinctive features of the scene. The first feature map 301 corresponds to the observed image OI, and the second feature map 302 corresponds to the target image TI. These feature maps will be used as reference data for subsequent correspondence search and depth estimation.

[0053] Feature extraction can be implemented using various machine learning algorithms, such as convolutional neural networks (CNNs), Transformer-based visual models, or traditional manual feature extractors, such as scale-invariant feature transforms (SIFTs) or histograms of oriented gradients (HOGs). In neural network implementations, hierarchical feature representations robust to changes in illumination, rotation, and scale can be learned by using multiple convolutional layers or attention modules, but this disclosure is not limited thereto.

[0054] In image segmentation step S32, multiple segmentation blocks 303 and 304 are determined from the observed image OI and the target image TI, respectively, through the image segmentation process. Each segmentation block corresponds to a pixel region with similar visual or statistical characteristics (e.g., color, texture, brightness, edge continuity, etc.). In some embodiments, the image segmentation blocks may also generate regions that roughly correspond to meaningful objects or surfaces (e.g., vehicles, pedestrians, buildings, or road areas).

[0055] In data representation, the image segmentation process divides an image into multiple labeled or indexed regions. Each pixel in the image can be associated with a segmentation block identifier, forming a segmentation map. Pixels with the same identifier in the segmentation map belong to the same segmentation block. For example, regions with different appearance or spatial attributes can be encoded using different indexes or color codes. In practice, segmentation blocks 303 and 304 can be stored as index maps or label matrices, where each pixel value identifies the segmentation block ID or classification label to which the pixel belongs.

[0056] Image segmentation can be implemented using a variety of algorithms. In one embodiment, a deep learning-based segmentation network can be used to perform pixel-level classification or instance-level delineation on the input image. For example, based on convolutional or Transformer models, semantic segmentation output can be generated, where each pixel is assigned a category label (e.g., "vehicle" or "pedestrian"), or instance segmentation output can be generated, where each object in the same category is assigned a unique identifier. In another embodiment, classic clustering-based or graph-based methods, such as mean-shift segmentation or region merging, can be used to divide the image into multiple consistent regions based on the similarity of visual features such as color, texture, or spatial continuity.

[0057] In summary, the term "image segmentation" as used in this specification broadly refers to any technique that segments an image into multiple coherent regions, including but not limited to semantic segmentation, instance segmentation, or unsupervised region-based segmentation. Such techniques can be combined with edge-preserving region delineation, multi-scale feature encoding, or cross-sensor feature correlation to ensure stable segmentation results across different environments and camera configurations. The segmentation output defines visually or semantically coherent image regions that provide region-level prior information for subsequent matching and depth estimation processes. These segmentation results can be represented as a label matrix or index map, where pixel values ​​label category labels, or, in the case of instance-level segmentation, as unique segmentation block identifiers.

[0058] The aforementioned feature extraction step S31 and image segmentation step S32 provide a reference basis for subsequent processing in the depth calculation flow (DCP). This reference information includes the first feature map 301, the second feature map 302, and segmentation blocks 303 and 304 extracted from the observed image OI and the target image TI, respectively. These data serve as complementary clues, jointly promoting accurate correspondence search and depth estimation. The following will refer to... Figure 3B Describe in detail the operation of the Depth Calculation Process (DCP) and explain how the DCP estimates the depth value 205 based on the aforementioned feature map and segmented blocks.

[0059] Figure 3B The data flow of the depth calculation process (DCP) according to an embodiment of this disclosure is described. For example... Figure 3B As shown, the depth calculation process DCP includes an epipolar calculation step S33, a segmentation block level matching step S34, a pixel level matching step S35, and a depth estimation step S36. These steps are performed for each observation point OP in the observed image OI to estimate its corresponding depth value 306. As referred to in this specification, the observation point OP refers to a pixel position selected from the observed image OI and located within the observed segmentation block OS, which is one of the segmentation blocks identified in the observed image through image segmentation. The processing unit 23 can iteratively execute the depth calculation process DCP repeatedly for multiple observation points in the observed image OI to generate multiple depth values ​​306, which together form Figure 3A The described depth value is 205.

[0060] An observation point (OP) represents the pixel location in an observation image (OI) where depth information needs to be inferred. Depending on the application, observation points can be selected in different ways. In dense stereo estimation tasks, almost all pixels in the OI can be considered observation points. In contrast, in sparse depth estimation or feature applications, only a subset of pixels (e.g., corners, edges, or regions of interest) may be selected as observation points. For example, in autonomous driving scenarios, observation points might correspond to detected obstacles or lane markings; while in augmented reality systems, observation points might correspond to key visual anchors used for spatial alignment.

[0061] In the polar calculation step S33, the processing unit 23 calculates the polarity based on the external parameters 305 of the imaging device (e.g., ...). Figure 2 The first photographic device 21 and the second photographic device 22 shown are used to calculate the epipolar constraint 307 corresponding to the observation point OP.

[0062] External parameter 305 defines the relative attitude between the two cameras, enabling the system to geometrically correlate a point in the observed image OI with the corresponding epipolar line in the target image TI. Specifically, external parameter 305 may include a rotation matrix and a translation vector to describe the transformation from the coordinate system of the first camera device 21 to the coordinate system of the second camera device 22. In some embodiments, these parameters can be obtained through a one-time calibration process or dynamically updated based on vehicle motion data or an inertial measurement unit (IMU). The accuracy of the external parameters directly affects the accuracy of the epipolar constraint, and consequently, the correctness of point matching between the two images.

[0063] Epipolar constraints are geometric properties used to establish relationships between corresponding points in images captured by cameras from different viewpoints. This constraint narrows the search space for pixel matching by imposing the requirement that "corresponding points must lie on a specific path defined by the relative positions of the cameras." Specifically, epipolar constraint 307 describes the geometric relationship between an observation point OP in the observation image OI and its potential corresponding point (hereinafter referred to as "target point") in the target image TI. Epipolar constraint 307 can be defined by the set of coefficients of the epipolar line, such as a linear equation. + + The set of coefficients with 0 is (a, b, c). Target points in the target image TI must be located on or near the epipolar line determined by camera geometry. This constraint will serve as a geometric prior for subsequent segmentation block-level and pixel-level matching steps.

[0064] In the segmentation block-level matching step S34, processing unit 23 determines the corresponding region of the target image TI, and searches for a matching relationship with the observation point OP in this region. Specifically, processing unit 23 first determines the observation segmentation block OS, which includes the observation point OP, among multiple segmentation blocks 303 of the observation image OI. The observation segmentation block OS represents a semantically or spatially consistent region surrounding the observation point, where pixels typically have similar depth or visual characteristics.

[0065] Next, processing unit 23 determines the target segmentation block TS corresponding to the observed segmentation block OS among multiple segmentation blocks 304 of the target image TI. The determination of the target segmentation block TS is guided by the epipolar constraint 307 obtained in the previous step (i.e., epipolar calculation step S33). Therefore, only segmentation blocks in the target image TI that intersect with the corresponding epipolar line are considered as candidate segmentation blocks. Thus, the segmentation block level matching step S34 provides a region-level correspondence architecture, which serves as a spatial constraint for the subsequent pixel-level matching step S35.

[0066] Figure 4An implementation example of segmentation block-level matching step S34 according to an embodiment of the present disclosure is provided. For example... Figure 4 As shown, the observed image OI comprises multiple segmentation blocks, such as elliptical segmentation block 401, star-shaped segmentation block 402, and square segmentation block 403. The observation point OP is located within elliptical segmentation block 401. The target image TI also includes corresponding segmentation blocks 411, 412, and 413, which represent regions with the same or similar semantic categories as those in the observed image OI, but captured by different camera perspectives.

[0067] exist Figure 4 In the example shown, processing unit 23 first determines the elliptical segmentation block 401 as the observation segmentation block OS, which includes the observation point OP. Then, under the epipolar constraint defined by the epipolar line EL, which indicates the pixel location that may correspond to the observation point OP, processing unit 23 searches for one or more candidate segmentation blocks in the target image TI that intersect with the epipolar line EL. In this example, only segmentation block 411 is determined as the target segmentation block TS because it is the only segmentation block that intersects with the epipolar line EL.

[0068] Reference Figure 3B In the pixel-level matching step S35, the processing unit 23 searches for a target point TP corresponding to the observation point OP within the target segmentation block TS determined in the previous step (i.e., the segmentation block-level matching step S34). In this step, the search is performed under the constraint of epipolar constraint 307, therefore the potential target point TP must be located on or close to the epipolar line associated with the observation point OP.

[0069] The matching evaluation between observation point OP and target point TP is based on the feature information included in the first feature map 301 and the second feature map 302. Specifically, the processing unit 23 compares the feature descriptors of pixels located on the epipolar line in the target segmentation block TS with the feature descriptors at observation point OP to find the pixel with the highest similarity or lowest matching cost as the target point TP.

[0070] By restricting the pixel-level search to the geometrically effective region (defined by epipolar constraint 307) and the semantically relevant region (defined by the results of segmentation block-level matching), the system establishes an accurate and robust correspondence between the observed image OI and the target image TI. The correspondence obtained between the observed point OP and the target point TP will serve as the basis for the subsequent depth estimation step S36.

[0071] Figure 5 An implementation example of pixel-level matching step S35 according to an embodiment of the present disclosure is described. For example... Figure 5As shown, observation point OP is located within the observation segmentation block OS in the observation image OI, and the target segmentation block TS corresponding to observation point OP in the target image TI has been determined by the segmentation block level matching step S34. Within the target segmentation block TS, multiple candidate points, such as C1 to C3, can be found along the epipolar line EL.

[0072] For each candidate point, processing unit 23 compares its local feature descriptor obtained from the second feature map 302 with the feature descriptor of the observation point OP obtained from the first feature map 301. A similarity value is calculated for each comparison, representing the probability that each candidate point corresponds to the same entity point observed by the two cameras. For example... Figure 5 As shown, the similarity values ​​of candidate points C1 to C3 are 30%, 95%, and 50%, respectively. Processing unit 23 selects the candidate point with the highest similarity (C2 in this example) as the target point TP corresponding to the observation point OP.

[0073] Reference Figure 3B In the depth estimation step S36, processing unit 23 estimates the depth value 306 of observation point OP based on the correspondence established between observation point OP and target point TP. It calculates the disparity between observation point OP in the observation image OI and target point TP in the target image TI based on their pixel coordinates. This disparity represents the apparent displacement of the same entity point in the fields of view of both cameras and is negatively correlated with the actual depth of the point relative to the camera.

[0074] In some embodiments, processing unit 23 uses known extrinsic parameters 305 and internal correction parameters of the first and second imaging devices to convert pixel parallax into depth values ​​306. This process can be implemented using geometric triangulation or an equivalent projection-based depth calculation model. In further embodiments, depth estimation step S36 can be further combined with local confidence weighting or feature-based consistency checks to improve the final depth accuracy.

[0075] By iteratively performing depth estimation step S36 on multiple observation points OP on the observed image OI, a series of depth values ​​306 are obtained and formed. Figure 3A The depth value shown is 205.

[0076] Figure 6A more detailed implementation of the segmentation block-level matching step S34 according to an embodiment of the present disclosure is described. As described above, in the segmentation block-level matching step S34, the processing unit 23 first determines one or more candidate segmentation blocks from multiple segmentation blocks of the target image TI based on epipolar constraints. To further refine the candidate segmentation blocks and more accurately determine the target segmentation block TS, the processing unit 23 may apply one or more filtering processes, such as a semantic category filter 61, an appearance similarity filter 62, and a bird's-eye view distance filter 63. The above filters may be selectively applied or applied together depending on the application requirements and the characteristics of the input data. Furthermore, the execution order of the filters is not limited here and may be dynamically adjusted or executed in parallel to improve computational efficiency or robustness.

[0077] The semantic category filter 61 filters candidate segmentation blocks in the target image TI by comparing the semantic labels of candidate segmentation blocks in TI with the semantic labels of the observed segmentation block OS. Specifically, only candidate segmentation blocks with the same semantic category as the observed segmentation block OS are retained as possible matches. This filtering process effectively eliminates semantically irrelevant regions, avoids erroneous matching results, thereby narrowing the search space and improving matching accuracy. The semantic labels used for filtering can be derived from, for example... Figure 3A The result of the image segmentation step S32 shown is obtained.

[0078] Figure 7 An implementation example of a semantic category filter 61 according to an embodiment of the present disclosure is described. For example... Figure 7 As shown, the observed image OI includes the observed segmentation block OS corresponding to "pedestrian", while the target image TI includes multiple segmentation blocks 711, 712 and 713 intersecting with the epipolar line EL. In the example described here, segmentation blocks 711, 712 and 713 are the segmentation results obtained based on image segmentation step S32, and are labeled as "pedestrian", "dog" and "vehicle", respectively.

[0079] When applying semantic category filter 61, processing unit 23 selects only candidate segmentation blocks in the target image TI whose semantic labels match the observed segmentation block OS. Figure 7 In the example shown, since the observed segmentation block OS is labeled as "pedestrian", only segmentation block 711 in the target image TI meets this condition and is retained as a valid candidate segmentation block, while other segmentation blocks 712 and 713 are excluded and will not be considered further.

[0080] The appearance similarity filter 62 refines the candidate segmentation blocks in the target image TI by evaluating the visual similarity between each candidate segmentation block in the target image TI and the observed segmentation block OS. Specifically, the processing unit 23 calculates the appearance similarity measurement value of each candidate segmentation block based on the feature values ​​(such as shape, color, texture, gradient distribution, or high-dimensional embedding vectors obtained by feature extraction) extracted in the feature extraction step S31.

[0081] Of all candidate segmentation blocks, only those whose appearance and observed segmentation block OS similarity exceed a preset similarity threshold will be retained as valid matches; the remaining segmentation blocks will be excluded. This similarity threshold can be adaptively determined based on the statistical characteristics of the feature map, or it can be set empirically based on specific application requirements.

[0082] Figure 8 An implementation example of an appearance similarity filter 62 according to an embodiment of the present disclosure is described. For example... Figure 8 As shown, the observed image OI includes an observed segmentation block OS, while the target image TI includes two candidate segmentation blocks 801 and 802 that intersect the epipolar line EL. The processing unit 23 compares the appearance features of each candidate segmentation block with the appearance features of the observed segmentation block OS and calculates their respective similarity values.

[0083] In the example illustrated here, candidate segmentation block 801 has a 95% appearance similarity to the observed segmentation block OS, while candidate segmentation block 802 has only a 50% similarity. Therefore, candidate segmentation block 801 is selected as the target segmentation block TS. Through this process, the appearance similarity filter 62 can effectively eliminate regions that belong to the same semantic category but have different visual characteristics, thereby improving the accuracy of the region hierarchical correspondence.

[0084] Figure 9 A flowchart illustrating an embodiment of this disclosure details the implementation of the bird's-eye view distance filter 63. Figure 9 As shown, the bird's-eye view distance filter 63 may include steps S901–S903, each of which will be described in detail below.

[0085] In step S901, the observation distance between the first photographic device and the ground point of the observation segment block is calculated in the bird's-eye view coordinate system. The bird's-eye view coordinate system represents a top-down spatial reference frame, in which the position of the object is represented relative to the ground plane. The ground point of the observation segment block can be defined as the representative contact point between the observed object and the ground, serving as a geometric reference for distance comparison.

[0086] In practice, the observation distance can be calculated in two ways: first, identify the ground points of the observation segment in the original image, and then project these ground points onto the bird's-eye view coordinate system; or, first project the entire observation segment onto the bird's-eye view coordinate system, and then determine its ground points within the projected area. This disclosure does not limit the specific order or manner of the above actions.

[0087] In step S902, for each candidate segmentation block located in the target image, the candidate distance between the first imaging device and the ground point of the candidate segmentation block is calculated in the bird's-eye view coordinate system. Similar to step S901, this calculation can be performed in two ways: first, identify the ground point, and then project the ground point onto the bird's-eye view coordinate system; or, first project the entire candidate segmentation block, and then determine its ground point in the bird's-eye view coordinate system.

[0088] By representing the observation distance and candidate distance in the same bird's-eye view coordinate system, geometric distortion caused by differences in camera perspective or orientation can be reduced, allowing for a more consistent comparison of the relative positions of segmented blocks in different images.

[0089] In step S903, the processing unit 23 selects the candidate segmentation block with the smallest difference between the candidate distance and the observation distance as the target segmentation block. This smallest difference represents the smallest positional deviation of the corresponding ground points of the observed segmentation block and the candidate segmentation block in the bird's-eye view coordinate system, thus indicating the highest probability of geometric correspondence between the two.

[0090] The selection criteria of the bird's-eye view distance filter 63 ensure that the final target segmentation block TS not only conforms to the epipolar constraint but is also geometrically aligned with the observed segmentation block in real space. Even if multiple candidate segmentation blocks have the same semantic category and similar visual appearance, the bird's-eye view distance filter 63 can still further distinguish them based on their relative distance to the ground. This allows the system to exclude erroneous matches that are semantically or visually similar but have significantly different spatial locations, thereby improving the overall reliability and geometric accuracy of the segmentation block hierarchical matching process in multi-view or cross-sensor configurations.

[0091] Figure 10 An example scenario illustrating the application of the bird's-eye view distance filter 63 according to an embodiment of this disclosure is described. For example... Figure 10 As shown, the observation image OI and the target image TI capture multiple students 1001, 1003, and 1005 crossing a pedestrian crossing. Since the students are all wearing the same uniform and have similar body types, their visual appearances are highly similar, making it difficult to distinguish them using only the semantic category filter 61 or the appearance similarity filter 62.

[0092] Each student corresponds to a ground point, which represents the position where the student's feet touch the ground. This point is marked as ground point 1002 in the observation image OI and ground point 1004 in the target image TI. The observation distance between the first imaging device and ground point 1002 is... This is calculated in the bird's-eye view coordinate system, as shown by the dashed arc below the observed image OI. Similarly, the target image TI includes multiple candidate segmentation blocks, each corresponding to a ground point (e.g., 1004 and 1006), and their respective distances from the bird's-eye view of the photographic device. and .

[0093] In this example, the two segmented blocks of students 1003 and 1005 are both located on the same epipolar line EL, and have similar semantic and visual features to the observed segmented block of student 1001. However, the bird's-eye view distance filter 63 will select the segmented block with the observation distance. =1m The candidate segmentation block with the smallest difference (corresponding to =1.1m), as the correct target segmentation block. This example illustrates that the bird's-eye view distance filter 63 can effectively distinguish geometrically different but visually similar objects, ensuring accurate correspondence even in challenging real-world scenarios such as pedestrian crossings or busy traffic environments.

[0094] Figure 11 This describes the advantages of a hierarchical matching strategy, which first performs segmented block-level matching and then pixel-level matching, according to an embodiment of this disclosure. Figure 11 As shown, the observed image OI comprises an observed segmentation block OS containing the observed point OP, while the target image TI comprises multiple candidate points C1 and C2 distributed along the corresponding epipolar line EL. In this example, candidate points C1 and C2 have almost the same feature similarity score (e.g., 95%) relative to the observed point OP, making it impossible to distinguish between them using a traditional pixel-level matching process that relies solely on feature comparison.

[0095] However, in the hierarchical matching strategy disclosed herein, the search space for pixel-level matching is first limited to the segmentation block 1101 (i.e., the target segmentation block) determined by the previous segmentation block hierarchical matching step. Therefore, only candidate points located within the target segmentation block (e.g., C1) are evaluated, while points located within irrelevant segmentation blocks (e.g., C2 within segmentation block 1102) are excluded from the matching process.

[0096] This hierarchical matching method not only reduces computational complexity by shrinking the search space along the epipolar line, but also significantly improves the robustness of the matching results by ensuring geometric and semantic consistency between corresponding image regions. Therefore, it effectively avoids erroneous matches caused by visually similar points with different contexts, resulting in more accurate and stable depth estimation results.

[0097] In one embodiment, after estimating the depth values ​​205 for each observation point through the depth computation process (DCP), the processing unit 23 can further refine these depth values ​​and integrate them into a complete and consistent depth map. This refinement can compensate for unreliable or incomplete depth estimations that often occur in texture-less or feature-sparse regions, thereby improving the overall quality of depth perception. Therefore, the refined and integrated depth information can be represented as a dense and geometrically coherent depth map for use in subsequent perception or reconstruction tasks. The following references... Figure 12 These additional processing steps are described in detail.

[0098] Figure 12 The data flow for generating a depth map 1205 according to an embodiment of this disclosure is described. For example... Figure 12 As shown, processing unit 23 can perform additional steps to generate depth map 1205, including a distinctiveness evaluation step S121, a depth refinement step S122, and a depth information integration step S123. The details of each step will be explained below. These improvements are based on... Figure 3A The depth value 205 generated by the depth calculation process DCP described herein is combined with the segmented blocks identified in the observed image and the target image. Both sets of segmented blocks undergo discriminability evaluation in the discriminability evaluation step S121 to determine whether they are textureless.

[0099] In the discriminability evaluation step S121, the processing unit 23 evaluates the discriminability scores of multiple segmentation blocks 303 and 304 in the observed image OI and the target image TI, and marks the segmentation blocks with discriminability scores lower than a preset discriminability threshold as "no texture". The discriminability score represents a quantitative indicator of local variation or visual richness within the segmentation block, reflecting whether the features of the region are easily uniquely matched between the two images.

[0100] Segmentation blocks with high discriminative scores typically correspond to regions with strong texture or obvious visual structure, such as vehicle bodies, building facades, and traffic signs. The local features of these regions can be reliably detected and matched. In contrast, segmentation blocks with low discriminative scores typically correspond to textureless or uniform regions, such as flat roads, the sky, or walls with extremely low contrast. These regions lack sufficient feature cues to establish effective correspondences.

[0101] Therefore, processing unit 23 divides the segmented blocks into two categories: segmented blocks 1201 without texture labels (i.e., regions with rich textures or obvious features) and segmented blocks 1202 with texture labels (i.e., regions without textures or with fewer features), such as... Figure 12 As shown.

[0102] In one embodiment, the discriminative score of each segmented block can be calculated based on the variance of feature values ​​within that segmented block. Feature values ​​can be obtained from at least one of a first feature map 301 and a second feature map 302. The variance of the feature values ​​quantifies the dispersion of local feature responses within the segmented block. A higher variance indicates that the segmented block has more diverse patterns or textures, thus generating a higher discriminative score. Conversely, a lower variance indicates less variation within the segmented block, potentially making it a textureless region and prone to confusion during correspondence estimation. Therefore, segmented blocks with variance below the discriminative threshold are labeled as textureless for subsequent refinement.

[0103] In the depth refinement step S122, the processing unit 23 directly uses the depth estimate corresponding to the segmentation block 1201 without texture labels to generate reliable depth information 1203, and simultaneously refines the depth values ​​of the segmentation block 1202 with and without texture labels to obtain refined depth information 1204. In this step, the segmentation block 1201 without texture labels provides reliable depth information 1203, while the segmentation block 1202 with and without texture labels needs to undergo interpolation or verification processes to generate refined depth information 1204.

[0104] This difference stems from the varying degrees of reliability corresponding to the two types of segmentation blocks. Segmentation block 1201 without texture labels typically provides robust feature correspondences and generates depth values ​​that can be considered reliable without further adjustments. In contrast, segmentation block 1202 with and without texture labels corresponds to texture defects or visual uniformity, potentially leading to undetermined matching or sparse regions with valid correspondences. Therefore, prior to integration, additional refinement of segmentation block 1202 with and without texture labels is required to improve the integrity and smoothness of its depth values.

[0105] In the depth information integration step S123, the processing unit 23 integrates reliable depth information 1203 and refined depth information 1204 to generate a complete depth map 1205. The integration process ensures that the two types of depth information are correctly aligned in space and seamlessly merged, thereby forming a dense and continuous depth representation covering the entire image area. The final depth map 1205 can provide accurate geometric information in both texture-rich and textureless regions, thus enabling stable downstream processing such as object detection, scene reconstruction, and path planning.

[0106] In one embodiment, for each segmentation block 1202 with or without texture labels, the processing unit 23 determines whether the segmentation block contains a sufficient number of matching points. A "matching point" refers to a pixel correspondence successfully established between the observed image and the target image within the segmentation block. A sufficient number of matching points indicates that the segmentation block has enough geometric evidence to support meaningful depth estimation. If the number of matching points in a segmentation block is less than a preset sufficient threshold, the processing unit 23 marks the depth value corresponding to that segmentation block as invalid in the refined depth information 1204. This mechanism can avoid unreliable or noise-dominated regions that introduce errors into the final depth map 1205 and helps maintain the structural consistency of the overall depth estimation.

[0107] In one embodiment, for each textureless segmentation block 1202 with a sufficient number of matching points, the processing unit 23 further determines whether the depth values ​​of these matching points are continuous within the segmentation block. The continuity assessment checks whether the depth values ​​exhibit a spatially coherent pattern consistent with the physical geometry of the observed object or surface. If discontinuities are found in the depth values ​​of matching points within the segmentation block, such as sudden jumps or irregular parallax, the processing unit 23 marks the depth values ​​of that segmentation block as invalid in the refined depth information. This mechanism prevents discontinuous or geometrically inconsistent estimates from affecting the reliability of the depth map 1205.

[0108] In another embodiment, for segmented blocks with textureless labels, containing a sufficient number of matching points, and where the matching points exhibit continuous depth values, processing unit 23 assigns interpolated depth values ​​to these segmented blocks and includes these interpolated depth values ​​in the refined depth information 1204. The interpolation process estimates a reasonable depth distribution within these textureless regions based on the surrounding valid depth values ​​to ensure smooth transitions between regions and maintain consistency with neighboring blocks. In this way, even for regions where direct feature correspondences are sparse or unreliable, the refined depth information can still provide dense and visually coherent depth estimates.

[0109] Figure 13The detailed implementation of the deep refinement step S122 according to an embodiment of the present disclosure is described. For example... Figure 13 As shown, the depth refinement step S122 refines the depth information of the segmented blocks 1202 with and without texture labels based on a series of condition evaluations. Two main conditions (condition 1301 and condition 1302) are applied to determine the reliability and continuity of the depth estimate for each segmented block.

[0110] Specifically, condition 1301 evaluates whether a given segmentation block without texture labels contains a sufficient number of matching points. If the number of valid matches does not reach a sufficient threshold, the corresponding depth value is classified as an invalid depth value 1303 in the refined depth information 1204, indicating that the segmentation block lacks sufficient corresponding evidence to support a meaningful depth estimate. Conversely, when condition 1301 is satisfied, the process continues to execute condition 1302.

[0111] Condition 1302 determines whether the depth values ​​of matching points in the segmented block exhibit spatial continuity. When the depth values ​​are spatially consistent and show a smooth variation throughout the segmented block, an interpolated depth value 1304 is generated to refine the depth information 1204 of the segmented block. Conversely, if the depth values ​​are discontinuous, indicating potential local differences or mismatches, the segmented block will be marked again as having invalid depth values ​​1303 to avoid problems in the subsequent depth information integration step S123.

[0112] Meanwhile, segmented blocks 1201 without texture labels directly provide reliable depth information 1203, derived from the estimated depth value of the segmented block. The reliable depth information 1203, together with refined depth information 1204 (including invalid depth values ​​1303 and interpolated depth values ​​1304), forms the comprehensive input set for the subsequent depth information integration step S123. This hierarchical refinement process ensures that textured regions, textureless regions, and undetermined regions are all properly represented, thereby improving the density and reliability of the final depth map 1205.

[0113] The depth estimation system and method disclosed herein integrate image segmentation, epipolar geometry, and hierarchical refinement techniques to achieve robust and reliable depth estimation in multi-view or cross-sensor configurations. By limiting the corresponding search space through segmentation block-level and pixel-level matching, the system of this disclosure effectively reduces computational complexity and minimizes false matches. Furthermore, the discriminative refinement mechanism improves the integrity and consistency of depth information, ensuring that both textured and textureless regions are accurately represented in the generated depth map. Through the combined use of semantic and geometric cues, the system of this disclosure maintains stable performance even in challenging scenarios, such as differences in viewpoint, varying lighting conditions, or differences in sensor modes, making it particularly suitable for applications such as autonomous navigation, obstacle detection, and scene reconstruction. Therefore, the system and method of this disclosure provide a practically deployable and scalable high-precision, highly robust depth estimation framework for real-world environments.

[0114] In some embodiments, the depth estimation system 20 can dynamically adjust its segmentation block selection and pixel matching behavior based on the characteristics of the input image and the operational context. For example, the processing unit 23 can prioritize candidate segmentation blocks that intersect with epipolar lines and have a high degree of semantic and visual similarity to the observed segmentation blocks, while minimizing geometric deviations in the bird's-eye view coordinate system. This multi-factor selection strategy enables the system to reduce false matches and improve depth estimation accuracy, especially in scenes containing blurry or textureless regions.

[0115] The depth estimation system 20 can further refine the depth values ​​by evaluating the feature variance of each segmentation block and applying conditional rules based on matching sufficiency and depth continuity. Segmentation blocks that do not meet these conditions can be marked as invalid depth values ​​or assigned interpolated depth values, depending on the specific circumstances. In this regard, the term "matching point" refers to the pixel in the target image that corresponds to the observation point in the observation image, determined by feature similarity subject to epipolar geometry constraints and segmentation block-level screening. The term "invalid" depth value refers to depth estimates at the pixel level or segmentation block level that are excluded before integration due to insufficient matching evidence or discontinuities, but this does not mean that the depth map 1205 as a whole is invalid. The term "continuous" depth value refers to a set of spatially coherent depth estimates within a segmentation block that exhibit smooth changes without sudden disparity jumps.

[0116] Depending on resource constraints or input variations, the filtering steps can be executed in parallel or dynamically adjusted in order. Intermediate results (e.g., first feature map 301, second feature map 302, and segmentation blocks 303, 304, or 1201) can be reused in multiple iterations to reduce computational load. The term "appearance similarity" refers to the similarity score between segmentation blocks calculated based on aggregated feature descriptors of feature maps 301 and 302, such as cosine similarity or L2 norm. The term "same semantic category" refers to the observed segmentation block and the candidate segmentation block being assigned the same category label, such as "vehicle," "pedestrian," or "road," according to a common classification criterion in the image segmentation process. The term "minimum difference" refers to the minimum absolute value of the difference between the observed distance and the candidate distance in the bird's-eye view coordinate system, calculated based on ground point projection. The system can also support auxiliary functions such as obstacle detection or scene reconstruction, which use the refined depth map 1205 but do not alter the core depth estimation process. These behaviors together enable the system to operate efficiently under various sensor configurations and environmental conditions.

[0117] The foregoing paragraphs describe this disclosure from multiple perspectives. Clearly, the teachings in this specification can be implemented in various ways. Specific structures or functions disclosed in any example are merely representative examples. Based on the teachings of this specification, those skilled in the art should understand that any aspect disclosed can be implemented alone, or two or more aspects can be combined and implemented.

[0118] While this disclosure has been described by way of example and in conjunction with preferred embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. Rather, this disclosure is intended to cover various modifications and similar configurations. Therefore, the scope of the appended claims should be given the broadest interpretation to cover all such modifications and similar arrangements.

Claims

1. A depth estimation system based on image segmentation, comprising: A first photographic device is configured to capture an observational image; A second photographic device is configured to capture an image of a target; One processing unit; and a storage unit coupled to the processing unit, which stores multiple instructions that, when executed by the processing unit, cause the processing unit to: Based on the observed image and the target image, a first feature map and a second feature map are generated respectively through feature extraction; Multiple segmentation blocks are identified in the observed image and the target image through an image segmentation process; For each observation point in this observation image: Based on multiple external parameters of the first and second imaging devices, a pair of polar constraints corresponding to the observation point are calculated. Among the multiple segmented blocks of the observed image, determine the observation segmented block in which the observation point is located; Based on this epipolar constraint, among the multiple segmentation blocks of the target image, a target segmentation block that matches the observed segmentation block is determined; Based on the first feature map, the second feature map, and the epipolar constraint, a target point that matches the observation point is searched within the target segmentation block; as well as Estimate a depth value based on the disparity between the observation point and the target point.

2. The depth estimation system of claim 1, wherein the processing unit determines the target segmentation block by performing the following steps: Based on the epipolar constraint, one or more candidate segmentation blocks are determined among the multiple segmentation blocks of the target image; as well as The candidate segmentation blocks are filtered by selecting only those blocks that share the same semantic category as the observed segmentation block.

3. The depth estimation system of claim 1, wherein, The processing unit determines the target segmentation block by performing the following steps: Based on this epipolar constraint, one or more candidate segmentation blocks are determined among the multiple segmentation blocks of the target image; and The candidate segmentation blocks are filtered by selecting only those blocks whose appearance similarity to the observed segmentation block is greater than a similarity threshold.

4. The depth estimation system of claim 3, wherein, The processing unit calculates the appearance similarity between the observed segmented block and each of the one or more candidate segmented blocks based on the first feature map and the second feature map.

5. The depth estimation system of claim 1, wherein, The processing unit determines the target segmentation block by performing the following steps: Based on the epipolar constraint, one or more candidate segmentation blocks are determined among the multiple segmentation blocks of the target image; In a bird's-eye view coordinate system, calculate an observation distance between the first photographic device and a ground point in the observation segmentation block; In the bird's-eye view coordinate system, for each of the one or more candidate segmentation blocks, calculate a candidate distance between the first photographic device and a ground point of each of the one or more candidate segmentation blocks; and The candidate segmentation block that has the smallest difference between the candidate distance and the observation distance is selected as the target segmentation block.

6. The depth estimation system of claim 1, wherein, The multiple instructions further enable the processing unit to: Evaluate a discriminability score for each of the plurality of segmented blocks in the observed image and the target image, and assign a textureless label to the plurality of segmented blocks whose discriminability scores are below a discriminability threshold; A reliable depth information is generated from the depth values ​​estimated by the plurality of segmented blocks without the textureless label, and the depth values ​​of the plurality of segmented blocks with the textureless label are refined to obtain a refined depth information. as well as The reliable depth information is integrated with the refined depth information to generate a depth map.

7. The depth estimation system as described in claim 6, wherein: The processing unit calculates the discriminability score for each of the multiple segmented blocks based on a variation of multiple feature values ​​within the multiple segmented blocks; wherein, The variance of the plurality of said feature values ​​is calculated based on at least one of the first feature map and the second feature map.

8. The depth estimation system of claim 6, wherein, The multiple instructions further enable the processing unit to: For each of the plurality of segmented blocks having the textureless label, determine whether the number of matching points in the segmented block is sufficient; and In the refined depth information, the depth values ​​corresponding to multiple segmented blocks with insufficient matching points are marked as invalid.

9. The depth estimation system of claim 8, wherein, The multiple instructions further enable the processing unit to: For each of the multiple segmented blocks that have a sufficient number of matching points and have the textureless label, determine whether the multiple depth values ​​of the matching points in the segmented block are continuous; as well as Multiple depth values ​​of multiple segmented blocks whose depth values ​​are discontinuous are marked as invalid in the refined depth information.

10. The depth estimation system of claim 9, wherein, The instructions further cause the processing unit to assign multiple interpolated depth values ​​to multiple segmented blocks that have the textureless label, a sufficient number of matching points, and multiple depth values ​​of the multiple matching points that are consecutive, and to incorporate the multiple interpolated depth values ​​into the refined depth information.

11. The depth estimation system as described in claim 1, wherein, The first photographic device is a pinhole camera, and the second photographic device is a fisheye camera.

12. The depth estimation system as described in claim 1, wherein, The multiple instructions further enable the processing unit, in at least one of an autonomous navigation system or an advanced driver assistance system, to apply the depth value estimated from the observation point and the target point to perform obstacle detection.

13. The depth estimation system as described in claim 1, wherein, The instructions further enable the processing unit to apply the depth value estimated from the observation point and the target point to perform scene reconstruction in an augmented reality system or a virtual reality system.

14. A depth estimation method based on image segmentation, executed by a processing unit, the method comprising: A first feature map and a second feature map are generated based on an observation image and a target image, respectively, through feature extraction, wherein the observation image is captured by a first photographic device and the target image is captured by a second photographic device. Multiple segmentation blocks are identified in the observed image and the target image through an image segmentation process; For each observation point in this observation image: Based on multiple external parameters of the first and second imaging devices, a pair of polar constraints corresponding to the observation point are calculated. Among the multiple segmented blocks of the observed image, identify the observation segmentation block in which the observation point is located; Based on this epipolar constraint, among the multiple segmentation blocks of the target image, a target segmentation block that matches the observed segmentation block is determined; Based on the first feature map, the second feature map, and the epipolar constraint, a target point that matches the observation point is searched within the target segmentation block; as well as Estimate a depth value based on the disparity between the observation point and the target point.

15. The depth estimation method as described in claim 14, wherein, Determining the target segmentation block includes: Based on this epipolar constraint, one or more candidate segmentation blocks are determined among the multiple segmentation blocks of the target image; and The candidate segmentation block is filtered by selecting only one or more candidate segmentation blocks that have the same semantic category as the observed segmentation block.

16. The depth estimation method as described in claim 14, wherein, The target segmentation block is determined to include: Based on this epipolar constraint, one or more candidate segmentation blocks are determined among the multiple segmentation blocks of the target image; and The candidate segmentation blocks are filtered by selecting only those blocks whose appearance similarity to the observed segmentation block is greater than a similarity threshold.

17. The depth estimation method as described in claim 16, wherein, Based on the first feature map and the second feature map, the appearance similarity between the observed segmented block and each of the one or more candidate segmented blocks is calculated.

18. The depth estimation method as described in claim 14, wherein, The processing unit determines the target segmentation block by performing the following steps: Based on the epipolar constraint, one or more candidate segmentation blocks are determined among the multiple segmentation blocks of the target image; In a bird's-eye view coordinate system, calculate an observation distance between the first photographic device and a ground point in the observation segmentation block; In the bird's-eye view coordinate system, for each of the one or more candidate segmentation blocks, calculate a candidate distance between the first photographic device and a ground point of each of the one or more candidate segmentation blocks; and The candidate segmentation block that has the smallest difference between the candidate distance and the observation distance is selected as the target segmentation block.

19. The depth estimation method as described in claim 14, further comprising: Evaluate a discriminability score for each of the plurality of segmented blocks in the observed image and the target image, and assign a textureless label to the plurality of segmented blocks whose discriminability scores are below a discriminability threshold; A reliable depth information is generated from the depth values ​​estimated by the plurality of segmented blocks without the textureless label, and the depth values ​​of the plurality of segmented blocks with the textureless label are refined to obtain a refined depth information. as well as The reliable depth information is integrated with the refined depth information to generate a depth map.

20. The depth estimation method as described in claim 19, wherein: The processing unit calculates the discriminability score for each of the multiple segmented blocks based on a variation of multiple feature values ​​within the multiple segmented blocks; The number of variations of the plurality of said feature values ​​is calculated based on at least one of the first feature map and the second feature map.

21. The depth estimation method as described in claim 19, further comprising: For each of the plurality of segmented blocks having the textureless label, determine whether the number of matching points in the segmented block is sufficient; as well as In the refined depth information, the depth values ​​corresponding to multiple segmented blocks with insufficient matching points are marked as invalid.

22. The depth estimation method of claim 21, further comprising: For each of the multiple segmented blocks that have a sufficient number of matching points and have the textureless label, determine whether the multiple depth values ​​of the matching points in the segmented block are continuous; as well as Multiple depth values ​​of multiple segmented blocks whose depth values ​​are discontinuous are marked as invalid in the refined depth information.

23. The depth estimation method as described in claim 22, further comprising: Multiple interpolated depth values ​​are assigned to multiple segmented blocks that have the textureless label, a sufficient number of matching points, and multiple depth values ​​of the multiple matching points are consecutive, and the multiple interpolated depth values ​​are incorporated into the refined depth information.

24. The depth estimation method as described in claim 19, wherein, The first photographic device is a pinhole camera, and the second photographic device is a fisheye camera.

25. The depth estimation method as described in claim 14, further comprising: In at least one of an autonomous navigation system or an advanced driver assistance system, the depth value estimated from the observation point and the target point is applied to perform obstacle detection.

26. The depth estimation method as described in claim 14, further comprising: In an augmented reality system or a virtual reality system, the depth value estimated from the observation point and the target point is applied to perform scene reconstruction.

Images