Human perception and distance measurement method and system based on deep learning

Abstract

This application discloses a deep learning-based method and system for human perception and distance measurement. The method acquires image pairs of non-rigid human body regions using a binocular camera, establishes initial pixel correspondences using dense optical flow estimation, and refines the corresponding point sets by combining semantic segmentation and feature enhancement processing. Then, it eliminates disparity anomalies using neighborhood consistency constraints, transforms the data into continuous depth information, and reconstructs a spatial point cloud through triangulation. Based on this, it constructs a fine-grained boundary depth field covering the entire human body region by dynamically tracking non-rigid deformations and fusing binocular geometric constraints. This application addresses the shortcomings of existing methods in sparse matching of non-rigid human body surfaces, depth discontinuities, and handling of occluded regions, achieving high-precision perception of the spatial distribution of various parts of the human body. This effectively supports safe collaboration and intelligent interaction of robots in complex dynamic scenarios.

Description

Technical Field

[0001] This application relates to the fields of computer vision intelligence and robot perception technology, specifically a method and system for human perception and distance measurement based on deep learning. Background Technology

[0002] In the fields of robot perception and human-robot collaboration, accurate measurement of the human body's spatial position is a fundamental prerequisite for safe collaborative operations. Existing solutions mainly rely on the geometric matching principle of binocular stereo vision, finding corresponding feature points in the left and right camera images and using triangulation to calculate the target distance. However, the human body is a typical non-rigid target, with uneven surface texture distribution and continuously changing shape with posture, which poses a serious challenge to traditional matching methods based on hand-designed features.

[0003] Traditional methods often acquire sparsely distributed parallax information, reliably establishing only a small number of corresponding point pairs in textured areas. This results in insufficient coverage of the human body surface, failing to fully depict the three-dimensional shape of the human body. In smooth-textured areas (such as a torso wearing solid-colored clothing) and occluded areas (such as an arm obscured by the body), traditional feature point detection algorithms are prone to failure, leading to large-scale loss or significant deviations in depth values. These sparsity and discontinuity issues limit the robot's refined perception of the spatial distribution of various parts of the human body.

[0004] On the other hand, human posture is constantly changing dynamically, and non-rigid deformation makes it difficult to maintain a stable correspondence between point clouds in adjacent frames. Existing tracking methods suffer from significant accumulation of tracking errors when dealing with large limb movements or rapid deformations, making it difficult to generate temporally continuous, high-precision depth sequences. Furthermore, when there are abrupt changes in depth near the depth boundary, discontinuity in the disparity values ​​introduces severe depth noise in the boundary region, further reducing the accuracy of 3D reconstruction.

[0005] In summary, existing human perception and distance measurement methods have significant shortcomings in key aspects such as dense depth estimation, non-rigid dynamic tracking, and occlusion region handling, making it difficult to meet the actual needs of robots for precise human perception in complex dynamic scenes.

[0006] Therefore, there is an urgent need to provide a high-precision measurement method that can stably acquire global depth information of the human body surface and has good robustness to occlusion and deformation. Summary of the Invention

[0007] To address the above problems, this invention provides a deep learning-based human perception and distance measurement method and system, which solves the core challenges of non-rigid human bodies under binocular vision, such as pixel-level dense matching failure, abrupt changes in depth boundaries, and sparse and discontinuous point clouds caused by texture loss, posture deformation, occlusion, and lighting interference. These challenges prevent robots from achieving high-precision, full-coverage, and robust human distance perception.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] According to a first aspect of this application, a deep learning-based method for human perception and distance measurement is provided, comprising: The left and right view images containing non-rigid human body regions are acquired by a binocular camera, and the left and right view images are divided into regions to generate a basic matching map. The region edge ambiguity is calculated based on the basic matching map. When the region edge ambiguity exceeds a preset threshold and the color gradient distribution is uneven, adaptive feature enhancement processing is performed on the region, and the corrected pixel corresponding point set is output. Based on the corrected pixel-corresponding point set, the neighborhood consistency constraint method is used to detect and smooth disparity aberration points and abrupt depth boundary regions to obtain the corrected matching result. The disparity values ​​in the corrected matching results are transformed into depth, and the neighborhood depth consistency is detected. When the segmentation accuracy of discontinuous regions is lower than the preset standard, the local interpolation adjustment is performed back to the low texture region to obtain a smooth depth distribution. The human body shape contour is segmented according to the smooth depth distribution, the occluded and missing areas are filled, and a spatial point cloud is generated using a triangulation mesh reconstruction algorithm. Track the dynamic deformation caused by non-rigid attitude changes in the spatial point cloud, update the local correspondence based on optical flow, adjust the constraint weights when the tracking error exceeds a preset threshold, and output a continuous depth sequence. By integrating the continuous depth sequence with binocular vision geometric constraints, a global depth field covering the human body boundary is constructed, and boundary refinement processing is performed through color gradient analysis to determine the fine boundary distribution required for robot perception.

[0010] Preferably, the specific method for dividing the left and right view images into regions includes: estimating the initial pixel correspondence between the left and right view images based on a dense optical flow algorithm, and dividing the left and right view images into regions according to the initial pixel correspondence, texture density, and local contrast.

[0011] Preferably, the estimation of the initial pixel correspondence based on the dense optical flow algorithm includes: A dense optical flow algorithm based on a multi-scale pyramid structure is adopted. The image is downsampled at each scale level, the displacement vector field between adjacent pixels is calculated, and the pixel displacement is refined layer by layer during the propagation process from coarse to fine in the pyramid to obtain a dense disparity estimate covering the entire image. The initial pixel correspondence is formed based on the dense disparity estimate.

[0012] Preferably, calculating the region edge ambiguity based on the basic matching map includes: The surface detail complexity features and light reflection difference features of the human body region are extracted from the base matching map, and the edge blurring of the region is calculated based on the surface detail complexity features and the light reflection difference features.

[0013] Preferably, the adaptive feature enhancement process includes: Adaptive histogram equalization is used to improve local contrast in blurred edge regions. For regions with uneven color gradient distribution, a semantic segmentation network is used to extract human body region masks, and edge-aware smoothing filtering is used to unify the gradient distribution. After enhancement, the corresponding point set is recalculated, which significantly improves the number of matching points and the matching confidence.

[0014] Preferably, the step of performing depth transformation on the disparity values ​​in the corrected matching results, detecting neighborhood depth consistency, and when the segmentation accuracy of discontinuous regions is lower than a preset standard, backtracking to low-texture regions for local interpolation adjustment to obtain a smooth depth distribution specifically includes: A neighborhood window centered on the target pixel is constructed, and the deviation between the disparity value of the target pixel and the neighborhood mean is calculated. When the deviation exceeds a preset standard deviation multiple, the target pixel is marked as an outlier and replaced with the neighborhood weighted average. At the same time, a distance attenuation weighted smoothing filter is applied to the abrupt depth boundary regions to ensure the spatial continuity of the depth values.

[0015] Preferably, the step of segmenting the human body shape contour based on the smooth depth distribution, filling in occluded and missing areas, and generating a spatial point cloud using a triangulation mesh reconstruction algorithm specifically includes: Delaunay triangulation is performed on the human foreground point cloud in the smooth depth distribution to generate a triangular mesh covering the human surface, and abnormal triangles with side lengths exceeding a preset threshold are removed; then the point cloud is voxelized and compressed to homogenize the point cloud density and reduce the computational cost of subsequent processing.

[0016] Preferably, the step of tracking the dynamic deformation caused by non-rigid attitude changes in the spatial point cloud, updating local correspondences based on optical flow, adjusting constraint weights when the tracking error exceeds a preset threshold, and outputting a continuous depth sequence specifically includes: Voxel segmentation is performed on the spatial point cloud, and the principal deformation direction of each voxel unit is extracted by principal component analysis. The displacement vector field of the point cloud between adjacent frames is calculated by an optical flow tracing algorithm based on the iterative nearest point. When the tracking error does not converge sufficiently, the deformation constraint weight is increased and the data is re-registered. The temporal resolution of the depth sequence is improved to the target frame rate by depth interpolation.

[0017] Preferably, the construction of the global depth field includes: Kalman filtering is used to denoise the continuous depth sequence in time; triangulation is used to fuse binocular parallax and geometric baseline parameters to generate a 3D point cloud; nearest neighbor interpolation is performed on sparse regions in the point cloud with a density below a preset threshold to make the depth field coverage reach a preset level; finally, the human body boundary is refined by weighted superposition of depth weight and gradient weight to improve the boundary positioning accuracy.

[0018] According to a second aspect of this application, a deep learning-based human perception and distance measurement system is provided, the system comprising: The matching map generation module is used to acquire left and right view images of non-rigid human body regions through a binocular camera, estimate the initial pixel correspondence based on the dense optical flow algorithm, and divide the region according to texture density and local contrast to generate a basic matching map. The corresponding point set correction module is used to extract surface detail complexity features and illumination reflection difference features from the basic matching map. When the edge ambiguity and color gradient non-uniformity meet the preset conditions, adaptive feature enhancement processing is performed, and the corrected pixel corresponding point set is output. The disparity correction module is used to smooth the disparity abnormal points and abrupt depth boundary regions in the corresponding point set based on the neighborhood consistency constraint, so as to obtain the correction matching result. The depth conversion module is used to convert the disparity values ​​in the correction matching results into continuous depth information, and to perform local interpolation adjustment on low-texture discontinuous areas to output a smooth depth distribution. The point cloud reconstruction module is used to segment the human body shape outline according to the smooth depth distribution, fill in the occluded and missing areas, and reconstruct a spatial point cloud through triangulation mesh. The depth sequence generation module is used to track the dynamic deformation in the spatial point cloud, update the local correspondence based on optical flow, adjust the constraint weight when the tracking error exceeds the limit, and output a continuous depth sequence. The depth field construction module is used to fuse the continuous depth sequence with the binocular vision geometric constraints to construct a global depth field, and output the fine boundary distribution required for robot perception through boundary refinement processing.

[0019] This application employs a seven-step progressive process: binocular image region segmentation, edge ambiguity-driven adaptive enhancement, neighborhood consistency correction, closed-loop interpolation optimization, triangular mesh reconstruction, dynamic deformation tracking, and global depth field construction. This process models non-rigid human perception and distance measurement into a four-stage collaborative system: structured perception, robust matching, continuous modeling, and precise localization. By leveraging five key technical mechanisms—dense optical flow initialization, semantic guidance enhancement, anisotropic smoothing, voxelized deformation modeling, and depth-gradient weighted boundary refinement—the application eliminates the coupling effects of texture loss, pose deformation, occlusion breaks, lighting interference, and boundary ambiguity. This enables the system to possess high precision, full coverage, strong robustness, and real-time performance without relying on dedicated hardware or manual annotation, fully supporting the safe collaboration and natural interaction of service robots in open environments. Attached Figure Description

[0020] Figure 1 A flowchart of a deep learning-based human perception and distance measurement method provided for this application.

[0021] Figure 2 This application provides a structural block diagram of a deep learning-based human perception and distance measurement system. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solution, the present invention will be described in detail below with reference to embodiments. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0023] Example 1:

[0024] The deep learning-based human perception and distance measurement method executes as follows at time T0:

[0025] Step 1: Acquire left and right view images containing non-rigid human body regions using a binocular camera, and divide the left and right view images into regions to generate a basic matching map;

[0026] In this step, the specific method for dividing the left and right view images into regions includes: estimating the initial pixel correspondence between the left and right view images based on the dense optical flow algorithm, and dividing the left and right view images into regions according to the initial pixel correspondence, texture density, and local contrast.

[0027] Among them, a binocular camera can refer to two synchronous imaging units with known intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and a fixed baseline distance (e.g., 10 cm) to acquire geometric complementary views of non-rigid human targets at the same time; a non-rigid human target area can refer to a human target area that has local deformations such as joint bending, limb swinging or clothing fluttering during the acquisition process, and its surface reflection characteristics and spatial structure change dynamically over time.

[0028] Region partitioning can refer to dividing the left and right views into high-reliability matching regions and low-reliability interpolation regions based on the differences in local texture density and contrast of the image, rather than adopting a globally unified matching strategy; the base matching map is a preliminary disparity map covering the entire image after initial correspondence estimation and regional disparity completion. It serves as the data basis for all subsequent enhancement and correction operations and is not directly used for depth calculation.

[0029] Image processing equipment controls a binocular camera to simultaneously acquire left and right view images containing non-rigid human body regions. The two sensors of the binocular camera are horizontally arranged at a preset baseline distance, and the baseline parameters are determined during the camera calibration stage. After acquisition, using the left image as a reference, a dense optical flow algorithm based on a multi-scale pyramid structure is used to calculate the corresponding position of each pixel in the right image. Specifically, a multi-layer Gaussian pyramid is constructed on the original image, the optical flow field is initialized at the coarsest level, and the displacement vector is refined layer by layer from top to bottom, finally obtaining a dense disparity estimate covering the entire image at the original resolution level.

[0030] During the generation of the basic matching map, the image processing device further performs regional attribute analysis on the image. Gradient operators are used to calculate the gradient magnitude map across the entire image to evaluate the texture density distribution within the neighborhood of each pixel; simultaneously, the local grayscale standard deviation is calculated to measure the local contrast level. Based on the joint determination of texture density and contrast, the image is divided into high-confidence matching regions and low-confidence matching regions. The disparity value of the former is directly preserved, while the disparity of the latter is completed through neighborhood interpolation, thereby generating a basic matching map with complete coverage.

[0031] Step 2: Calculate the region edge ambiguity based on the basic matching map. When the region edge ambiguity exceeds a preset threshold and the color gradient distribution is uneven, perform adaptive feature enhancement processing on the region and output the corrected pixel corresponding point set.

[0032] In this step, calculating the region edge ambiguity based on the basic matching map includes: extracting surface detail complexity features and light reflection difference features of the human body region from the basic matching map, and calculating the region edge ambiguity based on the surface detail complexity features and the light reflection difference features;

[0033] The adaptive feature enhancement process includes: applying adaptive histogram equalization to improve local contrast in blurred edge regions; extracting human body region masks using a semantic segmentation network for regions with uneven color gradient distribution; and unifying gradient distribution based on edge-aware smoothing filtering; and recalculating the corresponding point set after enhancement to significantly improve both the number of matching points and the matching confidence.

[0034] Among them, the region edge ambiguity is a composite index that integrates surface detail complexity features and illumination reflection difference features. It is used to quantitatively characterize the structural uncertainty of the human body region edge in the basic matching map. The surface detail complexity feature can refer to the response intensity distribution obtained by applying the Laplacian operator to the human body region mask edge. Its value ranges from 0 to 1. The higher the value, the richer the edge structure and the clearer the geometric details.

[0035] Light reflection difference features can refer to the statistical measure of the dispersion of gray-level distribution within the human body mask area. A standard deviation greater than 10 indicates significant uneven lighting or material reflection differences. Uneven color gradient distribution can refer to the variance of the color gradient map of the human body region in the basic matching map calculated using the Sobel operator being greater than 5, reflecting severe local color transitions and disordered texture direction, which is not conducive to stable matching. Adaptive feature enhancement processing can refer to applying differentiated enhancement methods to match degradation caused by different reasons, rather than a uniform filtering operation.

[0036] The specific method is as follows: Based on the basic matching map, the image processing device performs two types of analysis on the surface features of the human body region: First, it calculates the gray-level statistics in the neighborhood of local pixels in the human body region to measure the complexity of surface details; Second, it statistically analyzes the distribution pattern of the gray-level histogram in the region to assess the degree of difference in light reflection.

[0037] Adaptive feature enhancement processing is triggered when a sub-region is detected to meet the following dual conditions: the edge blur index (quantized by the proportion of pixels output by the edge detection operator) exceeds a preset threshold, and the color gradient distribution in the same region is uneven (determined by the gradient variance exceeding a preset range). The specific enhancement process is as follows:

[0038] First, a deep learning-based semantic segmentation network is used to extract the human body region mask from the left image to accurately locate the human body boundary;

[0039] Secondly, adaptive histogram equalization is applied within the mask area to improve local contrast;

[0040] Secondly, edge-aware smoothing filters are applied to areas with uneven gradient distribution to reduce interference from uneven gradient distribution on subsequent matching. After enhancement, the pixel-corresponding point set in this area is recalculated. The number of matching points and the matching confidence before and after correction can be verified by comparison with the reference frame, confirming the enhancement effect.

[0041] Step 3: Based on the corrected pixel-corresponding point set, the neighborhood consistency constraint method is used to detect and smooth disparity aberration points and abrupt depth boundary regions to obtain the corrected matching result;

[0042] In this step, the neighborhood consistency constraint method includes: constructing a neighborhood window centered on the target pixel, calculating the deviation between the disparity value of the target pixel and the neighborhood mean, marking the point as an outlier when the deviation exceeds a preset standard deviation multiple, and replacing it with the neighborhood weighted average value, while applying distance attenuation weighted smoothing filtering to the abrupt change region of the depth boundary to ensure the spatial continuity of the depth value.

[0043] Among them, the neighborhood consistency constraint method can refer to constructing a spatial neighborhood window centered on the target pixel, identifying anomalies by comparing the deviation of its disparity value from the neighborhood statistical features, and implementing differential smoothing based on spatial attenuation law, rather than global mean filtering; disparity anomalies can refer to isolated points within the neighborhood window whose disparity value deviates from the neighborhood mean by more than a preset standard deviation multiple (e.g., 2.5 times), which are mostly caused by mismatches or noise interference; depth boundary abrupt change regions can refer to regions in 3D space where the depth value undergoes a step jump, usually corresponding to the human contour edge or occlusion boundary, and pseudo-abrupt changes need to be suppressed while maintaining geometric realism; the corrected matching result is a disparity map after anomaly point removal and boundary region weighted smoothing, whose spatial continuity is significantly better than the basic matching map, but depth domain optimization has not yet been completed.

[0044] The specific method is as follows: Starting from the corrected set of corresponding pixel points, the image processing device uses a neighborhood consistency constraint method to eliminate disparity anomalies. Specifically, a neighborhood window is constructed centered on each target pixel, and the mean and standard deviation of the disparity values ​​within the window are statistically analyzed. When the difference between the disparity value of the target pixel and the neighborhood mean exceeds a certain number of standard deviations, the point is marked as a disparity anomaly point, and the original disparity value is replaced by the weighted average of the effective points in the neighborhood, with the weight decreasing as the distance from the target pixel increases.

[0045] For regions with abrupt changes in depth boundaries, the image processing device employs distance-attenuation weighted smoothing filtering: centering on the abrupt change point, depth values ​​within the filtering radius are assigned attenuation weights based on distance, resulting in a smooth transition of depth near the boundary and effectively reducing occlusion noise caused by viewing angle differences. After this processing, the output corrected matching result shows a significant reduction in depth error compared to before processing, improving matching accuracy.

[0046] Step 4: Perform depth transformation on the disparity values ​​in the corrected matching results, detect the neighborhood depth consistency, and when the segmentation accuracy of discontinuous regions is lower than the preset standard, backtrack to the low-texture region for local interpolation adjustment to obtain a smooth depth distribution;

[0047] In this step, the disparity value in the correction matching result is transformed into depth, and the neighborhood depth consistency is detected. When the segmentation accuracy of discontinuous regions is lower than the preset standard, the local interpolation adjustment is performed back to the low texture region to obtain a smooth depth distribution.

[0048] Depth conversion refers to converting disparity values ​​into physical spatial depth values ​​based on a binocular vision geometric model. The core formula is: Depth = (Baseline × Focal Length) / Disparity, where the baseline and focal length are known camera parameters. Neighborhood depth consistency refers to calculating the depth gradient within a 3×3 neighborhood. A gradient value greater than 0.2 m is considered a discontinuous region. The segmentation accuracy of discontinuous regions refers to the spatial position error between the boundary obtained by Canny edge detection and the true depth boundary. An error greater than 5 pixels is considered insufficient segmentation accuracy. Backtracking to low-texture regions refers to actively locating regions with texture entropy < 3.0 and Laplacian response > 50, rather than interpolating the entire image. Local interpolation adjustment refers to applying bilateral filtering (spatial σ = 5, range σ = 0.1) to the backtracked region to perform edge-preserving smoothing, reducing the depth gradient to below 0.15 m.

[0049] The specific method is as follows: The image processing device uses the standard depth calculation relationship of binocular vision to convert the disparity value in the correction matching result into continuous depth information. The depth value is inversely proportional to the disparity value, and the scaling factor is determined by the camera focal length and the binocular baseline distance.

[0050] After conversion, a neighborhood depth consistency check is performed on the depth map: the depth gradient is calculated within a sliding window of a preset size, and when the gradient exceeds a preset threshold, the pixel is marked as a potential discontinuity. The boundary segmentation accuracy of discontinuous regions is further evaluated. When the accuracy is lower than a preset standard, the process backtracks to regions with low texture complexity or poor texture correlation in the base matching map. For these low-confidence regions, a bilateral filter is applied for local depth adjustment. This filter considers both pixel spatial distance and depth value differences, smoothing noise while preserving true depth boundaries. The final output smooth depth distribution achieves the preset levels in both overall depth transition smoothness and boundary preservation capability.

[0051] Step 5: Segment the human body shape outline according to the smooth depth distribution, fill in the occluded and missing areas, and generate a spatial point cloud using a triangulation mesh reconstruction algorithm;

[0052] In this step, the triangulation mesh reconstruction algorithm includes: performing Delaunay triangulation on the human foreground point cloud in the smooth depth distribution to generate a triangular mesh covering the human surface, and removing abnormal triangles with side lengths exceeding a preset threshold; then performing voxel compression on the point cloud to homogenize the point cloud density and reduce the computational overhead of subsequent processing.

[0053] Among them, the human body contour can refer to the human foreground boundary extracted from the depth map by jointly using a depth threshold (e.g., 2.0–3.5 m) and a gradient threshold (>0.3 m / pixel); the occluded missing region can refer to the region with a depth value of 0 or an anomaly (e.g., NaN), usually caused by clothing occlusion, limb crossing, or sensor blind spots; the triangulation mesh reconstruction algorithm can refer to performing Delaunay triangulation on the foreground point cloud to generate topologically connected triangular patches, and removing abnormal triangles with a side length exceeding a preset threshold (e.g., 0.2 m) to ensure the geometric rationality of the mesh; the spatial point cloud can refer to the three-dimensional coordinate set generated by back projection of camera intrinsic parameters, with a coordinate accuracy ≤0.01 m.

[0054] The specific method is as follows: the image processing device segments the human body shape contour according to the smooth depth distribution. Specifically, the pixel area whose depth value falls within the preset human body distance range is marked as the foreground, and the shape contour of the human body region is obtained by combining the mask of the semantic segmentation network.

[0055] For regions lacking depth values ​​due to occlusion, the image processing device fills in the missing values ​​using neighborhood depth interpolation. The weighted average of the effective depth points surrounding the missing region is used as the filling value, and the interpolation range extends outward from the target missing point to a preset pixel radius. After establishing the complete contour, the foreground points of the human body in the depth map are back-projected into 3D space using the camera intrinsic parameter matrix to obtain a 3D point cloud. Delaunay triangulation is performed on the point cloud to generate a triangular mesh covering the human body surface, and abnormal triangles are removed based on a preset side length threshold. Finally, the point cloud is voxelized and compressed to homogenize the density and reduce the computational cost of subsequent processing, ultimately determining the spatial point cloud distribution.

[0056] Step 6: Track the dynamic deformation caused by non-rigid attitude changes in the spatial point cloud, update the local correspondence based on optical flow, adjust the constraint weights when the tracking error exceeds a preset threshold, and output a continuous depth sequence;

[0057] In this step, tracking dynamic deformation and outputting a continuous depth sequence includes: performing voxel segmentation on the spatial point cloud, extracting the main deformation direction of each voxel unit through principal component analysis; calculating the displacement vector field of the point cloud between adjacent frames using an optical flow tracing algorithm based on iterative nearest point; when the tracking error does not converge sufficiently through iteration, increasing the deformation constraint weight and re-registering; and improving the temporal resolution of the depth sequence to the target frame rate through depth interpolation.

[0058] Among them, dynamic deformation can refer to the local geometric deformation of the human body caused by joint movement, muscle contraction, or clothing movement in the time dimension; updating local correspondence based on optical flow can refer to calculating the pixel displacement vector field on the point cloud projection image of adjacent frames using the Lucas-Kanade optical flow method, with a time interval of 0.1 s, and marking the displacement deviation >0.03 m / s as the deformation active area; tracking error can refer to the average Euclidean distance error between point clouds during the iterative nearest point (ICP) registration process, with a preset threshold of 0.04 m; adjusting constraint weights can refer to increasing the weight coefficient of the deformation constraint term in the ICP objective function (for example, from 0.5 to 0.8) to enhance the modeling ability for non-rigid deformation; continuous depth sequence can refer to a depth map sequence with a temporal resolution of 30 fps and an average inter-frame depth map difference of <0.01 m.

[0059] The specific method is as follows: For a sequence of input frames, the image processing device uses a dynamic deformation tracking method based on optical flow, based on the point cloud data between the current frame and the reference frame. First, the 3D point cloud is segmented into voxels, and principal component analysis is used to extract the principal deformation direction of each voxel unit, thereby initially locating the deformation region and deformation amplitude. Then, based on the iterative nearest-point framework, the local displacement vector field of the point cloud between adjacent frames is calculated in combination with optical flow constraints, and the correspondence is updated frame by frame.

[0060] When the tracking error exceeds a preset threshold and the convergence of iterative optimization fails to meet the requirements, the image processing device automatically increases the deformation constraint weight, suppresses unreasonable offsets in displacement estimation, and re-executes point cloud registration until the error drops below the threshold or the maximum number of iterations is reached. At the temporal level, depth interpolation is applied to each frame's depth map sequence to increase the output frame rate to the target value. Simultaneously, the continuity of the sequence is verified through statistical analysis of depth differences between adjacent frames, ultimately outputting a continuous depth sequence that meets temporal consistency requirements.

[0061] Step 7: Integrate the continuous depth sequence with the binocular vision geometric constraints to construct a global depth field covering the human body boundary, and perform boundary refinement processing through color gradient analysis to determine the fine boundary distribution required for robot perception.

[0062] In this step, the construction of the global depth field includes: performing temporal denoising on the continuous depth sequence using Kalman filtering; generating a 3D point cloud by fusing binocular parallax and geometric baseline parameters using triangulation; performing nearest neighbor interpolation to complete sparse regions in the point cloud with a density lower than a preset threshold, so that the depth field coverage reaches a preset level; and finally refining the human body boundary by weighted superposition of depth weight and gradient weight to improve the boundary positioning accuracy.

[0063] Among them, the global depth field can refer to a three-dimensional depth representation that covers the entire human body boundary, generated by fusing temporal depth information and binocular geometric priors (baseline, focal length, triangulation model); color gradient analysis can refer to using the Sobel operator to extract the color gradient map of the RGB image corresponding to the depth map as a boundary sharpening guide signal; boundary refinement processing can refer to weighted superposition of depth weight (0.6) and gradient weight (0.4) to generate a sub-pixel level fine boundary map; the fine boundary distribution required for robot perception can refer to the boundary output with a positioning accuracy of ±2 pixels, which can be directly used for collision detection and path planning.

[0064] The specific method is as follows: The image processing device first uses Kalman filtering to perform temporal denoising on the continuous depth sequence. This filtering effectively suppresses random noise while preserving the true dynamic changes in depth through the iterative mechanism of maintaining state prediction and observation update, thus significantly improving the temporal stability of the depth sequence.

[0065] Subsequently, by integrating binocular vision geometric constraints and based on baseline distance and focal length parameters, the disparity data is transformed into a 3D point cloud through triangulation. At the same time, nearest neighbor interpolation is applied to sparse regions with density below a preset threshold to achieve the preset level of coverage of the global depth field.

[0066] In the boundary refinement stage, the image processing device uses gradient operators to extract color gradient maps from the RGB image and combines this with local contrast enhancement processing to further highlight the gradient response of the boundary region. Finally, a fine boundary map is generated by weighted superposition of the depth map and the gradient map. The depth weight reflects distance information in three-dimensional space, while the gradient weight reflects the boundary response at the color texture level. After the two are complemented and fused, the boundary localization accuracy and perception clarity are significantly better than those of a single data source, providing directly usable fine boundary distribution results for the robot perception system.

[0067] Using the above methods, this step acquires left and right view images of non-rigid human body regions using a binocular camera and performs texture and contrast-driven region segmentation to generate a structured basic matching map.

[0068] Then, based on the surface detail complexity and the difference in illumination reflection, the blur of the region edge is jointly quantized. When the blur exceeds the standard and the color gradient is uneven, CLAHE enhancement and semantic mask-guided gradient redistribution are performed to output a high-confidence corrected point set.

[0069] Furthermore, relying on neighborhood consistency constraints, we synchronously detect and smooth disparity anomalous points and abrupt depth boundary regions to ensure spatial continuity; then, through a backtracking interpolation mechanism driven by depth transformation and neighborhood consistency feedback, we obtain a highly smooth depth distribution.

[0070] Based on this, the human body contour is segmented, occlusions are filled, and Delaunay triangulation reconstruction is performed to generate a point cloud with reliable geometric quality. Then, through optical flow-driven dynamic deformation tracking and error-sensitive constraint weight adjustment, a highly stable continuous depth sequence is output.

[0071] Finally, the temporal depth and binocular geometric priors are fused to construct a global depth field, and color gradient analysis is used to refine the boundaries, outputting a sub-pixel level fine boundary distribution that meets the robot's perception requirements.

[0072] This seventh-order closed-loop link effectively overcomes the matching failure problem of non-rigid human bodies under texture loss, posture deformation, occlusion and lighting interference, and achieves full coverage, continuous, smooth and fine-grained depth perception capabilities.

[0073] Example 2:

[0074] Deep learning-based human perception and distance measurement systems, such as Figure 2 As shown, the intelligent control system includes: a matching map generation module, a corresponding point set correction module, a disparity correction module, a depth transformation module, a point cloud reconstruction module, a depth sequence generation module, and a depth field construction module, wherein:

[0075] The matching map generation module is used to acquire left and right view images of non-rigid human body regions using a binocular camera. It estimates the initial pixel correspondence based on a dense optical flow algorithm and divides the region according to texture density and local contrast to generate a basic matching map. The function of this module is to provide a stable and robust initial matching benchmark for subsequent modules. Its output basic matching map constitutes the starting point of the entire system's data flow and forms a tightly coupled input-output relationship with the corresponding point set correction module, ensuring that subsequent enhancement and correction operations have a clear target.

[0076] The corresponding point set correction module is used to extract surface detail complexity features and illumination reflection difference features from the basic matching map. When the edge blurring and color gradient non-uniformity meet the preset conditions, adaptive feature enhancement processing is performed, and the corrected pixel corresponding point set is output. It transfers image and mask data with the matching map generation module through shared memory, and outputs the matching point set in the form of a sparse point cloud coordinate array with the disparity correction module. The function of this module in the system is to improve the matching confidence of non-rigid regions and provide a highly reliable input for the disparity correction module. The two work together to solve the local matching drift problem caused by human body deformation.

[0077] The disparity correction module is used to smooth disparity aberration points and abrupt depth boundary regions in the corresponding point set based on neighborhood consistency constraints, and obtain the correction matching result. It transmits point set coordinates and disparity values ​​with the corresponding point set correction module through a DMA channel, and outputs the corrected dense disparity map to the depth conversion module. The function of this module is to suppress the propagation of disparity noise and ensure the spatial continuity of the input data of the depth conversion module. Its output directly determines the boundary integrity and naturalness of the subsequent depth map.

[0078] The depth conversion module is used to convert the disparity values ​​in the correction matching results into continuous depth information and perform local interpolation adjustment on low-texture discontinuous areas to output a smooth depth distribution. This module receives the corrected disparity map from the disparity correction module through memory mapping and outputs it to the point cloud reconstruction module in the form of a floating-point depth matrix. Its function is to establish a reliable mapping bridge from two-dimensional matching to three-dimensional geometry, supporting the physical consistency of subsequent morphological segmentation and point cloud reconstruction.

[0079] The point cloud reconstruction module is used to segment the human body shape contour according to the smooth depth distribution, fill in the occluded and missing areas, and reconstruct the spatial point cloud through triangulation mesh. This module and the depth transformation module access the depth map through CUDA unified memory access, and output the depth sequence generation module in the form of voxelized point cloud structure. Its function is to upgrade the depth information of a single frame into a three-dimensional geometric expression with topological structure, and provide a spatial carrier for dynamic deformation modeling.

[0080] The depth sequence generation module is used to track the dynamic deformation in the spatial point cloud, update the local correspondence based on optical flow, adjust the constraint weights when the tracking error exceeds the limit, and output a continuous depth sequence. It transmits point cloud data with the point cloud reconstruction module through a shared buffer, and outputs a depth map sequence in the form of timestamp aligned with the depth field construction module. Its function is to establish a depth evolution model in the time dimension to support the temporal stability and motion continuity of the global depth field.

[0081] The depth field construction module is used to fuse the continuous depth sequence with binocular vision geometric constraints to construct a global depth field, and output the fine boundary distribution required for robot perception through boundary refinement processing. It transmits depth map sequences with the depth sequence generation module through the DDS communication protocol, and outputs boundary distribution data with the external robot decision system through TCP / IP or CAN FD bus. Its functional positioning is to complete the output of spatiotemporally consistent, sharp-boundary, and robot-resolvable global perception from a single frame point cloud, and it is the final delivery interface of the entire system for downstream applications.

[0082] Through the above technical solution, it is achieved that: because the matching map generation module integrates dense optical flow and region division mechanism, it can provide a stable initial matching reference under non-rigid human posture changes;

[0083] Since the corresponding point set correction module integrates semantic mask and adaptive enhancement strategy, it can significantly improve the matching confidence of blurred edge and uneven lighting regions;

[0084] Because the parallax correction module introduces neighborhood consistency constraints and distance attenuation weighted filtering, it can effectively suppress depth boundary jaggedness and noise diffusion.

[0085] Since the depth transformation module supports local interpolation adjustment in low-texture areas, the continuity of the depth map in areas with weak textures such as clothing wrinkles and hair strands can be guaranteed; since the point cloud reconstruction module uses Delaunay triangulation and voxel compression, the computational cost of subsequent processing can be reduced while ensuring the integrity of the shape.

[0086] Because the depth sequence generation module has an error-driven constraint weight adaptive mechanism, it can cope with tracking failures caused by sudden large deformations.

[0087] Because the depth field construction module integrates temporal filtering, geometric constraints, and gradient weighted refinement, it can output a high-precision boundary distribution that the robot can directly analyze. The above modules work together to solve the technical problems of the algorithm being difficult to implement in engineering, high module coupling, and lack of standardized interfaces, enabling the system to have cross-platform deployment capabilities and real-time perception performance.

[0088] It should be noted that, in this document, the terms "comprising," "including," and any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Specific examples have been used in this document to illustrate the principles and implementation methods of the present invention. These examples are merely for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be pointed out that, due to the limitations of written expression and the objective existence of infinite specific structures, those skilled in the art can make several improvements, modifications, or variations without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, variations, or combinations, or the direct application of the concept and technical solution of the present invention to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based method for human perception and distance measurement, characterized in that, include: The left and right view images containing non-rigid human body regions are acquired by a binocular camera, and the left and right view images are divided into regions to generate a basic matching map. The region edge ambiguity is calculated based on the basic matching map. When the region edge ambiguity exceeds a preset threshold and the color gradient distribution is uneven, adaptive feature enhancement processing is performed on the region, and the corrected pixel corresponding point set is output. Based on the corrected pixel-corresponding point set, the neighborhood consistency constraint method is used to detect and smooth disparity aberration points and abrupt depth boundary regions to obtain the corrected matching result. The disparity values ​​in the corrected matching results are transformed into depth, and the neighborhood depth consistency is detected. When the segmentation accuracy of discontinuous regions is lower than the preset standard, the local interpolation adjustment is performed back to the low texture region to obtain a smooth depth distribution. The human body shape contour is segmented according to the smooth depth distribution, the occluded and missing areas are filled, and a spatial point cloud is generated using a triangulation mesh reconstruction algorithm. Track the dynamic deformation caused by non-rigid attitude changes in the spatial point cloud, update the local correspondence based on optical flow, adjust the constraint weights when the tracking error exceeds a preset threshold, and output a continuous depth sequence. By integrating the continuous depth sequence with binocular vision geometric constraints, a global depth field covering the human body boundary is constructed, and boundary refinement processing is performed through color gradient analysis to determine the fine boundary distribution required for robot perception.

2. The deep learning-based human perception and distance measurement method according to claim 1, characterized in that, The specific method for dividing the left and right view images into regions includes: estimating the initial pixel correspondence between the left and right view images based on the dense optical flow algorithm, and dividing the left and right view images into regions according to the initial pixel correspondence, texture density, and local contrast.

3. The deep learning-based human perception and distance measurement method according to claim 2, characterized in that, The estimation of the initial pixel correspondence based on the dense optical flow algorithm includes: A dense optical flow algorithm based on a multi-scale pyramid structure is adopted. The image is downsampled at each scale level, the displacement vector field between adjacent pixels is calculated, and the pixel displacement is refined layer by layer during the propagation process from coarse to fine in the pyramid to obtain a dense disparity estimate covering the entire image. The initial pixel correspondence is formed based on the dense disparity estimate.

4. The deep learning-based human perception and distance measurement method according to claim 1, characterized in that, Calculating the region edge ambiguity based on the base matching map includes: The surface detail complexity features and light reflection difference features of the human body region are extracted from the base matching map, and the edge blurring of the region is calculated based on the surface detail complexity features and the light reflection difference features.

5. The deep learning-based human perception and distance measurement method according to claim 4, characterized in that, The adaptive feature enhancement process includes: Adaptive histogram equalization is used to improve local contrast in blurred edge regions. For regions with uneven color gradient distribution, a semantic segmentation network is used to extract human body region masks, and edge-aware smoothing filtering is used to unify the gradient distribution. After enhancement, the corresponding point set is recalculated, which significantly improves the number of matching points and the matching confidence.

6. The deep learning-based human perception and distance measurement method according to claim 1, characterized in that, The process of performing depth transformation on the disparity values ​​in the corrected matching results, detecting neighborhood depth consistency, and when the segmentation accuracy of discontinuous regions is lower than a preset standard, backtracking to low-texture regions for local interpolation adjustment to obtain a smooth depth distribution specifically includes: A neighborhood window centered on the target pixel is constructed, and the deviation between the disparity value of the target pixel and the neighborhood mean is calculated. When the deviation exceeds a preset standard deviation multiple, the target pixel is marked as an outlier and replaced with the neighborhood weighted average. At the same time, a distance attenuation weighted smoothing filter is applied to the abrupt depth boundary regions to ensure the spatial continuity of the depth values.

7. The deep learning-based human perception and distance measurement method according to claim 1, characterized in that, The process of segmenting the human body shape contour based on the smooth depth distribution, filling in occluded and missing areas, and generating a spatial point cloud using a triangulation mesh reconstruction algorithm specifically includes: Delaunay triangulation is performed on the human foreground point cloud in the smooth depth distribution to generate a triangular mesh covering the human surface, and abnormal triangles with side lengths exceeding a preset threshold are removed; then the point cloud is voxelized and compressed to homogenize the point cloud density and reduce the computational cost of subsequent processing.

8. The deep learning-based human perception and distance measurement method according to claim 1, characterized in that, The process of tracking dynamic deformations caused by non-rigid attitude changes in the spatial point cloud, updating local correspondences based on optical flow, adjusting constraint weights when the tracking error exceeds a preset threshold, and outputting a continuous depth sequence specifically includes: Voxel segmentation is performed on the spatial point cloud, and the principal deformation direction of each voxel unit is extracted by principal component analysis. The displacement vector field of the point cloud between adjacent frames is calculated by an optical flow tracing algorithm based on the iterative nearest point. When the tracking error does not converge sufficiently, the deformation constraint weight is increased and the data is re-registered. The temporal resolution of the depth sequence is improved to the target frame rate by depth interpolation.

9. The deep learning-based human perception and distance measurement method according to claim 1, characterized in that, The construction of the global depth field includes: Kalman filtering is used to denoise the continuous depth sequence in time; triangulation is used to fuse binocular parallax and geometric baseline parameters to generate a 3D point cloud; nearest neighbor interpolation is performed on sparse regions in the point cloud with a density below a preset threshold to make the depth field coverage reach a preset level; finally, the human body boundary is refined by weighted superposition of depth weight and gradient weight to improve the boundary positioning accuracy.

10. A deep learning-based human perception and distance measurement system, characterized in that, The system includes: The matching map generation module is used to acquire left and right view images of non-rigid human body regions through a binocular camera, estimate the initial pixel correspondence based on the dense optical flow algorithm, and divide the region according to texture density and local contrast to generate a basic matching map. The corresponding point set correction module is used to extract surface detail complexity features and illumination reflection difference features from the basic matching map. When the edge ambiguity and color gradient non-uniformity meet the preset conditions, adaptive feature enhancement processing is performed, and the corrected pixel corresponding point set is output. The disparity correction module is used to smooth the disparity abnormal points and abrupt depth boundary regions in the corresponding point set based on the neighborhood consistency constraint, so as to obtain the correction matching result. The depth conversion module is used to convert the disparity values ​​in the correction matching results into continuous depth information, and to perform local interpolation adjustment on low-texture discontinuous areas to output a smooth depth distribution. The point cloud reconstruction module is used to segment the human body shape outline according to the smooth depth distribution, fill in the occluded and missing areas, and reconstruct a spatial point cloud through triangulation mesh. The depth sequence generation module is used to track the dynamic deformation in the spatial point cloud, update the local correspondence based on optical flow, adjust the constraint weight when the tracking error exceeds the limit, and output a continuous depth sequence. The depth field construction module is used to fuse the continuous depth sequence with the binocular vision geometric constraints to construct a global depth field, and output the fine boundary distribution required for robot perception through boundary refinement processing.

Images