Method and system for adaptive reconstruction of non-rigid object based on two-dimensional visual tracking

Abstract

A method and system for adaptive structured light reconstruction of non-rigid objects based on 2D visual tracking. The method includes: acquiring the 2D visual tracking results, 2D spatial mapping results, and cross-modal motion state information of the target; the cross-modal motion state information includes 3D spatial position, depth direction covariance, vibration amplitude, and motion mode classification results; determining the target depth value based on 2D target depth estimation, 3D spatial position, and preset priority rules; determining the working distance for structured light reconstruction based on the target depth value and target physical thickness; dynamically determining the depth search window based on the depth direction covariance and working distance; selecting a target resolution level from multiple preset reconstruction resolution levels based on the working distance, and suppressing frequent level switching through a continuous frame confirmation mechanism; dynamically adjusting the decoding confidence threshold of structured light stripes based on vibration amplitude; outputting structured light reconstruction strategy parameters; and performing adaptive reconstruction based on the structured light reconstruction strategy parameters.

Description

Technical Field

[0001] This invention relates to the field of structured light 3D reconstruction and cross-modal visual perception technology, specifically to a non-rigid object structured light adaptive reconstruction method and system based on 2D visual tracking, applicable to real-time 3D measurement scenarios where structured light reconstruction parameters are dynamically adjusted based on 2D visual tracking results and cross-modal motion states. Background Technology

[0002] With the rapid development of industrial structured light sensing and robotic flexible manipulation technologies, the demand for real-time 3D reconstruction of non-rigid objects such as flexible pipes, thin sheets, and elastic components is increasing. Such systems typically possess both high-frame-rate 2D fast visual inspection and tracking capabilities and low-frame-rate 3D structured light reconstruction capabilities, using 2D tracking for real-time localization and state estimation, and 3D reconstruction to provide absolute geometric information.

[0003] Existing structured light 3D reconstruction technologies primarily rely on pre-calibrated fixed parameters, including a fixed depth search range, a fixed reconstruction resolution, and a fixed stripe decoding confidence threshold. To adapt to different working distances, existing solutions typically preset several sets of parameter levels based on the application scenario during the offline phase, or perform a full-depth search to ensure no target is missed. However, these parameters are not adjusted according to the actual state of the target during runtime.

[0004] However, in real-world non-rigid object reconstruction scenarios, the aforementioned fixed-parameter methods still face numerous challenges. First, the motion or deformation of non-rigid objects causes their effective depth range to constantly change, making a fixed depth search range either too wide, wasting computational resources, or too narrow, resulting in missed targets. Second, the computational cost of full-depth search in structured light is significant, making it difficult to meet timing constraints in high-frame-rate real-time systems; furthermore, continuing to search the full depth range when depth estimation has converged to a high accuracy level leads to unnecessary efficiency waste. Third, a fixed reconstruction resolution struggles to balance the accuracy requirements at close range with the stability requirements at long range, and overly sensitive resolution switching can cause inter-frame discontinuities in the reconstruction results. Additionally, vibration or swaying of non-rigid objects can cause blurring or shifting of the structured light projection fringes during acquisition. A fixed decoding confidence threshold under vibration conditions can either be too strict, leading to insufficient effective decoding points and reconstruction failure, or too lenient, introducing substantial mismatch noise. Therefore, existing technologies struggle to simultaneously achieve efficient depth search, suitable reconstruction resolution, and robust fringe decoding under conditions of non-rigid object motion, deformation, and vibration. Summary of the Invention

[0005] In view of the above-mentioned problems of the prior art, the purpose of the present invention is to provide a non-rigid object structure light adaptive reconstruction method and system based on two-dimensional visual tracking. The method uses two-dimensional visual tracking results, two-dimensional spatial mapping results and cross-modal motion state to jointly and adaptively adjust the working distance, depth search window, reconstruction resolution level and decoding tolerance of structure light reconstruction, so as to balance reconstruction efficiency, accuracy and robustness.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The first aspect is a non-rigid object structured light adaptive reconstruction method based on two-dimensional visual tracking, which includes the following steps: S1. Obtain the target's two-dimensional visual tracking results, two-dimensional spatial mapping results, and cross-modal motion state information; the two-dimensional visual tracking results include the tracking status; the two-dimensional spatial mapping results include two-dimensional target depth estimation; the cross-modal motion state information includes three-dimensional spatial position, depth direction covariance, vibration amplitude, and motion pattern classification results; S2. Determine the target depth value based on the two-dimensional target depth estimation, three-dimensional spatial position, and preset priority rules; S3. Determine the working distance for structured light reconstruction based on the target depth value and the target physical thickness; S4. Dynamically determine the depth search window based on the depth direction covariance and working distance; S5. Select the target resolution level from multiple preset reconstruction resolution levels based on the working distance, and suppress frequent switching of the resolution level through a continuous frame confirmation mechanism. S6. Dynamically adjust the decoding confidence threshold of the structured light stripes according to the vibration amplitude; S7. Output the structured light reconstruction strategy parameters, including working distance, depth search window, reconstruction target level, and decoding confidence threshold; the structured light reconstruction module performs adaptive reconstruction based on the structured light reconstruction strategy parameters.

[0007] As a preferred option, in S2, the priority rule is as follows: when the two-dimensional spatial mapping result is valid, the two-dimensional target depth estimate is used as the target depth value; when the two-dimensional spatial mapping result is invalid but the three-dimensional spatial position is valid, the depth component of the three-dimensional spatial position is used as the target depth value; when both the two-dimensional spatial mapping result and the three-dimensional spatial position are invalid, the previous valid depth value is retained as the target depth value. Preferably, S3 includes: Working distance is calculated as follows: in, For the target depth value, For the target physical thickness, and These represent the minimum and maximum working distances allowed by the structured light hardware, respectively.

[0008] Preferably, S4 includes: The standard deviation of depth is the square root of the covariance in the depth direction. , depth standard deviation Multiply by window size Get half window width And apply upper and lower limit clamps, calculated as follows: Set the depth search window to ; When the depth direction covariance is less than the convergence threshold, the half-window width reaches its minimum value. When the depth direction covariance is greater than the divergence threshold or there is no valid depth information available, the depth search window reverts to the full depth range. To ensure that no target is overlooked.

[0009] As a preferred embodiment, S5 includes: When the working distance is greater than the high resolution switching threshold, the target resolution level is high resolution; when the working distance is less than the low resolution switching threshold, the target resolution level is low resolution; otherwise, it is medium resolution. When the target resolution level is different from the current resolution level, continuous frame counting is started. The target resolution level must appear continuously to reach the number of stabilization frames before switching can be performed. If the target level changes during this period, the count is reset to zero. After the target resolution level is successfully switched, it enters a cooling-off period, during which it will no longer respond to level change requests.

[0010] As a preferred embodiment, S6 includes: Based on vibration amplitude The confidence threshold and matching cost threshold for dynamically adjusting structured light stripe decoding are calculated as follows: in, The static vibration threshold, For strong vibration threshold, As the baseline decoding confidence threshold, The most lenient decoding confidence threshold. This is the threshold for matching cost.

[0011] As a preferred embodiment, S7 includes: When the structured light reconstruction fails in the previous cycle, the depth search window is automatically expanded to the full depth range in the current cycle. After successful reconstruction, the adaptive search window is restored based on the new depth information. When the tracking status is lost, the depth search window is automatically expanded to the full depth search, and the resolution level is reverted to medium resolution.

[0012] Secondly, a non-rigid object structured light adaptive reconstruction system based on two-dimensional visual tracking includes: The state acquisition module is used to acquire the target's two-dimensional visual tracking results, two-dimensional spatial mapping results, and cross-modal motion state information; the two-dimensional visual tracking results include the tracking state; the two-dimensional spatial mapping results include two-dimensional target depth estimation; and the cross-modal motion state information includes three-dimensional spatial position, depth direction covariance, vibration amplitude, and motion pattern classification results. The depth determination module is used to determine the target depth value based on the two-dimensional target depth estimation, the three-dimensional spatial position, and preset priority rules; The working distance module is used to determine the working distance for structured light reconstruction based on the target depth value and the target physical thickness. The search window module is used to dynamically determine the depth search window based on the depth direction covariance and working distance. The resolution level module is used to select the target resolution level from multiple preset reconstruction resolution levels based on the working distance, and to suppress frequent level switching through a continuous frame confirmation mechanism. The decoding tolerance module is used to dynamically adjust the decoding confidence threshold of the structured light stripes according to the vibration amplitude; The reconstruction strategy parameter output module is used to output structured light reconstruction strategy parameters, including working distance, depth search window, reconstruction target level, and decoding confidence threshold. The structured light reconstruction module is used to perform adaptive reconstruction based on the structured light reconstruction strategy parameters; The non-rigid object structure light adaptive reconstruction system based on two-dimensional visual tracking is used to implement the non-rigid object structure light adaptive reconstruction method and its steps based on two-dimensional visual tracking as described in the first aspect.

[0013] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention include at least the following: 1. This invention uses the target depth and depth direction covariance provided by two-dimensional spatial mapping and cross-modal motion state as a basis to dynamically determine the working distance and depth search window for structured light reconstruction. This allows the search range to be narrowed when the depth state converges to improve efficiency, and the search range to be expanded when uncertain to ensure reconstruction success rate. Furthermore, it automatically reverts to full depth search when there is no effective depth information. Thus, under the condition that the depth of non-rigid objects is constantly changing, it takes into account both reconstruction efficiency and success rate.

[0014] 2. This invention adaptively selects the reconstruction resolution level based on the working distance, and suppresses frequent switching of the resolution level through continuous frame confirmation and cooling period anti-shake mechanism, so that high resolution ensures reconstruction accuracy at close distances and low resolution ensures stability at long distances, while avoiding inter-frame discontinuity at the boundary.

[0015] 3. This invention dynamically adjusts the confidence threshold and matching cost threshold of structured light stripe decoding based on the vibration amplitude. When the vibration is severe, the tolerance is increased to increase the number of effective decoding points and avoid reconstruction failure. When the vibration is stable, the tolerance is tightened to suppress mismatch noise, thereby improving the reconstruction robustness under the vibration conditions of non-rigid objects. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the method framework of Embodiment 1 of the present invention; Figure 3 This is a schematic diagram illustrating the principle of depth-direction covariance-driven adaptive adjustment of the search window in Embodiment 1 of the present invention. Figure 4 This is a schematic diagram of the working distance and reconstruction resolution level switching and anti-shake mechanism in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the mapping relationship for vibration amplitude-driven decoding tolerance adjustment in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of the system framework of Embodiment 2 of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the technical ideas and implementation methods of the present invention and are not intended to limit the scope of protection of the present invention.

[0019] This embodiment uses a real-time detection system with both 2D fast visual tracking and 3D structured light reconstruction capabilities as its application background. It addresses issues such as low search efficiency due to changes in target depth and the difficulty of adapting fixed resolution and fixed decoding thresholds to motion and vibration conditions. Following the workflow of "state acquisition—target depth determination—working distance determination—depth search window adaptation—reconstruction resolution level selection and anti-shake—vibration sensing decoding tolerance adjustment—strategy parameter output," the structured light adaptive reconstruction method proposed in this invention is fully described. This embodiment can run on industrial control computers, edge computing devices, or embedded platforms, but the scope of protection of this invention is not limited to specific hardware platforms.

[0020] like Figure 1 , Figure 2 As shown, the structured light adaptive reconstruction process in this embodiment is deployed in a multimodal vision processing pipeline. Its inputs include: two-dimensional visual tracking results, including tracking status and tracking confidence; two-dimensional spatial mapping results, including mapping depth and its validity flags; cross-modal motion status, including spatial position, depth direction covariance, vibration amplitude, motion mode, and prediction confidence; and structured light hardware configuration, including the allowed minimum working distance, maximum working distance, and a projector preset list. The working distance refers to the physical distance from the structured light device to the geometric center of the target object. Its outputs are structured light reconstruction strategy parameters, including the working distance, upper and lower bounds of the depth search window, reconstruction resolution level number, decoding confidence threshold, and matching cost threshold. In this embodiment, the target physical thickness is 3.0 mm, the default working distance is 200 mm, the allowed minimum working distance for structured light is 100 mm, and the maximum working distance is 400 mm.

[0021] The preferred values ​​for each parameter in this embodiment are as follows, and all can be configured to be covered: window magnification is 3.0, minimum half-window is 5 mm, maximum half-window is 50 mm, depth direction covariance convergence threshold is 4.0 square millimeters (corresponding to a depth standard deviation of 2 mm), depth direction covariance divergence threshold is 100.0 square millimeters (corresponding to a depth standard deviation of 10 mm), high resolution switching threshold is 150 mm, low resolution switching threshold is 300 mm, anti-shake frame rate is 30 frames, switching cooldown period is 100 frames, baseline decoding confidence threshold is 0.8, most relaxed decoding confidence threshold is 0.4, static vibration threshold is 0.1 mm per frame, strong vibration threshold is 1.0 mm per frame, baseline matching cost threshold is 2.0, and most relaxed matching cost threshold is 5.0.

[0022] like Figure 1 , Figure 2 The non-rigid object structured light adaptive reconstruction method based on 2D visual tracking shown includes the following steps: Step S1: Obtain the state, obtain the two-dimensional visual tracking results, two-dimensional spatial mapping results and cross-modal motion state of the target. The motion state includes at least the target's spatial position, depth direction covariance, vibration amplitude and motion pattern. During the state acquisition phase, the system acquires 2D visual tracking results, 2D spatial mapping results, and cross-modal motion states from the multimodal pipeline. The 2D visual tracking results provide the tracking state (normal tracking, prediction, search, loss, etc.) and tracking confidence. The 2D spatial mapping results provide the target depth estimate and its validity indicators after backprojection of the 2D detection coordinates using calibration parameters. The cross-modal motion states provide the 3D spatial position, the depth component in the state covariance matrix, vibration amplitude, and motion mode output by the fusion estimator.

[0023] Step S2: Determine the depth and determine the target depth according to priority: when the two-dimensional spatial mapping result is valid, its depth is taken as the target depth; otherwise, the spatial position depth of the motion state is taken as the target depth. During the target depth determination stage, multi-source depth information is fused according to priority: when the two-dimensional spatial mapping result is valid, the absolute value of its mapping depth is taken and converted into millimeters as the target depth. This source is directly calculated based on two-dimensional detection and calibration parameters, resulting in a fast response speed. When the two-dimensional spatial mapping result is invalid, but the three-dimensional position of the motion state is valid, the absolute value of the depth component of the spatial position of the motion state is taken and converted into millimeters as the target depth. This source is smoothed by the fusion estimator, resulting in good stability but with a delay. When both are invalid, the target depth of the previous valid cycle remains unchanged. If no valid depth has ever been obtained, the default working distance is used.

[0024] Step S3: Determine the working distance. The working distance for structured light reconstruction is determined based on the target depth and the target's physical thickness. The working distance is the sum of half the target depth and half the physical thickness. During the working distance determination stage, compensation is performed by superimposing half the target's physical thickness on the target depth. The calculation relationship is as follows: in, For target depth, For the target physical thickness, and These represent the minimum and maximum working distances allowed by the structured light hardware, respectively. ( ) represents the limiting function. Physical thickness compensation directs the working distance toward the geometric center of the target rather than the front surface, thereby improving the symmetry of the depth search window and the reconstruction hit rate.

[0025] Step S4: Determine the search window. The depth search window is determined based on the depth direction covariance. When the covariance representing the depth state has converged, the search window is reduced. When the covariance representing the depth state is uncertain, the search window is expanded. When there is no effective depth information, the search window is rolled back to full depth search. During the depth search window adaptation phase, such as Figure 3 As shown, the dynamic scaling of the search window is driven by the depth direction covariance. The depth direction covariance is taken. The square root of the depth standard deviation Multiply it by the window size. Get half window width And apply upper and lower limit clamps, the calculation relationship is as follows: The search window is When the depth direction covariance is less than the preset convergence threshold ( The half-window width is taken as the minimum value. At this point, the depth estimation has converged sufficiently, and the narrowest window is used to obtain the highest search efficiency. When the depth direction covariance is greater than the divergence threshold ( If no valid depth information is currently available, the search window will revert to the full depth range. This ensures that no target is missed. Therefore, the search window automatically expands and contracts according to the depth estimation accuracy: a narrow search is used during convergence to save computational resources, a wide search is used when uncertain to ensure success rate, and a full search is used as a fallback in case of failure.

[0026] Step S5: Determine the resolution level, select the current level from multiple preset reconstruction resolution levels based on the working distance, and suppress frequent level switching through a continuous frame confirmation mechanism; During the reconstruction resolution setting and image stabilization stage, such as Figure 4 As shown, three preset reconstruction resolution levels are provided: high resolution (projector preset number 0) for close-range precision reconstruction, medium resolution (projector preset number 1) for medium-range reconstruction, and low resolution (projector preset number 2) for long-range stable reconstruction. The target resolution level is determined based on the working distance: when the working distance is less than the high resolution switching threshold, the target resolution is high; when the working distance is greater than the low resolution switching threshold, the target resolution is low; and in between, it is medium resolution. When the target resolution differs from the current resolution, continuous frame counting is initiated: the target resolution must appear consecutively for a certain number of anti-shake frames before switching can proceed; if the target resolution changes during this period, the count is reset to zero. After a successful switch, a cooldown period begins, during which resolution change requests are no longer responded to. Therefore, the resolution level will not frequently flip due to minor fluctuations in the working distance around the threshold, ensuring the inter-frame continuity of the reconstruction results.

[0027] Step S6: Determine the decoding tolerance and dynamically adjust the confidence threshold for structured light stripe decoding based on the vibration amplitude. The larger the vibration amplitude, the looser the confidence threshold. In the vibration sensing decoding tolerance adjustment stage, such as Figure 5 As shown, based on the vibration amplitude The confidence threshold and matching cost threshold for structured light fringe decoding are dynamically adjusted. The decoding confidence threshold is calculated as follows: in The static vibration threshold, For strong vibration threshold, As the baseline decoding confidence threshold, This represents the most lenient decoding confidence threshold. Note that a smaller confidence threshold indicates greater leniency (accepting decoding results with lower confidence levels). The matching cost threshold is set using the same interpolation coefficient. The baseline matching cost threshold is linearly interpolated to the most lenient matching cost threshold; the more lenient the threshold, the higher the allowed matching cost. Thus, when the vibration is stable, a strict threshold is used to suppress noise, while when the vibration is severe, the tolerance is relaxed to increase the number of effective decoding points and avoid reconstruction failure.

[0028] Step S7: Strategy output, assemble the working distance, depth search window, reconstruction resolution level and decoding confidence threshold into structured light reconstruction strategy parameters for the structured light reconstruction module to perform adaptive reconstruction.

[0029] During the strategy parameter output phase, the working distance, depth search window upper and lower bounds, reconstruction resolution level number, decoding confidence threshold, and matching cost threshold are assembled into structured light reconstruction strategy parameters for output. If reconstruction fails in the previous cycle, the search window is automatically expanded to the full depth range in the current cycle. After successful reconstruction, the adaptive search window is restored based on the new depth information. When the tracking status is lost, the search window automatically expands to the full depth search, and the resolution level reverts to medium resolution. The downstream structured light reconstruction module configures the projector presets, sets the depth search range, and selects the decoding confidence threshold based on the above strategy parameters, thereby completing the cross-modal inverse adaptive adjustment of 2D tracking and motion state to 3D reconstruction.

[0030] Example 2: like Figure 6 The non-rigid object structured light adaptive reconstruction system based on 2D vision tracking shown includes: The state acquisition module is used to acquire the target's two-dimensional visual tracking results, two-dimensional spatial mapping results, and cross-modal motion state information; the two-dimensional visual tracking results include the tracking state; the two-dimensional spatial mapping results include two-dimensional target depth estimation; and the cross-modal motion state information includes three-dimensional spatial position, depth direction covariance, vibration amplitude, and motion pattern classification results. The depth determination module is used to determine the target depth value based on the two-dimensional target depth estimation, the three-dimensional spatial position, and preset priority rules; The working distance module is used to determine the working distance for structured light reconstruction based on the target depth value and the target physical thickness. The search window module is used to dynamically determine the depth search window based on the depth direction covariance and working distance. The resolution level module is used to select the target resolution level from multiple preset reconstruction resolution levels based on the working distance, and to suppress frequent level switching through a continuous frame confirmation mechanism. The decoding tolerance module is used to dynamically adjust the decoding confidence threshold of the structured light stripes according to the vibration amplitude; The reconstruction strategy parameter output module is used to output structured light reconstruction strategy parameters, including working distance, depth search window, reconstruction target level, and decoding confidence threshold. The structured light reconstruction module is used to perform adaptive reconstruction based on the structured light reconstruction strategy parameters.

[0031] It should be noted that the present invention is not limited to the specific values ​​and forms mentioned above: the window magnification, minimum / maximum half window, convergence / divergence threshold, gear switching threshold and anti-shake frame rate, cooldown period, vibration threshold mapping parameters and the baseline and leniency values ​​of decoding / matching threshold can all be adjusted according to the scenario; the number of resolution gears can be expanded to more than three; the depth direction covariance can be replaced with other uncertainty measures; the vibration amplitude can be replaced with vibration level or spectrum analysis results; the interpolation method of confidence threshold can be changed from linear to piecewise or nonlinear mapping.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-rigid object structured light adaptive reconstruction method based on two-dimensional visual tracking, characterized in that, Includes the following steps: S1. Obtain the target's two-dimensional visual tracking results, two-dimensional spatial mapping results, and cross-modal motion state information; the two-dimensional visual tracking results include the tracking state; The two-dimensional spatial mapping results include two-dimensional target depth estimation; Cross-modal motion state information includes three-dimensional spatial position, depth direction covariance, vibration amplitude, and motion mode classification results; S2. Determine the target depth value based on the two-dimensional target depth estimation, three-dimensional spatial position, and preset priority rules; S3. Determine the working distance for structured light reconstruction based on the target depth value and the target physical thickness; S4. Dynamically determine the depth search window based on the depth direction covariance and working distance; S5. Select the target resolution level from multiple preset reconstruction resolution levels based on the working distance, and suppress frequent switching of the resolution level through a continuous frame confirmation mechanism. S6. Dynamically adjust the decoding confidence threshold of the structured light stripes according to the vibration amplitude; S7. Output the structured light reconstruction strategy parameters, including working distance, depth search window, reconstruction target level, and decoding confidence threshold; the structured light reconstruction module performs adaptive reconstruction based on the structured light reconstruction strategy parameters.

2. The non-rigid object structure light adaptive reconstruction method based on two-dimensional visual tracking according to claim 1, characterized in that, In S2, the priority rule is: when the two-dimensional spatial mapping result is valid, the two-dimensional target depth estimate is used as the target depth value; When the two-dimensional spatial mapping result is invalid but the three-dimensional spatial position is valid, the depth component of the three-dimensional spatial position is used as the target depth value. When both the 2D spatial mapping result and the 3D spatial position are invalid, the previous valid depth value is retained as the target depth value.

3. The non-rigid object structure light adaptive reconstruction method based on two-dimensional visual tracking according to claim 1, characterized in that, S3 include: Working distance is calculated as follows: in, For the target depth value, For the target physical thickness, and These represent the minimum and maximum working distances allowed by the structured light hardware, respectively.

4. The non-rigid object structure light adaptive reconstruction method based on two-dimensional visual tracking according to claim 1, characterized in that, S4 include: The standard deviation of depth is the square root of the covariance in the depth direction. , depth standard deviation Multiply by window size Get half window width And apply upper and lower limit clamps, calculated as follows: Set the depth search window to ; When the depth direction covariance is less than the convergence threshold, the half-window width reaches its minimum value. When the depth direction covariance is greater than the divergence threshold or there is no valid depth information available, the depth search window reverts to the full depth range. To ensure that no target is overlooked.

5. The non-rigid object structure light adaptive reconstruction method based on two-dimensional visual tracking according to claim 1, characterized in that, S5 include: When the working distance is greater than the high resolution switching threshold, the target resolution level is high resolution; when the working distance is less than the low resolution switching threshold, the target resolution level is low resolution; otherwise, it is medium resolution. When the target resolution level is different from the current resolution level, continuous frame counting is started. The target resolution level must appear continuously to reach the number of stabilization frames before switching can be performed. If the target level changes during this period, the count is reset to zero. After the target resolution level is successfully switched, it enters a cooling-off period, during which it will no longer respond to level change requests.

6. The non-rigid object structured light adaptive reconstruction method based on two-dimensional visual tracking according to claim 1, characterized in that, S6 include: Based on vibration amplitude The confidence threshold and matching cost threshold for dynamically adjusting structured light stripe decoding are calculated as follows: in, The static vibration threshold, For strong vibration threshold, As the baseline decoding confidence threshold, The most lenient decoding confidence threshold. This is the threshold for matching cost.

7. The non-rigid object structure light adaptive reconstruction method based on two-dimensional visual tracking according to claim 1, characterized in that, S7 includes: When the structured light reconstruction fails in the previous cycle, the depth search window is automatically expanded to the full depth range in the current cycle. After successful reconstruction, the adaptive search window is restored based on the new depth information. When the tracking status is lost, the depth search window is automatically expanded to the full depth search, and the resolution level is reverted to medium resolution.

8. A non-rigid object structured light adaptive reconstruction system based on two-dimensional visual tracking, characterized in that, include: The state acquisition module is used to acquire the target's two-dimensional visual tracking results, two-dimensional spatial mapping results, and cross-modal motion state information; Two-dimensional visual tracking results include tracking status; The two-dimensional spatial mapping results include two-dimensional target depth estimation; Cross-modal motion state information includes three-dimensional spatial position, depth direction covariance, vibration amplitude, and motion mode classification results; The depth determination module is used to determine the target depth value based on the two-dimensional target depth estimation, the three-dimensional spatial position, and preset priority rules; The working distance module is used to determine the working distance for structured light reconstruction based on the target depth value and the target physical thickness. The search window module is used to dynamically determine the depth search window based on the depth direction covariance and working distance. The resolution level module is used to select the target resolution level from multiple preset reconstruction resolution levels based on the working distance, and to suppress frequent level switching through a continuous frame confirmation mechanism. The decoding tolerance module is used to dynamically adjust the decoding confidence threshold of the structured light stripes according to the vibration amplitude; The reconstruction strategy parameter output module is used to output structured light reconstruction strategy parameters, including working distance, depth search window, reconstruction target level, and decoding confidence threshold. The structured light reconstruction module is used to perform adaptive reconstruction based on the structured light reconstruction strategy parameters; The non-rigid object structure light adaptive reconstruction system based on two-dimensional visual tracking is used to implement the non-rigid object structure light adaptive reconstruction method and its steps based on two-dimensional visual tracking as described in claim 1.

Images