A two-dimensional visual detection enhancement method and system for non-rigid objects driven by three-dimensional spatial priors
By combining 3D structured light reconstruction with cross-modal motion state fusion to calculate the prior and perspective eccentricity compensation of 2D elliptical shape, the recognition bias caused by deformation and tilt in non-rigid object detection is solved, achieving high-precision and robust 2D visual detection.
Patent Information
- Application Number
- CN202611233887.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-17
AI Technical Summary
Existing two-dimensional vision detection technologies struggle to enhance detection accuracy by utilizing three-dimensional absolute geometric information in the detection of non-rigid objects. Furthermore, they are susceptible to erroneous prior knowledge contamination caused by target deformation, surface tilt, and unstable states, leading to recognition biases.
By fusing the 3D structured light reconstruction results with cross-modal motion states, the prior of the 2D elliptical shape and the compensation amount of perspective eccentricity are calculated. Combined with prior confidence adjustment, the results are back-injected into the 2D vision detection module to improve the fitting success rate and accuracy, and avoid contamination by unreliable priors.
It improves the shape fitting success rate and accuracy of non-rigid object detection, eliminates systematic central bias, and enhances the robustness and accuracy of detection.
Smart Images

Figure CN123118482A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of machine vision and cross-modal perception technology, specifically to a method and system for enhancing the two-dimensional visual detection of non-rigid objects driven by three-dimensional spatial priors. It is applicable to real-time detection scenarios where two-dimensional visual detection is enhanced by inversely using three-dimensional structured light reconstruction results and cross-modal motion states. Background Technology
[0002] With the rapid development of industrial structured light sensing and robotic flexible manipulation technologies, the demand for real-time visual inspection of non-rigid objects such as flexible pipes, sheets, and elastic components is increasing. Such systems typically possess both low-frame-rate 3D structured light reconstruction capabilities and high-frame-rate 2D rapid visual inspection capabilities, using 2D inspection for real-time positioning and 3D reconstruction to provide absolute geometric information.
[0003] Existing 2D visual detection techniques primarily rely on the image's own texture, contours, or grayscale gradients to locate targets within the image plane through template matching, contour fitting, or deep learning-based object detection, and to fit marker points with circular or elliptical models when needed. To improve robustness, existing schemes typically limit prior information to the 2D plane, such as constraining the search in the current frame with the scale of the detection box from the previous frame, or constraining the fitting with a fixed shape model, while rarely utilizing the absolute spatial information provided by the 3D modality.
[0004] However, in practical non-rigid object detection scenarios, the aforementioned methods relying solely on two-dimensional information still face numerous challenges. First, the local deformation and surface tilt of non-rigid objects cause the projected shape of the target on the image plane to constantly change, making it difficult for fixed two-dimensional shape models to adapt, leading to fitting failures or decreased accuracy. Second, when the target surface normal vector is not perpendicular to the camera's optical axis, its circular marker surface appears as an ellipse after perspective projection, and the visual center obtained by fitting the ellipse does not coincide with the projection of the three-dimensional geometric center of the marker surface. Directly using the visual center as the localization result will introduce systematic bias, which is difficult for two-dimensional information itself to detect and correct. Furthermore, relying solely on two-dimensional textures or contours is prone to false detection in cases of reflection, texture loss, or partial occlusion; while introducing priors indiscriminately can contaminate the two-dimensional detection results with erroneous priors when the target's motion state is unstable or the three-dimensional estimation is unreliable. Therefore, existing technologies struggle to both enhance two-dimensional detection using three-dimensional absolute geometry and avoid recognition bias caused by unreliable priors under conditions of non-rigid object deformation, surface tilt, and state fluctuations. Summary of the Invention
[0005] In view of the above-mentioned problems of the prior art, the purpose of the present invention is to provide a three-dimensional spatial prior-driven method and system for enhancing two-dimensional visual detection of non-rigid objects. By using the three-dimensional structured light reconstruction results and cross-modal motion state to inject spatial priors into the two-dimensional visual detection mode, the technical problems of target surface tilt causing changes in projection shape, the offset of the two-dimensional visual center relative to the three-dimensional geometric center, and the detection of erroneous prior contamination when the state is unstable are solved.
[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Firstly, a two-dimensional visual detection enhancement method for non-rigid objects driven by three-dimensional spatial priors includes the following steps: S1. Obtain the 3D structured light reconstruction results and cross-modal motion state of the target; based on the 3D structured light reconstruction results and cross-modal motion state, determine the surface normal vector and working distance according to the preset multi-level priority fusion strategy; S2. Calculate the prior of the two-dimensional ellipse shape based on the projection radius of the surface normal vector on the image plane; the prior of the two-dimensional ellipse shape includes the major axis, minor axis and tilt angle of the ellipse. S3. Calculate the eccentricity based on the tilt angle of the surface normal vector relative to the camera optical axis, the target physical radius, and the working distance. This eccentricity is used to characterize the offset of the visual center of the ellipse obtained by perspective projection of the circular mark surface of the target relative to its three-dimensional geometric center projection. Determine the eccentricity direction along the projection direction of the surface normal vector on the image plane, and then obtain the perspective eccentricity compensation amount. S4. Calculate the prior confidence level based on the three-dimensional prior of the cross-modal motion state; apply attenuation to the prior confidence level when the preset confidence conditions are not met; S5. Input the prior of the two-dimensional elliptical shape, the perspective eccentricity compensation amount, and the prior confidence into the two-dimensional vision detection module to enhance the two-dimensional detection of non-rigid objects.
[0007] Preferably, the 3D structured light reconstruction results include the target surface normal vector, geometric center and its valid indicators; the cross-modal motion state includes spatial position, velocity, surface normal vector, state covariance, motion mode, prediction confidence and 3D prior valid indicators.
[0008] As a preferred option, in S1, the multi-level priority fusion strategy includes: When the 3D structured light reconstruction result is valid and its geometric center is valid, the surface normal vector of the 3D structured light reconstruction result is used, and the absolute value of the geometric center coordinates of the 3D structured light reconstruction result is used as the working distance. When the 3D structured light reconstruction result is valid but the geometric center is invalid, and the 3D prior of the cross-modal motion state is valid, the surface normal vector of the 3D structured light reconstruction result is used and the depth of the spatial position of the cross-modal motion state is taken as the working distance. When there are no effective 3D structured light reconstruction results and both the 3D prior and surface normal vector of the cross-modal motion state are effective, the surface normal vector of the cross-modal motion state is used and the depth of its spatial position is taken as the working distance.
[0009] As a preferred embodiment, in S2, the projection radius is the product of the camera focal length and the target physical radius, divided by the working distance; the major axis of the ellipse is twice the projection radius; the inclination cosine is the absolute value of the component of the surface normal vector in the optical axis direction and clamped between the lower limit and 1; the ellipse angle is the arctangent of the two components of the surface normal vector on the image plane; and the minor axis of the ellipse is the product of the major axis of the ellipse and the absolute value of the inclination cosine.
[0010] Preferably, in S2, the camera focal length is dynamically obtained from the epipolar correction buffer calibrated by the camera, and the calculation of the two-dimensional elliptical shape prior and perspective eccentricity compensation is skipped when the camera focal length is not ready; the physical radius of the target is the measured radius of the preset target marking surface.
[0011] Preferably, the prior confidence calculation in S4 based on the three-dimensional prior of the cross-modal motion state includes: The trace of the state covariance matrix is taken, normalized according to a preset upper limit of covariance, clamped to the interval [0,1], and then complemented to obtain the covariance score; the motion mode is assigned a value in descending order of static, linear, oscillating, accelerating, and random to obtain the motion mode score; the velocity is normalized according to a preset velocity dead zone and velocity saturation upper limit, clamped, and then complemented. The velocity score is full when the velocity does not exceed the dead zone and drops to 0 when the velocity reaches the saturation upper limit; the prediction score is obtained from the prediction confidence; the basic confidence is obtained by weighted summation of the covariance score, motion mode score, velocity score, and prediction score.
[0012] As a preferred option, the preset confidence conditions in S4 are that the motion mode is stationary or linear, the target speed is less than the preset relative stationary threshold, the trace of the covariance is less than the preset convergence threshold, and the out-of-bounds prediction score is greater than the preset prediction threshold. When the confidence conditions are met, the basic confidence is used as the prior confidence; otherwise, the basic confidence is multiplied by the attenuation coefficient as the prior confidence.
[0013] As a preferred option, S4 also includes: The uncertainty of the 3D structured light reconstruction result or cross-modal motion state is mapped to the filter scaling factor of the 2D filter intensity.
[0014] As a preferred option, S5 also includes: The two-dimensional elliptical shape prior, perspective eccentricity compensation, prior confidence, and filtering scaling factor are assembled into spatial prior parameters. The two-dimensional vision detection module constrains ellipse fitting based on the two-dimensional elliptical shape prior, corrects the positioning reference of candidate selection based on the perspective eccentricity compensation, and decides whether to skip the complete detection by accelerating the prior based on the prior confidence. The filtering intensity is adjusted according to the filtering scaling factor, thereby completing the cross-modal enhancement of two-dimensional detection by three-dimensional spatial prior.
[0015] Secondly, a two-dimensional visual detection enhancement system for non-rigid objects driven by three-dimensional spatial priors includes: The prior acquisition module is used to acquire the three-dimensional structured light reconstruction results and cross-modal motion state of the target. Based on the three-dimensional structured light reconstruction results and cross-modal motion state, the surface normal vector and working distance are determined according to the preset multi-level priority fusion strategy. The shape prior module is used to calculate the shape prior of a two-dimensional ellipse based on the projection radius of the surface normal vector onto the image plane; the two-dimensional ellipse shape prior includes the major axis, minor axis and tilt angle of the ellipse. The eccentricity compensation module is used to calculate the eccentricity based on the tilt angle of the surface normal vector relative to the camera optical axis, the target physical radius, and the working distance. It is used to characterize the offset of the visual center of the ellipse obtained by perspective projection of the circular mark surface of the target relative to its three-dimensional geometric center projection; and to determine the eccentricity direction along the projection direction of the surface normal vector on the image plane, thereby obtaining the perspective eccentricity compensation amount. The prior confidence module is used to calculate the prior confidence based on the three-dimensional prior of the cross-modal motion state; and to apply attenuation to the prior confidence when the preset confidence conditions are not met. The two-dimensional vision detection module is used to input the two-dimensional elliptical shape prior, perspective eccentricity compensation amount and prior confidence into the two-dimensional vision detection module to enhance the two-dimensional detection of non-rigid objects. The aforementioned three-dimensional space prior-driven two-dimensional visual detection enhancement system for non-rigid objects is used to implement the three-dimensional space prior-driven two-dimensional visual detection enhancement method and its steps as described in the first aspect.
[0016] Compared with the prior art, the beneficial effects of the present invention include at least the following: 1. Based on the surface normal vector, working distance and target physical scale provided by the three-dimensional structured light reconstruction results and cross-modal motion state, the present invention analyzes the back projection of a two-dimensional elliptical shape prior through perspective projection, so that the two-dimensional visual detection still has shape constraints that conform to the current three-dimensional posture when the surface of a non-rigid object is tilted or the projected shape changes, thereby improving the success rate and accuracy of fitting.
[0017] 2. This invention addresses the problem that the visual center of the projected ellipse of a circular marker surface shifts relative to its three-dimensional geometric center when the target surface normal vector is not perpendicular to the optical axis. It analytically calculates the perspective eccentricity compensation based on the normal tilt angle and the target physical scale, and corrects the positioning reference point along the projection direction of the normal vector on the image plane, thereby eliminating the systematic center deviation that cannot be detected by two-dimensional fitting alone.
[0018] 3. This invention determines the prior confidence level based on the convergence, motion mode, velocity and prediction confidence level represented by the state covariance. When the motion state is unstable or the three-dimensional estimation is unreliable, the prior confidence level is attenuated. Thus, while enhancing the two-dimensional detection with the three-dimensional prior, the detection results are not contaminated by unreliable priors, thus balancing the enhancement effect and robustness. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 This is a schematic diagram of the method structure provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the method flow provided in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram illustrating the principle that the three-dimensional circular marker surface of Embodiment 1 of the present invention is back-projected into a two-dimensional elliptical shape through perspective projection. Figure 4 This is a schematic diagram illustrating the perspective eccentricity compensation principle of the elliptical visual center projected off relative to the three-dimensional geometric center due to the tilt of the surface normal vector in Embodiment 1 of the present invention. Figure 5 This is a schematic diagram of the prior confidence calculation and weight decay when the state is unstable in Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of the system framework of Embodiment 2 of the present invention. Detailed Implementation
[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the technical ideas and implementation methods of the present invention and are not intended to limit the scope of protection of the present invention.
[0022] This embodiment takes a real-time detection system with both 3D structured light reconstruction and 2D rapid visual detection capabilities as its application background. It addresses issues such as changes in projected shape due to target surface tilt, the offset of the 2D visual center relative to the 3D geometric center, and prior contamination during state fluctuations. Following the process of "prior acquisition—elliptical shape prior generation—perspective eccentricity compensation—prior confidence assessment and attenuation—environmental adaptation—prior injection," it provides a complete description of the proposed 3D spatial prior-driven 2D visual detection enhancement method. This embodiment can run on industrial control computers, edge computing devices, or embedded platforms, but the scope of protection of this invention is not limited to specific hardware platforms.
[0023] Example 1: like Figure 1 The illustrated method for enhancing 2D visual detection driven by 3D spatial priors includes the following steps: The 3D spatial prior processing flow in this embodiment is deployed in a multimodal vision processing pipeline. Its inputs include: 3D structured light reconstruction results, including the target surface normal vector, geometric center, and its valid markers; cross-modal motion states, including spatial position, velocity, surface normal vector, state covariance, motion mode, prediction confidence, and 3D prior valid markers; and the camera focal length dynamically acquired from the epipolar correction cache calibrated by the camera. Its outputs are spatial prior parameters, including the major axis, minor axis, and angle of the 2D elliptical shape prior, their valid markers, perspective eccentricity compensation, filtering scaling factor, their valid markers, prior confidence, and motion mode. These parameters are injected into the resource center via task synchronization for use in the circle fitting stage of 2D vision detection. In this embodiment, the target physical radius is 2.5 mm (corresponding to the measured radius of the marked surface), the camera focal length during operation is the calibrated and corrected lateral focal length, with a default value of approximately 1200 pixels, and the default working distance is 200 mm.
[0024] Step S1: Obtain prior information by acquiring the 3D structured light reconstruction results and cross-modal motion state of the industrial target to be inspected. The motion state includes at least the target's surface normal vector, spatial position, state covariance, motion mode, and prediction confidence. Also, obtain the camera focal length to determine the surface normal vector and working distance used for prior calculation. In the prior acquisition phase, a priority fusion of three-dimensional absolute observation and motion state is established: When the 3D structured light reconstruction result is valid and its geometric center is valid, the surface normal vector of the 3D structured light reconstruction result is used, and its geometric center is taken. The absolute value of the coordinates, converted to millimeters, is used as the working distance; When the 3D structured light reconstruction result is valid but the geometric center is invalid, and the 3D prior of the motion state is valid, the surface normal vector of the 3D structured light reconstruction result is used and the depth of the spatial position of the motion state is taken as the working distance to solve the transition in the early stage of accurate fitting; the 3D prior of the motion state specifically refers to the spatial position and surface normal vector used for prior calculation in the motion state. When no valid 3D reconstruction results are available, but the 3D prior and normal vectors of the motion state are valid, the system reverts to using the surface normal vectors of the motion state and takes their spatial depth as the working distance. The normal vectors are then normalized for subsequent calculations.
[0025] "Valid" means that the corresponding data has been confirmed to be usable for prior calculation after quality inspection by its generation module, and is represented by a valid flag output along with the data. Specifically, the valid flag of the 3D structured light reconstruction result and its geometric center is given by the 3D structured light reconstruction module: it is set to valid when the structured light point cloud of the target surface is successfully fitted, the fitting residual is lower than the preset residual threshold, and the number of valid points is not less than the preset number of points threshold; otherwise, it is set to invalid. Step S2: Shape Prior. Calculate the projection radius based on the camera focal length, the target's physical radius, and the working distance. Determine the major axis of the two-dimensional elliptical shape prior using the projection radius. Determine the minor axis by compressing the projection radius using the cosine of the angle between the surface normal vector and the camera optical axis. Determine the ellipse angle using the projection direction of the surface normal vector on the image plane to obtain the two-dimensional elliptical shape prior. In the elliptical shape prior generation stage, the calculation of the two-dimensional elliptical shape prior is as follows: the projection radius is the product of the camera focal length and the physical radius, divided by the working distance; the major axis is twice the projection radius; the minor axis is the product of the major axis and the absolute value of the tilt angle cosine; the tilt angle cosine is the absolute value of the component of the surface normal vector in the optical axis direction and clamped between the lower limit and one to avoid singularity; the elliptical angle is the arctangent of the two components of the surface normal vector on the image plane.
[0026] like Figure 3 As shown, in this embodiment, a physical radius is The disk, in the camera coordinate system, has a unit normal vector of... Working distance is The focal length is After perspective projection, it appears as an ellipse on the image plane. The system takes the absolute value of the component of the normal vector along the optical axis as the cosine of the tilt angle. clamp located To avoid singularities, the calculation is as follows: This yields a priori two-dimensional elliptical shape that matches the current three-dimensional pose; when facing the camera directly, the minor axis equals the major axis, and the more tilted the surface, the more pronounced the compression of the minor axis relative to the major axis.
[0027] Step S3: Eccentricity compensation. The eccentricity is calculated based on the tilt angle of the surface normal vector relative to the camera optical axis, the physical radius of the target, and the working distance. The eccentricity represents the offset of the visual center of the ellipse obtained by perspective projection of the circular mark surface of the target relative to its three-dimensional geometric center projection. The eccentricity direction is determined along the projection direction of the surface normal vector on the image plane to obtain the perspective eccentricity compensation amount. like Figure 4 As shown, when the normal vector is not perpendicular to the optical axis, the two endpoints of the disk along the tilt direction... , Due to the different distances to the camera, the midpoint of the projections of the two objects (i.e., the visual center obtained by ellipse fitting) shifts relative to the projection of the three-dimensional geometric center of the disk.
[0028] set up ,exist In the first-order approximation, the projections of the two endpoints are as follows: Elliptical visual center: Three-dimensional geometric center projection Therefore, the magnitude of the deviation is: The eccentricity direction is taken as the projection direction of the normal vector onto the image plane, that is, let... Then the perspective eccentricity compensation amount is: , When selecting candidate targets in the 2D vision detection module, the positioning reference point is moved from the center of the region of interest along... The offset aligns the selected candidate boxes with the 3D projection position, thus eliminating systematic centering biases that are imperceptible with 2D fitting alone. This stage depends on the camera focal length; the calculation is skipped if the focal length is not ready.
[0029] Step S4: Prior confidence: Calculate the prior confidence based on the convergence, motion pattern, target velocity and prediction confidence represented by the state covariance, and apply attenuation to the prior confidence when the preset confidence conditions are not met. As an important improvement of this invention, the prior confidence is calculated as follows: the covariance score of the convergence of the expression is obtained from the trace of the state covariance; the motion mode score is obtained by taking the motion mode in descending order of stationary, linear, oscillating, accelerating, and random; the velocity score is obtained from the target velocity, which decreases as the velocity increases; and the prediction score is obtained from the prediction confidence. The basic confidence is obtained by weighted summation of the above scores. The preset confidence conditions are that the motion mode is stationary or linear, the target velocity is less than the relative stationary threshold, the trace of the covariance is less than the convergence threshold, and the prediction score is greater than the prediction threshold. When the confidence conditions are met, the basic confidence is taken as the prior confidence; otherwise, the basic confidence is multiplied by a decay coefficient less than one as the prior confidence, thereby suppressing the influence of the prior when the target motion state is unstable or the three-dimensional estimation is unreliable.
[0030] like Figure 5 As shown, in this embodiment, the prior confidence level is determined by integrating multiple dimensions. First, the covariance score representing the convergence of the expression is obtained from the state covariance: the trace of the state covariance matrix (i.e., the sum of the variances of each position, representing the overall uncertainty of the position estimation) is taken, normalized according to a preset upper limit of covariance, clamped to the interval [0,1], and then complemented to obtain the covariance score, where clamp(x, a, b) means clamping x to the interval [a, b]; the smaller the uncertainty, the higher the score, and the score drops to 0 when the upper limit of covariance is reached. Second, the motion mode score is obtained from the motion mode, and the values are assigned in descending order of stationary, linear, oscillating, accelerating, and random, with the more stable the motion, the higher the score. Third, the velocity score is obtained from the target velocity: the velocity is normalized according to a preset velocity dead zone and velocity saturation upper limit, clamped, and then complemented; the score is full when the velocity does not exceed the dead zone, and drops to 0 when the saturation upper limit is reached. Fourth, the prediction score is obtained from the prediction confidence level. The scores are weighted and summed according to preset weights to obtain the basic confidence level. Then, a security gate is applied: the basic confidence level is taken as the prior confidence level only when the motion pattern is a stationary mode (stationary or linear in this embodiment), the speed is less than the relative stationary threshold, the trace of the covariance is less than the convergence threshold, and the predicted score is greater than the prediction threshold; otherwise, the basic confidence level is multiplied by a preset attenuation coefficient less than one and used as the prior confidence level. This ensures that when the state is unstable or the 3D estimation is unreliable, 2D detection is forced for highly fault-tolerant localization, avoiding erroneous priors from contaminating the recognition results.
[0031] Furthermore, it also includes an environment adaptation step: based on the uncertainty represented by the 3D reconstruction results or state covariance, a filter scaling factor is calculated via logarithmic mapping and clamped between a lower limit and an upper limit. The filter scaling factor is then input into the 2D vision detection module to adjust its filtering intensity, so that filtering is enhanced when the uncertainty is large and weakened when the uncertainty is small.
[0032] During the environment adaptation phase, the system maps the uncertainty of 3D reconstruction or motion estimation to a scaling factor of the 2D filter intensity. This embodiment uses the trace of the state covariance as a proxy for uncertainty. The filter scaling factor is obtained through linear mapping and clamping. When the uncertainty is large, the scaling factor is large and the filtering is enhanced to suppress noise; when the uncertainty is small, the scaling factor is small and the filtering is weakened to ensure reliable detection.
[0033] Step S5: Inject Enhancement. Input the prior of the two-dimensional elliptical shape, the perspective eccentricity compensation amount, and the prior confidence into the two-dimensional vision detection module to guide its candidate selection, positioning reference, and shape fitting, thereby enhancing the two-dimensional detection of non-rigid objects.
[0034] During the prior injection phase, the aforementioned two-dimensional elliptical shape prior, perspective eccentricity compensation, prior confidence, and filtering scaling factor are assembled into spatial prior parameter outputs. The overall effectiveness is determined by the validity of any one of the shape prior, eccentricity compensation, or filtering scaling. The downstream two-dimensional vision detection module constrains ellipse fitting based on the two-dimensional elliptical shape prior, corrects the positioning reference of candidate selection based on the perspective eccentricity compensation, determines whether to skip complete detection due to prior acceleration based on the prior confidence, and adjusts the filtering intensity based on the filtering scaling factor, thereby completing the cross-modal enhancement of two-dimensional detection by the three-dimensional spatial prior.
[0035] The downstream 2D vision detection module consumes the aforementioned spatial prior parameters as follows: First, during the ellipse fitting stage, the detection module uses the prior major axis, minor axis, and angle of the 2D ellipse shape as fitting constraints. It prioritizes candidates whose deviation from the prior shape is less than a preset shape threshold among the candidate contours, and uses the prior shape as an initial value or boundary constraint during the fitting solution, thereby improving the success rate and accuracy of fitting on inclined surfaces. Second, when selecting from multiple candidate targets, the detection module shifts the positioning reference point from the center of the region of interest along the direction of the perspective eccentricity compensation by a corresponding magnitude. It then uses the corrected positioning reference for candidate selection, aligning the 2D positioning result with the true 3D geometric center projection of the target. Third, when the prior confidence level is higher than a preset acceleration threshold, the detection module accelerates using the prior positioning result and skips the complete 2D detection process; otherwise, it performs the complete detection, reducing computational overhead when the state is reliable and maintaining high fault tolerance when the state is unreliable. Fourth, the detection module scales the filtering coefficients of its own filtering stage according to a filtering scaling factor, enhancing smoothness when uncertainty is high and maintaining response speed when uncertainty is low. The basic algorithms used for ellipse fitting and filtering are all existing technologies in the field. The improvement of this invention lies in constraining and correcting them with prior parameters in three-dimensional space.
[0036] Preferably, the camera focal length is dynamically obtained from the epipolar correction buffer calibrated by the camera, and the calculation of the two-dimensional elliptical shape prior and perspective eccentricity compensation is skipped when the camera focal length is not ready; the physical radius of the target is the measured radius of the preset target marking surface.
[0037] It should be noted that this invention is not limited to the specific values and forms mentioned above: the target physical radius, the lower limit of the cosine of the tilt angle, the weights and thresholds of each score, the attenuation coefficient, and the mapping range of the filter scaling can all be adjusted according to the scenario; the eccentricity can be approximated using the above first-order approximation, or it can be approximated using its denominator containing... A more precise form of the correction term; the uncertainty proxy can be replaced by other metrics such as the 3D reconstruction signal-to-noise ratio; this method can be used for circular marking surfaces and can also be extended to markings of other known physical shapes.
[0038] Example 2: like Figure 6 The illustrated system is a two-dimensional vision detection enhancement system for non-rigid objects driven by prior knowledge in three-dimensional space, comprising: The prior acquisition module is used to acquire the three-dimensional structured light reconstruction results and cross-modal motion state of the target. Based on the three-dimensional structured light reconstruction results and cross-modal motion state, the surface normal vector and working distance are determined according to the preset multi-level priority fusion strategy. The shape prior module is used to calculate the shape prior of a two-dimensional ellipse based on the projection radius of the surface normal vector onto the image plane; the two-dimensional ellipse shape prior includes the major axis, minor axis and tilt angle of the ellipse. The eccentricity compensation module is used to calculate the eccentricity based on the tilt angle of the surface normal vector relative to the camera optical axis, the target physical radius, and the working distance. It is used to characterize the offset of the visual center of the ellipse obtained by perspective projection of the circular mark surface of the target relative to its three-dimensional geometric center projection; and to determine the eccentricity direction along the projection direction of the surface normal vector on the image plane, thereby obtaining the perspective eccentricity compensation amount. The prior confidence module is used to calculate the prior confidence based on the three-dimensional prior of the cross-modal motion state; and to apply attenuation to the prior confidence when the preset confidence conditions are not met. The 2D vision detection module is used to input the prior of the 2D elliptical shape, the perspective eccentricity compensation amount, and the prior confidence into the 2D vision detection module to enhance the 2D detection of non-rigid objects.
[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A two-dimensional visual detection enhancement method for non-rigid objects driven by three-dimensional spatial priors, characterized in that, Includes the following steps: S1. Obtain the three-dimensional structured light reconstruction results and cross-modal motion state of the target; Based on the 3D structured light reconstruction results and cross-modal motion state, the surface normal vector and working distance are determined according to a preset multi-level priority fusion strategy. S2. Calculate the prior of the two-dimensional ellipse shape based on the projection radius of the surface normal vector on the image plane; the prior of the two-dimensional ellipse shape includes the major axis, minor axis and tilt angle of the ellipse. S3. Calculate the eccentricity based on the tilt angle of the surface normal vector relative to the camera optical axis, the target physical radius, and the working distance. This eccentricity is used to characterize the offset of the visual center of the ellipse obtained by perspective projection of the circular mark surface of the target relative to its three-dimensional geometric center projection. Determine the eccentricity direction along the projection direction of the surface normal vector on the image plane, and then obtain the perspective eccentricity compensation amount. S4. Calculate the prior confidence based on the three-dimensional prior of the cross-modal motion state; When the preset confidence conditions are not met, the prior confidence level is attenuated. S5. Input the prior of the two-dimensional elliptical shape, the perspective eccentricity compensation amount, and the prior confidence into the two-dimensional vision detection module to enhance the two-dimensional detection of non-rigid objects.
2. The method for enhancing two-dimensional visual detection of non-rigid objects driven by prior knowledge in three-dimensional space according to claim 1, characterized in that, The 3D structured light reconstruction results include the target surface normal vector, geometric center and its valid indicators; the cross-modal motion state includes spatial position, velocity, surface normal vector, state covariance, motion mode, prediction confidence and 3D prior valid indicators.
3. The two-dimensional visual detection enhancement method for non-rigid objects driven by prior knowledge in three-dimensional space according to claim 2, characterized in that, In S1, the multi-level priority fusion strategy includes: When the 3D structured light reconstruction result is valid and its geometric center is valid, the surface normal vector of the 3D structured light reconstruction result is used, and the absolute value of the geometric center coordinates of the 3D structured light reconstruction result is used as the working distance. When the 3D structured light reconstruction result is valid but the geometric center is invalid, and the 3D prior of the cross-modal motion state is valid, the surface normal vector of the 3D structured light reconstruction result is used and the depth of the spatial position of the cross-modal motion state is taken as the working distance. When there are no effective 3D structured light reconstruction results and both the 3D prior and surface normal vector of the cross-modal motion state are effective, the surface normal vector of the cross-modal motion state is used and the depth of its spatial position is taken as the working distance.
4. The two-dimensional visual detection enhancement method for non-rigid objects driven by three-dimensional spatial priors according to claim 1, characterized in that, In S2, the projection radius is the product of the camera focal length and the target physical radius, divided by the working distance; the major axis of the ellipse is twice the projection radius; the inclination cosine is the absolute value of the component of the surface normal vector in the optical axis direction and clamped between the lower limit and 1; the ellipse angle is the arctangent of the two components of the surface normal vector on the image plane; and the minor axis of the ellipse is the product of the major axis of the ellipse and the absolute value of the inclination cosine.
5. The two-dimensional visual detection enhancement method for non-rigid objects driven by prior knowledge in three-dimensional space according to claim 4, characterized in that, In S2, the camera focal length is dynamically obtained from the epipolar correction buffer calibrated by the camera. When the camera focal length is not ready, the calculation of the two-dimensional elliptical shape prior and perspective eccentricity compensation is skipped. The physical radius of the target is the measured radius of the preset target mark surface.
6. The method for enhancing two-dimensional visual detection of non-rigid objects driven by prior knowledge in three-dimensional space according to claim 1, characterized in that, The prior confidence calculation in S4 based on the three-dimensional prior of cross-modal motion states includes: The trace of the state covariance matrix is taken, normalized according to a preset upper limit of covariance, clamped to the interval [0,1], and then complemented to obtain the covariance score; the motion mode is assigned a value in descending order of static, linear, oscillating, accelerating, and random to obtain the motion mode score; the velocity is normalized according to a preset velocity dead zone and velocity saturation upper limit, clamped, and then complemented. The velocity score is full when the velocity does not exceed the dead zone and drops to 0 when the velocity reaches the saturation upper limit; the prediction score is obtained from the prediction confidence; the basic confidence is obtained by weighted summation of the covariance score, motion mode score, velocity score, and prediction score.
7. The two-dimensional visual detection enhancement method for non-rigid objects driven by prior knowledge in three-dimensional space according to claim 1, characterized in that, In S4, the preset confidence conditions are that the motion mode is static or linear, the target speed is less than the preset relative static threshold, the trace of the covariance is less than the preset convergence threshold, and the out-of-bounds prediction score is greater than the preset prediction threshold. When the confidence conditions are met, the basic confidence is used as the prior confidence; otherwise, the basic confidence is multiplied by the attenuation coefficient as the prior confidence.
8. The two-dimensional visual detection enhancement method for non-rigid objects driven by prior knowledge in three-dimensional space according to claim 1, characterized in that, S4 also includes: The uncertainty of the 3D structured light reconstruction result or cross-modal motion state is mapped to the filtering scaling factor of the 2D filtering intensity.
9. The two-dimensional visual detection enhancement method for non-rigid objects driven by prior knowledge in three-dimensional space according to claim 1, characterized in that, S5 also includes: The two-dimensional elliptical shape prior, perspective eccentricity compensation, prior confidence, and filtering scaling factor are assembled into spatial prior parameters. The two-dimensional vision detection module constrains ellipse fitting based on the two-dimensional elliptical shape prior, corrects the positioning reference of candidate selection based on the perspective eccentricity compensation, and decides whether to skip the complete detection by accelerating the prior based on the prior confidence. The filtering intensity is adjusted according to the filtering scaling factor, thereby completing the cross-modal enhancement of two-dimensional detection by three-dimensional spatial prior.
10. A two-dimensional visual detection enhancement system for non-rigid objects driven by three-dimensional spatial priors, characterized in that, include: The prior acquisition module is used to acquire the three-dimensional structured light reconstruction results and cross-modal motion state of the target. Based on the three-dimensional structured light reconstruction results and cross-modal motion state, the surface normal vector and working distance are determined according to the preset multi-level priority fusion strategy. The shape prior module is used to calculate the shape prior of a two-dimensional ellipse based on the projection radius of the surface normal vector onto the image plane; the two-dimensional ellipse shape prior includes the major axis, minor axis and tilt angle of the ellipse. The eccentricity compensation module is used to calculate the eccentricity based on the tilt angle of the surface normal vector relative to the camera optical axis, the target physical radius, and the working distance. It is used to characterize the offset of the visual center of the ellipse obtained by perspective projection of the circular mark surface of the target relative to its three-dimensional geometric center projection; and to determine the eccentricity direction along the projection direction of the surface normal vector on the image plane, thereby obtaining the perspective eccentricity compensation amount. The prior confidence module is used to calculate the prior confidence based on the three-dimensional prior of the cross-modal motion state; When the preset confidence conditions are not met, the prior confidence level is attenuated. The two-dimensional vision detection module is used to input the two-dimensional elliptical shape prior, perspective eccentricity compensation amount and prior confidence into the two-dimensional vision detection module to enhance the two-dimensional detection of non-rigid objects. The aforementioned three-dimensional space prior-driven two-dimensional visual detection enhancement system for non-rigid objects is used to implement the three-dimensional space prior-driven two-dimensional visual detection enhancement method and its steps as described in claim 1.
