Pose perception method and device for zero-shot adaptation and storage medium

Abstract

The application discloses a zero-sample adaptive pose perception method and device and a storage medium, and relates to the technical field of machine vision perception. The application first divides local geometric units based on a color image, a depth image and an object mask image, and establishes geometric, texture and depth three-dimensional alignment constraints. Then, the local geometric units are aligned with a rendering area to improve the reliability of pose estimation. The virtual rendering data and the real observation data are mapped to a unified feature space through a shared parameter feature extraction module to improve the accuracy of consistency error calculation. Finally, the pose increment is physically constrained and numerically normalized, and finally a stable, accurate and engineering usable 6D pose is output. In this way, the pose perception method realizes stable, reliable and engineering actual pose solving for unknown objects under local observation conditions such as occlusion or stacking in a zero-sample scene.

Description

Technical Field

[0001] This application relates to the field of machine vision perception technology, specifically to a zero-sample adaptive pose perception method, device, and storage medium. Background Technology

[0002] In the field of industrial automation, accurate six-degree-of-freedom (6D) pose perception of target objects by robots is a core prerequisite for achieving stable grasping, assembly, and various automated operations, directly determining the operational accuracy and scene adaptability of industrial robots. With the diversification of target object types and the accelerated update frequency in industrial production, and the fact that objects in actual working environments are often distributed in a disordered stacking and mutual occlusion manner, only a local visible area of ​​the object can be obtained from a single viewpoint, creating an urgent need for universal, zero-sample pose perception technology.

[0003] Traditional 6D object pose perception methods mostly rely on pre-training processes for specific objects. These methods generally assume that the target object is in a known state during the training phase and require the acquisition of an accurate 3D model of the object or a large number of labeled training samples in advance. Pose solutions are obtained through global pose regression, overall model matching, and other methods. However, in real-world industrial scenarios, when faced with unknown objects that did not appear during the training phase, or when only occluded / stacked objects in local areas can be observed, these methods struggle to achieve zero-shot adaptation. They are prone to problems such as multiple pose solutions, unstable estimation results, or even complete failure, severely limiting their generalization ability.

[0004] Meanwhile, depth sensors commonly used in industrial vision systems are limited by hardware performance, exhibiting objective deficiencies in resolution, noise levels, and data integrity. Different object materials and ambient lighting conditions also lead to significant differences in depth data quality, making it difficult to use depth information as a unified and reliable geometric input for global 6D pose solving. Furthermore, while some pose-aware methods attempt to combine virtual rendering with real-world observation for pose optimization, they lack effective constraint mechanisms for local observation scenarios and fail to address the interference from imaging differences between real-world observations and idealized rendering results. In addition, the lack of constraints on the physical laws of rigid body motion during pose updates easily leads to optimization divergence and results that do not meet the requirements of practical engineering applications, further restricting the applicability of pose-aware methods in complex industrial scenarios.

[0005] In summary, traditional 6D pose perception methods generally suffer from technical defects such as model dependence, lack of zero-sample adaptation capability, poor solution stability in local observation scenarios, and insufficient robustness in complex imaging environments. These defects cannot meet the pose perception needs of industrial robots for unknown objects in complex scenarios such as occlusion and stacking. Summary of the Invention

[0006] The purpose of this application is to provide a zero-sample adaptive pose sensing method, device and storage medium to solve the problem that traditional pose sensing methods are difficult to achieve stable, reliable and engineering-realistic pose solving for unknown objects under local observation conditions such as occlusion or stacking in zero-sample scenarios.

[0007] To achieve the above objectives, the first aspect of this application provides a zero-sample adaptive pose-aware method, comprising:

[0008] The sensor observation data of the target object is acquired, and the visible area of ​​the target object is divided into multiple local geometric units based on the sensor observation data. The sensor observation data includes a color image, a depth image, and an object mask image.

[0009] Based on the current pose assumption, virtual rendering data of the target object corresponding to the current viewpoint is generated, rendering local regions corresponding to each local geometric unit are extracted, and three-dimensional alignment constraints between the local geometric unit and the rendering local region are established. The three-dimensional alignment constraints include spatial geometric feature constraints, appearance texture feature constraints, and depth information constraints.

[0010] The virtual rendering data and the sensor observation data are mapped to a unified feature space through the feature extraction module with shared parameters. In the unified feature space, the consistency error is calculated and non-geometric interference is weakened.

[0011] Based on the consistency error, the pose increment is predicted, and physical constraint processing and numerical normalization are performed on the pose increment. The current pose assumption is iteratively updated until all local geometric units satisfy the three-dimensional alignment constraint, and the 6D pose of the target object is output.

[0012] A second aspect of this application provides a zero-sample adaptive pose sensing device, comprising:

[0013] The segmentation module is used to acquire sensor observation data of the target object and divide the visible area of ​​the target object into multiple local geometric units based on the sensor observation data. The sensor observation data includes a color image, a depth image, and an object mask image.

[0014] The alignment module is used to generate virtual rendering data of the target object corresponding to the current viewpoint based on the current pose assumption, extract the rendering local region corresponding to each local geometric unit, and establish three-dimensional alignment constraints between the local geometric unit and the rendering local region. The three-dimensional alignment constraints include spatial geometric feature constraints, appearance texture feature constraints, and depth information constraints.

[0015] The mapping module is used to map the virtual rendering data and the sensor observation data to a unified feature space through the feature extraction module with shared parameters, and to calculate the consistency error and weaken non-geometric interference in the unified feature space.

[0016] The output module is used to predict the pose increment based on the consistency error, perform physical constraint processing and numerical normalization on the pose increment, iteratively update the current pose assumption until all local geometric units satisfy the three-dimensional alignment constraint, and output the 6D pose of the target object.

[0017] A third aspect of this application provides a computer-readable storage medium storing a program that can be loaded by a processor and executed as described above for zero-sample adaptive pose awareness.

[0018] The beneficial effects of this application are:

[0019] This application first divides local geometric units based on color images, depth maps, and object mask images, and establishes 3D alignment constraints for geometry, texture, and depth. This achieves zero-sample pose perception for unknown objects without requiring complete object observation, pre-trained models, or a large number of labeled samples, thus enhancing its versatility. Then, by aligning local geometric units with the rendering region, it effectively overcomes the problems of multiple pose solutions and instability caused by occlusion and stacking in local observations, improving the reliability of pose estimation. A shared parameter feature extraction module maps virtual rendering data and real observation data to a unified feature space, weakening non-geometric interference such as lighting and noise, and improving the accuracy of consistency error calculation. Finally, physical constraints and numerical normalization are applied to the pose increment to ensure that the pose update conforms to the laws of rigid body motion, reducing optimization oscillations and divergences, ultimately outputting a stable, accurate, and engineering-usable 6D pose. In summary, through local geometric unit division, 3D alignment constraints, unified feature space mapping, and physically constrained pose iterative updates, the pose perception method achieves stable, reliable, and engineering-practical pose solutions for unknown objects under local observation conditions such as occlusion or stacking in zero-sample scenarios.

[0020] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a zero-sample adaptive pose perception method provided in an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of the structure of a zero-sample adaptive pose sensing device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified. Details are set forth in the following description for illustrative purposes. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid unnecessarily obscuring the description of this application. Therefore, this application is not intended to be limited to the embodiments shown, but rather to be consistent with the broadest scope of the principles and features disclosed herein.

[0025] Figure 1 This is a flowchart illustrating a zero-sample adaptive pose awareness method provided in an embodiment of this application. Figure 1 As shown, this pose sensing method may include steps 101-104, which will be described in detail below.

[0026] Step 101: Obtain sensor observation data of the target object, and divide the visible area of ​​the target object into multiple local geometric units based on the sensor observation data. The sensor observation data includes color image, depth image and object mask image.

[0027] A local geometric unit refers to an independent region formed by dividing the sensor's visible area of ​​a target object from the current viewpoint according to the inherent characteristics of its geometric structure. It is the basic unit for pose inference in this application embodiment. This application embodiment uses an industrial vision sensor to collect three types of sensor observation data of the target object: color image, depth image, and object mask image. Based on these three types of data, the visible area of ​​the target object is defined and the local geometric units are divided.

[0028] Specifically, industrial vision sensors are used to acquire color images, depth maps, and object mask images of the target object. The object mask image is used to distinguish the target object from the background region, the color image reflects the appearance texture distribution characteristics of the target object, and the depth map represents the spatial distance and geometric structure information of the target object. Based on the object mask image, the background region is removed, and the visible area of ​​the target object from the current viewpoint is locked. Combining the texture distribution of the color image and the spatial distance information of the depth image, the visible area is divided into multiple local geometric units. Each local geometric unit retains independent geometric features, and the spatial and geometric information correlation between adjacent units is maintained, laying the foundation for subsequent pose constraint verification.

[0029] Step 102: Based on the current pose assumption, generate virtual rendering data of the target object corresponding to the current viewpoint, extract the rendering local area corresponding to each local geometric unit, and establish three-dimensional alignment constraints between the local geometric unit and the rendering local area.

[0030] Virtual rendering data refers to the collective term for rendered images and depth projection data generated in a virtual environment based on the current pose assumption, combined with the 3D model of the target object and the intrinsic parameters of the current viewpoint camera, and matched with the real observation viewpoint. This application's embodiments, based on the current pose assumption, complete the generation of virtual rendering data, extraction of local rendering regions, and construct a three-dimensional alignment constraint composed of spatial geometric feature constraints, apparent texture feature constraints, and depth information constraints. The three-dimensional alignment constraint refers to the consistency verification constraint established between the real observed local geometric unit and the corresponding virtual rendered local region, which can include three dimensions: spatial geometric feature constraints, apparent texture feature constraints, and depth information constraints, used to evaluate the rationality of the pose assumption.

[0031] Specifically, the current pose assumption of the target object is initialized. Combined with the translation and rotation parameters of this pose assumption and the geometric information of the target object's 3D model, the camera intrinsics of the current viewpoint are matched to generate virtual rendering data consistent with the real observation viewpoint in the virtual environment. Based on the spatial position and boundary information of each local geometric unit, a correspondence between the real observation and the virtual rendering data is established. Rendering local regions corresponding one-to-one with each local geometric unit are extracted from the virtual rendering data. For each local geometric unit and its corresponding rendering local region, three sub-constraints for 3D alignment are constructed: comparing their contour morphology, topology, etc., to construct spatial geometric feature constraints; based on the texture distribution patterns of the color image, comparing their texture details and brightness variation trends to construct apparent texture feature constraints; and combining the spatial distance data of the depth map to verify the correlation of depth values ​​at corresponding positions, constructing depth information constraints. These three sub-constraints are then integrated to form a complete 3D alignment constraint.

[0032] Using local geometric units as the basic units for pose inference, the global pose of the target object is constrained by joint 3D alignment constraints of multiple local regions. This reduces the under-constraint problem caused by directly solving the global pose and effectively solves the problems of multiple pose solutions and unstable estimation caused by disordered stacking and mutual occlusion of objects in industrial scenes. Even when only a local visible area of ​​the object can be obtained from a single viewpoint, the pose can still be solved stably and reliably.

[0033] Step 103: Through the feature extraction module with shared parameters, the virtual rendering data and sensor observation data are mapped to a unified feature space. In the unified feature space, the consistency error is calculated and non-geometric interference is weakened.

[0034] By constructing a unified feature space through a feature extraction module with shared parameters, feature mapping between virtual and real data can be achieved. This space allows for the accurate calculation of consistency errors reflecting pose deviations, while simultaneously eliminating non-geometric interference. The unified feature space refers to the feature representation space constructed by the feature extraction module with shared parameters. This space enables virtual rendering data and sensor observation data to complete feature mapping according to consistent rules, achieving comparability between the two at a high-level feature level.

[0035] Specifically, a feature extraction module with shared parameters is configured to ensure that virtual rendering data and sensor observation data follow consistent feature mapping rules. The virtual rendering data and sensor observation data are input into this feature extraction module respectively, completing the mapping of the two types of data to a unified feature space. During the mapping process, core features such as geometric structure and contour distribution related to pose are preserved. In the unified feature space, the features of the virtual rendering data and sensor observation data are compared, and the feature differences between the two are calculated using a feature similarity measurement method, initially generating an error signal. The initial error signal is processed to weaken spurious errors caused by non-geometric factors such as changes in illumination intensity, sensor random noise, and background interference. The processed error signal is then quantized and integrated to generate a consistency error that truly reflects the deviation between the current pose assumption and the actual pose of the target object.

[0036] By constructing a unified feature space through a feature extraction module with shared parameters, virtual rendering data and sensor observation data are compared at a high-level feature level, rather than calculating differences directly at the pixel level. This effectively preserves the core geometric information related to pose, weakens the influence of non-geometric interference such as illumination changes, sensor noise, and background interference, reduces the possibility of mistaking differences in imaging conditions for pose deviations, provides a stable and reliable error signal for pose optimization, and improves the accuracy of pose deviation assessment.

[0037] Step 104: Based on the consistency error, predict the pose increment, perform physical constraint processing and numerical normalization on the pose increment, iteratively update the current pose assumption until all local geometric units satisfy the 3D alignment constraint, and output the 6D pose of the target object.

[0038] Pose increment refers to the local parameter update amount generated based on the consistency error, used to correct the current pose assumption. It includes rotation increment and translation increment, representing the direction and magnitude of pose correction. In this embodiment, the consistency error is used as a feedback signal to complete the prediction and processing of pose increment, and the pose assumption is gradually converged to the true value through iterative optimization, finally outputting the 6D pose of the target object.

[0039] Specifically, based on the generated consistency error, a pose increment is predicted to correct the current pose hypothesis. This pose increment includes rotation and translation increments. Physical constraints are applied to the pose increment to ensure that the rotation increment conforms to the physical laws of 3D rigid body motion and that the translation increment's variation is within a reasonable range, avoiding non-rigid body rotation or unreasonable large displacements. Numerical normalization is performed on the physically constrained pose increment to unify the numerical scales of rotation and translation parameters, balancing the update magnitudes of pose parameters in different dimensions and improving iteration stability. The processed pose increment is then superimposed onto the current pose hypothesis, completing a single pose iteration update and generating a new pose hypothesis. Based on the new pose hypothesis, steps 102 to 104 are executed iteratively to continuously verify whether all local geometric units satisfy the 3D alignment constraints. When the 3D alignment results of all local geometric units and the corresponding rendered local regions reach a stable matching state, the iteration terminates, and the final pose hypothesis is output as the 6D pose of the target object.

[0040] By adopting an incremental pose update method instead of absolute pose regression, combined with physical constraint processing and numerical normalization, the pose update results are guaranteed to always conform to the physical laws of three-dimensional rigid body motion, reducing problems such as numeric degradation and non-rigid body rotation. It also effectively prevents optimization divergence and oscillation caused by excessive parameter update amplitude, making the pose optimization process numerically stable and convergent. The output 6D pose can be directly applied to actual engineering operations such as industrial robot grasping and assembly, and has extremely high engineering practical value.

[0041] This application first divides local geometric units based on color images, depth maps, and object mask images, and establishes 3D alignment constraints for geometry, texture, and depth. This achieves zero-sample pose perception for unknown objects without requiring complete object observation, pre-trained models, or a large number of labeled samples, thus enhancing its versatility. Then, by aligning local geometric units with the rendering region, it effectively overcomes the problems of multiple pose solutions and instability caused by local observations such as occlusion and stacking, improving the reliability of pose estimation. A shared parameter feature extraction module maps virtual rendering data and real observation data to a unified feature space, weakening non-geometric interferences such as lighting and noise, and improving the accuracy of consistency error calculation. Finally, physical constraints and numerical normalization are applied to the pose increment to ensure that the pose update conforms to the laws of rigid body motion, reducing optimization oscillations and divergences, ultimately outputting a stable, accurate, and engineering-usable 6D pose. In summary, through local geometric unit division, 3D alignment constraints, unified feature space mapping, and physically constrained pose iterative updates, the pose perception method achieves stable, reliable, and engineering-practical pose solutions for unknown objects under local observation conditions such as occlusion or stacking in zero-sample scenarios.

[0042] In step 101, the background region is first removed based on the object mask image to lock the visible area edge of the target object. The object mask image is a binarized image used to accurately define the boundary between the target object and the background region in the observation image of an industrial vision sensor. It enables pixel-level segmentation of the target object, eliminating background interference to lock the effective observation area of ​​the target object. This clarifies the effective range of subsequent geometric region division and reduces the interference of the background region on the analysis of the object's geometric features.

[0043] Then, by combining the appearance texture distribution of the color image and the spatial distance information of the depth image, the geometric structure partitions within the visible area are initially identified. Geometric structure partitioning identifies regions with similar geometric features based on the appearance texture distribution and spatial distance information of the visible area of ​​the target object, such as continuous structural regions like convex, concave, flat surfaces, and contour edges. For example, texture abrupt changes and light-dark boundaries in the color image are used to identify the structural contours and texture partitions of the object's surface, while pixel value changes in the depth image are used to identify spatial height differences, convex and concave structures, and spatial distance distributions of the object's surface. Combining the analysis results of these two types of data, the geometric structure partitions within the visible area are initially identified, distinguishing continuous regions with different geometric features, such as flat surfaces, convex structures, concave structures, and contour edge zones, providing a geometric feature basis for subsequent preliminary partitioning.

[0044] Based on the spatial distribution characteristics and geometric structural partitioning of the visible area, the visible area is divided according to the correlation and integrity of the geometric structure, resulting in preliminary partitions. Preliminary partitions are large-scale regions formed by dividing the visible area of ​​the target object according to the correlation and integrity of its geometric structure. Each region contains a continuous and unbroken object surface structure, serving as the basis for the refinement and subdivision of local geometric units. Each preliminary partition contains a continuous object surface structure. During the partitioning process, it is ensured that each preliminary partition contains a continuous and unbroken object surface structure, and that the geometric features within the same preliminary partition are highly correlated, while the geometric features between different preliminary partitions are clearly distinguishable. This makes each preliminary partition a large-scale region with independent geometric attributes, reducing the breakage of object geometric structural information caused by improper partitioning.

[0045] The initial partitioning is further refined and subdivided. During this subdivision process, the size range of local geometric units is determined by combining sensor observation accuracy with the resolution of the depth map, ensuring that the size of each local geometric unit is adapted to the sensor observation accuracy. Sensor observation accuracy refers to the spatial resolution and detail capture capability of industrial vision sensors for target objects, and it is the core basis for determining the size of local geometric units, ensuring that the geometric features within the unit can be effectively identified and verified by the sensor. In this way, the size is neither too small, causing the sensor to be unable to capture effective geometric features, nor too large, causing the geometric features within the unit to be too complex to be independently verified.

[0046] After splitting, each local geometric unit contains independent and verifiable geometric features, while preserving the spatial positional relationship between adjacent local geometric units, as well as the continuity of texture and depth information. This ensures that when multiple local geometric units are jointly used to construct pose constraints, the overall geometric features of the visible area of ​​the target object can be restored, reducing the loss of geometric information caused by unit splitting.

[0047] This method employs a hierarchical partitioning approach, initially dividing the object into regions based on geometric structural correlation and integrity to ensure the continuous geometric structure of the object's surface remains intact. Then, it refines the region into local geometric units based on sensor precision, ensuring each unit possesses independently verifiable geometric features. This achieves a balance between geometric structural integrity and unit-level independent verifiability. During the refinement process, the spatial positional relationships and the continuity of texture and depth information between adjacent local geometric units are preserved. This ensures that multiple local geometric units are not isolated but can jointly reconstruct the overall geometric features of the visible area of ​​the target object. This allows for subsequent constraint of the target object's global pose through the joint constraints of multiple local geometric units, effectively solving the problem of under-constrained pose under local observation conditions and laying a core foundation for stable pose inference in zero-shot scenarios. The region partitioning is based solely on the geometric features of the target object's visible area and sensor observation data, without relying on prior category information, pre-trained models, or manually labeled information. This meets the core requirements of zero-shot pose perception and is adaptable to the geometric region partitioning of various unknown objects, enhancing the method's versatility and generalization ability.

[0048] In zero-shot general object pose perception scenarios, systems typically cannot obtain prior training samples for the target object when performing pose estimation, nor can they rely on the appearance or geometric features of specific object categories for supervised learning. Furthermore, for unknown or unmodeled objects, traditional pose estimation methods that rely on labeled data or category-specific network parameters are difficult to apply directly.

[0049] Under the aforementioned conditions, the pose error of the target object lacks clear and reliable supervision signals, making it difficult to judge the merits of the current pose assumption during the pose optimization process. This can easily lead to slow convergence, unstable results, or failure to converge. Therefore, how to construct an effective feedback signal for the pose optimization process in the absence of manual annotation and object priors is another key technical challenge in achieving zero-shot pose perception.

[0050] To address the aforementioned issues, step 102 proposes an iterative optimization strategy based on rendering-observation consistency. This strategy generates virtual rendering results corresponding to real observations under the current pose assumptions, thereby constructing a self-supervised optimization mechanism that does not require manual annotation.

[0051] In step 102, the translation and rotation parameters of the current pose assumption and the geometric information of the target object's 3D model are first combined to match the camera intrinsic parameters of the current viewpoint, generating a virtual rendered image and depth projection data that include the object's geometric structure and surface texture. The virtual rendered image and depth projection data together constitute the virtual rendered data. The current pose assumption is an initial estimate of the target object's 6D pose, which may include translation parameters along the X / Y / Z axes and rotation parameters around the X / Y / Z axes in 3D space. It is the fundamental parameter for generating virtual rendered data and the starting state for pose iteration optimization. The camera intrinsic parameters are inherent parameters of the industrial vision sensor, representing the geometric projection relationship of the camera imaging. They are the core parameters for achieving matching between the real observation viewpoint and the virtual rendered viewpoint, and ensuring spatial consistency between the virtual rendered data and the real observation data.

[0052] Specifically, based on the assumed current pose of the target object, its 6D pose information, including translation and rotation parameters, is extracted. Combined with the geometric information of the target object's 3D model, the camera intrinsic parameters of the industrial vision sensor's current viewpoint are precisely matched. Following the imaging projection rules of real observation, a rendering operation is performed in a virtual environment to generate a virtual rendered image and depth projection data that are completely consistent with the real observation viewpoint. The virtual rendered image restores the appearance texture and geometric contours of the target object under the assumed current pose, while the depth projection data restores the spatial distance distribution of the target object under the assumed current pose. Together, they constitute the virtual rendering data, achieving a virtual simulation of the real observation scene.

[0053] Based on the spatial coordinate range and partition boundaries of local geometric units, a coordinate mapping relationship between real observation and virtual rendering data is established to locate the corresponding region of each local geometric unit in the virtual rendering image. This region retains texture and depth information related to 3D alignment constraints and serves as a virtual comparison object for verifying the rationality of pose assumptions. For example, based on the spatial coordinate range and physical partition boundaries of each partitioned local geometric unit, combined with the projection mapping rules of camera intrinsic parameters, a coordinate transformation relationship between the real observation space and the virtual rendering space is established. This further derives the precise coordinate mapping relationship between each local geometric unit in real observation and the virtual rendering data. Through this relationship, the precise spatial location of the real local geometric unit in the virtual rendering image can be achieved.

[0054] Then, the corresponding local rendering regions are extracted according to the coordinate mapping relationship, so that the spatial range and geometric structure of each local rendering region correspond one-to-one with the local geometric units in the actual observation, while retaining the texture information and depth information associated with the 3D alignment constraints. The 3D alignment constraints are a set of consistency verification constraints constructed from three dimensions—spatial geometry, apparent texture, and depth information—for local geometric units and their corresponding local rendering regions. They are the core basis for evaluating the deviation of the current pose assumption and include three sub-constraints: spatial geometric feature constraints, apparent texture feature constraints, and depth information constraints.

[0055] Based on the established coordinate mapping relationship, the rendering local area corresponding to each local geometric unit is extracted one by one from the virtual rendering data. During the extraction process, the principle of one-to-one correspondence between spatial range and geometric structure is strictly followed to ensure that the spatial scale and geometric contour of each rendering local area are highly matched with the corresponding local geometric unit in real observation. At the same time, the texture information and depth information related to the 3D alignment constraints in the rendering local area are completely preserved, providing complete virtual feature data for subsequent multi-dimensional consistency verification.

[0056] Spatial geometric feature constraints are constructed by comparing the contour morphology, topology, and spatial distribution relationship between local geometric units and their corresponding rendered local regions. These constraints represent the degree of matching between the two in terms of spatial geometry. For example, a consistency comparison of spatial geometric features is performed on each local geometric unit and its corresponding rendered local region. Core geometric features such as contour morphology, topology, and spatial distribution relationship are extracted, and the consistency of the two types of regions in key geometric features is verified using a feature matching algorithm. Based on the verification results, spatial geometric feature constraints are constructed. These constraints are used to quantify the matching deviation between the local geometric unit and the rendered local region in terms of spatial geometry.

[0057] Based on the texture distribution patterns in the color image, the texture details and brightness variation trends of local geometric units and rendered local areas are compared, and the interference caused by lighting differences is weakened to construct apparent texture feature constraints. Apparent texture feature constraints are texture consistency constraints constructed based on the texture distribution patterns of the color image, comparing the texture details and brightness variation trends of local geometric units and rendered local areas while weakening lighting interference. These constraints characterize the degree of matching between the two in terms of appearance texture. For example, based on the texture distribution patterns of a real-world color image, apparent texture features such as texture details and brightness variation trends are extracted from local geometric units and corresponding rendered local areas. During feature comparison, feature normalization is used to weaken the interference caused by changes in ambient light intensity and lighting angle differences, retaining only the core consistency verification results of texture layout and texture details. Based on these results, apparent texture feature constraints are constructed, which are used to quantify the matching deviation between the two in terms of appearance texture.

[0058] By combining spatial distance data from the depth map, the correlation between the depth values ​​of local geometric units and rendered local regions at corresponding locations is verified, thus constructing depth information constraints. Depth information constraints are depth consistency constraints constructed after verifying the correlation between the depth values ​​of local geometric units and rendered local regions at corresponding locations using spatial distance data from the depth map. They characterize the degree of matching between the two in terms of spatial distance distribution. For example, by combining spatial distance data from real-world depth maps, the depth values ​​of local geometric units and corresponding rendered local regions at each corresponding spatial location are extracted. Correlation analysis is used to verify the correlation between the depth values ​​at corresponding locations, checking the degree of matching between the two types of regions in terms of spatial distance distribution. Based on the verification results, depth information constraints are constructed, which are used to quantify the matching deviation between the two in spatial depth.

[0059] By integrating spatial geometric feature constraints, appearance texture feature constraints, and depth information constraints, a three-dimensional alignment constraint is formed, which ensures that local geometric units and local rendering regions meet the consistency requirements in the three dimensions of geometry, texture, and depth. The three-dimensional alignment constraints of multiple local geometric units together constitute the overall verification basis for the current pose assumption.

[0060] This step, serving as a consistency verification step between rendering and observation in the zero-sample adaptive pose perception method, addresses issues such as mismatch between virtual and real data, single constraint dimension, and susceptibility to lighting interference in traditional pose perception methods through virtual rendering with viewpoint matching, accurate regional coordinate mapping, and the construction of multi-dimensional alignment constraints. It provides a reliable verification basis for subsequent pose deviation calculation.

[0061] In pose optimization based on render-observation consistency, the system needs to align and compare the virtual rendering results with the real sensor observation data. However, there are significant differences between the real observation images and the rendered images in terms of data generation mechanisms, specifically in aspects such as lighting conditions, texture detail representation, and noise distribution characteristics.

[0062] Real-world observed images are typically affected by factors such as changes in ambient lighting, sensor noise, imaging blur, and background interference, while rendered images are generated from idealized imaging models, resulting in relatively regular appearance characteristics and less noise. When aligning or calculating differences between these two types of images directly at the pixel level, it is easy to mistake appearance changes caused by differences in imaging conditions for pose errors, leading to instability in the optimization process, reduced convergence speed, and even incorrect pose update directions.

[0063] Therefore, how to reduce the interference introduced by differences in imaging conditions while ensuring sufficient preservation of pose-related geometric information is a key technical challenge in achieving stable rendering-observation consistency optimization. To address this issue, step 103 proposes a difference-sensitive modeling method in a shared feature space. By employing a unified feature extraction and representation mechanism for both the real observation image and the rendered image, the two can be aligned and compared within the same feature space.

[0064] In step 103, a shared-parameter encoder-decoder feature extraction module is first configured to ensure that the virtual rendering data and sensor observation data use consistent feature mapping rules. The shared-parameter encoder-decoder feature extraction module employs an encoder-decoder network structure, and all network parameters are globally shared, enabling different input data to follow completely consistent feature mapping rules, achieving the same feature extraction rules for virtual and real data. Globally sharing the configuration of all network parameters in the module ensures that subsequent virtual rendering data and sensor observation data follow completely consistent feature mapping and extraction rules when input into the module. This eliminates comparison biases caused by different feature extraction rules between the two types of data from the bottom layer, laying the foundation for the comparability of their features.

[0065] The virtual rendered image from the aggregated virtual rendering data and the color image from the sensor observation data are then input into the feature extraction module to extract high-level features containing geometric contours, spatial layout, and texture distribution. High-level features are abstract features extracted from the virtual rendered image and color image, containing the geometric contours, spatial layout, and texture distribution of the target object. They are a higher-order representation of the pixel information of the original image and better reflect the essential visual characteristics of the object. The virtual rendered image from the virtual rendering data and the color image from the sensor observation data are used as two independent inputs, respectively, and fed into the pre-configured shared parameter feature extraction module. The module's encoding layer reduces and abstracts the original image information, while the decoding layer restores and optimizes the abstract features. Finally, high-level features containing the geometric contours, spatial layout, and texture distribution of the target object are extracted from the two inputs. These features discard redundant pixel information from the original image and retain the core visual features.

[0066] Next, the depth projection data from the virtual rendering data and the depth map from the sensor observation data are simultaneously input into the feature extraction module to extract spatial distance correlation features. Spatial distance correlation features are features extracted from the depth projection data and depth map that characterize the relative spatial distances and positional relationships of different parts of the target object, reflecting the object's three-dimensional geometric spatial distribution characteristics. By simultaneously inputting the depth projection data from the virtual rendering data and the depth map from the sensor observation data into this shared parameter feature extraction module, based on the module's shared mapping rules and targeting the spatial distance characteristics of depth-type data, spatial distance correlation features that characterize the relative spatial distances and positional relationships of different parts of the target object are extracted, reflecting the core distribution information of the object's three-dimensional geometric space.

[0067] Then, a feature fusion mechanism is used to integrate high-level features and spatial distance correlation features into a unified-dimensional feature vector, completing the mapping of virtual rendering data and sensor observation data to a unified feature space. The feature fusion mechanism is an algorithm that integrates features of different types and dimensions to generate a unified-dimensional feature vector, achieving effective fusion and complementarity of multimodal features. The unified feature space is constructed by a shared parameter feature extraction module, ensuring that virtual rendering data and sensor observation data, after feature extraction and fusion, are mapped to the same feature representation space, guaranteeing direct comparability of the features of the two types of data.

[0068] Feature fusion mechanisms such as feature concatenation and weighted fusion are employed to integrate extracted high-level features with extracted spatial distance correlation features, eliminating dimensional differences between different features and generating a feature vector of uniform dimension. Through this fusion operation, virtual rendering data and sensor observation data are mapped to the same unified feature space, enabling the two types of data to form a one-to-one correspondence of feature representations at the high-level feature level, thus achieving direct comparison between the two.

[0069] Further focusing on pose-related features in a unified feature space, a feature weight allocation mechanism is constructed to enhance the verification weights of geometric contours, spatial layout, and depth-related features. This mechanism assigns different verification weights to various features in the unified feature space based on their relevance to the pose, strengthening the role of core pose-related features and reducing interference from irrelevant features. Specifically, in the unified feature space, all features are categorized and filtered according to their relevance to the target object's pose, focusing on core pose-related features such as geometric contours, spatial layout, and depth-related features. A feature weight allocation mechanism is then constructed for these filtered features, assigning higher verification weights to core pose-related features and reducing the weights of features unrelated to pose, such as texture details and pixel noise, thus strengthening the dominant role of core features in error calculation.

[0070] Based on the assigned feature vectors, a vector similarity metric is used to calculate the degree of difference between the corresponding feature vectors of the virtual rendering data and the sensor observation data, generating an initial error signal. Vector similarity metric algorithms can include cosine similarity or Euclidean distance, etc. The initial error signal is the original difference signal between the corresponding feature vectors of the virtual rendering data and the sensor observation data, calculated using vector similarity metric in a unified feature space. It includes feature differences caused by pose deviations and non-geometric disturbances.

[0071] For the initial error signal, the target error signal is obtained by filtering out the feature differences corresponding to non-geometric interferences. Non-geometric interferences are feature differences caused by factors unrelated to the target object's pose, such as changes in illumination intensity, sensor random noise, and background interference. These can interfere with the judgment of the true pose deviation, including weakening the effects of changes in illumination intensity, sensor random noise, and background interference. The target error signal is obtained by filtering the initial error signal to remove feature differences corresponding to non-geometric interferences, thus more closely resembling the true pose deviation features. For example, using filtering algorithms for interference removal, feature differences corresponding to non-geometric factors such as changes in illumination intensity, sensor random noise, and background interference are accurately identified and removed, retaining only the effective feature differences caused by the deviation between the current pose assumption and the true pose. The target error signal obtained after filtering eliminates the influence of non-geometric interferences on error judgment, thus more closely resembling the true pose deviation.

[0072] Finally, the target error signal is quantized and integrated to generate a consistency error that accurately reflects the pose deviation. The consistency error, generated after quantizing and integrating the target error signal, is a quantitative indicator that accurately reflects the degree of deviation between the current pose assumption and the actual pose of the target object; it is the core basis for pose increment prediction. The filtered target error signal is quantized and integrated in multiple dimensions, normalizing and weighting the target errors from different local geometric units and different feature dimensions to generate a comprehensive consistency error that accurately reflects the degree of deviation between the current pose assumption and the actual pose of the target object. The consistency error is a scalar or standardized vector and is the direct and core basis for pose increment prediction in subsequent steps.

[0073] By employing a design that incorporates shared parameter feature extraction, multimodal feature fusion, weight allocation to focus on the core features, and filtering to remove interference, this approach effectively addresses issues in existing pose perception methods, such as the incomparability of virtual and real data features, the significant impact of non-geometric interference on error calculation, and the lack of prominence of core features. This provides accurate and reliable deviation data for subsequent pose optimization.

[0074] In this embodiment, both the real-world observed image and the virtual rendered image are input into the same feature extraction module. The feature extraction module uses shared parameter configurations to ensure that the two types of input data follow consistent mapping rules during feature extraction. Through this shared feature extraction mechanism, the real-world observed image and the rendered image are mapped to a unified feature representation space, thereby achieving comparability at a high-level feature level. In this unified feature space, key feature information related to the geometric structure, contour distribution, and spatial layout of the target object is retained, while sensitivity to non-geometric factors such as illumination changes, texture details, and random noise is reduced. Based on this feature representation, the system performs alignment, difference calculation, and consistency evaluation between the rendered result and the real-world observation in the feature space rather than the pixel space, thus providing a more stable and reliable error signal for subsequent pose optimization.

[0075] In zero-sample general object 6D pose perception, the pose optimization process faces dual challenges in terms of numerical stability and physical executability due to the unknown target object category, complex observation conditions, and difficulty in accurately controlling the initial pose error. On the one hand, directly predicting or regressing the absolute pose parameters of the target object can easily lead to optimization divergence or convergence to a non-optimal solution due to excessively large update amplitudes. On the other hand, even if the optimization process converges numerically, its prediction results may still violate the physical constraints of rigid body motion, making it difficult to directly use for subsequent robot execution or control tasks. Therefore, ensuring that the pose update results always meet the requirements of physical rationality and numerical stability while guaranteeing the asymptotic stability of the pose optimization process is a crucial technical problem in realizing an engineering-grade pose perception system.

[0076] In step 104, an optimized gradient can be generated based on the consistency error. The direction and magnitude of this optimized gradient are used to predict the pose increment for pose correction. The optimized gradient is a gradient vector generated based on the consistency error using a gradient propagation algorithm. It characterizes the direction and magnitude of pose parameter correction and reflects the changing trend of the consistency error with the pose parameters, serving as the core basis for pose increment prediction. Using the consistency error, which accurately reflects the pose deviation, as a loss signal, the partial derivatives of this consistency error with respect to each translation and rotation parameter in the current pose assumption are calculated using a backpropagation algorithm to generate the optimized gradient. The direction of this optimized gradient represents the optimal correction direction for the pose parameters, and the magnitude of the gradient characterizes the correction strength of each parameter, providing a quantitative basis for accurate prediction of subsequent pose increments. Based on the generated optimization gradient, and combined with the step size configuration of the gradient descent optimization algorithm, the pose increment used to correct the current pose assumption is calculated along the correction direction indicated by the gradient. The pose increment includes two parts: rotation increment and translation increment. The rotation increment corresponds to the correction value of the rotation parameter around the X / Y / Z axis in three-dimensional space, and the translation increment corresponds to the correction value of the translation parameter along the X / Y / Z axis in three-dimensional space. Together, they define the specific correction scheme for the current pose assumption.

[0077] Then, rigid body motion compliance constraints are applied to the rotation increment in the pose increment to ensure that the rotation parameters conform to the physical laws of 3D rigid body motion. Rigid body motion compliance constraints are constraint rules applied to the rotation increment that conform to the physical laws of 3D rigid body motion. These constraints ensure that the rotation parameters satisfy orthogonality and determinant constraints, reducing the occurrence of non-rigid body rotations or numerically degenerate pose results. For example, through an orthogonalization correction algorithm, the rotation matrix corresponding to the rotation increment is ensured to satisfy the orthogonality constraint of 3D rigid body motion and the constraint that the determinant is 1. This eliminates numerical components in the rotation parameters that do not conform to the physical laws of rigid body motion, reducing problems such as non-rigid body rotations and spatial pose distortions in the updated pose caused by rotation increment distortion, and ensuring the physical executability of the pose update.

[0078] For the translation increments within the rotation and pose increments after compliance constraints, a uniform-scale numerical normalization is performed to balance the update magnitudes of pose parameters in different dimensions, generating the pose increment to be updated. Numerical normalization is a uniform-scale processing operation performed on the rotation and translation increments after rigid body motion compliance constraints. It is used to balance the update magnitudes of pose parameters in different dimensions and improve the numerical stability of iterative optimization. For example, based on the physical meaning and numerical scale of the pose parameters, the rotation and translation increments are mapped to the same numerical range, balancing the update magnitudes of pose parameters in different dimensions and reducing optimization oscillations, slow convergence, or even divergence caused by excessively large update magnitudes of single parameters. After processing, a pose increment to be updated that combines physical compliance and numerical stability is generated. The pose increment to be updated is the final pose increment that, after rigid body motion compliance constraints and numerical normalization, can be directly used for pose correction, possessing both physical compliance and numerical stability.

[0079] The pose increment to be updated is superimposed on the current pose hypothesis to complete a single pose iteration update and generate a new pose hypothesis. For example, translation parameters are directly superimposed with translation increments, and rotation parameters are fused with rotation increments through matrix multiplication to complete a single pose iteration update and generate a new pose hypothesis, which serves as the initial pose for the next round of optimization.

[0080] Based on the new pose assumption, the process of virtual rendering data generation, local rendering region extraction, and 3D alignment constraint establishment is executed iteratively to verify the 3D alignment constraint satisfaction status of multiple local geometric units. For example, the process of virtual rendering data generation, local rendering region extraction, 3D alignment constraint establishment, unified feature space mapping and consistency error calculation, pose increment prediction and constraint normalization is completed sequentially, continuously verifying the 3D alignment constraint satisfaction status of multiple local geometric units and their corresponding local rendering regions.

[0081] When the 3D alignment results of multiple local geometric units reach a stable matching state, the iteration terminates and the final 6D pose is output. The stable matching state is the state in which the 3D alignment constraints of multiple local geometric units and the corresponding rendered local regions meet the consistency requirements, and the pose update amount of continuous iterations tends to converge without significant fluctuations. This is the core criterion for terminating the iteration.

[0082] In the final iterative optimization and result output stage, through the design of gradient prediction increment, rigid body constraint to ensure compliance, normalization to stabilize amplitude, and iterative convergence to promote convergence, the problems of easy divergence in absolute pose regression, pose update violating physical laws, poor stability of iterative optimization, and inaccurate convergence judgment under local observation in existing pose perception methods are specifically solved, thus achieving stable and accurate solution of 6D pose of unknown objects.

[0083] Figure 2This is a schematic diagram of the structure of a zero-sample adaptive pose sensing device 200 provided in an embodiment of this application. Figure 2 As shown, the zero-sample adaptive pose sensing device 200 may include a segmentation module 201, an alignment module 202, a mapping module 203, and an output module 204.

[0084] The segmentation module 201 is used to acquire sensor observation data of the target object and divide the visible area of ​​the target object into multiple local geometric units based on the sensor observation data. The sensor observation data includes color images, depth images, and object mask images.

[0085] The alignment module 202 is used to generate virtual rendering data of the target object corresponding to the current viewpoint based on the current pose assumption, extract the rendering local area corresponding to each local geometric unit, and establish three-dimensional alignment constraints between the local geometric unit and the rendering local area. The three-dimensional alignment constraints include spatial geometric feature constraints, appearance texture feature constraints and depth information constraints.

[0086] The mapping module 203 is used to map virtual rendering data and sensor observation data to a unified feature space through the feature extraction module with shared parameters, calculate consistency error and weaken non-geometric interference in the unified feature space.

[0087] The output module 204 is used to predict the pose increment based on the consistency error, perform physical constraint processing and numerical normalization on the pose increment, iteratively update the current pose assumption until all local geometric units satisfy the three-dimensional alignment constraint, and output the 6D pose of the target object.

[0088] The partitioning module 201, alignment module 202, mapping module 203, and output module 204 can be used to execute steps 101-104 in the embodiments of the zero-sample adaptive pose perception method described above. For the specific implementation of these modules and more details, please refer to the corresponding method section, which will not be elaborated here.

[0089] This application also provides a computer-readable storage medium storing a program that can be loaded by a processor and executed by any of the zero-sample adaptive pose awareness methods in this application.

[0090] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0091] The above examples illustrate this application only to aid understanding and are not intended to limit its scope. Those skilled in the art to which this application pertains can make various simple deductions, modifications, or substitutions based on the ideas presented.

Claims

1. A zero-sample adaptive pose sensing method, characterized in that, include: The sensor observation data of the target object is acquired, and the visible area of ​​the target object is divided into multiple local geometric units based on the sensor observation data. The sensor observation data includes a color image, a depth image, and an object mask image. Based on the current pose assumption, virtual rendering data of the target object corresponding to the current viewpoint is generated. Rendering local regions corresponding to each local geometric unit are extracted. The contour shape, topology, and spatial distribution relationship between the local geometric unit and the corresponding rendering local region are compared to construct spatial geometric feature constraints. Based on the texture distribution pattern in the color image, the texture details and brightness variation trends of the local geometric unit and the rendering local region are compared to weaken the interference caused by lighting differences, thus constructing apparent texture feature constraints. Combining the spatial distance data in the depth image, the correlation of depth values ​​between the local geometric unit and the rendering local region at corresponding positions is verified, thus constructing depth information constraints. Integrating the spatial geometric feature constraints, the apparent texture feature constraints, and the depth information constraints, a three-dimensional alignment constraint is formed, ensuring that the local geometric unit and the rendering local region meet consistency requirements in geometry, texture, and depth dimensions. By using a feature extraction module with shared parameters, the virtual rendering data and the sensor observation data are mapped to a unified feature space. The pose-related features in the unified feature space are focused on, and a feature weight allocation mechanism is constructed to improve the verification weight of geometric contour, spatial layout and depth-related features. A vector similarity metric is used to calculate the degree of difference between the feature vectors corresponding to the virtual rendering data and the sensor observation data, and an initial error signal is generated. The target error signal is obtained by filtering the feature differences corresponding to non-geometric interference in the initial error signal. The non-geometric interference includes weakened light intensity changes, sensor random noise, and background interference. The target error is integrated to generate a consistent error that can truly reflect the pose deviation; Based on the consistency error, an optimized gradient is generated. According to the direction and magnitude of the optimized gradient, the pose increment used to correct the pose is predicted. Rigid body motion compliance constraints are applied to the rotation increment in the pose increment to ensure that the rotation parameters conform to the physical laws of three-dimensional rigid body motion. For the rotation increment after compliance constraints and the translation increment in the pose increment, a uniform-scale numerical normalization is performed to balance the update magnitude of pose parameters in different dimensions, generate the pose increment to be updated, iteratively update the current pose assumption until all local geometric units satisfy the three-dimensional alignment constraints, and output the 6D pose of the target object.

2. The zero-sample adaptive pose sensing method according to claim 1, characterized in that, The process of dividing the visible region of the target object into multiple local geometric units based on the sensor observation data includes: Based on the object mask image, the background area is removed, the visible area boundary of the target object is locked, and the geometric structure partitions within the visible area are initially identified by combining the appearance texture distribution of the color image and the spatial distance information of the depth image. Based on the spatial distribution characteristics and geometric structure partitioning of the visible area, the visible area is divided according to the correlation and integrity of the geometric structure to obtain preliminary partitions, each of which contains a continuous object surface structure. The initial partition is further subdivided, and the size range of the local geometric unit is determined by combining the sensor observation accuracy and referring to the resolution of the depth map, so that the size of each local geometric unit is adapted to the sensor observation accuracy and contains independent and verifiable geometric features. The spatial positional relationship between adjacent local geometric units, as well as the continuity of texture and depth information, are preserved.

3. The zero-sample adaptive pose sensing method according to claim 1, characterized in that, The process of generating virtual rendering data of the target object corresponding to the current viewpoint based on the current pose assumption, and extracting the rendering local region corresponding to each local geometric unit, includes: By combining the translation and rotation parameters of the current pose assumption and the geometric information of the three-dimensional model of the target object, and matching the camera intrinsic parameters of the current viewpoint, a virtual rendering image and depth projection data containing the object's geometric structure and surface texture are generated. The virtual rendering image and the depth projection data together constitute the virtual rendering data. Based on the spatial coordinate range and division boundary of the local geometric unit, a coordinate mapping relationship between real observation and virtual rendering data is established, and the corresponding region of each local geometric unit in the virtual rendering image is located. Extract the corresponding local rendering regions according to the coordinate mapping relationship, so that the spatial range and geometric structure of each local rendering region correspond one-to-one with the local geometric units in the actual observation, and retain the texture information and depth information associated with the three-dimensional alignment constraints.

4. The zero-sample adaptive pose sensing method according to claim 1, characterized in that, The feature extraction module using shared parameters maps the virtual rendering data and the sensor observation data to a unified feature space, including: Configure an encoding-decoding type feature extraction module with shared parameters to ensure that the virtual rendering data and the sensor observation data use the same feature mapping rules; The virtual rendering image, which is a collection of the virtual rendering data, and the color image from the sensor observation data are respectively input into the feature extraction module to extract high-level features including geometric contours, spatial layout, and texture distribution. The depth projection data in the virtual rendering data and the depth map in the sensor observation data are synchronously input into the feature extraction module to extract spatial distance correlation features; The high-level features and the spatial distance correlation features are integrated into a feature vector of a unified dimension through a feature fusion mechanism, thereby completing the mapping of the virtual rendering data and the sensor observation data to the unified feature space.

5. The zero-sample adaptive pose sensing method according to claim 1, characterized in that, The iterative update of the current pose assumption until all local geometric units satisfy the 3D alignment constraint, and the output of the 6D pose of the target object, includes: The pose increment to be updated is superimposed on the current pose hypothesis to complete a single pose iteration update and generate a new pose hypothesis; Based on the new pose assumption, the virtual rendering data generation, rendering local region extraction and 3D alignment constraint establishment process are executed cyclically to verify the 3D alignment constraint satisfaction status of multiple local geometric units. When the 3D alignment results of multiple local geometric units all reach a stable matching state, the iteration terminates and the final 6D pose is output.

6. A zero-sample adaptive pose sensing device, characterized in that, include: The segmentation module is used to acquire sensor observation data of the target object and divide the visible area of ​​the target object into multiple local geometric units based on the sensor observation data. The sensor observation data includes a color image, a depth image, and an object mask image. The alignment module is used to generate virtual rendering data of the target object corresponding to the current viewpoint based on the current pose assumption, extract the rendering local region corresponding to each local geometric unit, compare the contour shape, topology and spatial distribution relationship of the local geometric unit and the corresponding rendering local region, and construct spatial geometric feature constraints; based on the texture distribution pattern in the color image, compare the texture details and brightness change trends of the local geometric unit and the rendering local region, weaken the interference caused by lighting differences, and construct appearance texture feature constraints; combined with the spatial distance data in the depth image, verify the correlation of the depth values ​​of the local geometric unit and the rendering local region at corresponding positions, and construct depth information constraints; integrate the spatial geometric feature constraints, the appearance texture feature constraints and the depth information constraints to form a three-dimensional alignment constraint, so that the local geometric unit and the rendering local region meet the consistency requirements in the three dimensions of geometry, texture and depth. The mapping module is used to map the virtual rendering data and the sensor observation data to a unified feature space through the feature extraction module with shared parameters, focus on the pose-related features in the unified feature space, construct a feature weight allocation mechanism, and improve the verification weight of geometric contour, spatial layout and depth-related features. A vector similarity metric is used to calculate the degree of difference between the feature vectors corresponding to the virtual rendering data and the sensor observation data, and an initial error signal is generated. The target error signal is obtained by filtering the feature differences corresponding to non-geometric interference in the initial error signal. The non-geometric interference includes weakened light intensity changes, sensor random noise, and background interference. The target error is integrated to generate a consistent error that can truly reflect the pose deviation; The output module is used to generate an optimized gradient based on the consistency error, predict the pose increment for correcting the pose according to the direction and magnitude of the optimized gradient, perform rigid body motion compliance constraints on the rotation increment in the pose increment to ensure that the rotation parameters conform to the physical laws of three-dimensional rigid body motion, perform uniform-scale numerical normalization on the rotation increment after compliance constraints and the translation increment in the pose increment to balance the update magnitude of pose parameters in different dimensions, generate the pose increment to be updated, iteratively update the current pose assumption until all local geometric units satisfy the three-dimensional alignment constraints, and output the 6D pose of the target object.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that can be loaded by a processor and executed as described in any one of claims 1 to 5, the pose-aware method of zero-sample adaptation.

Images