Weak supervision accessibility positioning robot grabbing method based on self-adaption during testing

Abstract

The method comprises the following steps: initializing a teacher model and a student model which are consistent in structure and are used for online adaptive learning in a test stage based on an adaptive weak supervision affordability positioning robot grabbing method during test; utilizing the frozen teacher model to generate an affordability prediction map, and performing significant knowledge screening and normalization processing; performing feature normalization processing on a first person view angle image acquired by the robot online; in a set updating period, generating an affordability prediction map by using a student model, and constraining the uncertainty of the affordability prediction map based on the information entropy of the prediction map; based on the significant knowledge of the teacher model and the student model prediction result, constructing a joint target of knowledge migration loss and entropy constraint loss, and updating student model parameters; updating teacher model parameters according to the student model parameters; and restraining the grabbing detection network by using an affordability graph output by the student model, and generating a robot grabbing pose conforming to task semantics. The function positioning precision and grabbing stability of the robot to the unknown object can be improved.

Description

Technical Field

[0001] This invention relates to a localization robot grasping method, specifically a weakly supervised availability-based localization robot grasping method based on test-time adaptive approach, belonging to the field of intelligent robot visual perception, availability perception and grasping control technology. Background Technology

[0002] In recent years, significant progress has been made in embodied intelligence in robots, enabling them to overcome technological limitations and integrate into daily life. Most importantly, robots should be able to manipulate objects flexibly using limited observational data. However, an object may possess multiple inherent motion possibilities, posing a considerable challenge to current robotic systems. In contrast, humans excel at imitation and generalization, learning to use objects through limited external observations of their use. Therefore, endowing robots with this functional learning ability has become a crucial step and a popular research topic in realizing embodied manipulation in robots.

[0003] To address the high cost associated with large-scale, finely labeled data, weakly supervised functional localization methods have emerged. These methods rely solely on image-level labels to learn the functional regions of objects, eliminating the need for labeling every pixel or local area. This significantly reduces the labor intensity of data preparation and promotes the engineering application of the technology. Existing weakly supervised methods have achieved good performance in specific scenarios by mining the interaction features of first-person and third-person images and fusing multimodal information such as text descriptions, providing a lightweight perception solution for robot grasping tasks.

[0004] Existing weakly supervised functional localization methods, in practical applications of robot grasping tasks, are limited by data characteristics and model architecture defects, making it difficult to achieve stable adaptation in dynamic scenarios. On the one hand, real-world environmental data is scarce and dynamically distributed, and existing weakly supervised models cannot effectively utilize unlabeled data generated in real time during the testing phase to complete online parameter optimization. On the other hand, traditional models lack efficient knowledge transfer mechanisms, and the general functional knowledge acquired during the pre-training phase is difficult to adapt to dynamic testing scenarios. Their generalization ability is insufficient when faced with unseen objects or environmental shifts, directly leading to a decline in functional localization accuracy and affecting the robot's grasping success rate. Furthermore, the successful experience of test-time adaptive techniques in image classification cannot be directly transferred to weakly supervised functional localization tasks. Because existing test-time adaptive methods do not consider the spatial locality requirements of functional localization and only focus on the overall optimization of classification results, they cannot accurately adapt to the spatial distribution characteristics of object functional regions. Therefore, their application to functional localization only brings minor performance improvements and cannot fundamentally solve the model-data distribution shift problem during the testing phase. In addition, existing methods lack a selective filtering mechanism for teacher model knowledge, easily introducing noisy knowledge to interfere with student model optimization, and failing to effectively constrain the prediction space, further reducing the accuracy of functional localization. Summary of the Invention

[0005] To address the aforementioned technical shortcomings, the purpose of this invention is to provide a weakly supervised availability-based robot grasping method based on test-time adaptive adaptation. This method solves key problems in existing weakly supervised functional localization models, such as insufficient generalization ability, poor adaptability of test-time adaptive techniques, noise interference during knowledge transfer, and lack of prediction space constraints. By using online unlabeled data to adaptively optimize the model during the testing phase, this invention achieves accurate perception and stable prediction of the object's operating area, realizes efficient utilization of unlabeled data in dynamic scenarios, and improves the robot's functional localization accuracy and grasping stability for unknown objects.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A test-time adaptive weakly supervised availability localization robot grasping method includes the following steps: S1. Initialize the teacher-student model; Based on the pre-trained weakly supervised availability localization model, construct a teacher model and a student model with consistent structure and initial parameters, respectively. The teacher model is used to provide significant availability knowledge constraints to the student model during the adaptive process at test time, and the student model is used to perform online optimization on the test data and generate an availability prediction graph. S2. Obtain the normalized feature representation; After deploying the pre-trained weakly supervised availability localization model to the robot system, the robot continuously collects first-person perspective RGB images online to obtain test data without real labels. The test data is preprocessed, and then visual features are extracted from it. Feature normalization is performed on the extracted visual features to obtain the normalized feature representation, so as to reduce the distribution difference between the test data and the training data, which is used for subsequent joint inference and online adaptive optimization of the teacher model and the student model. S3. The teacher model predicts and filters significant availability knowledge. The normalized feature representation is simultaneously input into the teacher model and the student model. The teacher model parameters are kept frozen, and its forward inference process generates an original availability prediction map to characterize the correlation between each pixel region in the image and the semantics of the target operation. The original availability prediction map is processed by the significant availability filtering module to filter out prediction maps with low confidence, resulting in a filtered prediction map. Then, by performing max-min normalization on the filtered prediction map, high-quality teacher significant availability knowledge is obtained to guide the online learning of the student model. S4. The student model predicts and applies entropy constraints. Within a set update period, the normalized feature representation is input into the student model. The student model generates an original availability prediction map through forward inference. Its form is consistent with the teacher model output and is used to characterize the correlation between the position of each pixel in the image and the semantics of the target operation. Based on this, an information entropy constraint based on the normalized student availability prediction map is introduced. By minimizing the information entropy of the prediction map, the uncertainty of the student model's prediction distribution is limited, guiding it to generate an availability prediction map with a more concentrated and definite spatial distribution for subsequent joint optimization and grasping decision. S5. Joint loss optimization of the student model: Based on the significant availability knowledge output by the teacher model and the availability prediction graph generated by the student model under entropy constraints, a joint optimization objective consisting of knowledge transfer loss and entropy constraint loss is constructed. The student model parameters are updated online by minimizing the joint optimization objective. After the parameter update is completed, the updated student model is used to perform forward inference on the current test sample to generate the final availability prediction graph. S6. Exponential sliding average update of the teacher model: After the student model parameters are updated, the teacher model parameters are updated smoothly using an exponential sliding average strategy. Through the smooth update method, the effective availability representations obtained by the student model during the adaptive process during testing are gradually accumulated into the teacher model to ensure that the teacher model can continuously provide stable and reliable significant availability knowledge in subsequent test samples, which is used to constrain the online adaptive optimization process of the student model. S7. Generate grasping pose based on the final availability prediction graph; use the final availability prediction graph output by the student model after adaptive optimization during the test in S5 to constrain the grasping detection network and generate a robot grasping pose that conforms to the task semantics.

[0007] Preferably, the pre-trained weakly supervised availability localization model in step S1 of the present invention is trained based on source training data containing first-person view images and their corresponding image-level availability labels. Let the pre-trained weakly supervised availability localization model be... Its parameters are At the start of the testing phase, both the teacher and student models are initialized to this pre-trained weakly supervised availability localization model, i.e.: in, Represents the teacher model parameters at the initial moment; Represents the student model parameters at the initial moment; During subsequent adaptive testing, the teacher model parameters will remain frozen for inference, while the student model parameters will be optimized and updated based on online test data.

[0008] Preferably, step S2 of the present invention specifically involves: the robot continuously acquiring RGB images from a first-person perspective via a vision sensor to form a test data stream. ,in Indicates the first One test sample, Indicates the total number of test samples; These test data are not properly labeled, and their distribution may differ from that of the training data. For each test sample arriving online First, preprocessing is performed: the image is cropped and scaled to a fixed size (224×224 pixels), then input into a pre-trained visual encoder (DINO-ViT-S) to extract visual features. Next, feature normalization is performed on the extracted visual features, and the transformation formula is as follows: in, This represents a predefined feature normalization transformation operation; This represents the normalized feature representation, which is used for subsequent joint processing of the teacher and student models.

[0009] Preferably, step S3 of the present invention specifically comprises: Normalized feature representation Simultaneously input into both the teacher and student models, with the teacher model parameters frozen, its forward inference process generating the original availability prediction graph: in, This represents the forward reasoning process of the teacher model; Represents the original availability prediction map output by the teacher model; used to characterize the degree of correlation between each pixel region in the image and the semantics of the target operation; To ensure the quality of knowledge provided by the teacher model, a salient knowledge filtering module is used to process the original availability prediction graph; This module first bases on a preset threshold. The low-confidence pixel locations in the original availability prediction map are suppressed using the following transformation formula: in, This represents the pixel location index in the original availability prediction map; This indicates the saliency screening threshold, controlling the strictness of knowledge screening; Subsequently, the original availability prediction graph of the teacher model, after saliency screening, is subjected to max-min normalization to redistribute feature values ​​and enhance global information: in, Represents the maximum-minimum normalization function; For numerical stability terms, set to ; After the above processing, high-quality significant availability knowledge of teachers is obtained. It is used to constrain and guide the online learning of student models during the adaptive process of testing.

[0010] Preferably, step S4 of the present invention specifically comprises: The student model receives the same normalized feature representation. The original availability prediction graph is generated through forward propagation: in, This represents the forward reasoning process of the student model; Subsequently, the original availability prediction graph output by the student model is subjected to max-min normalization: To reduce the uncertainty of student model predictions, an uncertainty constraint based on the normalized student availability prediction graph is introduced as an entropy constraint term, and its information entropy is calculated as follows: in, This represents the pixel position index in the normalized student availability prediction map; The information entropy represents the distribution predicted by the student model and is used to measure the uncertainty of the distribution predicted by the student model.

[0011] By minimizing this entropy value This guides students to generate normalized student availability prediction maps with a more concentrated and defined spatial distribution.

[0012] Preferably, step S5 of the present invention specifically comprises: During the testing phase, when the cumulative number of test samples reaches the preset update interval, the joint optimization objective of the student model is calculated based on the current sample set. Specifically, firstly, based on the normalized availability prediction graph of the student model and the salient availability prediction graph output by the teacher model, the knowledge transfer loss is constructed, and its calculation formula is as follows: in, Indicates the number of samples used for updating; This represents the normalized availability prediction graph output by the student model. This represents the significant availability prediction results output by the teacher model; Meanwhile, an uncertainty constraint based on the prediction graph is introduced as the entropy loss, which is defined as follows: Here, represents the information entropy of the student model's predicted distribution, which is used to measure the uncertainty of the predicted distribution, and the student model parameters are updated by minimizing the entropy loss during the joint optimization process; Based on this, the knowledge transfer loss and entropy loss are weighted and combined to form the final joint optimization objective, the calculation formula of which is as follows: in, These are weighting coefficients used to balance entropy constraint loss and knowledge transfer loss; The Adam optimizer is used with a learning rate of Minimize joint loss Update student model parameters .

[0013] Preferably, step S6 of the present invention specifically comprises: The exponential moving average strategy is as follows: in, The coefficients of the exponential moving average control the degree of conservatism in updating the teacher model parameters. This update method ensures that the teacher model can slowly absorb the knowledge learned by the student model, while maintaining parameter stability and avoiding drastic fluctuations due to noise in a single batch of data.

[0014] Preferably, step S7 of the present invention specifically comprises: After all test samples have been processed, the student model completes its final update. For a new query image, the optimized student model is used to generate the final availability prediction map. The availability prediction graph As a spatial attention mask, with RGB images and depth images Combined with the pre-trained grasping and detection network middle: in, The generated grasping pose is parameterized as follows: : To capture the coordinates of the center in the image coordinate system; The rotation angle is about the Z-axis; This refers to the gripper width; To capture quality scores; Finally, the grasping pose in the image coordinate system is determined using the camera-robot hand-eye calibration matrix. The system is switched to the robot's base coordinate system, which drives the robotic arm to perform precise grasping operations.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a weakly supervised availability localization robot grasping method based on test-time adaptive approach. By using online unlabeled data to adaptively optimize the model during the testing phase, it achieves accurate perception and stable prediction of the object's operating area, effectively alleviating the performance degradation problem of weakly supervised availability localization methods in real-world scenarios caused by changes in data distribution, and improving the practicality and robustness of the robot grasping system in complex environments.

[0016] The teacher-student knowledge transfer mechanism introduced in this invention enables the continuous transfer and updating of high-quality available knowledge without the need for test labels. By performing significance screening and normalization on the teacher's prediction results, the accumulation of noise information is effectively suppressed, enhancing the stability and reliability of the model in the online learning process.

[0017] The entropy-enhanced availability localization constraint proposed in this invention models and constrains the uncertainty of the availability prediction graph, guiding the model to generate a more concentrated and consistent spatial prediction distribution. This improves the accuracy and consistency of availability region localization and enhances the model's adaptability to complex scenarios and multi-task instructions.

[0018] This invention combines the optimized availability prediction graph with the grasping detection process, enabling the robot to generate a reasonable grasping posture while meeting operational semantic constraints. This avoids operational deviations caused by relying solely on geometric grasping scores, significantly improving the robot's grasping success rate and overall system stability in multi-task operation scenarios. It has good engineering application value and promotion prospects. Attached Figure Description

[0019] Figure 1 Examples of robot availability for different operational purposes; Figure 2 This is a framework diagram of the availability positioning robot grasping according to the present invention; Figure 3 This section provides an overview of a typical multimodal availability localization module, whose parameters are optimized using the entropy values ​​of the classification results. Figure 4This is an example diagram illustrating the qualitative results of the intermediate process of knowledge transfer in this invention. Figure 5 This is a qualitative result of the present invention regarding the availability of various functions; Figure 6 Qualitative comparison of the present invention with LOCATE and WSMA on AGD20K-Seen, AGD20K-Unse, and HICO; Figure 7 Screenshots of real-world robot grasping experiments targeting various objects with different functions; Figure 8 This is a qualitative result of the availability of the present invention for various objects and functions in the real world. Detailed Implementation

[0020] The invention will now be further described with reference to the accompanying drawings.

[0021] This invention aims to address the performance degradation of weakly supervised availability localization methods in real-world robotic applications due to environmental changes and data distribution shifts. By introducing an adaptive learning mechanism during the testing phase, it achieves stable perception of the object's operational area and guides the robot to complete grasping tasks that conform to the operational semantics. The overall system structure is as follows: Figure 2 As shown, it includes an availability positioning model, a teacher-student adaptive module, a salient knowledge filtering module, an entropy constraint optimization module, and a crawling decision module.

[0022] The present invention provides a weakly supervised availability localization robot grasping method based on test-time adaptation, comprising the following steps: S1. Initialize the teacher-student model; based on the pre-trained weakly supervised availability localization model, construct the teacher model and student model respectively. The teacher model and student model use completely identical network structures and initial parameters, such as... Figure 2 As shown, each model includes a visual encoding module, a multimodal interaction module, and an availability prediction module, which are used to generate corresponding availability prediction results based on the input first-person perspective image and operation instructions. The pre-trained weakly supervised availability localization model in step S1 is trained based on source training data containing first-person view images and their corresponding image-level availability labels. Let the pre-trained weakly supervised availability localization model be... Its parameters are At the start of the testing phase, both the teacher and student models are initialized to this pre-trained weakly supervised availability localization model, i.e.: in, Represents the teacher model parameters at the initial moment; Represents the student model parameters at the initial moment; During subsequent adaptive testing, the teacher model parameters will remain frozen for inference, while the student model parameters will be optimized and updated based on online test data.

[0023] S2. Obtain the normalized feature representation; After deploying the pre-trained weakly supervised availability localization model to the robot system, the robot continuously collects first-person perspective RGB images online to obtain test data without real labels. The test data is preprocessed, and then visual features are extracted from it. Feature normalization is performed on the extracted visual features to obtain the normalized feature representation, so as to reduce the distribution difference between the test data and the training data, which is used for subsequent joint inference and online adaptive optimization of the teacher model and the student model. Step S2 specifically involves the robot continuously acquiring first-person perspective RGB images online via a vision sensor, forming a test data stream. ,in Indicates the first One test sample, Indicates the total number of test samples; These test data are not properly labeled, and their distribution may differ from that of the training data. For each test sample arriving online First, preprocessing is performed: the image is cropped and scaled to a fixed size (224×224 pixels), then input into a pre-trained visual encoder (DINO-ViT-S) to extract visual features. Next, feature normalization is performed on the extracted visual features, and the transformation formula is as follows: in, This represents a predefined feature normalization transformation operation; This represents the normalized feature representation, which is used for subsequent joint processing of the teacher and student models.

[0024] S3. The teacher model predicts and filters significant availability knowledge. The normalized feature representation is simultaneously input into the teacher model and the student model. The teacher model parameters are kept frozen, and its forward inference process generates an original availability prediction map to characterize the correlation between each pixel region in the image and the semantics of the target operation. The original availability prediction map is processed by the significant availability filtering module to filter out prediction maps with low confidence, resulting in a filtered prediction map. Then, by performing max-min normalization on the filtered prediction map, high-quality teacher significant availability knowledge is obtained to guide the online learning of the student model. Step S3 is as follows: Normalized feature representation Simultaneously input into both the teacher and student models, with the teacher model parameters frozen, its forward inference process generating the original availability prediction graph: in, This represents the forward reasoning process of the teacher model; This represents the original availability prediction map output by the teacher model, used to characterize the degree of correlation between each pixel region in the image and the semantics of the target operation; To ensure the quality of knowledge provided by the teacher model, a salient knowledge filtering module is used to process the original availability prediction graph output by the teacher model. This module first bases on a preset threshold. The low-confidence pixel locations in the original availability prediction map are suppressed using the following transformation formula: in, This represents the pixel location index in the original availability prediction map; This indicates the saliency screening threshold, controlling the strictness of knowledge screening; Subsequently, the original availability prediction graph of the teacher model, after saliency screening, is subjected to max-min normalization to redistribute feature values ​​and enhance global information: in, Represents the maximum-minimum normalization function; For numerical stability terms, set to ; After the above processing, high-quality significant availability knowledge of teachers is obtained. It is used to constrain and guide the online learning of student models during the adaptive process of testing.

[0025] S4. The student model predicts and applies entropy constraints. Within a set update period, the normalized feature representation is input into the student model. The student model generates an original availability prediction map through forward inference. Its form is consistent with the teacher model output and is used to characterize the correlation between the position of each pixel in the image and the semantics of the target operation. Based on this, an information entropy constraint based on the normalized student availability prediction map is introduced. By minimizing the information entropy of the prediction map, the uncertainty of the student model's prediction distribution is limited, guiding it to generate an availability prediction map with a more concentrated and definite spatial distribution for subsequent joint optimization and grasping decision. Step S4 is as follows: The student model receives the same normalized feature representation. The original availability prediction graph is generated through forward propagation: in, This represents the forward reasoning process of the student model; Subsequently, the original availability prediction graph output by the student model is subjected to max-min normalization: To reduce the uncertainty of student model predictions, an uncertainty constraint based on the normalized student availability prediction graph is introduced as an entropy constraint term, and its information entropy is calculated as follows: in, Represents the pixel location index in the availability prediction map; Information entropy represents the distribution predicted by the student model, and is used to measure the uncertainty of the distribution predicted by the student model. By minimizing this entropy value This guides students to generate normalized student availability prediction maps with a more concentrated and defined spatial distribution.

[0026] S5. Joint loss optimization of the student model: Based on the significant availability knowledge output by the teacher model and the availability prediction graph generated by the student model under entropy constraints, a joint optimization objective consisting of knowledge transfer loss and entropy constraint loss is constructed. The student model parameters are updated online by minimizing the joint loss. After the parameter update is completed, the updated student model is used to perform forward reasoning on the current test sample to generate the final availability prediction graph. Step S5 is as follows: During the testing phase, when the cumulative number of test samples reaches the preset update interval, the joint optimization objective of the student model is calculated based on the current sample set. First, based on the normalized availability prediction graph of the student model and the significant availability prediction graph output by the teacher model, the knowledge transfer loss is constructed, and its calculation formula is as follows: in, Indicates the number of samples used for updating; This represents the normalized availability prediction graph output by the student model. This represents the significant availability prediction map output by the teacher model after significant filtering and normalization. Meanwhile, an uncertainty constraint based on the prediction graph is introduced as the entropy loss, which is defined as follows: in, The information entropy representing the predicted distribution of the student model is used to measure the uncertainty of the predicted distribution, and the student model parameters are updated by minimizing the entropy loss during the joint optimization process. Based on this, the knowledge transfer loss and entropy loss are weighted and combined to form the final joint optimization objective, the calculation formula of which is as follows: in, These are weighting coefficients used to balance entropy constraint loss and knowledge transfer loss; The Adam optimizer is used with a learning rate of Minimize joint loss Update student model parameters .

[0027] S6. Exponential sliding average update of the teacher model: After the student model parameters are updated, the teacher model parameters are updated smoothly using an exponential sliding average strategy. Through the smooth update method, the effective availability representations obtained by the student model during the adaptive process during testing are gradually accumulated into the teacher model to ensure that the teacher model can continuously provide stable and reliable significant availability knowledge in subsequent test samples, which is used to constrain the online adaptive optimization process of the student model. Step S6 is as follows: The exponential moving average strategy is as follows: in, The coefficients of the exponential moving average control the degree of conservatism in updating the teacher model parameters. This update method ensures that the teacher model can slowly absorb the knowledge learned by the student model, while maintaining parameter stability and avoiding drastic fluctuations due to noise in a single batch of data.

[0028] S7. Generate grasping pose based on the final availability prediction graph; use the final availability prediction graph output by the student model after adaptive optimization during test completion in S5 to constrain the grasping detection network and generate a robot grasping pose that conforms to the task semantics.

[0029] Step S7 is as follows: After all test samples have been processed, the student model completes its final update. For a new query image, the optimized student model is used to generate the final availability prediction map. The availability prediction graph As a spatial attention mask, with RGB images and depth images Combined with the pre-trained grasping and detection network middle: in, The generated grasping pose is parameterized as follows: : To capture the coordinates of the center in the image coordinate system; The rotation angle is about the Z-axis; This refers to the gripper width; To capture quality scores; Finally, the grasping pose in the image coordinate system is determined using the camera-robot hand-eye calibration matrix. The robot is then switched to its base coordinate system, and the robotic arm is driven to perform a precise grasping operation. This operation is a standard procedure for those skilled in the art and will not be described in detail here.

[0030] Example This invention aims to address the problem of decreased prediction performance of weakly supervised availability localization methods in real-world robotic applications due to environmental changes and data distribution shifts. By introducing an adaptive learning mechanism during the testing phase, it achieves stable perception of the object's operational area and guides the robot to complete grasping tasks that conform to operational semantics. Figure 1 Examples of availability for different operational purposes. Figure 2 This is a diagram of the overall system structure of the present invention, including an availability localization model, a teacher-student adaptive module, a saliency knowledge filtering module, an entropy constraint optimization module, and a crawling decision module. Figure 3 This section provides an overview of a typical multimodal availability localization module, whose parameters are optimized using the entropy values ​​of the classification results.

[0031] Visual images and textual instructions from different scenes in the AGD20K and HICO datasets are used to improve the accuracy of availability localization, enhance the model's online adaptability during testing, and maintain prediction robustness in data distribution shifts and dynamic environments. The overall network architecture is fixed as a teacher-student knowledge transfer framework, with optimized configurations for the salient knowledge filtering module, the entropy-enhanced availability localization module, and the exponential moving average update strategy. Figure 4 Intermediate visualization results of the knowledge transfer process are presented, showing that although the teacher model's predictions contain noise, the student model's predictions are significantly improved in detail after being filtered and constrained by the knowledge selection module. This verifies that the method can effectively transfer knowledge, reduce error accumulation, and thus enhance the model's predictive ability. Figure 5 The invention demonstrates qualitative prediction results of availability in multiple scenarios, showing that it can accurately predict the availability regions of various objects based on different instructions. Even in complex multi-object scenarios such as "typing", the prediction results are still highly consistent with the true values. Figure 6Qualitative comparisons with existing advanced methods LOCATE and WSMA further demonstrate that although WSMA, which incorporates textual context, outperforms the purely visual method LOCATE, it still suffers from insufficient fine-grained localization. In contrast, this invention exhibits superior localization consistency and accuracy across various scenarios. In real-world robot grasping verification, Figure 7 and Figure 8 The results of grasping execution sequences based on availability prediction and various physical tests are presented respectively: the robot can accurately grasp the target part according to the instructions (such as grasping the handle instead of the blade in the screw-tightening task) and maintain stable performance in different objects and tasks, demonstrating the effectiveness and generalization ability of the method in real-world scenarios.

[0032] This invention effectively adapts to data distribution shifts in dynamic scenarios by introducing a teacher-student knowledge transfer mechanism, a collaborative design of significant knowledge screening and entropy constraint optimization, suppresses the accumulation of noisy knowledge and reduces prediction uncertainty, improves the accuracy and consistency of weakly supervised availability localization, and has good engineering practicality. It can be directly embedded into robot grasping systems, significantly enhancing the robustness of grasping unknown objects in complex environments and the success rate of tasks.

[0033] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A weakly supervised availability-based localization robot grasping method based on test-time adaptation, characterized in that, Includes the following steps: S1. Initialize the teacher-student model; Based on the pre-trained weakly supervised availability localization model, construct a teacher model and a student model with consistent structure and initial parameters, respectively. The teacher model is used to provide significant availability knowledge constraints to the student model during the adaptive process at test time, and the student model is used to perform online optimization on the test data and generate an availability prediction graph. S2. Obtain the normalized feature representation; After deploying the pre-trained weakly supervised availability localization model to the robot system, the robot continuously collects first-person perspective RGB images online to obtain test data without real labels. The test data is preprocessed, and then visual features are extracted from it. Feature normalization is performed on the extracted visual features to obtain the normalized feature representation, which is used for subsequent joint inference and online adaptive optimization of the teacher model and student model. S3. The teacher model predicts and filters significant availability knowledge. The normalized feature representation is simultaneously input into the teacher model and the student model. The teacher model parameters are kept frozen, and its forward inference process generates an original availability prediction map to characterize the correlation between each pixel region in the image and the semantics of the target operation. The original availability prediction map is processed by the significant availability filtering module to filter out prediction maps with low confidence, resulting in a filtered prediction map. Then, by performing max-min normalization on the filtered prediction map, high-quality teacher significant availability knowledge is obtained to guide the online learning of the student model. S4. The student model predicts and applies entropy constraints. Within a set update period, the normalized feature representation is input into the student model. The student model generates an original availability prediction map through forward inference. Its form is consistent with the teacher model output and is used to characterize the correlation between the position of each pixel in the image and the semantics of the target operation. Based on this, an information entropy constraint based on the normalized student availability prediction map is introduced. By minimizing the information entropy of the prediction map, the uncertainty of the student model's prediction distribution is limited, guiding it to generate an availability prediction map with a more concentrated and definite spatial distribution for subsequent joint optimization and grasping decision. S5. Joint loss optimization of the student model: Based on the significant availability knowledge output by the teacher model and the availability prediction graph generated by the student model under entropy constraints, a joint optimization objective consisting of knowledge transfer loss and entropy constraint loss is constructed. The student model parameters are updated online by minimizing the joint optimization objective. After the parameter update is completed, the updated student model is used to perform forward inference on the current test sample to generate the final availability prediction graph. S6. Exponential sliding average update of the teacher model: After the student model parameters are updated, the teacher model parameters are updated smoothly using an exponential sliding average strategy. Through the smooth update method, the effective availability representations obtained by the student model during the adaptive process during testing are gradually accumulated into the teacher model to ensure that the teacher model can continuously provide stable and reliable significant availability knowledge in subsequent test samples, which is used to constrain the online adaptive optimization process of the student model. S7. Generate grasping pose based on the final availability prediction graph; use the final availability prediction graph output by the student model after adaptive optimization during the test in S5 to constrain the grasping detection network and generate a robot grasping pose that conforms to the task semantics.

2. The weakly supervised availability localization robot grasping method based on test-time adaptive approach as described in claim 1, characterized in that, The pre-trained weakly supervised availability localization model in step S1 is trained based on source training data containing first-person view images and their corresponding image-level availability labels. Let the pre-trained weakly supervised availability localization model be... Its parameters are At the start of the testing phase, both the teacher and student models are initialized to the pre-trained weakly supervised availability localization model, i.e.: in, Represents the teacher model parameters at the initial moment; Represents the student model parameters at the initial moment; During subsequent adaptive testing, the teacher model parameters will remain frozen for inference, while the student model parameters will be optimized and updated based on online test data.

3. The weakly supervised availability localization robot grasping method based on test-time adaptive approach as described in claim 2, characterized in that, Step S2 specifically involves the robot continuously acquiring first-person perspective RGB images online via a vision sensor, forming a test data stream. ,in Indicates the first One test sample, Indicates the total number of test samples; These test data are not properly labeled, and their distribution differs from that of the training data; For each test sample arriving online First, preprocessing is performed: the image is cropped and scaled to a fixed size, then input into a pre-trained visual encoder to extract visual features. Next, feature normalization is performed on the extracted visual features, and the transformation formula is as follows: in, This represents a predefined feature normalization transformation operation; This represents the normalized feature representation, which is used for subsequent joint processing of the teacher and student models.

4. The weakly supervised availability localization robot grasping method based on test-time adaptive approach as described in claim 1, characterized in that, Step S3 is as follows: Normalized feature representation Simultaneously input into both the teacher and student models, with the teacher model parameters frozen, its forward inference process generating the original availability prediction graph: in, This represents the forward reasoning process of the teacher model; This represents the original availability prediction graph output by the teacher model; The original availability prediction graph output by the teacher model is processed using a salient knowledge filtering module. This module first bases on a preset threshold. The low-confidence pixel locations in the original availability prediction map are suppressed using the following transformation formula: in, This represents the pixel location index in the original availability prediction map; This indicates the saliency screening threshold, controlling the strictness of knowledge screening; Subsequently, the original availability prediction graph of the teacher model, after saliency screening, is subjected to max-min normalization to redistribute feature values ​​and enhance global information: in, Represents the maximum-minimum normalization function; For numerical stability terms, set to ; After the above processing, high-quality significant availability knowledge of teachers is obtained. It is used to constrain and guide the online learning of student models during the adaptive process of testing.

5. The weakly supervised availability localization robot grasping method based on test-time adaptive approach as described in claim 1, characterized in that, Step S4 is as follows: The student model receives the same normalized feature representation. The original availability prediction graph for the student model is generated through forward propagation: in, This represents the forward reasoning process of the student model; Subsequently, the original availability prediction graph output by the student model is subjected to max-min normalization: Based on the uncertainty constraint of the normalized student availability prediction graph as an entropy constraint term, its information entropy is calculated as follows: in, This represents the pixel position index in the normalized student availability prediction map; Information entropy represents the distribution predicted by the student model, and is used to measure the uncertainty of the distribution predicted by the student model. By minimizing this entropy value This guides students to generate normalized student availability prediction maps with a more concentrated and defined spatial distribution.

6. The weakly supervised availability localization robot grasping method based on test-time adaptive approach as described in claim 5, characterized in that, Step S5 is as follows: During the testing phase, when the cumulative number of test samples reaches the preset update interval, the joint optimization objective of the student model is calculated based on the current sample set. First, based on the normalized availability prediction graph of the student model and the significant availability prediction graph output by the teacher model, the knowledge transfer loss is constructed, and its calculation formula is as follows: in, Indicates the number of samples used for updating; This represents the normalized availability prediction graph output by the student model. This represents the significant availability prediction map output by the teacher model after significant filtering and normalization. Meanwhile, an uncertainty constraint based on the prediction graph is introduced as the entropy loss, which is defined as follows: in, The information entropy representing the predicted distribution of the student model is used to measure the uncertainty of the predicted distribution, and the student model parameters are updated by minimizing the entropy loss during the joint optimization process. Based on this, the knowledge transfer loss and entropy loss are weighted and combined to form the final joint optimization objective, the calculation formula of which is as follows: in, These are weighting coefficients used to balance entropy constraint loss and knowledge transfer loss; The Adam optimizer is used with a learning rate of Minimize joint loss Update student model parameters .

7. The weakly supervised availability localization robot grasping method based on test-time adaptive approach as described in claim 5, characterized in that, Step S6 is as follows: The exponential moving average strategy is as follows: in, The coefficients of the exponential moving average control the degree of conservatism in updating the teacher model parameters. This update method ensures that the teacher model can slowly absorb the knowledge learned by the student model, while maintaining parameter stability and avoiding drastic fluctuations due to noise in a single batch of data.

8. The weakly supervised availability localization robot grasping method based on test-time adaptive approach as described in claim 5, characterized in that, Step S7 is as follows: After all test samples have been processed, the student model completes its final update. For a new query image, the optimized student model is used to generate the final availability prediction map. The availability prediction graph As a spatial attention mask, with RGB images and depth images Combined with the pre-trained grasping and detection network middle: in, The generated grasping pose is parameterized as follows: : To capture the coordinates of the center in the image coordinate system; The rotation angle is about the Z-axis; This refers to the gripper width; To capture quality scores; Finally, the grasping pose in the image coordinate system is determined using the camera-robot hand-eye calibration matrix. The system is switched to the robot's base coordinate system, which drives the robotic arm to perform precise grasping operations.

Images