Object reachability detection method and apparatus for robots

Abstract

The application relates to the technical field of service robots, and provides a robot object availability detection method and device, wherein an initial feature of a complex availability text is subjected to self-attention reconstruction by introducing a learnable parameter to obtain a refined text feature, in a decoding stage, a selective weight generated by global context is used for attention weighting in parallel, and a threshold truncation mechanism based on a preset quantile is used for filtering weakly related local attention, efficient alignment and decoding of cross-modal features are finally realized, high-quality feature extraction and high-precision visual alignment of a complex and lengthy text instruction are realized, and the accuracy, robustness and adaptability in a complex scene of small sample object availability detection of the robot when facing a semantic-intensive open-set environment instruction are significantly improved, so that the detection performance is improved.

Description

Technical Field

[0001] This invention relates to the field of service robot technology, and in particular to a method and apparatus for detecting the availability of objects for a robot. Background Technology

[0002] Object availability describes the functional attributes of an object and is fundamental for robots to understand their environment and achieve precise interaction; therefore, its research is of great significance. Early methods relied on large-scale availability datasets to train object availability detection networks, thereby obtaining perception results of object functional components. However, in real-world scenarios, constructing large-scale datasets for each availability is tedious and even difficult. This premise severely hinders the practical application of robots. Therefore, researchers have begun to focus on few-shot object availability detection, with One-shot Open Affordance Learning (OOAL) being a representative work. Although existing few-shot object availability detection methods can reduce the need for large amounts of data annotation, they still suffer from poor adaptability in open-set scenarios. An open-set scenario refers to a scenario where the training phase uses availability words as input, while the testing phase can use more verbose and semantically dense complex text as input. For example, the training text might be "Hold," but the test text might be "Carefully pick up the bat." While this more human-like expression improves the flexibility of practical applications, it also increases the difficulty of availability detection.

[0003] Therefore, further research is needed to improve the detection performance of existing small-sample object availability detection methods in open-set environments. Summary of the Invention

[0004] This invention provides a method and apparatus for detecting object availability in robots, which addresses the shortcomings of existing methods for detecting object availability in small sample environments, where the detection performance is poor.

[0005] This invention provides a method for object availability detection in a robot, comprising: Acquire the target object's image to be detected and the available text of the target object in the scene where the robot is located; Global context features and local visual block features of the image to be detected are extracted; The initial text features of the available text are extracted, and then, based on multiple cascaded transformer layers, learnable parameters are concatenated into the initial text features and self-attention reconstruction is performed to obtain refined text features. Based on multiple cascaded decoders, feature alignment is performed on the local visual block features, the global context features, and the query features to obtain decoded features. The query features of the first decoder are the refined text features, and the query features of subsequent decoders are the decoded features output by the previous decoder. In each decoder, a selective weight generated by the interaction between the global context features and the local visual block features is used to weight the first attention score between the local visual block features and the query features to obtain a second attention score. Furthermore, based on a filtering threshold determined by a preset quantile, the intermediate process features between the local visual block features and the query features are filtered and truncated to obtain a third attention score. Decoding is then performed based on the second attention score and the third attention score to obtain the decoded features. The decoded features output by the last decoder are multiplied by the local visual block features to obtain the availability detection result of the target object.

[0006] According to the present invention, a robot object availability detection method is provided, wherein the refined text features are obtained by concatenating learnable parameters and performing self-attention reconstruction based on multiple cascaded transformer layers, including: The initial text features are used as the input text features for the first transformer layer; For each transformer layer, learnable parameters corresponding to each transformer layer are concatenated into the input text features to obtain concatenated features. The concatenated features are then normalized and processed by multi-head attention, and then added to the concatenated features. After normalization and linear mapping, the concatenated features are added again, and the top few vectors corresponding to the number of text categories in the added results are selected as output text features. The output text features are then used as the input text features for the next transformer layer. The output text features of the last transformer layer are linearly mapped to obtain mapped text features. The mapped text features are then added to the initial text features to obtain the refined text features.

[0007] According to the present invention, a robot object availability detection method includes a second attention score obtained by weighting a first attention score between the local visual block features and the query features using a selective weight generated by the interaction of the global context features and the local visual block features. After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed, followed by normalization to obtain the first attention score. The global context features and the linearly mapped local visual block features are multiplied by a matrix and scaled to obtain intermediate process features. The intermediate process features are averaged and then mapped to a preset interval to obtain the selective weights. The second attention score is obtained by multiplying the selective weight by the first attention score.

[0008] According to the object availability detection method for robots provided by the present invention, the step of filtering and truncating intermediate process features between the local visual block features and the query features based on a filtering threshold determined by a preset quantile to obtain a third attention score includes: After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed to obtain the intermediate process features; The intermediate process features are flattened and arranged in ascending order, and the filtering threshold is determined according to the preset quantile. Elements in the intermediate process features that are less than the filtering threshold are set to zero to obtain filtered features. The filtered features are then normalized to obtain the third attention score.

[0009] According to the present invention, a method for detecting object availability in a robot, wherein decoding is performed based on a second attention score and a third attention score to obtain the decoded features, comprising: The second attention score and the third attention score are weighted and summed, and then regularized to obtain the process attention score. The process attention score is multiplied by the linearly mapped local visual block features to obtain the cross-modal fusion features; The cross-modal fusion features are processed by the forward unit, added to the query features, and normalized to obtain the decoded features.

[0010] According to the object availability detection method for robots provided by the present invention, global contextual features of the image to be detected are extracted, including: The first visual features of the image to be detected are extracted by a self-supervised visual model, and the first visual features are linearly transformed to obtain the first global visual features. The second visual global features of the image to be detected are extracted using an image encoder; The first visual global feature and the second visual global feature are weighted and summed to obtain the global context feature.

[0011] According to the object availability detection method for robots provided by the present invention, local visual block features of the image to be detected are extracted, including: Visual block features based on multi-layer output of the image to be detected are extracted by a self-supervised visual model. Each visual block feature is linearly mapped and multiplied by its corresponding learnable weight and then summed to obtain the merged visual block features. The merged visual block features are sequentially processed by normalization, linear mapping, activation, and regularization to obtain the local visual block features.

[0012] According to the present invention, a method for detecting object availability in a robot acquires a target image of a target object in the scene where the robot is located, comprising: The robot acquires scene images via RGB sensors; The scene image is segmented into objects, and the image to be detected of the target object is obtained from the segmentation result based on the category name of the target object.

[0013] According to the object availability detection method for a robot provided by the present invention, the number of the plurality of cascaded transformer layers is 12.

[0014] The present invention also provides an object availability detection device for a robot, comprising: The acquisition unit acquires the image of the target object to be detected in the scene where the robot is located, as well as the availability text of the target object; The visual feature extraction unit extracts the global context features and local visual block features of the image to be detected; The text feature extraction unit extracts the initial text features of the available text, and, based on multiple cascaded transformer layers, concatenates learnable parameters into the initial text features and performs self-attention reconstruction to obtain refined text features. The decoding unit, based on multiple cascaded decoders, aligns the local visual block features, the global context features, and the query features to obtain decoded features. The query features of the first decoder are the refined text features, and the query features of subsequent decoders are the decoded features output by the previous decoder. In each decoder, a selective weight generated by the interaction between the global context features and the local visual block features is used to weight the first attention score between the local visual block features and the query features to obtain a second attention score. Furthermore, based on a filtering threshold determined by a preset quantile, the intermediate process features between the local visual block features and the query features are filtered and truncated to obtain a third attention score. Finally, decoding is performed based on the second attention score and the third attention score to obtain the decoded features. The detection output unit multiplies the decoded features output by the last decoder with the local visual block features to obtain the availability detection result of the target object.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the object availability detection method for any of the robots described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the object availability detection method for a robot as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements an object availability detection method for a robot as described above.

[0018] The robot object availability detection method and apparatus provided by this invention introduces learnable parameters to perform self-attention reconstruction of the initial features of complex availability text to obtain refined text features. In the decoding stage, selective weights generated by global context are used in parallel for attention weighting, and a threshold truncation mechanism based on preset quantiles is used to filter weakly correlated local attention. Finally, efficient alignment and decoding of cross-modal features are achieved, realizing high-quality feature extraction and high-precision visual alignment of complex and lengthy text instructions. This significantly improves the accuracy, robustness, and adaptability of robots in small-sample object availability detection when facing semantically dense open-set environment instructions, thereby improving detection performance. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the object availability detection method for robots provided by the present invention. Figure 2 This is a schematic diagram of the object availability detection device for the robot provided by the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] It should be understood that, in the description of the embodiments of the present invention, "multiple" means two or more, unless otherwise explicitly specified. Furthermore, all actions involving the acquisition of signals, information, or data in this application are performed in accordance with the applicable data protection laws and regulations of the locality and with permission granted by the owner of the respective device.

[0023] To address the poor detection performance of existing small-sample object availability detection methods in open-set environments, this invention provides a robot object availability detection method to improve feature alignment quality under complex text instructions, thereby enhancing detection accuracy. Figure 1 This is a flowchart illustrating the object availability detection method for robots provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Obtain the image of the target object to be detected in the scene where the robot is located, as well as the availability text of the target object.

[0024] Here, "robot" refers to an intelligent entity that performs interactive tasks in a real-world open-set environment. The target object is the entity in the scene whose functional attributes and interaction methods need to be perceived. The image to be detected is the visual image information containing the target object, which can be used to provide basic visual features. Availability text is a linguistic instruction describing the functional attributes of the target object; in an open-set scenario, it can be a complex, long text containing rich semantics and action details.

[0025] Specifically, the robot's vision sensors can collect visual information about the scene environment, perform preliminary foreground extraction or instance segmentation to identify target objects, and then capture or acquire the corresponding image to be detected. Simultaneously, it can receive user input or availability text from system planning as text guidance instructions for the current detection task.

[0026] Step 120: Extract the global context features and local visual block features of the image to be detected.

[0027] Here, global context features aim to characterize the overall visual scene information and macroscopic semantics of the image to be detected, while local visual block features aim to characterize the detailed texture and structural information of each local region in the image to be detected.

[0028] Specifically, the image to be detected is processed by a pre-trained image feature extraction network. Through different branches or receptive fields of the network, global context features reflecting macroscopic vision and local visual block features reflecting microscopic details are extracted, providing visual representations of different granularities for subsequent cross-modal interactions.

[0029] Step 130: Extract the initial text features of the available text, and, based on multiple cascaded transformer layers, concatenate learnable parameters into the initial text features and perform self-attention reconstruction to obtain refined text features.

[0030] Here, the initial text features refer to the feature representation obtained by initially vectorizing the available text using a text encoder. Multiple cascaded transformer layers constitute a text transformer unit, which in turn forms a text feature adapter. This adapter processes the initial text information layer by layer to obtain refined text features. The learnable parameters refer to the weight vectors that can be adaptively updated during network training, used to supplement the text features with additional task-driven information. The refined text features are high-quality text features output after multi-layer feature interaction and reconstruction, which are more closely aligned with the complex semantics of open-set scenarios.

[0031] Specifically, the initial text features of the available text are first obtained through a pre-trained encoding model. Then, when inputting into each transformer layer, the current text features and the corresponding learnable parameters are concatenated along the text sequence dimension. The attention mechanism is used to promote full information interaction and self-attention reconstruction between different parts of the text and between the text and the learnable parameters. Finally, the refined text features after deep optimization are output.

[0032] It should be noted that by introducing learnable parameters and performing self-attention reconstruction, the feature representation of the input text can be effectively refined, thereby significantly enhancing the model's adaptability and understanding of complex instructions that are lengthy and semantically dense in open-set scenarios.

[0033] Step 140: Based on multiple cascaded decoders, feature alignment is performed on the local visual block features, the global context features, and the query features to obtain decoded features.

[0034] The query feature of the first decoder is the refined text feature, and the query feature of subsequent decoders is the decoding feature output by the previous decoder. In each decoder, the selective weight generated by the interaction between the global context feature and the local visual block feature is used to weight the first attention score between the local visual block feature and the query feature to obtain the second attention score. Based on the filtering threshold determined by the preset quantile, the intermediate process feature between the local visual block feature and the query feature is filtered and truncated to obtain the third attention score. The decoded feature is obtained based on the second attention score and the third attention score.

[0035] Here, multiple cascaded decoders constitute a text-visual feature alignment module, used to achieve deep fusion and progressive refinement of cross-modal features. Here, query features refer to the reference input used during the decoding process to guide visual feature aggregation and extract target-related information. Here, decoded features are the high-level features output by each decoder layer, incorporating cross-modal alignment information.

[0036] Specifically, the multi-level decoders operate in a cascade manner. The query features of the first decoder are initialized with the refined text features obtained above, while the query features of each subsequent decoder are updated with the decoded features output by the previous decoder.

[0037] In each decoder, a second attention score is obtained by weighting the first attention score between the local visual patch features and the query features using selective weights generated from the interaction of global context features and local visual patch features. These selective weights measure the relative importance of each local visual patch feature within the global context. Here, the first attention score represents the preliminary relevance matrix between the local visual patch and the text query features. The second attention score, however, is a reinforced relevance matrix after being guided and filtered by global information.

[0038] In each decoder, the attention mapping between local visual block features and the current query feature is calculated in detail to obtain a first attention score. Simultaneously, the global context features and local visual block features are interactively processed and non-linearly mapped to obtain selective weights. These selective weights are then used to modulate the first attention score to obtain a second attention score.

[0039] It should be noted that using global visual context to guide the alignment of text queries with local visual blocks can effectively suppress background noise and highlight target-related areas with global relevance.

[0040] Additionally, each decoder includes a filtering threshold determined based on a preset quantile to truncate intermediate features between local visual block features and query features, resulting in a third attention score. Decoding is then performed based on the second and third attention scores to obtain the decoded features. Here, the intermediate features are the unnormalized matrix generated when calculating the similarity between query features and local visual block features. The preset quantile is a hyperparameter used to control the filtering ratio, dynamically locating truncation points among the sorted feature elements to determine the filtering threshold. The third attention score is a sparsed relevance matrix obtained after truncating irrelevant or weakly relevant features.

[0041] In each decoder, the intermediate process features are calculated in detail by multiplying the inner product or multiplication of the local visual block features and the query features. These intermediate features are then sorted by numerical value, and a corresponding filtering threshold is found by combining them with a preset quantile. Feature elements below the filtering threshold are forcibly set to zero to filter out weakly correlated features, and then normalized to obtain the third attention score. Finally, the second attention score, which contains global guidance information, and the third attention score, which contains local filtering information, are fused and processed through feedforward to output the decoding features of this layer of the decoder.

[0042] It should be noted that by explicitly discarding weakly matched visual blocks, attention is forced to focus on local areas that are truly highly relevant to the available text, further enhancing the fine-grained and precise alignment between text and visual features.

[0043] Step 150: Multiply the decoded features output by the last decoder with the local visual block features to obtain the availability detection result of the target object.

[0044] Specifically, the high-level decoding features rich in cross-modal alignment information output from the last-level decoder are extracted. These features are then multiplied with the lower-level local visual block features containing rich spatial details using a matrix multiplication operation, mapping them back to the original visual space dimension. This generates the final object availability detection result, guiding the robot to perform interactions. Here, the availability detection result is typically a distribution matrix or pixel-by-pixel detection heatmap representing the probability of specific functional attributes corresponding to each pixel or local region of the target object.

[0045] The method provided in this invention introduces learnable parameters to perform self-attention reconstruction on the initial features of complex availability text to obtain refined text features. In the decoding stage, selective weights generated by the global context are used in parallel for attention weighting, and a threshold truncation mechanism based on preset quantiles is used to filter weakly correlated local attention. Finally, efficient alignment and decoding of cross-modal features are achieved, realizing high-quality feature extraction and high-precision visual alignment of complex and lengthy text instructions. This significantly improves the accuracy, robustness, and adaptability of robots in small-sample object availability detection when facing semantically dense open-set environment instructions, thereby improving detection performance.

[0046] Based on any of the above embodiments, in step 130, based on multiple cascaded transformer layers, learnable parameters are concatenated into the initial text features and self-attention reconstruction is performed to obtain refined text features, including: The initial text features are used as the input text features for the first transformer layer; For each transformer layer, learnable parameters corresponding to each transformer layer are concatenated into the input text features to obtain concatenated features. The concatenated features are then normalized and processed by multi-head attention, and then added to the concatenated features. After normalization and linear mapping, the concatenated features are added again, and the top few vectors corresponding to the number of text categories in the added results are selected as output text features. The output text features are then used as the input text features for the next transformer layer. The output text features of the last transformer layer are linearly mapped to obtain mapped text features. The mapped text features are then added to the initial text features to obtain the refined text features.

[0047] In practice, the initial text features are first used as the input text features for the first transformer layer. These initial text features are the basic text feature representations extracted by the Contrastive Language-Image Pre-training (CLIP) text encoder.

[0048] Specifically, initial text features of size K×512 are extracted, where K represents the number of availability categories. These initial text features are first processed by a linear layer to increase their dimensionality, resulting in category-level text features of size K×768. Then, they are transformed into feature vectors of size 1×K×768 through a dimensionality reshaping operation, and used as the input text features of the first Transformer layer.

[0049] Then, for each transformer layer, the learnable parameters corresponding to each transformer layer are concatenated into the input text features to obtain concatenated features. These concatenated features are then normalized and processed with multi-head attention before being added to the previous layer. After further normalization and linear mapping, they are added again, and the top few vectors corresponding to the number of text categories in the summation are selected as the output text features. These output text features are then used as the input text features for the next transformer layer. It should be noted that the concatenated features here refer to high-dimensional features formed by combining the text features of the current layer with a sequence of learnable parameters with specific dimensions set independently for that layer.

[0050] Specifically, the lightweight text feature adapter comprises 12 cascaded transformer layers. Each transformer layer has a learnable parameter of size 1×M×768, initialized using a normal distribution and updated during training. M is a preset parameter, preferably 10. The learnable parameter is concatenated with the input text features in the second dimension to obtain a concatenated feature of size 1×(K+M)×768. This concatenated feature is then passed through a normalization layer and a multi-head attention mechanism with eight attention heads for self-attention reconstruction. The updated feature is then summed with the residuals of the original concatenated feature. Next, the feature passes through the next normalization layer, a linear mapping layer, a non-linear activation function, and another linear mapping, followed by a second residual summation to obtain the deeply reconstructed text features. Finally, the first K vectors from the second dimension of this feature are extracted as output text features of size 1×K×768 and passed to the next transformer layer for iterative processing.

[0051] Furthermore, the output text features of the last transformer layer are linearly mapped to obtain mapped text features. The mapped text features are then added to the initial text features to obtain refined text features.

[0052] Specifically, the features output from the 12th transformer layer are fed into a linear layer for dimensionality reduction, and then their size is restored to K×512 through a dimensionality reshaping operation to obtain mapped text features. Finally, this mapped text feature is fused with the initial text feature, which is also K×512 in size, at a 1:1 ratio using residual addition, to obtain refined text features that contain deep context and retain the original basic semantics.

[0053] The method provided in this invention, by concatenating learnable parameters of a custom dimension in cascaded transformer layers and adding the mapped features to the initial features using a residual network mechanism, completes the deep reconstruction of text features and compensation of initial information. This effectively enhances the model's ability to mine the detailed features of complex operation instructions, helps to preserve the initial semantics and enhance the ability to express complex text features.

[0054] In one embodiment, the initial text features can be... The input is fed into a lightweight text feature adapter, which consists of network layers such as text transformer units, where each text transformer unit is composed of multiple cascaded transformer layers. Within the lightweight text feature adapter... First, it passes through the seventh linear layer, Linear_7. D , D m Processing to obtain category-level text features The size is K ×768. It should be noted that here ( D , D m ) indicates that the number of input channels of this network layer is D The number of output channels is D m In the following embodiments, this format represents the number of input channels and the number of output channels of the network layer, which can be considered as (number of input channels, number of output channels), and will not be explained again below. The text features are then obtained after a reshape operation. , size is 1× K ×768, As a text transformer unit, transformer_ text The input features, i.e., as text transformer units transformer_ text The input features of the first transformer layer in the middle. The transformer_ text The Middle l A transformer layer l The processing procedure is denoted as a function. Therefore, ,in For transformer_ l The output of is denoted as the th . l Output text features, size 1× K ×768; For transformer_ lThe input, denoted as the first... l splicing features, when hour, The size is 1×( K+M )×768, of which This indicates a concatenation operation in the second dimension. This represents the learnable parameters corresponding to the first transformer layer; when hour, , Indicates the first The output features of the layer Indicates the first The learnable parameters for each transformer layer are of size 1×( K+M )×768. Each layer of transformer_ l For the first l splicing features The processing procedure is the same, among which =1,2,…,12.

[0055] Specifically, First, the updated text features are obtained after sequential processing through the third normalization layer (LayerNorm_3) and multihead attention. The size is 1×( K+M )×768. (This appears to be a typo and can be left as is.) and The sum is used to obtain the second updated text features. The size is 1×( K+M )×768. Then it goes through the fourth normalization layer LayerNorm_4 and the ninth linear layer Linear_9. D m , D m ), Gaussian Error Linear Unit (GELU) activation function and the tenth linear layer Linear_10 ( D m , D m After processing in sequence, the third updated text feature is obtained. The size is 1×( K+M )×768. Finally, and The sum is used to obtain the fourth updated text feature. The size is 1×( K+ M )×768. In Before selecting the second dimensionK The nth vector is used as the nth vector l Output text features , size is 1× K ×768. (This appears to be a typo and can be left as is.) Substitute the values ​​1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 into the corresponding numbers. Finally obtained =12 , size is 1× K ×768, will As a transformer text The output. Furthermore, Send to the eighth linear layer Linear_8 ( D m , D The first mapped text features are obtained. , size is 1× K ×512, and then a reshape operation is performed to obtain the second mapped text features. The size is K ×512. Map the second text features. and initial text features The refined text features are obtained by adding them at a 1:1 ratio. The size is K ×512.

[0056] Here, the specific structure of the lightweight text feature adapter is explained. The lightweight text feature adapter consists of the seventh linear layer, Linear_7. D , D m ), reshape operation, a text transformer unit transformer_ text The eighth linear layer, Linear_8 ( D m , D It consists of a cascade of reshape operations. The seventh linear layer, Linear_7, is one such layer. D , D m ) and the eighth linear layer Linear_8 ( D m , D The text transformer unit is implemented using the `nn.Linear` function in the `torch` library. The `reshape` operation is implemented using the `reshape` function in the `torch` library. text It consists of 12 transformer layers (denoted as transformer_ l, =1,2,…,12) are cascaded together. Because of transformer_ l The components have the same structure. Below is a description of transformer_ l The composition and structure are described in a unified manner.

[0057] In detail, transformer_ l Including the l Learnable parameters The third normalized layer (LayerNorm_3), multihead attention, the fourth normalized layer (LayerNorm_4), and the ninth linear layer (Linear_9). D m , D m ), GELU activation function and the tenth linear layer Linear_10 ( D m , D m ) cascaded together. Among them, the first l Learnable parameters Implemented using the nn.Parameter function from the torch library, with a size of 1× M ×768, of which, M It is a preset parameter. M The preferred value is 10. Use the `nn.init.normal()` function from the torch library to... Initialization: the mean parameter in the function is 0.0, and the bias parameter is 1.0. The third normalization layer (LayerNorm_3) and the fourth normalization layer (LayerNorm_4) are implemented using `nn.LayerNorm` from the torch library; multihead attention is implemented using the `nn.MultiheadAttention` function from the torch library, where the number of attention heads is 8; the ninth linear layer (Linear_9...) D m , D m ) and the tenth linear layer Linear_10 ( D m , D m The GELU activation function is implemented using the `nn.Linear` function from the `torch` library; the GELU activation function is implemented using the `nn.GELU` function from the `torch` library. Substitute the values ​​1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12 into the transformer_ function in sequence. l, the structures of transformer_1, transformer_2, transformer_3, transformer_4, transformer_5, transformer_6, transformer_7, transformer_8, transformer_9, transformer_10, transformer_11, transformer_12 can be obtained.

[0058] Based on any of the above embodiments, in step 140, the first attention score between the local visual block feature and the query feature is weighted using the selective weight generated by the interaction between the global context feature and the local visual block feature to obtain a second attention score, including: After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed, followed by normalization to obtain the first attention score. The global context features and the linearly mapped local visual block features are multiplied by a matrix and scaled to obtain intermediate process features. The intermediate process features are averaged and then mapped to a preset interval to obtain the selective weights. The second attention score is obtained by multiplying the selective weight by the first attention score.

[0059] In practice, the local visual block features and query features are first linearly mapped, then matrix multiplication and scaling are performed, and then normalization is applied to obtain the first attention score.

[0060] Specifically, the local visual block features are fed into the eleventh linear layer to obtain key features of size 256×512, and the query features are fed into the twelfth linear layer to obtain a query feature vector of size K×512. After dimensionality reshaping and permutation operations on both, matrix multiplication is performed, and the result is multiplied by a preset scaling factor of 0.125. Finally, the softmax activation function is applied for normalization to obtain the first attention score of size 8×K×256. It should be noted that the linear mapping here is used to project the input visual and text features onto the same metric space, and the scaling operation is used to prevent the gradient distribution from becoming extreme due to excessively large high-dimensional dot product values.

[0061] Then, the global context features and the linearly mapped local visual block features are multiplied by a matrix and scaled to obtain intermediate features. The intermediate features are averaged and mapped to a preset interval to obtain selective weights. Here, the intermediate features reflect the response distribution of each local region of the image in the context of the global macro-semantic background, and the selective weights are global guiding identifiers used as attention switches after numerical compression.

[0062] Specifically, the global context features of size 1×512 are dimensionally reshaped and permuted, then multiplied by the previously generated local visual key features, and multiplied by a scaling factor of 0.125 to obtain the first intermediate process features. Next, an averaging operation is performed on this feature in the first dimension, and the sigmoid function is used to map the averaged result to a probability value range of 0 to 1, resulting in selective weights of size 1×1×256.

[0063] Finally, the selective weights are multiplied by the first attention score to obtain the second attention score. Specifically, the selective weights, which represent the global matching confidence, are directly multiplied element-wise by the first attention score, which reflects the local image-text alignment. This preserves high-confidence local attention and suppresses low-confidence background noise, resulting in a second attention score with a size of 8×K×256.

[0064] The method provided in this invention generates selective weights for continuous interval mapping by interacting global context features and local visual features, and uses these weights to softly modulate the first attention response matrix of text and local vision. It makes full use of the global macro view to guide cross-modal alignment, effectively avoids the mismatch of text features to isolated and non-functional local texture regions in complex scenes, and significantly enhances the anti-environment interference capability of the detected target.

[0065] Based on any of the above embodiments, in step 140, the intermediate process features between the local visual block features and the query features are filtered and truncated based on a filtering threshold determined by a preset quantile to obtain a third attention score, including: After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed to obtain the intermediate process features; The intermediate process features are flattened and arranged in ascending order, and the filtering threshold is determined according to the preset quantile. Elements in the intermediate process features that are less than the filtering threshold are set to zero to obtain filtered features. The filtered features are then normalized to obtain the third attention score.

[0066] In practice, firstly, the local visual block features and query features are linearly mapped, then matrix multiplication and scaling are performed to obtain intermediate process features. Specifically, the local visual block features and query features are fed into the corresponding thirteenth and fourteenth linear layers to generate key feature and query feature vectors, respectively. After dimensionality reshaping and permutation, matrix multiplication is performed and multiplied by a scaling factor of 0.125 to obtain the second intermediate process features with a size of 8×K×256.

[0067] Then, the intermediate process features are flattened and sorted in ascending order, and the filtering threshold is determined according to a preset quantile. In this embodiment, the preset quantile is preferably 0.4. Specifically, the generated second intermediate process features are flattened to convert the tensor into a one-dimensional data sequence and sorted in ascending order according to the element values. Then, the quantile calculation method is called to accurately locate and extract the response boundary values ​​in the sequence based on the preset quantile value of 0.4, and the boundary value is set as the filtering threshold in the current matching environment.

[0068] Finally, elements in the intermediate process features smaller than the filtering threshold are set to zero to obtain the filtered features. These filtered features are then normalized to obtain the third attention score. Here, the filtered features are sparse feature matrices with partially truncated responses. Specifically, the extracted filtering threshold is used to iterate through and compare all elements of the second intermediate process features. Weakly correlated elements with values ​​smaller than the threshold are forcibly set to 0, while elements with values ​​greater than or equal to the threshold retain their original values, thus obtaining the third intermediate process features with truncated interference terms. Subsequently, the softmax function is applied to these features for normalization, generating the final third attention score of size 8×K×256.

[0069] The method provided in this invention uses flattening sorting and preset quantiles to accurately locate the dynamic filtering threshold, and performs a sparsity truncation operation on intermediate process features that are directly set to zero below the threshold. This forcibly and accurately cuts off the information divergence of text query features to irrelevant visual backgrounds, greatly improving the model's ability to focus on strongly related local regions and the sensitivity of attention alignment, thereby outputting high-precision availability detection results.

[0070] Based on any of the above embodiments, in step 140, decoding is performed based on the second attention score and the third attention score to obtain the decoding features, including: The second attention score and the third attention score are weighted and summed, and then regularized to obtain the process attention score. The process attention score is multiplied by the linearly mapped local visual block features to obtain the cross-modal fusion features; The cross-modal fusion features are processed by the forward unit, added to the query features, and normalized to obtain the decoded features.

[0071] In practice, the second and third attention scores are first weighted and summed, and then regularized to obtain the process attention score. The second attention score contains global guidance information, while the third attention score is highly relevant local attention information after sparse truncation. The process attention score is the final mapping weight matrix used to perform cross-modal feature aggregation after the two are fused.

[0072] Specifically, the second and third attention scores are weighted and summed in a 1:6 ratio to enhance the guiding role of strongly correlated local responses, resulting in a strongly correlated attention score of size 8×K×256. This strongly correlated attention score is then fed into an attention regularization layer for a dropout mechanism to prevent overfitting, thus obtaining the process attention score.

[0073] Then, the process attention score is multiplied by the linearly mapped local visual block features to obtain the cross-modal fusion feature. Here, the cross-modal fusion feature refers to the comprehensive feature vector extracted from the original local visual space by using the text query intent as an attention guide.

[0074] Specifically, the local visual block features are fed into the fifteenth linear layer for feature space transformation to obtain the first-value features. Then, the dimensions are adjusted to 8×256×64 through dimension reshaping and permutation operations to obtain the third-value features. Next, the process attention score is multiplied by the third-value features to complete the aggregation of cross-modal features. After another dimension permutation and reshaping operation, a cross-modal fusion feature with a size of K×512 is obtained.

[0075] Finally, the cross-modal fused features are processed by the feedforward unit, added to the query features, and normalized to obtain the decoded features. The feedforward unit here is a residual network module containing multiple layers of linear mapping and nonlinear activation, used for deep processing of the cross-modal aggregated features.

[0076] Specifically, the cross-modal fused features are fed into the feedforward unit, and then processed sequentially through the sixteenth linear layer and a regularization layer to obtain intermediate features. These intermediate features are added to the input query features to form the first residual fusion. Subsequently, this feature is processed sequentially through a normalization layer, a seventeenth linear layer, a Rectified Linear Unit (ReLU), a regularization layer, an eighteenth linear layer, and a second regularization layer. The result is again added to the query features to form the second residual fusion, and finally processed by a normalization layer to output decoded features of size K×512.

[0077] The method provided in this invention combines a globally guided second attention score with a sparsely truncated third attention score in a specific ratio, and utilizes a deep forward unit combined with a multiple residual addition mechanism to achieve feature reconstruction. This approach takes into account the fusion advantages of the global macro background and the local strongly correlated details, while effectively preventing the gradient vanishing problem in deep networks by using residual addition, thus significantly improving the multimodal representation capability of decoded features under the guidance of complex text.

[0078] In one embodiment, a text-visual feature alignment module can be constructed by cascading two decoders, a first decoder Decoder_1 and a second decoder Decoder_2, to align local visual block features, global context features, and query features to obtain decoded features. The first decoder and the second decoder have the same structure and use Decoder_2. d ( d =1,2) Unified description. Decoder_ d This includes global guided attention branches, selective local attention branches, and cross-modal decoding branches. It should be noted that... D , D This format represents the number of input channels and the number of output channels of the network layer, which can be considered as (number of input channels, number of output channels). The explanation will not be repeated below.

[0079] Among them, the global guidance attention branch is composed of the eleventh linear layer Linear_11 ( D , D ), the twelfth linear layer Linear_12 ( D , D The selective local attention branch consists of the thirteenth linear layer, Linear_13, which includes reshape, permute, softmax, and sigmoid operations. D , D ), the fourteenth linear layer Linear_14 D , D It consists of operations such as reshape, permute, flatten, quantile, and softmax.

[0080] The cross-modal decoding branch includes the attention regularization layer Att_Dropout and the fifteenth linear layer Linear_15. D , D ), reshape operation, permute operation, and feedforward_block, the feedforward_block is composed of the sixteenth linear layer Linear_16 ( D ,D ), third regularization layer Dropout_3, fifth normalization layer LayerNorm_5, seventeenth linear layer Linear_17 ( D , D ), ReLU activation function, fourth regularization layer Dropout_4, eighteenth linear layer Linear_18 ( D , D The fifth regularization layer (Dropout_5) and the sixth normalization layer (LayerNorm_6) are cascaded together. The fifteenth linear layer is Linear_15. D , D ), the eleventh linear layer Linear_11 D , D ), the twelfth linear layer Linear_12 ( D , D ), the thirteenth linear layer Linear_13 ( D , D ), the fourteenth linear layer Linear_14 D , D ), the sixteenth linear layer Linear_16 ( D , D ), the seventeenth linear layer Linear_17 ( D , D ), the eighteenth linear layer Linear_18 ( D , D All of the above are implemented using the `nn.linear` function in the `torch` library; the attention regularization layer `Att_Dropout`, the third regularization layer `Dropout_3`, the fourth regularization layer `Dropout_4`, and the fifth regularization layer `Dropout_5` are all implemented using the `nn.Dropout` function in the `torch` library; the fifth normalization layer `LayerNorm_5` and the sixth normalization layer `LayerNorm_6` are both implemented using the `nn.LayerNorm` function in the `torch` library; the permute operation is implemented using the `permute` function in the `torch` library; the softmax operation is implemented using the `softmax` function in the `torch` library; the sigmoid operation is implemented using the `sigmoid` function in the `torch` library; the flatten operation is implemented using the `flatten` function in the `torch` library; the quantile operation is implemented using the `quantile` function in the `torch` library; and the ReLU activation function is implemented using the `nn.ReLU()` function in the `torch` library. d Substituting 1 and 2 into Decoder_ d Thus, the structures of the first decoder Decoder_1 and the second decoder Decoder_2 can be obtained.

[0081] The specific data processing flow includes: extracting local visual block features Global context features and refined text features The data is fed into the text-visual feature alignment module, where the first decoder, Decoder_1, receives local visual block features. Refining text features and global context features The first decoded feature is output after processing. The second decoder, Decoder_2, receives local visual block features. First decoding feature and global context features The second decoded feature is output after processing. The second decoding feature and local visual block features transpose Multiplying these together yields the final availability detection heatmap. This is the output of the robot's object availability detection network.

[0082] Among them, Decoder_ d The processing procedure is denoted as a function. ,in d =1,2. Function The input consists of three parts: local visual patch features. Global context features and query features ,in, d When =1, query features To refine text features The size is K ×512; d When =2, query features The first decoding feature The size is K ×512; function The output is The size is K ×512. Therefore, .

[0083] Here, The processing procedure consists of three parts: global guided attention branch processing, selective local attention branch processing, and cross-modal decoding branch processing.

[0084] In the global guided attention branch processing, local visual block features are first processed. Send to the eleventh linear layer Linear_11 ( D, D The first key feature is obtained from ) The size is 256×512; at the same time, the query features will be included. Send to the twelfth linear layer Linear_12 ( D , D The first query feature is obtained from ) The size is K ×512. Then the first key feature. and the first query feature The second bond features were obtained by performing the reshape operation. Second query features The dimensions are 256×8×64 and 256×8×64 respectively. K ×8×64. Next, the second bond feature... Second query features The dimensions are adjusted using the permute operation to obtain the third-key features. and third query features The dimensions are 8×64×256 and 8× K ×64. Furthermore, the third query feature... and third bond features The first attention score is obtained by multiplying the result by a matrix multiplication operation, multiplying it by a scaling factor of 0.125, and then applying a softmax operation. The dimensions are 8× K ×256. Global context features. The first context features are obtained using the reshape operation. The dimensions are 1×8×64; Second context features are obtained using the permute operation. The dimensions are 8×1×64. Next, the second context feature... and third bond features Perform matrix multiplication, and multiply the result by a scaling factor of 0.125 to obtain the first intermediate process feature. The dimensions are 8×1×256. The first intermediate process features... Averaging is performed on the first dimension, and the result is mapped to the [0,1] range using a sigmoid operation to obtain the selective weights. The dimensions are 1×1×256. Selective weights will be applied. With first attention score Multiply to obtain the second attention score. The dimensions are 8× K ×256.

[0085] In selective local attention branching, local visual block features are first processed. Send to the thirteenth linear layer Linear_13 ( D , D The fourth bond feature is obtained from ) The size is 256×512; at the same time, the query features will be included. Send to the fourteenth linear layer Linear_14 ( D , D The fourth query feature is obtained from ) The size is K ×512. Then the fourth bond feature. and the fourth query feature The fifth bond feature was obtained by performing the reshape operation. and the fifth query feature The dimensions are 256×8×64 and 256×8×64 respectively. K ×8×64. Next, the fifth bond feature... and the fifth query feature The sixth-key feature is obtained by adjusting the dimensions using the permute operation. and the sixth query feature The dimensions are 8×64×256 and 8× K ×64. Furthermore, the sixth query feature... and sixth bond features The second intermediate process feature is obtained by multiplying the result of matrix multiplication by a scaling factor of 0.125. The dimensions are 8× K ×256. Characteristics of the second intermediate process. The flatten operation is used to flatten the tensors and sort them in ascending order, and the quantile operation is used to sort them according to the given quantiles. Determine the filtering threshold ,in, The preferred value is 0.4. Use the filtering threshold. Features of the second intermediate process Filter all elements, less than The element is set to 0, and the element greater than 0 is set to 0. The elements remain unchanged, thus obtaining the characteristics of the third intermediate process. The dimensions are 8× K ×256, then the third attention score is obtained after a softmax operation. The dimensions are 8× K ×256.

[0086] In cross-modal decoding branch processing, the second attention score and third attention score The highly relevant attention scores are output by weighting the sums in a 1:6 ratio. The dimensions are 8× K ×256. The strongly correlated attention score. The process attention score is obtained by feeding the data into the attention regularization layer Att_Dropout. The dimensions are 8× K ×256. Local visual block features. Send to the fifteenth linear layer Linear_15 ( D , D The first value feature is obtained from ) The dimensions are 256×512. Then the first value feature... Second-value features are obtained using the reshape operation. The dimensions are 256×8×64. Next, the second value feature... Use the permute operation to adjust the dimensions and obtain the tertiary features. The dimensions are 8×256×64. The process attention score... and third-value features Matrix multiplication is performed to obtain the first cross-modal fusion feature. The dimensions are 8× K ×64, then the first cross-modal fusion features The dimensions are adjusted using the permute operation to obtain the second cross-modal fusion feature. The size is K ×8×64. Next, the second cross-modal fusion features... The third cross-modal fusion feature is obtained using the reshape operation. The size is K ×512. The feedforward_block receives the third cross-modal fusion feature. After passing through the sixteenth linear layer, Linear_16 ( D , D The fourth cross-modal fusion feature was obtained. The size is K ×512; then, the fifth cross-modal fusion feature is obtained by passing it through the third regularization layer Dropout_3. The size is K ×512; Incorporate the fifth cross-modal fusion feature and query features Adding them together yields the sixth cross-modal fusion feature. The size is K ×512; Then through LayerNorm_5, Linear_17 ( D , D), ReLU activation function, Dropout_4, Linear_18 ( D , D The features are processed sequentially using Dropout_5 to obtain the seventh cross-modal fusion feature. The size is K ×512; The seventh cross-modal fusion feature and query features Adding them together yields the eighth cross-modal fusion feature. The size is K ×512; The eighth cross-modal fusion feature Decoding features are obtained after LayerNorm_6. The size is K ×512. (The rest of the text appears to be incomplete and requires further context.) d =1 Substitute The processing steps yield decoded features. ,Will This is called the first decoding feature; combined with the first decoding feature ,Will d =2 Substitute The processing steps yield decoded features. ,Will This is called the second decoding feature. The second decoding feature... and local visual block features transpose Multiplying these together yields the final availability detection heatmap. The size is K ×256, representing the output of the robot's object availability detection network.

[0087] Based on any of the above embodiments, in step 120, the global context features of the image to be detected are extracted, including: The first visual features of the image to be detected are extracted by a self-supervised visual model, and the first visual features are linearly transformed to obtain the first global visual features. The second visual global features of the image to be detected are extracted using an image encoder; The first visual global feature and the second visual global feature are weighted and summed to obtain the global context feature.

[0088] In practice, firstly, a self-supervised visual model is used to extract the first visual features of the image to be detected, and then a linear transformation is performed on the first visual features to obtain the first global visual features. This self-supervised visual model has the ability to learn robust visual representations on unlabeled data and can capture high-quality basic image features.

[0089] Specifically, the input image is fed into a self-supervised visual model, such as the Self-Distillation with No Labels Vision Transformer (DiNO-ViT) model, and the features output from the last layer of this model are extracted as the first visual feature, with a size of 1×768. Then, the sixth linear layer is used to perform a dimensionality transformation on this first visual feature to obtain a first global visual feature with a size of 1×512.

[0090] Then, the second visual global features of the image to be detected are extracted through an image encoder. Here, the image encoder specifically refers to the visual encoding end in the contrastive language image pre-trained model, which has natural multimodal alignment prior knowledge, such as the CLIP image encoder.

[0091] Specifically, the image to be detected is simultaneously input into the image encoder for processing, and the 1×512 feature vector output by the encoder is directly used as the second visual global feature.

[0092] Finally, the first and second visual global features are weighted and summed to obtain the global context features. Specifically, the first visual global features, which contain unsupervised robust representations, and the second visual global features, which contain multimodal alignment priors, are weighted and summed according to a specific weight ratio of 1:0.1 to obtain global context features that combine rich visual details and semantic alignment capabilities.

[0093] The method provided in this invention utilizes a self-supervised model and a multimodal image encoder to extract global features respectively, and then effectively fuses the two through a weighted summation operation to construct global contextual features. This cleverly combines the robustness of fine-grained self-supervised visual features with the strong semantic alignment capability of a pre-trained visual encoder, providing more comprehensive and accurate macro-semantic support for global vision guidance in the subsequent cross-modal decoding process.

[0094] In one embodiment, the image to be detected can be input into an image feature extraction module. The global vision branch of the image feature extraction module then extracts features from the image to obtain global context features. Here, the global vision branch of the image feature extraction module includes a self-supervised vision model DiNO-ViT and a sixth linear layer (Linear_6). D m , D It consists of a DiNO-ViT and a CLIP image encoder. The input image is processed by the DiNO-ViT and CLIP image encoder. Global visual features are extracted from the image to be detected, resulting in the first visual feature and the second global visual feature. The sixth linear layer (Linear_6)D m , D The dimensionality transformation is performed on the first visual feature, which is achieved using the nn.linear function in the torch library.

[0095] The specific data processing flow includes: input image After processing with DiNO-ViT, the global visual features output from the last layer of DiNO-ViT are used as the first visual feature, with a size of 1×768. Then, the sixth linear layer, Linear_6 ( D m , D A linear transformation is performed on the first visual features to obtain the first visual global features. The input image is 1×512 pixels. It is also processed by the CLIP image encoder, and the result is used as the second visual global feature. The size is 1×512. Then, the first visual global features are... Second visual global features The global context features are obtained by summing the weighted values ​​in a 1:0.1 ratio. The dimensions are 1×512.

[0096] Based on any of the above embodiments, in step 120, extracting the local visual block features of the image to be detected includes: Visual block features based on multi-layer output of the image to be detected are extracted by a self-supervised visual model. Each visual block feature is linearly mapped and multiplied by its corresponding learnable weight and then summed to obtain the merged visual block features. The merged visual block features are sequentially processed by normalization, linear mapping, activation, and regularization to obtain the local visual block features.

[0097] In practice, firstly, visual block features based on multi-layer output of the image to be detected are extracted by a self-supervised visual model. Each visual block feature is then linearly mapped, multiplied by its corresponding learnable weight, and summed to obtain the merged visual block features.

[0098] Specifically, the image to be detected is fed into a self-supervised visual model, and the local visual features of the last three layers are extracted, denoted as the first visual block feature, the second visual block feature, and the third visual block feature, each with a size of 256×768. These three features are then processed independently using the first, second, and third linear layers, respectively, and multiplied by the first, second, and third learnable weights automatically optimized by the network. The multiplication results are then summed to obtain the seventh visual block feature with a size of 256×768, i.e., the merged visual block feature. It should be noted that the visual block features based on multi-layer outputs here encompass shallow textures and deep semantics at different network depths; the merged visual block feature is a comprehensive expression resulting from the adaptive aggregation of information from each layer.

[0099] Then, the merged visual block features are processed sequentially by normalization, linear mapping, activation, and regularization to obtain local visual block features.

[0100] Specifically, the merged visual block features obtained above are fed into the local feedforward unit, and in the order of forward propagation, they are sequentially processed by the first normalization layer, the fourth linear layer for dimensionality reduction, GELU activation, the first regularization layer to prevent overfitting, the fifth linear layer, the second regularization layer, and the second normalization layer, and finally output highly refined local visual block features with a size of 256×512.

[0101] The method provided in this invention utilizes a self-supervised visual model multi-layer feature extraction mechanism, combines learnable weights to adaptively aggregate multi-scale visual block features, and further supplements it with local feedforward units containing various nonlinear and regularization processes for deep processing. This greatly enriches the multi-scale expression of local details and texture features, effectively overcomes the limitations of a single network layer in feature capture, and thus achieves accurate availability detection for tiny objects.

[0102] In one embodiment, the image to be detected can be input into an image feature extraction module. The local vision branch within the image feature extraction module extracts local features from the image to be detected, obtaining local visual block features. Here, the local vision branch consists of a self-supervised vision model DiNO-ViT and a first linear layer Linear_1. D m , D m ), second linear layer Linear_2 ( D m , D m ), third linear layer Linear_3 ( D m , D m First learnable weight W 1. Second Learnable Weight W2. Third Learnable Weights W It consists of 3 and a local feedforward unit. In the embodiments of the present invention... D m Preferably 768; the local feedforward unit consists of the first normalized layer LayerNorm_1 and the fourth linear layer Linear_4. D m , D ), GELU activation function, first regularization layer Dropout_1, fifth linear layer Linear_5 ( D , D The second regularization layer Dropout_2 and the second normalization layer LayerNorm_2 are cascaded together, as described in this embodiment of the invention. D The preferred value is 512.

[0103] Among them, the first linear layer Linear_1 ( D m , D m ), second linear layer Linear_2 ( D m , D m ), third linear layer Linear_3 ( D m , D m ), the fourth linear layer Linear_4 ( D m , D ) and the fifth linear layer Linear_5 ( D , D All are implemented using the nn.linear function in the torch library; the first learnable weight W 1. Second Learnable Weight W 2. Third Learnable Weights W 3 is implemented using the nn.Parameter function in the torch library; the first normalization layer LayerNorm_1 and the second normalization layer LayerNorm_2 are implemented using the nn.LayerNorm function in the torch library; the GELU activation function is implemented using the nn.GELU function in the torch library; the first regularization layer Dropout_1 and the second regularization layer Dropout_2 are implemented using the nn.Dropout function in the torch library.

[0104] The specific data processing flow includes: input image The image to be detected is processed by the self-supervised visual model DiNO-ViT. The local visual features output from the last three layers of DiNO-ViT are then recorded as the first visual block features. Second visual block features and third visual block features All dimensions are 256×768. Then, , and After passing through Linear_1( D m , D m ), Linear_2( D m , D m ) and Linear_3( D m , D m The fourth visual block feature is obtained through processing. Fifth visual block features and sixth visual block features All dimensions are 256×768. Next, the fourth visual block features... Fifth visual block features Sixth visual block features Each with the first learnable weight W 1. Second Learnable Weight W 2. Third Learnable Weights W Multiply by 3, and sum the results to obtain the seventh visual block feature. The dimensions are 256×768. Finally, the features of the seventh visual block are received through a local feedforward unit. It passes through the first normalization layer LayerNorm_1 and the fourth linear layer Linear_4 in sequence. D m , D ), GELU activation function, first regularization layer Dropout_1, fifth linear layer Linear_5 ( D , D After processing by the second regularization layer Dropout_2 and the second normalization layer LayerNorm_2, the output is local visual block features. The dimensions are 256×512.

[0105] Based on any of the above embodiments, step 110, obtaining the image of the target object to be detected in the scene where the robot is located, includes: The robot acquires scene images via RGB sensors; The scene image is segmented into objects, and the image to be detected of the target object is obtained from the segmentation result based on the category name of the target object.

[0106] In practice, the first step is to acquire scene images captured by the robot using RGB sensors. Here, RGB sensors refer to visual hardware devices capable of capturing color information in the real environment, while scene images are color digital images that contain the full view of the complex working environment in which the robot is currently located.

[0107] Specifically, a red-green-blue (RGB) sensor mounted on the robot can be used to collect RGB color visual information of the working environment in real time, forming an initial scene image input. In this embodiment, the sensor is preferably a Kinect v2 device.

[0108] Then, object segmentation is performed on the scene image, and the target object's image to be detected is obtained from the segmentation result based on the target object's category name. The object segmentation here aims to accurately isolate individual entities from complex background information in the scene, and the category name serves as a semantic label to guide the robot in locating specific task objects.

[0109] Specifically, a pre-trained high-precision instance segmentation network, such as a mask region-based convolutional neural network (MRCNN) or Mask R-CNN, is used to perform pixel-level object instance segmentation on the entire captured scene image. Then, using the object category name given in the task instructions as a retrieval matching condition, the RGB region image containing only the features of that target object is selected from numerous segmentation mask results and used as a high-quality detection image for subsequent network inference.

[0110] The method provided in this invention uses a high-resolution color hardware sensor to acquire environmental images and performs precise image matting and extraction of targets with specific category names. This effectively eliminates the visual interference caused by the complex and varied background in real open-set environments on availability detection, providing highly focused target features and clean background image data for subsequent feature extraction and cross-modal alignment.

[0111] Based on any of the above embodiments, the number of the plurality of cascaded transformer layers is 12.

[0112] The 12 transformer layers here constitute a lightweight text feature adapter with appropriate depth and a reasonable number of parameters. Specifically, within the lightweight text feature adapter of the text feature extraction module, 12 identical transformer layers are configured in series. After the initial text features are input into this cascaded network, they will be passed forward layer by layer through these 12 layers. In each layer, the features undergo multi-head attention interaction, residual fusion, and non-linear mapping, thereby peeling away the surface semantics layer by layer and extracting the complex action guidance details contained in long texts.

[0113] The method provided in this invention explicitly sets the number of cascaded transformer layers used for self-attention reconstruction and feature refinement to 12. While ensuring that the model has a sufficiently deep parameter expression space to fully explore the implicit semantics of complex open-set instructions, it effectively avoids network bloat, excessive computational overhead, and the risk of deep gradient vanishing caused by too many layers, achieving an excellent balance between the depth of text semantic mining and the computational efficiency of the robot's edge.

[0114] In one embodiment, a robot object availability detection network is provided, comprising an image feature extraction module, a text feature extraction module, and a text-visual feature alignment module. During the training phase of the robot's object availability detection network, the training set in the Seen folder of the AGD20K dataset can be used as training data. The training process employs the SGD optimizer with a learning rate of 0.01. Batch training is used, with a batch size of 32 samples and 20k generations of training. In the image feature extraction module, the self-supervised visual model DiNO-ViT uses dinov2_vitb14 weights to extract first visual block features, second visual block features, third visual block features, and first visual features. The CLIP image encoder in the image feature extraction module uses ViT-B / 16 weights to extract second visual global features. The CLIP text encoder in the text feature extraction module uses ViT-B / 16 weights to extract initial text features. During training, the parameters of DiNO-ViT, the CLIP image encoder, and the CLIP text encoder are frozen, while the parameters of other parts of the robot's object availability detection network are trained normally. The training loss function uses the binary cross-entropy loss function, as shown in the following formula: ; In the formula, This represents the loss function of the robot's object availability detection network during the training phase. N This indicates the number of pixels in the input image. and They represent the first i The pixel corresponds to the first kThe true probability and predicted probability of class availability.

[0115] Therefore, the trained robot object availability detection network can be applied to robot object availability detection methods. In the application phase, the robot object availability detection method specifically includes: First, the robot acquires an RGB image of the scene using its own RGB sensors. Then, it performs object segmentation on the RGB image of the scene, obtaining the RGB image of the object to be detected, i.e., the detection image, based on the category name of the object.

[0116] Then, the RGB image of the object to be detected and the complex text with availability semantics (i.e., availability text) are input into the trained robot's object availability detection network to obtain the availability detection results of the target object output by the object availability detection network.

[0117] The specific process includes: inputting the RGB image of the object to be detected into the image feature extraction module of the trained robot's object availability detection network to extract the global contextual features and local visual block features corresponding to the RGB image of the object to be detected. Additionally, complex text with availability semantics, such as "Carefully pick up thebat," is input into the text feature extraction module of the trained robot's object availability detection network to extract refined text features corresponding to the complex text with availability semantics. Then, the local visual block features, global contextual features, and refined text features are simultaneously input into the text-visual feature alignment module of the trained robot's object availability detection network for processing, outputting a pixel-by-pixel availability detection heatmap as the availability detection result for the target object.

[0118] Based on any of the above embodiments Figure 2 This is a schematic diagram of the object availability detection device for the robot provided by the present invention, as shown below. Figure 2 As shown, the device includes: The acquisition unit 210 acquires the image of the target object to be detected in the scene where the robot is located, as well as the availability text of the target object; The visual feature extraction unit 220 extracts the global context features and local visual block features of the image to be detected; The text feature extraction unit 230 extracts the initial text features of the available text, and, based on multiple cascaded transformer layers, concatenates learnable parameters into the initial text features and performs self-attention reconstruction to obtain refined text features. Decoding unit 240, based on multiple cascaded decoders, performs feature alignment on the local visual block features, the global context features, and the query features to obtain decoded features; wherein, the query features of the first decoder are the refined text features, and the query features of subsequent decoders are the decoded features output by the previous decoder; in each decoder, a selective weight generated by the interaction between the global context features and the local visual block features is used to weight the first attention score between the local visual block features and the query features to obtain a second attention score, and an intermediate process feature between the local visual block features and the query features is filtered and truncated based on a filtering threshold determined by a preset quantile to obtain a third attention score, and the decoded features are obtained based on the second attention score and the third attention score; The detection output unit 250 multiplies the decoded features output by the last decoder with the local visual block features to obtain the availability detection result of the target object.

[0119] The apparatus provided in this invention introduces learnable parameters to perform self-attention reconstruction on the initial features of complex availability text to obtain refined text features. In the decoding stage, selective weights generated by the global context are used in parallel for attention weighting, and a threshold truncation mechanism based on preset quantiles is used to filter weakly correlated local attention. Finally, efficient alignment and decoding of cross-modal features are achieved, realizing high-quality feature extraction and high-precision visual alignment of complex and lengthy text instructions. This significantly improves the accuracy, robustness, and adaptability of robots in small-sample object availability detection when facing semantically dense open-set environment instructions, thereby improving detection performance.

[0120] Based on any of the above embodiments, the text feature extraction unit is specifically used for: The initial text features are used as the input text features for the first transformer layer; For each transformer layer, learnable parameters corresponding to each transformer layer are concatenated into the input text features to obtain concatenated features. The concatenated features are then normalized and processed by multi-head attention, and then added to the concatenated features. After normalization and linear mapping, the concatenated features are added again, and the top few vectors corresponding to the number of text categories in the added results are selected as output text features. The output text features are then used as the input text features for the next transformer layer. The output text features of the last transformer layer are linearly mapped to obtain mapped text features. The mapped text features are then added to the initial text features to obtain the refined text features.

[0121] Based on any of the above embodiments, the decoding unit is specifically used for: After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed, followed by normalization to obtain the first attention score. The global context features and the linearly mapped local visual block features are multiplied by a matrix and scaled to obtain intermediate process features. The intermediate process features are averaged and then mapped to a preset interval to obtain the selective weights. The second attention score is obtained by multiplying the selective weight by the first attention score.

[0122] Based on any of the above embodiments, the decoding unit is further specifically used for: After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed to obtain the intermediate process features; The intermediate process features are flattened and arranged in ascending order, and the filtering threshold is determined according to the preset quantile. Elements in the intermediate process features that are less than the filtering threshold are set to zero to obtain filtered features. The filtered features are then normalized to obtain the third attention score.

[0123] Based on any of the above embodiments, the decoding unit is further specifically used for: The second attention score and the third attention score are weighted and summed, and then regularized to obtain the process attention score. The process attention score is multiplied by the linearly mapped local visual block features to obtain the cross-modal fusion features; The cross-modal fusion features are processed by the forward unit, added to the query features, and normalized to obtain the decoded features.

[0124] Based on any of the above embodiments, the visual feature extraction unit is specifically used for: The first visual features of the image to be detected are extracted by a self-supervised visual model, and the first visual features are linearly transformed to obtain the first global visual features. The second visual global features of the image to be detected are extracted using an image encoder; The first visual global feature and the second visual global feature are weighted and summed to obtain the global context feature.

[0125] Based on any of the above embodiments, the visual feature extraction unit is further specifically used for: Visual block features based on multi-layer output of the image to be detected are extracted by a self-supervised visual model. Each visual block feature is linearly mapped and multiplied by its corresponding learnable weight and then summed to obtain the merged visual block features. The merged visual block features are sequentially processed by normalization, linear mapping, activation, and regularization to obtain the local visual block features.

[0126] Based on any of the above embodiments, the acquisition unit is specifically used for: The robot acquires scene images via RGB sensors; The scene image is segmented into objects, and the image to be detected of the target object is obtained from the segmentation result based on the category name of the target object.

[0127] Based on any of the above embodiments, the number of the plurality of cascaded transformer layers is 12.

[0128] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a robot object availability detection method. This method includes: acquiring a target image of a target object in the robot's scene and availability text of the target object; extracting global context features and local visual block features of the target image; extracting initial text features of the availability text; and, based on multiple cascaded transformer layers, concatenating learnable parameters and performing self-attention reconstruction on the initial text features to obtain refined text features; and, based on multiple cascaded decoders, performing feature alignment on the local visual block features, the global context features, and the query features to obtain decoded features; wherein the query feature of the first decoder is the refined text feature. The text features are modified, and the query features of subsequent decoders are the decoded features output by the previous decoder. In each decoder, the selective weights generated by the interaction between the global context features and the local visual block features are used to weight the first attention score between the local visual block features and the query features to obtain a second attention score. Based on a filtering threshold determined by a preset quantile, the intermediate process features between the local visual block features and the query features are filtered and truncated to obtain a third attention score. Decoding is performed based on the second attention score and the third attention score to obtain the decoded features. The decoded features output by the last decoder are multiplied by the local visual block features to obtain the availability detection result of the target object.

[0129] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the robot object availability detection method provided by the above methods. The method includes: acquiring a target image of a target object in the robot's scene and availability text of the target object; extracting global context features and local visual block features of the target image; extracting initial text features of the availability text; and, based on multiple cascaded transformer layers, concatenating learnable parameters and performing self-attention reconstruction on the initial text features to obtain refined text features; and, based on multiple cascaded decoders, processing the local visual block features, the global context features, and the query... Features are aligned to obtain decoded features; the query feature of the first decoder is the refined text feature, and the query feature of subsequent decoders is the decoded feature output by the previous decoder; in each decoder, a selective weight generated by the interaction between the global context feature and the local visual block feature is used to weight the first attention score between the local visual block feature and the query feature to obtain a second attention score; and, based on a filtering threshold determined by a preset quantile, the intermediate process feature between the local visual block feature and the query feature is filtered and truncated to obtain a third attention score, and decoding is performed based on the second attention score and the third attention score to obtain the decoded feature; the decoded feature output by the last decoder is multiplied by the local visual block feature to obtain the availability detection result of the target object.

[0131] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the robot object availability detection method provided by the methods described above. This method includes: acquiring a target image of a target object in a scene where the robot is located and availability text of the target object; extracting global context features and local visual block features of the target image; extracting initial text features of the availability text; and, based on multiple cascaded transformer layers, concatenating learnable parameters and performing self-attention reconstruction on the initial text features to obtain refined text features; and, based on multiple cascaded decoders, performing feature alignment on the local visual block features, the global context features, and the query features to obtain decoded features. The query feature of the first decoder is the refined text feature, and the query feature of subsequent decoders is the decoded feature output by the previous decoder. In each decoder, a selective weight generated by the interaction between the global context feature and the local visual block feature is used to weight the first attention score between the local visual block feature and the query feature to obtain a second attention score. Based on a filtering threshold determined by a preset quantile, the intermediate process feature between the local visual block feature and the query feature is filtered and truncated to obtain a third attention score. Decoding is performed based on the second attention score and the third attention score to obtain the decoded feature. The decoded feature output by the last decoder is multiplied by the local visual block feature to obtain the availability detection result of the target object.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting object availability in a robot, characterized in that, include: Acquire the target object's image to be detected and the available text of the target object in the scene where the robot is located; Global context features and local visual block features of the image to be detected are extracted; The initial text features of the available text are extracted, and then, based on multiple cascaded transformer layers, learnable parameters are concatenated into the initial text features and self-attention reconstruction is performed to obtain refined text features. Based on multiple cascaded decoders, feature alignment is performed on the local visual block features, the global context features, and the query features to obtain decoded features. The query features of the first decoder are the refined text features, and the query features of subsequent decoders are the decoded features output by the previous decoder. In each decoder, a selective weight generated by the interaction between the global context features and the local visual block features is used to weight the first attention score between the local visual block features and the query features to obtain a second attention score. Furthermore, based on a filtering threshold determined by a preset quantile, the intermediate process features between the local visual block features and the query features are filtered and truncated to obtain a third attention score. Decoding is then performed based on the second attention score and the third attention score to obtain the decoded features. The decoded features output by the last decoder are multiplied by the local visual block features to obtain the availability detection result of the target object; The aforementioned transformer layers, based on multiple cascaded layers, concatenate learnable parameters into the initial text features and perform self-attention reconstruction to obtain refined text features, including: The initial text features are used as the input text features for the first transformer layer; For each transformer layer, learnable parameters corresponding to each transformer layer are concatenated into the input text features to obtain concatenated features. The concatenated features are then normalized and processed with multi-head attention before being added to the concatenated features to obtain the second updated text features. The third updated text features, obtained by normalizing and linearly mapping the second updated text features, are then added to the second updated text features. The top few vectors corresponding to the number of text categories in the addition result are selected as output text features, and the output text features are used as the input text features for the next transformer layer. The output text features of the last transformer layer are linearly mapped to obtain mapped text features. The mapped text features are then added to the initial text features to obtain the refined text features. The selective weights generated by the interaction between the global context features and the local visual block features are used to weight the first attention score between the local visual block features and the query features to obtain a second attention score, including: After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed, followed by normalization to obtain the first attention score. The global context features and the linearly mapped local visual block features are multiplied by a matrix and scaled to obtain intermediate process features. The intermediate process features are averaged and then mapped to a preset interval to obtain the selective weights. The second attention score is obtained by multiplying the selectivity weight by the first attention score; The filtering threshold determined based on a preset quantile is used to filter and truncate the intermediate process features between the local visual block features and the query features to obtain a third attention score, including: After linearly mapping the local visual block features and the query features respectively, matrix multiplication and scaling are performed to obtain the intermediate process features; The intermediate process features are flattened and arranged in ascending order, and the filtering threshold is determined according to the preset quantile. Elements in the intermediate process features that are less than the filtering threshold are set to zero to obtain filtered features. The filtered features are then normalized to obtain the third attention score.

2. The robot object availability detection method according to claim 1, characterized in that, The decoding based on the second attention score and the third attention score to obtain the decoding features includes: The second attention score and the third attention score are weighted and summed, and then regularized to obtain the process attention score. The process attention score is multiplied by the linearly mapped local visual block features to obtain the cross-modal fusion features; The cross-modal fusion features are processed by the forward unit, added to the query features, and normalized to obtain the decoded features.

3. The robot object availability detection method according to claim 1, characterized in that, The global contextual features of the image to be detected are extracted, including: The first visual features of the image to be detected are extracted by a self-supervised visual model, and the first visual features are linearly transformed to obtain the first global visual features. The second visual global features of the image to be detected are extracted using an image encoder; The first visual global feature and the second visual global feature are weighted and summed to obtain the global context feature.

4. The robot object availability detection method according to claim 1, characterized in that, Extracting local visual block features from the image to be detected includes: Visual block features based on multi-layer output of the image to be detected are extracted by a self-supervised visual model. Each visual block feature is linearly mapped and multiplied by its corresponding learnable weight and then summed to obtain the merged visual block features. The merged visual block features are sequentially processed by normalization, linear mapping, activation, and regularization to obtain the local visual block features.

5. The robot object availability detection method according to claim 1, characterized in that, Acquire the target object's image to be detected in the robot's scene, including: The robot acquires scene images via RGB sensors; The scene image is segmented into objects, and the image to be detected of the target object is obtained from the segmentation result based on the category name of the target object.

6. The robot object availability detection method according to claim 1, characterized in that, The number of cascaded transformer layers is 12.

7. An object availability detection device for a robot based on the object availability detection method for a robot according to any one of claims 1 to 6, characterized in that, include: The acquisition unit acquires the image of the target object to be detected in the scene where the robot is located, as well as the availability text of the target object; The visual feature extraction unit extracts the global context features and local visual block features of the image to be detected; The text feature extraction unit extracts the initial text features of the available text, and, based on multiple cascaded transformer layers, concatenates learnable parameters into the initial text features and performs self-attention reconstruction to obtain refined text features. The decoding unit, based on multiple cascaded decoders, aligns the local visual block features, the global context features, and the query features to obtain decoded features. The query features of the first decoder are the refined text features, and the query features of subsequent decoders are the decoded features output by the previous decoder. In each decoder, a selective weight generated by the interaction between the global context features and the local visual block features is used to weight the first attention score between the local visual block features and the query features to obtain a second attention score. Furthermore, based on a filtering threshold determined by a preset quantile, the intermediate process features between the local visual block features and the query features are filtered and truncated to obtain a third attention score. Finally, decoding is performed based on the second attention score and the third attention score to obtain the decoded features. The detection output unit multiplies the decoded features output by the last decoder with the local visual block features to obtain the availability detection result of the target object.

Images