Robust fine-grained hand action detection method based on multi-modal context enhancement

Abstract

This invention provides a robust fine-grained hand action detection method based on multimodal context enhancement, belonging to the field of computer vision technology. The method includes: acquiring global text features, context-enhanced text features, and global visual features; performing cascaded cross-modal fusion on the global visual features, global text features, and context-enhanced text features to generate enhanced cross-modal visual features and enhanced cross-modal text features respectively; generating text-guided queries and context-guided queries in parallel based on the enhanced cross-modal visual features and enhanced cross-modal text features, and concatenating the text-guided queries and context-guided queries to form a cross-modal decoding query. This invention effectively improves the ability to distinguish fine-grained hand actions that are highly similar in both visual and semantic aspects, thereby achieving higher accuracy and stronger robustness in fine-grained hand action recognition and localization.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a robust fine-grained hand motion detection method based on multimodal context enhancement. Background Technology

[0002] Fine-grained hand motion detection aims to accurately identify and spatially locate subtle hand interactions in images or videos, and has wide applications in human-computer interaction, robotics, and other fields. Existing technologies have made some progress in improving detection accuracy, such as by constructing fine-grained datasets or introducing open-vocabulary detection tasks to enhance the model's generalization ability.

[0003] However, existing technologies primarily rely on the semantic information of the actions themselves (usually verbs) for modeling, neglecting crucial contextual information during hand interactions, such as the objects being interacted with, the areas being interacted with, and their spatial relationships. Since many fine-grained hand actions are highly similar in visual appearance and semantic expression (e.g., "cut" vs. "slice down"), relying solely on verb-level semantic information is insufficient to adequately distinguish these easily confused action categories. This leads to misclassification of similar actions by the model, thus limiting its discriminative ability and generalization performance. Therefore, existing technologies suffer from a deficiency in effectively utilizing contextual information, resulting in difficulty in accurately distinguishing and locating visually and semantically highly similar fine-grained hand actions. Summary of the Invention

[0004] This invention provides a robust fine-grained hand motion detection method based on multimodal context enhancement, which addresses the shortcomings of existing technologies that lack effective utilization of interactive context information, making it difficult to accurately distinguish and locate fine-grained hand motions that are highly similar in both vision and semantics.

[0005] In a first aspect, the present invention provides a robust fine-grained hand motion detection method based on multimodal context enhancement, comprising: Obtain global text features, context-enhanced text features, and global visual features; wherein, the global text features represent the category of fine-grained hand movements, and the context-enhanced text features represent interactive context description information related to the fine-grained hand movements; The global visual features, global text features, and context-enhanced text features are cascaded and fused across modally to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. Based on the enhanced cross-modal visual features and enhanced cross-modal text features, text-guided queries and context-guided queries are generated in parallel, and the text-guided queries and context-guided queries are concatenated to form cross-modal decoding queries; The cross-modal decoding query is decoded using a context-guided decoder that includes a context-guided cross-attention mechanism to generate the bounding box and category prediction results of the fine-grained hand movements.

[0006] According to the present invention, a robust fine-grained hand action detection method based on multimodal context enhancement acquires global text features, context-enhanced text features, and global visual features, including: A text encoder is used to encode verb texts of fine-grained hand gesture categories to obtain global text features; Obtain interactive context description information related to fine-grained hand movements, construct context-enhanced text based on preset templates, and extract context-enhanced text features through a text encoder; A visual encoder is used to extract features from the input video frames or images to obtain global visual features.

[0007] According to the present invention, a robust fine-grained hand motion detection method based on multimodal context enhancement obtains interaction context description information related to fine-grained hand motions, constructs context-enhanced text based on a preset template, and extracts context-enhanced text features through a text encoder, including: Based on the contextual description information related to interactive verbs and interactive areas, pre-set templates are combined to generate context-enhanced text; The context-enhanced text features are obtained by extracting features from the context-enhanced text using a text encoder. The context description information includes the interactive objects and the interaction relationships between them.

[0008] According to the present invention, a robust fine-grained hand motion detection method based on multimodal context enhancement performs cascaded cross-modal fusion of global visual features, global text features, and context-enhanced text features through a preset cross-modal fusion module, including: First stage: Perform bidirectional cross-attention fusion between the global visual features and the context-enhanced text features to generate context-enhanced visual features; The second stage involves performing bidirectional cross-attention fusion between the context-enhanced visual features and the global text features to generate enhanced cross-modal visual features and enhanced cross-modal text features.

[0009] According to the present invention, a robust fine-grained hand motion detection method based on multimodal context enhancement is provided, wherein the text-guided query and the context-guided query are generated through a pre-set dual query generation mechanism; The dual query generation mechanism includes a text-guided query generation module and a context-guided query generation module; and both the text-guided query generation module and the context-guided query generation module contain a content part and a position part; the content part uses a randomly initialized static query vector as a learnable parameter, and the position part generates the query position information through a language-guided query selection mechanism.

[0010] According to the present invention, a robust fine-grained hand motion detection method based on multimodal context enhancement generates text-guided queries and context-guided queries in parallel based on the enhanced cross-modal visual features and enhanced cross-modal text features, including: Calculate the similarity between the cross-modal visual features and the cross-modal text features to obtain the similarity calculation result; Based on the similarity calculation results, the cross-modal visual feature most relevant to the text semantics is selected as the candidate feature; Dynamic anchor points are constructed using the candidate features to determine the location information of the text-guided query and the context-guided query, and the text-guided query and the context-guided query are generated.

[0011] According to the present invention, a robust fine-grained hand action detection method based on multimodal context enhancement is provided. The context-guided decoder includes a multi-layer decoding structure. Each layer of the decoding structure includes: a self-attention sublayer, a context-guided cross-attention sublayer, a text cross-attention sublayer, a visual cross-attention sublayer, and a feedforward network sublayer. The context-guided cross-attention sublayer utilizes the context-enhanced text features to guide the cross-modal decoding query.

[0012] According to the present invention, a robust fine-grained hand action detection method based on multimodal context enhancement is provided, wherein the text encoder, the cross-modal fusion module and the context-guided decoder are jointly optimized through an end-to-end training strategy and verified under closed-set and open-vocabulary detection task settings, and verified and evaluated through preset detection performance indicators.

[0013] Secondly, the present invention also provides a robust fine-grained hand motion detection device based on multimodal context enhancement, comprising: The acquisition module is used to acquire global text features, context-enhanced text features, and global visual features; wherein, the global text features represent the category of fine-grained hand movements, and the context-enhanced text features represent the interaction context description information related to the fine-grained hand movements; The fusion module is used to perform cascaded cross-modal fusion of the global visual features, global text features, and context-enhanced text features to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. The processing module is used to generate text-guided queries and context-guided queries in parallel based on the enhanced cross-modal visual features and enhanced cross-modal text features, and to concatenate the text-guided queries and context-guided queries to form a cross-modal decoding query; The detection module is used to decode the cross-modal decoding query using a context-guided decoder that includes a context-guided cross-attention mechanism, so as to generate the bounding box and category prediction results of the fine-grained hand action.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robust fine-grained hand motion detection method based on multimodal context enhancement as described in the first aspect above.

[0015] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robust fine-grained hand motion detection method based on multimodal context enhancement as described in the first aspect above.

[0016] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robust fine-grained hand motion detection method based on multimodal context enhancement as described in the first aspect above.

[0017] Compared with the prior art, the present invention has the following beneficial effects: The robust fine-grained hand action detection method based on multimodal context enhancement provided by this invention introduces dynamic instance-level contextual information to construct discriminative context-enhanced text features, and combines global visual and textual semantic information for cascaded cross-modal fusion to gradually strengthen the discriminative boundary between vision and semantics. At the same time, a dual-query generation mechanism selects the visual features most relevant to the text and context to generate robust cross-modal queries, and a context-guided attention mechanism is used in the decoding stage to focus on key interaction areas, effectively improving the ability to distinguish fine-grained hand actions that are highly similar in vision and semantics, thereby achieving higher accuracy and stronger robustness in fine-grained hand action recognition and localization. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the robust fine-grained hand motion detection method based on multimodal context enhancement provided by the present invention; Figure 2 This is a schematic diagram of the overall module architecture in an embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-modal fusion module in an embodiment of the present invention; Figure 4 This is a schematic diagram of the dual query generation module in an embodiment of the present invention; Figure 5 This is a schematic diagram of the context-guided decoding module in an embodiment of the present invention; Figure 6 This is a schematic diagram of the visualization results under the closed set setting in the experiment of this invention; Figure 7 This is a schematic diagram of the visualization results under the open vocabulary setting in the experiment of this invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0021] Please see Figure 1 This application provides a robust, fine-grained hand action detection method based on multimodal context enhancement. Existing technologies primarily rely on the semantic information of the action itself for modeling, neglecting crucial contextual information during hand interactions. This leads to difficulty in accurately distinguishing visually or semantically similar action categories, such as "cut" and "slice down," limiting the model's discriminative ability. This application aims to solve the above problems by fusing multimodal information and introducing context, thereby improving the accuracy and robustness of detection. The method specifically includes the following steps: Step S101: Obtain global text features, context-enhanced text features, and global visual features; wherein, global text features represent the category of fine-grained hand movements, and context-enhanced text features represent the interactive context description information related to fine-grained hand movements; Step S102: Perform cascaded cross-modal fusion on global visual features, global text features, and context-enhanced text features to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. Step S103: Based on enhanced cross-modal visual features and enhanced cross-modal text features, generate text-guided queries and context-guided queries in parallel, and concatenate the text-guided queries and context-guided queries to form a cross-modal decoding query; Step S104: Use a context-guided decoder that includes a context-guided cross-attention mechanism to decode the cross-modal decoding query to generate fine-grained bounding boxes and category prediction results for hand movements.

[0022] The method described above will now be illustrated with detailed embodiments. This embodiment employs a three-encoder-single-decoder structure, comprising a visual encoder, a text encoder, a dynamic contextual encoder (DCE), a cascaded cross-modal fusion module (CMF), a dual query generator (DQG), and a context-guided decoder (CGD), as follows: Figure 2 As shown, Figure 2 This is a schematic diagram of the overall module architecture in an embodiment of the present invention. The method provided in this application first performs the steps of acquiring global text features, context-enhanced text features, and global visual features. Global text features are used to characterize the core category of fine-grained hand actions, such as the verb "cut"; while context-enhanced text features characterize the interactive context description information related to the action, such as "cut an orange with a knife," providing richer scene details. The purpose of this step is to capture semantic information from different granularities, providing a foundation for subsequently distinguishing similar actions. By simultaneously acquiring category-level and context-level text information, the problem of insufficient information caused by relying solely on single verb-level semantic information in the background technology is solved, providing the model with more discriminative input.

[0023] In one specific implementation, the process of obtaining the above features includes: encoding the verb text of fine-grained hand gesture categories using a text encoder to obtain global text features. , n Indicates the number of text tokens.d This represents the dimension of the token vector; it obtains interaction context description information related to fine-grained hand movements, constructs context-enhanced text based on a preset template, and then extracts context-enhanced text features through a text encoder; simultaneously, it uses a visual encoder to extract features from the input video frames or images to obtain global visual features. , k This represents the number of visual tokens. This separate encoding method ensures that features from different modalities and semantic levels can be extracted independently and effectively, providing a high-quality feature source for subsequent fusion steps. In this way, this scheme transforms abstract textual concepts and concrete visual content into a unified, computable feature representation. In this embodiment, the multimodal input data, consisting of video frames, verb text, and corresponding interactive context descriptions, can be represented as: (Frame, Text, Context). Preferably, the visual encoder uses a Transformer-based visual backbone network (such as SwinTransformer), and the text encoder is a BERT-based language model. Furthermore, in a preferred embodiment, the process of constructing context-enhanced text is further refined. Specifically, based on the contextual description information related to the interactive verb and the interactive area, a preset template is used to combine and generate context-enhanced text. Here, the contextual description information explicitly includes the interactive objects and the interactive relationships between them. For example, the template can be in the form of "verb: contextual description," generating text such as "slice: slice the head of an orange with the blade of a utility knife." This design aims to bind ambiguous action intentions with specific interactive scenarios. By explicitly introducing interactive objects ("utility knife," "orange") and relationships ("slice"), it greatly enriches semantic information, thereby solving the problem in the background technology where similar actions such as "cutting" and "slicing" cannot be distinguished due to semantic ambiguity, and significantly enhancing the semantic expressive power of fine-grained actions.

[0024] For example, to address the problem that existing fine-grained hand action detection methods mainly rely on verb-level text semantics and ignore key contextual information during hand-object interaction, thus making it difficult to distinguish visually and semantically highly similar actions, this embodiment proposes a dynamic context encoding module. This module is used to construct context-enhanced text features for interaction awareness, thereby improving fine-grained semantic expression capabilities. Specifically, the dynamic context encoding module dynamically models contextual information related to the hand interaction area and incorporates it into the text features to enhance the discriminative ability of fine-grained actions. Contextual information includes the interaction object and its relationship information; this type of information has strong discriminative power and can explicitly characterize the differences between similar fine-grained actions.

[0025] In this implementation, the context description set is defined as follows:

[0026] in, Represents a set of context descriptions. Indicates the first c Contextual descriptive information (e.g., “utilityknife body cut slice orange head”, “hand left press on apple middle”, etc.) is used to characterize the interactive semantics of the hand-object interaction process.

[0027] Define the verb text set as:

[0028] in, T Represents a collection of verb texts. Indicates the first v A series of interactive verbs (e.g., “cut slice”, “presson”, “grab”...).

[0029] More specifically, contextual description information can be obtained in the following ways: 1. Obtained directly from the interactive annotation information provided in the dataset; 2. In the absence of annotations, interactive description information is generated using a multimodal large model or a text generation model. In this embodiment, interactive description information provided by the dataset is preferred.

[0030] Furthermore, based on the context description set With verb-level text collections Context-enhanced text sets are constructed using preset prompt templates:

[0031] in, This represents a collection of context-enhanced text, which takes the following structure: . For example: "cut slice : utility knife body cut slice orangehead", "press on : hand left press on apple middle". in, c , v and ce These represent the number of categories for context description, verb text, and context-enhanced text, respectively. Subsequently, the context-enhanced text is input into a text encoder to obtain context-enhanced text features. , i This represents the number of context-enhanced text tokens. Preferably, the text encoder is a language model based on the BERT architecture. The context-enhanced text features constructed in the above manner can introduce discriminative semantic information related to the interaction, enabling the model to effectively suppress irrelevant semantic noise while maintaining semantic integrity, thereby improving its ability to distinguish fine-grained hand movements.

[0032] Compared with existing context-based methods, the DCE module in this embodiment does not rely on global scene semantics or general descriptive information. Instead, it dynamically models contextual information aligned with the corresponding interaction area for each fine-grained hand action category, thereby generating more refined and discriminative text features and significantly improving the ability to recognize easily confused fine-grained actions.

[0033] After acquiring multimodal features, this method then performs cascaded cross-modal fusion of global visual features, global text features, and context-enhanced text features to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. The purpose of this step is to achieve alignment and interaction between different modal information, allowing visual features to "understand" text semantics and text features to "perceive" visual content. Through this fusion, the model can establish a connection between visual patterns and semantic descriptions, overcoming the shortcomings of the previous techniques in separating and effectively combining visual and linguistic information, thus achieving cross-modal feature enhancement.

[0034] In another preferred embodiment, cascaded cross-modal fusion is performed through a pre-defined cross-modal fusion module, which is used to progressively refine the boundaries between visual and semantic representations through multi-stage cross-modal interaction, comprising two stages. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of the cross-modal fusion module in this embodiment of the invention. In the first stage, bidirectional cross-attention fusion is performed between global visual features and context-enhanced text features to generate context-enhanced visual features. The purpose of this stage is to initially filter and enhance the interaction-related parts of the visual features using detailed contextual information. In the second stage, the context-enhanced visual features generated in the first stage are fused with global text features through bidirectional cross-attention fusion to generate the final enhanced cross-modal visual features and enhanced cross-modal text features. This two-stage, coarse-to-fine cascaded strategy gradually strengthens the discriminative boundary between visual and semantic information, avoids interference caused by direct mixing of semantic information at different levels, and achieves more accurate feature alignment and enhancement.

[0035] For example, the cross-modal fusion module includes two cascaded fusion stages: a "visual-context fusion stage" and a "visual-text fusion stage," with each stage containing multiple feature fusion layers to progressively enhance cross-modal feature representation capabilities. In the visual-context fusion stage, contextual features are enhanced into textual features. With global visual features V Cross-modal fusion processing specifically includes, firstly, performing intra-modal modeling on features of different modalities, and secondly, enhancing text features based on context. Apply standard self-attention mechanisms to visual features V A deformable self-attention mechanism is applied to enhance the modeling ability of visual features on local regions. Subsequently, a bidirectional cross-attention mechanism is introduced between the visual features and the context-enhanced text features, including "visual-to-context cross-attention" and "context-to-visual cross-attention." Through the above bidirectional interaction, context-enhanced cross-modal visual features are obtained. and enhanced cross-modal text features The calculation process is as follows:

[0036]

[0037] in, , , These represent the visual query vector and the context text key-value vector, respectively. , , Represents contextual text queries and visual key-value vectors; This is the attention scaling factor. Through this stage of fusion, contextual information is injected into visual features, thereby enhancing the ability to perceive fine-grained differences.

[0038] In the visual-text fusion stage, the context-enhanced visual features obtained in the previous stage are used. With global text features T Further cross-modal fusion is carried out, specifically including, firstly, intra-modal enhancement of the features, and then enhancing the visual features. Apply deformable self-attention mechanism; apply global text features A standard self-attention mechanism is applied; subsequently, the same bidirectional cross-attention strategy as in the first stage is employed, including "visual-to-text cross-attention" and "text-to-visual cross-attention," to obtain secondary enhanced visual features of the text. and enhanced text features The calculation process is as follows:

[0039]

[0040] in, , , These represent the visual query vector and text key-value vector after the first stage of enhancement, respectively. , , This represents the text query and the visual key-value vector after the first stage of enhancement; This serves as an attention scaling factor. Through this stage of fusion, the context-enhanced extended modal visual features are further aligned and fused with the global text representation, thereby propagating discriminative cues and strengthening the semantic boundaries between similar hand gestures.

[0041] After feature fusion, this method generates text-guided queries and context-guided queries in parallel based on enhanced cross-modal visual features and enhanced cross-modal text features. These two types of queries are then concatenated to form a cross-modal decoded query. This step aims to provide the subsequent decoder with precise "search instructions." Text-guided queries focus on regions related to the core verb, while context-guided queries focus on regions related to specific interaction scenarios. By generating and concatenating these two complementary queries in parallel, the semantic ambiguity that may exist in a single query is resolved, enabling the decoder to locate and identify target actions simultaneously at both the category and instance levels, thus achieving more robust query construction.

[0042] Figure 4 This is a schematic diagram of the dual query generation module in an embodiment of the present invention, as shown below. Figure 4 As shown, in one optional implementation, text-guided queries and context-guided queries are generated through a pre-defined dual-query generation mechanism. This mechanism includes a text-guided query generation module and a context-guided query generation module. The two query generation modules are based on enhanced text features, respectively. and context-enhanced text features This generates query vectors that are highly semantically relevant to the input text. Specifically, both modules include content and location components.

[0043] Furthermore, in a preferred embodiment, the process of generating query location information is specified. First, the similarity between cross-modal visual features and cross-modal text features is calculated. Then, based on the similarity calculation results, the cross-modal visual features most relevant to the text semantics are selected as candidate features. Finally, dynamic anchors are constructed using these candidate features to determine the location information for text-guided queries and context-guided queries. This mechanism ensures that the query location is no longer fixed or preset, but dynamically generated based on the current image and text content, adaptively focusing on the most relevant region in the image, greatly improving the accuracy of localization.

[0044] Specifically, the content part uses randomly initialized static query vectors as learnable parameters, responsible for encoding the semantic content of the query. These vectors are used to extract fine-grained action semantic information from textual or contextual semantics, thus forming the semantic representation basis of the query. Preferably, the content query vectors are initialized and trained following a query modeling approach based on the Transformer detection framework.

[0045] The location information is generated through a language-guided query selection mechanism, which includes the following steps: 1. Similarity calculation based on enhanced fusion of cross-modal visual features Corresponding text features ( or ), calculate the similarity between the visual token and the text token to obtain the similarity matrix. ,in, b Indicates batch size, v Indicates the number of visual tokens. t Indicates the number of text tokens; 2. Relevance Aggregation: For each visual token, the maximum similarity value is selected in its corresponding text dimension to obtain the strongest semantic correlation between the visual token and the text, resulting in a relevance vector. ; 3. Candidate feature selection: Based on the relevance vector, select the Top-K visual tokens that are most relevant to the text semantics from the visual features to form a candidate feature set; 4. Dynamic anchor point generation: Based on the candidate feature set, the corresponding spatial index position is selected and a dynamic anchor point box is constructed to provide location information initialization for the query. Through the above language-guided feature selection mechanism, the query can focus on the visual region most relevant to the current semantics, thereby improving the query positioning capability.

[0046] Based on the content and location information mentioned above, text-guided queries are generated respectively. and context-guided queries Among them, text-guided queries represent visual information related to verb semantics, while context-guided queries represent visual information related to the semantics of the interaction context. Further, these two types of queries are concatenated to obtain the final decoded query. This design, which separates content from location, allows the semantics and spatial positioning of queries to be decoupled and optimized separately, improving the flexibility and accuracy of queries.

[0047] Finally, this method utilizes a context-guided decoding module that includes a context-guided cross-attention mechanism to decode the cross-modal decoding query generated in the previous step, in order to generate fine-grained bounding boxes and category prediction results for hand movements, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the context-guided decoding module in an embodiment of the present invention. This step is the final detection stage, and its core lies in using the decoder to "translate" the abstract query into specific detection boxes and category labels. By introducing context-guided cross-attention, the decoder can more intelligently utilize contextual information to assist decision-making, solving the problem of insufficient information in traditional decoders when processing similar targets, thereby achieving high-precision detection output.

[0048] In another optional implementation, the context-guided decoding module includes a multi-layer decoding structure, preferably a six-layer decoding structure. Each layer of the decoding structure includes a self-attention sublayer, a context-guided cross-attention sublayer, a text cross-attention sublayer, a visual cross-attention sublayer, and a feedforward network sublayer. The context-guided cross-attention sublayer utilizes context-enhanced text features to guide cross-modal decoding queries. This multi-layered, multi-information-source design allows the decoding query to fully interact and update information with itself, the context, global text, and visual features at each layer. Through layer-by-layer refinement, the model's ability to focus on relevant interactive regions in the input is ultimately improved, thereby enhancing its ability to distinguish fine-grained actions.

[0049] For example, cross-modal queries Feature updates and fusion are performed sequentially through each sublayer in the order described above. The context-guided cross-attention sublayer is used to introduce context-enhanced text features to guide the decoding query, thereby enhancing the model's ability to focus on relevant input regions. Its calculation process is as follows:

[0050] in, This indicates context-guided cross-attention output. This indicates a cross-modal query output by the dual query generation module. This indicates context-enhanced text features. This indicates a self-attention operation. , These represent the projection matrices of the query, key, and value, respectively. This represents the attention scaling factor. Through the above mechanism, contextual semantic information is directly introduced into the query update process, enabling the decoder to perceive key region features that are highly relevant to the current input semantics.

[0051] In each decoding structure, cross-modal queries are updated in the following order: a self-attention sublayer is used to globally model the query to enhance the internal information interaction of the query; a context-guided cross-attention sublayer is used to introduce context-enhanced text features into the query to achieve semantic guidance; a text cross-attention sublayer is used to fuse global text semantic information into the query; a visual cross-attention sublayer is used to extract query-related regional features from visual features; and a feedforward network is used to perform nonlinear transformation and feature enhancement on the sublayer outputs. Through the synergistic effect of the above multiple sublayers, the query features are refined layer by layer in the semantic and visual spaces.

[0052] Furthermore, in a preferred embodiment, the text encoder, cross-modal fusion module, and context-guided decoding module are all jointly optimized using an end-to-end training strategy. This means that all components of the model are trained as a whole, allowing gradients to propagate freely throughout the network, thereby collaboratively adjusting all parameters. Simultaneously, the model is validated in both closed-set and open-vocabulary detection tasks and evaluated using preset detection performance metrics (such as mAP). This joint optimization and multi-scenario validation strategy ensures that the modules work collaboratively to achieve global optimum and verifies the effectiveness and generalization ability of the method in different application scenarios, solving the suboptimal solution problem that may arise from traditional staged training.

[0053] For example, during the model validation phase, validation was performed under both closed-set and open-vocabulary detection task settings. The experimental results demonstrated superior detection performance and generalization ability, effectively improving the robustness and accuracy of fine-grained hand action detection. In the model evaluation process, mean average precision (mAP) or AP50 was used as the detection performance metric, and the harmonic mean metric was used to balance the detection performance of easily confused and easily distinguishable categories, as well as the basic and unseen categories. AP50 represents the average precision calculated under the condition that the intersection-over-union (IoU) threshold between the predicted and ground truth bounding boxes is 0.5.

[0054] To verify the robustness and generalization ability of the above method, experiments were conducted under two task settings: closed-set fine-grained hand action detection and open-vocabulary fine-grained hand action detection.

[0055] In the closed-set setting, the FHA-Kitchens dataset was used. This dataset contains 130 classes of fine-grained action verbs and 880 classes of action triples. To improve training stability, the original training and validation sets were merged into a new training set, while the test set was retained for evaluation. Simultaneously, 77 classes of valid interaction verbs were selected, resulting in 46,521 training samples and 13,000 validation samples. In the open-vocabulary setting, the FHA-Kitchens OVD dataset was used. This dataset contains 61 classes of fine-grained verbs, divided into 46 basic categories and 15 novel categories. The model was trained only on the basic categories (35,351 samples) and evaluated on a validation set (9,361 samples) containing both basic and novel categories.

[0056] In the closed-set setting, mean precision (mAP) was used as the primary evaluation metric. In the open-vocabulary setting, AP50 was used as the primary evaluation metric, and the performance of both base and novel categories was evaluated simultaneously in the generalized setting. Furthermore, to evaluate the ability to distinguish easily confused actions, verb categories were divided into 38 easily confused categories (CV) and 39 easily distinguishable categories (DV) based on visual and semantic similarity. The harmonic mean (HM) was used to measure the balance of model performance, including the balance between easily confused and easily distinguishable categories. ) and the balance between basic and novel categories ( ):

[0057]

[0058] in, The mAP represents the CV category. Indicates mAP for DV category, AP50 represents the base category. AP50 indicates the novelty category.

[0059] This method is implemented based on the MMDetection framework. The visual encoder uses the Win Transformer (including Win-T, Win-B, and Win-L), and the text encoder uses a pre-trained BERT model. The Adam optimizer is used during training, with an initial learning rate of [missing information]. The weight decays to 1. The learning rate scaling factor was set to 0.1 for both the visual and text encoders. Loss functions included classification loss, L1 bounding box loss, and GIoU loss. The model was trained on four NVIDIA RTX 4090 GPUs for a default of 12 epochs.

[0060] The experimental results under the FHA-Kitchens closed set setting are shown in Table 1.

[0061] Table 1 Comparison results on the FHA-Kitchens validation set under the closed-set action detection setting.

[0062] Experimental results show that, under the Swin-T backbone, this method achieves 66.80% mAP across all categories, a 17.80% improvement over the current best method, Open-FGHA; and a 25.27% improvement over mm-GDINO in easily confused categories. Furthermore, this method... The higher metric values ​​indicate a better performance balance between easily confused and easily distinguishable categories. Furthermore, the method maintains a significant performance advantage even on a larger-scale backbone (Swin-L), demonstrating its good scalability and stability.

[0063] The experimental results of this experiment under the FHA-Kitchens OVD setting are shown in Table 2.

[0064] Table 2 Comparison results on the FHA-Kitchens OVD validation set under the open-vocabulary action detection setting.

[0065] Experimental results show that, under the Swin-T backbone, this method achieves 42.52% AP50 in the novelty category, a 14.02% improvement over existing methods; under the Swin-L backbone, the novelty category performance reaches 45.18%, a 15.50% improvement. Meanwhile, this method also... The results show that it significantly outperforms existing methods, indicating that it has better generalization and balance capabilities between basic and novel categories.

[0066] Figure 6 This is a schematic diagram of the visualization results under the closed-set setting in the experiment of this invention. Actions within the left-hand-object interaction, right-hand-object interaction, and object-object interaction areas are marked with light blue, light purple, and green, respectively. Red boxes and text indicate misclassified categories, while red dashed boxes indicate undetected areas. As shown in the figure, this method can accurately detect left-hand-object, right-hand-object, and object-object interaction areas and effectively distinguish fine-grained actions that are visually or semantically similar.

[0067] Figure 7This is a visualization of the experimental results under an open vocabulary setting. Basic category detection results are represented by blue boxes and text, novel category detection results by pink boxes and text, and incorrect category results by red boxes and text. As shown in the figure, this method can accurately identify and locate novel categories, while effectively distinguishing visually or semantically similar actions (such as "cut in" vs. "peel off", "grab at" vs. "grab up"), outperforming existing methods.

[0068] The above experiments demonstrate that our proposed method significantly outperforms existing methods under both closed-set and open-vocabulary settings. It also shows clear advantages in easily confused fine-grained action recognition, cross-class generalization ability, and performance balance, proving the effectiveness and practical value of the proposed multimodal context enhancement mechanism in fine-grained hand action detection tasks. This invention also provides a robust fine-grained hand motion detection device based on multimodal context enhancement. The following describes the robust fine-grained hand motion detection device based on multimodal context enhancement provided by this invention. The robust fine-grained hand motion detection device based on multimodal context enhancement described below can be referred to in correspondence with the robust fine-grained hand motion detection method based on multimodal context enhancement described above. The device includes: The acquisition module is used to acquire global text features, context-enhanced text features, and global visual features. Among them, global text features represent the category of fine-grained hand movements, and context-enhanced text features represent the interactive context description information related to fine-grained hand movements. The fusion module is used to perform cascaded cross-modal fusion of global visual features, global text features, and context-enhanced text features to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. The processing module is used to generate text-guided queries and context-guided queries in parallel based on enhanced cross-modal visual features and enhanced cross-modal text features, and to concatenate the text-guided queries and context-guided queries to form cross-modal decoded queries; The detection module is used to decode cross-modal decoding queries using a context-guided decoder that includes a context-guided cross-attention mechanism, in order to generate fine-grained bounding boxes and category prediction results for hand movements.

[0069] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, communication interface 802, and memory 803 communicate with each other via the communication bus 804. The processor 801 can call logical instructions in the memory 803 to execute a robust fine-grained hand motion detection method based on multimodal context enhancement. This method includes: Obtain global text features, context-enhanced text features, and global visual features; among them, global text features represent the categories of fine-grained hand movements, and context-enhanced text features represent interactive context description information related to fine-grained hand movements; Cascaded cross-modal fusion of global visual features, global text features, and context-enhanced text features is performed to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. Based on enhanced cross-modal visual features and enhanced cross-modal text features, text-guided queries and context-guided queries are generated in parallel, and the text-guided queries and context-guided queries are concatenated to form cross-modal decoding queries; A context-guided decoder with a context-guided cross-attention mechanism is used to decode cross-modal decoding queries to generate fine-grained bounding boxes and category predictions for hand movements.

[0070] Furthermore, the logical instructions in the aforementioned memory 803 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the robust fine-grained hand motion detection method based on multimodal context enhancement provided by the above methods, the method comprising: Obtain global text features, context-enhanced text features, and global visual features; among them, global text features represent the categories of fine-grained hand movements, and context-enhanced text features represent interactive context description information related to fine-grained hand movements; Cascaded cross-modal fusion of global visual features, global text features, and context-enhanced text features is performed to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. Based on enhanced cross-modal visual features and enhanced cross-modal text features, text-guided queries and context-guided queries are generated in parallel, and the text-guided queries and context-guided queries are concatenated to form cross-modal decoding queries; A context-guided decoder with a context-guided cross-attention mechanism is used to decode cross-modal decoding queries to generate fine-grained bounding boxes and category predictions for hand movements.

[0072] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robust fine-grained hand motion detection method based on multimodal context enhancement provided by the methods described above, the method comprising: Obtain global text features, context-enhanced text features, and global visual features; among them, global text features represent the categories of fine-grained hand movements, and context-enhanced text features represent interactive context description information related to fine-grained hand movements; Cascaded cross-modal fusion of global visual features, global text features, and context-enhanced text features is performed to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. Based on enhanced cross-modal visual features and enhanced cross-modal text features, text-guided queries and context-guided queries are generated in parallel, and the text-guided queries and context-guided queries are concatenated to form cross-modal decoding queries; A context-guided decoder with a context-guided cross-attention mechanism is used to decode cross-modal decoding queries to generate fine-grained bounding boxes and category predictions for hand movements.

[0073] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robust fine-grained hand motion detection method based on multimodal context enhancement, characterized in that, include: Obtain global text features, context-enhanced text features, and global visual features; wherein, the global text features represent the category of fine-grained hand movements, and the context-enhanced text features represent interactive context description information related to the fine-grained hand movements; The global visual features, global text features, and context-enhanced text features are cascaded and fused across modally to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. Based on the enhanced cross-modal visual features and enhanced cross-modal text features, text-guided queries and context-guided queries are generated in parallel, and the text-guided queries and context-guided queries are concatenated to form cross-modal decoding queries; The cross-modal decoding query is decoded using a context-guided decoder that includes a context-guided cross-attention mechanism to generate the bounding box and category prediction results of the fine-grained hand movements.

2. The robust fine-grained hand motion detection method based on multimodal context enhancement according to claim 1, characterized in that, Obtain global text features, context-enhanced text features, and global visual features, including: A text encoder is used to encode verb texts of fine-grained hand gesture categories to obtain global text features; Obtain interactive context description information related to fine-grained hand movements, construct context-enhanced text based on preset templates, and extract context-enhanced text features through a text encoder; A visual encoder is used to extract features from the input video frames or images to obtain global visual features.

3. The robust fine-grained hand motion detection method based on multimodal context enhancement according to claim 2, characterized in that, Obtain interaction context description information related to fine-grained hand movements, construct context-enhanced text based on preset templates, and extract context-enhanced text features through a text encoder, including: Based on the contextual description information related to interactive verbs and interactive areas, pre-set templates are combined to generate context-enhanced text; The context-enhanced text features are obtained by extracting features from the context-enhanced text using a text encoder. The context description information includes the interactive objects and the interaction relationships between them.

4. The robust fine-grained hand motion detection method based on multimodal context enhancement according to claim 2, characterized in that, The cascaded cross-modal fusion of the global visual features, global text features, and context-enhanced text features is performed through a preset cross-modal fusion module, including: First stage: Perform bidirectional cross-attention fusion between the global visual features and the context-enhanced text features to generate context-enhanced visual features; The second stage involves performing bidirectional cross-attention fusion between the context-enhanced visual features and the global text features to generate enhanced cross-modal visual features and enhanced cross-modal text features.

5. The robust fine-grained hand motion detection method based on multimodal context enhancement according to claim 1, characterized in that, The text-guided query and the context-guided query are generated through a pre-defined dual-query generation mechanism; The dual query generation mechanism includes a text-guided query generation module and a context-guided query generation module; and both the text-guided query generation module and the context-guided query generation module contain a content part and a position part; the content part uses a randomly initialized static query vector as a learnable parameter, and the position part generates the query position information through a language-guided query selection mechanism.

6. The robust fine-grained hand motion detection method based on multimodal context enhancement according to claim 5, characterized in that, Based on the enhanced cross-modal visual features and enhanced cross-modal text features, text-guided queries and context-guided queries are generated in parallel, including: Calculate the similarity between the cross-modal visual features and the cross-modal text features to obtain the similarity calculation result; Based on the similarity calculation results, the cross-modal visual feature most relevant to the text semantics is selected as the candidate feature; Dynamic anchor points are constructed using the candidate features to determine the location information of the text-guided query and the context-guided query, and the text-guided query and the context-guided query are generated.

7. The robust fine-grained hand motion detection method based on multimodal context enhancement according to claim 1, characterized in that, The context-guided decoder includes a multi-layer decoding structure, and each layer of the decoding structure includes: a self-attention sublayer, a context-guided cross-attention sublayer, a text cross-attention sublayer, a visual cross-attention sublayer, and a feedforward network sublayer; The context-guided cross-attention sublayer utilizes the context-enhanced text features to guide the cross-modal decoding query.

8. The robust fine-grained hand motion detection method based on multimodal context enhancement according to claim 4, characterized in that, The text encoder, the cross-modal fusion module, and the context-guided decoder are all jointly optimized using an end-to-end training strategy and verified under closed-set and open-vocabulary detection task settings, respectively, and evaluated using preset detection performance indicators.

9. A robust fine-grained hand motion detection device based on multimodal context enhancement, characterized in that, include: The acquisition module is used to acquire global text features, context-enhanced text features, and global visual features; wherein, the global text features represent the category of fine-grained hand movements, and the context-enhanced text features represent the interaction context description information related to the fine-grained hand movements; The fusion module is used to perform cascaded cross-modal fusion of the global visual features, global text features, and context-enhanced text features to generate enhanced cross-modal visual features and enhanced cross-modal text features, respectively. The processing module is used to generate text-guided queries and context-guided queries in parallel based on the enhanced cross-modal visual features and enhanced cross-modal text features, and to concatenate the text-guided queries and context-guided queries to form a cross-modal decoding query; The detection module is used to decode the cross-modal decoding query using a context-guided decoder that includes a context-guided cross-attention mechanism, so as to generate the bounding box and category prediction results of the fine-grained hand action.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the robust fine-grained hand motion detection method based on multimodal context enhancement as described in any one of claims 1 to 8.

Images