A method for character interaction detection based on Mamba architecture

Abstract

This invention discloses a person interaction detection method based on the Mamba architecture, comprising: 1. Global feature encoding: inputting the image to be detected into a global feature encoder for feature extraction to obtain a global visual representation; 2. Multi-view feature aggregation: inputting the global visual representation and a set of queries into two cascaded CEM units to obtain shallow shared semantics and deep shared semantics, while simultaneously realizing information exchange between human and object branches; 3. Human and object branches: obtaining low, medium, high, and comprehensive detection features through a four-stage feature extraction strategy to acquire rich detection information; 4. Interaction branch: constructing a multi-stage, adaptive, dynamically weighted, and comprehensive progressive learning strategy to obtain advanced visual understanding and improve the model's recognition performance on rare and complex interaction categories. This invention can fully extract rich and comprehensive interaction semantics to drive the detection of human-object pairs, thereby helping to discover more instances of person interaction detection.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision, object detection, and human interaction detection, specifically a human interaction detection method based on the Mamba architecture. Background Technology

[0002] Human interaction detection aims to detect person-object pairs interacting in an image and identify the type of interaction between them. Thanks to its importance in scene understanding and relationship reasoning, human interaction detection plays a crucial role in a wide range of applications, including image retrieval, robotics, and visual question answering. Therefore, developing an efficient and accurate human interaction detection method is essential for advancing computer vision technology and promoting computer perception of complex real-world scenes.

[0003] Previous methods for human interaction detection can be divided into two paradigms: those based on convolutional neural networks (CNNs) and those based on the Transformer architecture. Due to the limitations of the local receptive field of the convolutional kernel, CNN-based methods struggle to identify complex interaction categories. However, the Transformer architecture excels in modeling long-range interactions, leading to significant performance improvements in methods based on it, thus making it the mainstream approach for current human interaction detection tasks.

[0004] Existing Transformer-based methods can be categorized into single-branch, two-branch, and three-branch methods based on the number of decoder branches. However, these methods all suffer from several issues: Single-branch methods use a single decoder to handle three sub-tasks simultaneously: human detection, object detection, and interaction classification. While structurally simple, a single decoder struggles to achieve a good performance trade-off among these three sub-tasks. Two-branch methods further decompose human interaction detection into two sub-tasks: instance detection (human detection and object detection) and interaction classification, using two separate decoders for each. Although this method decouples object detection and interaction classification, coupling still exists between the human and object detection sub-tasks, limiting further performance improvements. Three-branch methods completely decouple the human, object, and interaction branches, allowing each branch to explicitly use different parameters for each sub-task. However, the interaction branch lacks effective prior knowledge, resulting in longer training times and slower convergence. Furthermore, the additional decoder branch parameters introduced by the Transformer architecture incur significant training costs, hindering further development of human interaction detection technology.

[0005] At the same time, current methods neglect the need to capture higher-level visual expressions to identify rare and complex interaction categories, resulting in insufficiently comprehensive and rich captured interaction semantics. This limits the model's ability to further discover and identify more instances of human interaction, making it difficult to meet the needs of practical applications. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a human interaction detection method based on the Mamba architecture. This method aims to capture advanced and comprehensive visual semantics to drive the detection of human-object pairs, alleviate the problem of identifying rare and complex interaction categories, and thus help discover more human interaction detection instances and achieve more accurate and efficient human interaction detection.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] The characteristic of this invention's character interaction detection method based on the Mamba architecture is that it is performed according to the following steps:

[0009] Step 1: Obtain the image dataset for human interaction detection and perform preprocessing to obtain the preprocessed image dataset. ,in, Indicates the first Zhang represents the preprocessed human interaction detection image; N represents the total number of human interaction detection images; Indicates the length of the image used for detecting human interaction. This represents the width of the image used for detecting human interaction. This indicates the number of channels in the image used for detecting human interaction.

[0010] Step 2: Construct a human interaction detection network based on the Mamba architecture, including: a global feature encoding module, a multi-view feature aggregation module, a human body branch, an object branch, and an interaction branch; wherein, the global feature encoding module includes: a ResNet-50 model, a feature mapping unit, and a Transformer encoder; the multi-view feature aggregation module includes: two CEM units; the human body branch and the object branch each include: two cascaded LoRA units, one Router unit for dynamic weighting, one detection context propagation module, and a fully connected layer; the interaction branch includes: two cascaded LoRA units, two Router units for dynamic weighting, and a fully connected layer;

[0011] Step 3: The global feature encoding module... Processing is performed to obtain the first... A global visual representation ;

[0012] Step 4: The multi-view feature aggregation module... Processing yields the first... A shallow shared semantic and the Deep shared semantics These are then input together into the human body branch, object branch, and interaction branch for processing, resulting in the output of the human body branch. Human bounding box prediction results and its confidence level , object branch output Object category prediction results Object bounding box prediction results and its confidence level and the output of interactive branches Interaction category prediction results and its confidence level ;

[0013] Step 5, based on , , And the actual results of human body bounding boxes The actual result of the object bounding box Actual results of object categories Interaction Category Real Results Construct the total loss function of the character interaction detection network. This was used to train the character interaction detection network until the total loss function was reached. The process continues until convergence, thus obtaining the optimal human interaction detection model;

[0014] Step Six: During the reasoning process, the first... Zhang's image to be tested The input is processed by the optimal character interaction detection model to obtain... The first in Individual body bounding box prediction results and its detection confidence Object bounding box prediction results and its detection confidence Object category prediction results Interaction category prediction results and its confidence level ; , express The total number of predicted results;

[0015] Will , , After addition, it becomes the first The confidence score of the prediction results of individual interactions is used to obtain... The top few prediction results with high confidence scores are used as... The final prediction result.

[0016] The characteristic of the Mamba-based character interaction detection method described in this invention is that step 3 includes:

[0017] Step 3.1, Inputting the data into a ResNet-50 model for processing yields the... Initial visual features , Indicates the length of visual features, Width representing visual features The number of channels representing visual features;

[0018] Step 3.2: The feature mapping unit uses convolution operations to... The channel dimension is reduced to Then, the reduced visual features are unfolded using a flattening operator to obtain the first... Intermediate visual features ; Indicates the number of channels after the dimension;

[0019] Step 3.3: Encode the location and After being superimposed, the result is input into a Transformer encoder for processing to obtain the [number of layers]. A global visual representation .

[0020] Furthermore, step 4 includes:

[0021] Step 4.1: Randomly generate A query vector to be learned ; and will and The first CEM cell is input for processing to obtain the second cell. A shallow shared semantic ;in, Indicates the number of query vectors; Indicates the number of channels in the query vector;

[0022] Step 4.2, The input is processed in the first LoRA unit of the human body branch to obtain the... Low-level detection features of human body ;

[0023] Will The input is processed in the first LoRA unit of the object branch to obtain the... Low-level detection features of individual objects ;

[0024] Step 4.3, and After superposition, with Together, they are input into the second CEM cell for processing to obtain the... Deep shared semantics ;

[0025] Step 4.4, Feature enhancement processing is performed on the second LoRA unit of the input human branch to obtain the... Intermediate detection features of individual human body ;

[0026] Will Feature enhancement processing is performed in the second LoRA unit of the input object branch to obtain the... Intermediate detection features of individual objects ;

[0027] Step 4.5, and Adaptive dynamic weighting is performed on the Router unit of the input human body branch to obtain the first... Advanced Personal Body Detection Features ;

[0028] Will and Adaptive dynamic weighting is performed on the Router unit in the input object branch to obtain the first... Advanced detection features of individual objects ;

[0029] Step 4.6, the first Router unit in the interaction branch , , Perform adaptive dynamic weighting to generate the first Low-level interaction features ;

[0030] Step 4.7, the first LoRA unit pair in the interaction branch Perform feature enhancement to obtain the first Intermediate interaction features ;

[0031] Step 4.8, the second Router unit in the interaction branch , , , Perform adaptive dynamic weighting to obtain the first... Advanced interactive features ;

[0032] Step 4.9, the second LoRA unit pair in the interaction branch Perform feature enhancement to generate the first A comprehensive interactive feature ;

[0033] Step 4.10, the detection context propagation module in the human body branch... and Processing yields the first... Personal comprehensive physical characteristics ;

[0034] The object branch detection context propagation module for and Processing yields the first... Comprehensive detection features of individual objects ;

[0035] Step 4.11, the fully connected layers in the human body branches Processing is performed to obtain Human bounding box prediction results Detection confidence of human body bounding box , express Number of people;

[0036] The fully connected layer pairs of the object branches Processing is performed to obtain Object bounding box prediction results Detection confidence of object bounding boxes and object category prediction results , This indicates the number of objects detected. This indicates the number of categories of the detected objects;

[0037] The fully connected layer pairs in the interaction branch Processing is performed to obtain Interaction category prediction results and interaction category confidence ; Indicates the number of interactions detected; This indicates the number of categories of interactions detected.

[0038] Furthermore, each CEM unit includes: two self-Mamba blocks, two feature integration blocks, and two cross-Mamba blocks; the first CEM unit in step 4.1 is obtained according to the following process. A shallow shared semantic :

[0039] Step 4.1.1, The feature enhancement is performed on a self-Mamba block input into the first CEM cell, and the second feature enhancement is obtained using equations (1)-(2). Step 1 global visual state quantity and the Step 1 global visual output Thus, the first Step 1 global visual output As the first One-stage global enhancement of visual features :

[0040] (1)

[0041] (2)

[0042] In equations (1)-(2), Indicates the first The first-stage global visual state quantity of the step, when k=1, let = , Indicates the total number of steps; express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix;

[0043] Step 4.1.2, The input is another Mamba block in the first CEM cell for feature enhancement, and the first Mamba block is obtained using equations (3)-(4). The first-stage query state quantity and the The first-stage query output volume Thus, the first The first-stage query output volume As a first-stage enhanced query vector :

[0044] (3)

[0045] (4)

[0046] In equations (3)-(4), Indicates the first The first-stage query state variable of the step, when k=1, let = , express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix;

[0047] Step 4.1.3: The first feature integration block is obtained using equations (5)-(6). A two-stage enhancement of global visual features and two-stage enhanced query vector :

[0048] (5)

[0049] (6)

[0050] In equations (5)-(6), Norm represents a LayNorm layer;

[0051] Step 4.1.4, and Input into a cross Mamba block in the first CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (7)-(8). Three-stage global visual state quantity and the The three-stage global visual output of the step Thus, the first The three-stage global visual output of the step As the first Three-stage enhancement of visual features :

[0052] (7)

[0053] (8)

[0054] In equations (7)-(8), Indicates the first The three-stage global visual state variables of the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (9). express for The influence matrix is ​​obtained from equation (10). express for Influence matrix Indicates hyperparameters;

[0055] (9)

[0056] (10)

[0057] In equations (9)-(10), Indicates a fully connected layer. Indicates hyperparameters;

[0058] Step 4.1.5, and Input into another cross Mamba block in the first CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (11)-(12). The three-stage query state quantity of the step and the The output volume of the three-stage query Thus, the first The output volume of the three-stage query As a three-stage enhanced query vector :

[0059] (11)

[0060] (12)

[0061] In equations (11)-(12), Indicates the first The three-phase query state variable of the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (13). express for The influence matrix is ​​obtained from equation (14). express for The influence matrix;

[0062] (13)

[0063] (14)

[0064] Step 4.1.6: The second feature integration block uses equations (15)-(16) to obtain the first... Four-stage enhancement of global visual features and four-stage enhanced query vector :

[0065] (15)

[0066] (16)

[0067] Step 4.1.7: Use equation (17) to obtain the first... A shallow shared semantic :

[0068] (17)

[0069] In equation (17), This indicates channel merging.

[0070] Furthermore, each detection context propagation module includes: two self-Mamba blocks and two cross-Mamba blocks. The detection context propagation module in the human branch in step 4.10 is obtained according to the following process: Personal comprehensive physical characteristics :

[0071] Step 4.10.1, the detection of human branches, a self-Mamba block in the context propagation module, according to equations (1)-(2) Processing yields the first... The first phase strengthens human detection features ;

[0072] Step 4.10.2, another Mamba block in the human branch detection context propagation module, according to equations (3)-(4), performs... Processing yields the first... Each stage enhances interactive features ;

[0073] Step 4.10.3, the detection of human branches: a cross Mamba block in the context propagation module is used according to equations (7)-(8) to... and To process, in order to Perform adaptive feature enhancement to obtain the first Two-stage enhanced human detection features ;

[0074] Step 4.10.4, another cross Mamba block in the human branch detection context propagation module, according to equations (11)-(12), performs... as well as and The product obtained by multiplication Interactive human detection features To process, in order to Perform adaptive feature enhancement to obtain the first Three-stage enhanced human detection features ;

[0075] Step 4.10.5, and Add them together to obtain the comprehensive human body detection characteristics. .

[0076] Furthermore, in step five, the total loss function is constructed using equation (16). :

[0077] (16)

[0078] In equation (16), , , and There are 4 hyperparameters. The GIOU loss represents the human body bounding box. Indicates the L1 loss of the human body bounding box; The GIOU loss represents the object bounding box. Represents the L1 loss of the object's bounding box; Represents the cross-entropy loss of object categories. This represents the Focal loss for the interaction category.

[0079] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor in executing any of the human interaction detection methods, and the processor is configured to execute the program stored in the memory.

[0080] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the human interaction detection method.

[0081] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0082] 1. This invention utilizes the Mamba architecture to drive the human interaction detection task. With a small number of parameters, it achieves excellent end-to-end detection performance and exhibits better recognition performance on various datasets compared to existing methods. Experimental results show that the proposed method outperforms state-of-the-art methods on both the HICO-DET and V-COCO datasets.

[0083] 2. This invention designs a decoder based on the Mamba architecture, effectively absorbing the advantages of existing single-branch, dual-branch, and triple-branch methods while alleviating common problems in previous approaches. Regarding parameter count and inference speed, each branch is designed using cascaded low-rank adaptation, maintaining the simplicity of the single-branch method. Human and object features provide good priors for the interaction branch, inheriting the fast convergence advantage of the dual-branch method. By dividing the human interaction detection task into three sub-tasks—human detection, object detection, and interaction classification—the advantage of complete decoupling in the triple-branch method is absorbed. Furthermore, this invention sets independent branches for each sub-task to explicitly isolate different parameters and trains specific Router units for each sub-task to dynamically allocate weight combinations, achieving implicit gradient separation. This effectively alleviates conflicts between sub-tasks from both explicit and implicit perspectives.

[0084] 3. This invention designs a novel comprehensive progressive learning strategy to facilitate the recognition of rare and complex interaction categories. By designing CEM units and a detection context propagation module to extract more expressive feature representations, the detection performance of the model is effectively improved. Attached Figure Description

[0085] Figure 1 This is the overall flowchart of the present invention;

[0086] Figure 2 This is a structural diagram of the Mamba block, the cross Mamba block, the CEM unit, and the detection context propagation module of the present invention. Detailed Implementation

[0087] In this embodiment, the overall process of a character interaction detection method based on the Mamba architecture is as follows: Figure 1 Specifically, it is done in the following steps:

[0088] Step 1: Obtain the image dataset for human interaction detection and perform preprocessing to obtain the preprocessed image dataset. ,in, Indicates the first Zhang represents the preprocessed human interaction detection image; N represents the total number of human interaction detection images; Indicates the length of the image used for detecting human interaction. This represents the width of the image used for detecting human interaction. This indicates the number of channels in the image used for detecting human interaction.

[0089] Step 2, as follows Figure 1 As shown, a human interaction detection network based on the Mamba architecture is constructed, including: a global feature encoding module, a multi-view feature aggregation module, a human branch, an object branch, and an interaction branch. The global feature encoding module includes: a ResNet-50 model, feature mapping units, and a Transformer encoder; the multi-view feature aggregation module includes: two CEM units; both the human and object branches include: two cascaded LoRA units, one Router unit for dynamic weighting, one detection context propagation module, and a fully connected layer; the interaction branch includes: two cascaded LoRA units, two Router units for dynamic weighting, and a fully connected layer.

[0090] Step 3: The global feature encoding module... Processing yields the first... A global visual representation ;

[0091] Step 3.1, Inputting the data into a ResNet-50 model for processing yields the... Initial visual features , Indicates the length of visual features, Width representing visual features The number of channels representing visual features;

[0092] Step 3.2: The feature mapping unit uses convolution operations to... The channel dimension is reduced to Then, the reduced visual features are unfolded using a flattening operator to obtain the first... Intermediate visual features ; The dimension represents the number of channels. In this embodiment, the global feature encoding module uses a DETR model pre-trained on the MS-COCO dataset (Nicolas Carion, et al. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213–229. Springer, 2020) to initialize the model parameters. Take 256.

[0093] Step 3.3: Encode the location and After being superimposed, the result is input into a Transformer encoder for processing to obtain the [number of layers]. A global visual representation .

[0094] Step 4: Multi-view feature aggregation module Processing is performed to obtain the first... A shallow shared semantic and the Deep shared semantics These are then input together into the human body branch, object branch, and interaction branch for processing, resulting in the corresponding output of the human body branch. Human bounding box prediction results and its confidence level , object branch output Object category prediction results Object bounding box prediction results and its confidence level and the output of interactive branches Interaction category prediction results and its confidence level .

[0095] Step 4.1: Randomly generate A query vector to be learned ; and will and The first CEM cell is input for processing to obtain the second cell. A shallow shared semantic ;in, Indicates the number of query vectors; This represents the number of channels in the query vector. In this embodiment, and Taking 64 and 256 respectively, the first CEM cell includes: two self-Mamba blocks, two feature integration blocks, and two cross-Mamba blocks. The first CEM cell is obtained as follows: A shallow shared semantic ,like Figure 2 As shown:

[0096] Step 4.1.1, The feature enhancement is performed on a self-Mamba block input into the first CEM cell, and the second feature enhancement is obtained using equations (1)-(2). Step 1 global visual state quantity and the Step 1 global visual output Thus, the first Step 1 global visual output As the first One-stage global enhancement of visual features :

[0097] (1)

[0098] (2)

[0099] In equations (1)-(2), Indicates the first The first-stage global visual state quantity of the step, when k=1, let = , Indicates the total number of steps; express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0100] Step 4.1.2, The input is another Mamba block in the first CEM cell for feature enhancement, and the first Mamba block is obtained using equations (3)-(4). The first-stage query state quantity and the The first-stage query output volume Thus, the first The first-stage query output volume As a first-stage enhanced query vector :

[0101] (3)

[0102] (4)

[0103] In equations (3)-(4), Indicates the first The first-stage query state variable of the step, when k=1, let = , express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0104] Step 4.1.3: The first feature integration block of the first CEM unit is obtained using equations (5)-(6). A two-stage enhancement of global visual features and two-stage enhanced query vector :

[0105] (5)

[0106] (6)

[0107] In equations (5)-(6), Norm represents a LayNorm layer.

[0108] Step 4.1.4, and Input into a cross Mamba block in the first CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (7)-(8). Three-stage global visual state quantity and the The three-stage global visual output of the step Thus, the first The three-stage global visual output of the step As the first Three-stage enhancement of visual features :

[0109] (7)

[0110] (8)

[0111] In equations (7)-(8), Indicates the first The three-stage global visual state variables of the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (9). express for The influence matrix is ​​obtained from equation (10). express for Influence matrix In this embodiment, hyperparameters are represented. Take 0.75.

[0112] (9)

[0113] (10)

[0114] In equations (9)-(10), Indicates a fully connected layer. In this embodiment, hyperparameters are used to represent hyperparameters. Take 0.50.

[0115] Step 4.1.5, and Input into another cross Mamba block in the first CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (11)-(12). The three-stage query state quantity of the step and the The output volume of the three-stage query Thus, the first The output volume of the three-stage query As a three-stage enhanced query vector :

[0116] (11)

[0117] (12)

[0118] In equations (11)-(12), Indicates the first The three-phase query state variable of the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (13). express for The influence matrix is ​​obtained from equation (14). express for The influence matrix.

[0119] (13)

[0120] (14)

[0121] Step 4.1.6: The second feature integration block of the first CEM unit is obtained using equations (15)-(16). Four-stage enhancement of global visual features and four-stage enhanced query vector :

[0122] (15)

[0123] (16)

[0124] Step 4.1.7: Use equation (17) to obtain the first... A shallow shared semantic :

[0125] (17)

[0126] In equation (17), This indicates channel merging.

[0127] Step 4.2, The input is processed in the first LoRA unit of the human body branch to obtain the... Low-level detection features of human body ;

[0128] Will The input is processed in the first LoRA unit of the object branch to obtain the... Low-level detection features of individual objects ;

[0129] Step 4.3, and After superposition, with Together, they are input into the second CEM unit for processing, enabling the exchange of contextual information between people and object branches, as well as shallow shared semantics. The enhancement yields the first Deep shared semantics The second CEM unit comprises: two self-Mamba blocks, two feature integration blocks, and two cross-Mamba blocks. The second CEM unit is obtained as follows: Deep shared semantics .

[0130] Step 4.3.1, The input is a self-Mamba block in the second CEM cell for feature enhancement, and the first feature is obtained using equations (18)-(19). The first-stage shared semantic state quantity and the The first-stage shared semantic output of the step Thus, the first The first-stage shared semantic output of the step As the first One-stage shared semantics :

[0131] (18)

[0132] (19)

[0133] In equations (18)-(19), Indicates the first The first-stage shared semantic state of the step, when k=1, let = express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0134] Step 4.3.2, will and Superposition yields the first An interactive character detects features. The input is fed into another Mamba block in the second CEM cell for feature enhancement, and the first feature is obtained using equations (20)-(21). The first stage of the interaction between the character and the state quantity and the The first stage of the interaction between the character and the output volume Thus, the first The first stage of the interaction between the character and the output volume As the first Each stage of interaction involves the detection of features. :

[0135] (20)

[0136] (twenty one)

[0137] In equations (20)-(21), Indicates the first In the first phase of the interaction between the characters, the state variables are defined. When k=1, let... = , express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0138] Step 4.3.3: The first feature integration block of the second CEM unit is obtained using equations (22)-(23). Two-stage interactive characters for detecting features and the Two-stage shared semantics :

[0139] (twenty two)

[0140] (twenty three)

[0141] Step 4.3.4, will and Input into a cross Mamba block in the second CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (24)-(25). Three-phase shared semantic state of the step and the Three-stage shared semantic output of the step Thus, the first Three-stage shared semantic output of the step As the first Three-stage shared semantics :

[0142] (twenty four)

[0143] (25)

[0144] In equations (24)-(25), Indicates the first The three-phase shared semantic state of the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (26). express for The influence matrix is ​​obtained from equation (27). express for Influence matrix This represents hyperparameters.

[0145] (26)

[0146] (27)

[0147] Step 4.3.5, will and Input into another cross Mamba block in the second CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (28)-(29). The three-stage interactive character interaction and the The three-stage interactive character output of the step Thus, the first The three-stage interactive character output of the step As a three-stage interactive figure, the detection features :

[0148] (28)

[0149] (29)

[0150] In equations (28)-(29), Indicates the first The three-stage interaction between the characters in the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (30). express for The influence matrix is ​​obtained from equation (31). express for The influence matrix.

[0151] (30)

[0152] (31)

[0153] Step 4.3.6: The second feature integration block of the second CEM unit is obtained using equations (32)-(33). Four-stage shared semantics and the A four-stage interactive character for detecting features. :

[0154] (32)

[0155] (33)

[0156] Step 4.3.7: Use equation (34) to obtain the first... Deep shared semantics :

[0157] (34)

[0158] Step 4.4, Feature enhancement processing is performed on the second LoRA unit of the input human branch to obtain the... Intermediate detection features of individual human body ;

[0159] Will Feature enhancement processing is performed in the second LoRA unit of the input object branch to obtain the... Intermediate detection features of individual objects .

[0160] Step 4.5, and Adaptive dynamic weighting is performed on the Router unit of the input human body branch to obtain the first... Advanced Personal Body Detection Features ;

[0161] Will and Adaptive dynamic weighting is performed on the Router unit in the input object branch to obtain the first... Advanced detection features of individual objects .

[0162] Step 4.6: Improving the performance of the human interaction detection algorithm on rare and complex interaction categories is crucial for practical applications. Inspired by the fact that the human brain requires a gradual process to learn complex things, a comprehensive progressive learning strategy for interaction classification is constructed. This strategy acquires low-level, mid-level, high-level, and comprehensive interaction features in four stages to achieve a higher level of interaction semantic understanding. The first Router unit in the interaction branch... , , Perform adaptive dynamic weighting to generate the first Low-level interaction features .

[0163] Step 4.7, the first LoRA unit pair in the interaction branch Perform feature enhancement to obtain the first Intermediate interaction features ;

[0164] Step 4.8, the second Router unit in the interaction branch , , , Perform adaptive dynamic weighting to obtain the first... Advanced interactive features ;

[0165] Step 4.9, the second LoRA unit pair in the interaction branch Perform feature enhancement to generate the first A comprehensive interactive feature .

[0166] Step 4.10: In order to pass information that is helpful for human detection from the interaction branch to the human branch, the detection context propagation module in the human branch... and Processing is performed to obtain the first... Personal comprehensive physical characteristics ;

[0167] In order to pass information that is beneficial for object detection from the interaction branch to the object branch, the object branch's detection context propagation module... and Processing is performed to obtain the first... Comprehensive detection features of individual objects ;

[0168] Each detection context propagation module includes: two self-Mamba blocks and two cross-Mamba blocks. The detection context propagation module in the human branch is obtained as follows: Personal comprehensive physical characteristics :

[0169] Step 4.10.1, The input to the human branch is processed by a Mamba block for feature enhancement, and the first feature is obtained using equations (35)-(36). Step 1: Human body detection status measurement and the Step 1 human body detection output Thus, the first Step 1 human body detection output As the first The first phase strengthens human detection features :

[0170] (35)

[0171] (36)

[0172] In equations (35)-(36), Indicates the first The first-stage human detection state quantity of step, when k=1, let = . express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0173] Step 4.10.2, will The input to the human branch detection context propagation module is further enhanced by another Mamba block, and the first feature is obtained using equations (37)-(38). Step 1 human interaction state quantity and the Step 1 human interaction output Thus, the first Step 1 human interaction output As the first The first stage enhances human interaction features. :

[0174] (37)

[0175] (38)

[0176] In equations (37)-(38), Indicates the first The first-stage human interaction state quantity of the step, when k=1, let = , express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0177] Step 4.10.3, will and A cross-Mamba block in the context propagation module for detecting human branches. Adaptive feature enhancement is performed, and the first feature is obtained using equations (39)-(40). Step Two-stage human body detection status measurement and the Step 2 human body detection output Thus, the first Step 2 human body detection output As the first Two-stage enhanced human detection features :

[0178] (39)

[0179] (40)

[0180] In equations (39)-(40), Indicates the first The second-stage human detection state quantity of the step, when k=1, let = express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (41). express for The influence matrix is ​​obtained from equation (42). express for The influence matrix.

[0181] (41)

[0182] (42)

[0183] Step 4.10.4, will and Multiply to obtain interactive human detection features ,Will and Another cross-Mamba block in the context propagation module for detecting human branches is for... To perform guided feature enhancement, use equations (43)-(44) to obtain the first... Three-stage human body detection status measurement and the The output of the three-stage human body detection step Thus, the first The output of the three-stage human body detection step As the first Three-stage enhanced human detection features :

[0184] (43)

[0185] (44)

[0186] In equation (43)-(44), Indicates the first The three-stage human detection state variables of the step, when k=1, let = express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (45). express for The influence matrix is ​​obtained from equation (46). express for The influence matrix.

[0187] (45)

[0188] (46)

[0189] Step 4.10.5, and Add them together to get the first one. Personal comprehensive physical characteristics ;

[0190] The detection context propagation module in the object branch obtains the first result as follows: Comprehensive detection features of individual objects :

[0191] Step 4.10.6, will The input to the object branch is enhanced by a Mamba block, and the first feature is obtained using equations (47)-(48). The first-stage object detection state quantity and the The output of the first-stage object detection step Thus, the first The output of the first-stage object detection step As the first One-stage enhanced object detection features :

[0192] (47)

[0193] (48)

[0194] In equations (47)-(48), Indicates the first The first-stage object detection state variable of the step, when k=1, let = . express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0195] Step 4.10.7, will The input to the object branch is processed by another Mamba block for feature enhancement, and the first feature is obtained using equations (49)-(50). The first-stage object interaction state quantity and the Step-1 object interaction output Thus, the first Step-1 object interaction output As the first One stage enhances object interaction features :

[0196] (49)

[0197] (50)

[0198] In equations (49)-(50), Indicates the first The first-stage object interaction state variables, when k=1, let = , express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix.

[0199] Step 4.10.8, will and A cross-Mamba block in the context propagation module for detecting input object branches. Adaptive feature enhancement is performed, and the first feature is obtained using equations (51)-(52). Two-stage object detection state variables and the The output of the two-stage object detection step Thus, the first The output of the two-stage object detection step As the first Two-stage enhanced object detection features :

[0200] (51)

[0201] (52)

[0202] In equations (51)-(52), Indicates the first The two-stage object detection state variables of the step, when k=1, let = express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (19). express for The influence matrix is ​​obtained from equation (20). express for The influence matrix;

[0203] (53)

[0204] (54)

[0205] Step 4.10.9, will and Multiply to obtain interactively perceived object detection features. ,Will and Another cross-Mamba block in the context propagation module for detecting input object branches. To perform guided feature enhancement, use equations (55)-(56) to obtain the first... Three-stage object detection state variables and the The output of the three-stage object detection step Thus, the first The output of the three-stage object detection step As the first Three-stage enhanced object detection features :

[0206] (55)

[0207] (56)

[0208] In equation (55)-(56), Indicates the first The three-stage object detection state variables of the step, when k=1, let = express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (57). express for The influence matrix is ​​obtained from equation (58). express for The influence matrix;

[0209] (57)

[0210] (58)

[0211] Step 4.10.10, will and Add them together to get the first one. Comprehensive detection features of individual objects .

[0212] Step 4.11, Fully Connected Layers in Human Body Branches Processing is performed to obtain Human bounding box prediction results Detection confidence of human body bounding box , express Number of people;

[0213] Fully connected layer pairs of object branches Processing is performed to obtain Object bounding box prediction results Detection confidence of object bounding boxes and object category prediction results , This indicates the number of objects detected. This indicates the number of categories of the detected objects;

[0214] Fully connected layer pairs in interactive branches Processing is performed to obtain Interaction category prediction results and interaction category confidence ; Indicates the number of interactions detected; This indicates the number of categories of interactions detected.

[0215] Step 5, based on , , And the actual results of human body bounding boxes The actual result of the object bounding box Actual results of object categories Interaction Category Real Results Construct the total loss function of the character interaction detection network. This was used to train the character interaction detection network until the total loss function was reached. The process continues until convergence, thus obtaining the optimal human interaction detection model and the total loss function. It is established according to the following steps:

[0216] Step 5.1, based on and The GIOU loss of the human detection box is constructed using equation (59). :

[0217] (59)

[0218] In equation (59), This represents the area of ​​intersection between the predicted human body bounding box and the actual human body bounding box. This represents the area of ​​the union between the predicted human bounding box and the actual human bounding box. This represents the area of ​​the smallest rectangle that contains both the predicted and actual human bounding box results.

[0219] Step 5.2, based on and The L1 loss of the human detection box is constructed using equation (60). :

[0220] (60)

[0221] In equation (60), , , , They represent The x-coordinate of the center point, the y-coordinate of the center point, the width of the human bounding box prediction result, and the height of the human bounding box prediction result; , , , They represent The x-coordinate of the center point, the y-coordinate of the center point, the width of the actual human bounding box, and the height of the actual human bounding box;

[0222] Step 5.3, based on and The GIOU loss of the object detection box is constructed using equation (61). :

[0223] (61)

[0224] In equation (61), This represents the area of ​​intersection between the predicted and actual bounding boxes of an object. This represents the area of ​​the union between the predicted and actual bounding boxes of an object. This represents the area of ​​the smallest rectangle that contains both the predicted and actual bounding box results of the object.

[0225] Step 5.4, based on and The L1 loss of the object detection box is constructed using equation (62). :

[0226] (62)

[0227] In equation (62), , , , They represent The x-coordinate of the center point, the y-coordinate of the center point, the width of the predicted object bounding box, and the height of the predicted object bounding box. , , , They represent The x-coordinate of the center point, the y-coordinate of the center point, the width of the actual bounding box of the object, and the height of the actual bounding box of the object.

[0228] Step 5.5, based on and The cross-entropy loss of object categories is constructed using equation (63). :

[0229] (1- (63)

[0230] In equation (63), Indicated by A logarithmic function with base 0.

[0231] Step 5.6, based on and The Focal loss of the interaction category is constructed using equation (64). :

[0232] (64)

[0233] Step 5.7: Construct the total loss function using equation (65). :

[0234] (65)

[0235] In equation (65), , , and There are 4 hyperparameters, in this embodiment, , , and Take values ​​of 3, 1, 1.25, and 1 respectively.

[0236] Step Six: During the reasoning process, the first... Zhang's image to be tested The input is processed in the optimal human interaction detection model to obtain... The first in Individual body bounding box prediction results and its detection confidence Object bounding box prediction results and its detection confidence Object category prediction results Interaction category prediction results and its confidence level ; , express The total number of predicted results;

[0237] Will , , After addition, it becomes the first The confidence score of the prediction results of individual interactions is used to obtain... The top few prediction results with high confidence scores are used as... The final prediction result.

[0238] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0239] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0240] Example

[0241] To verify the effectiveness of the HOIMamba method of this invention, this embodiment uses the common HICO-DET dataset (Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. workshop on applications of computer vision, 2017.) and the V-COCO dataset (Saurabh Gupta and Jitendra Malik. Visualsemantic role labeling. arXiv: Computer Vision and Pattern Recognition, 2015.) for training and testing, and compares it with state-of-the-art methods based on the Transformer architecture, namely QPIC (Tamura, M.; Ohashi, H.; and Yoshinaga, T. 2021. QPIC: Query-based pairwise human-object interaction detection with image-wide contextual information. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 10410–10419.) and UPT (Zhang, FZ; Campbell, D.; andGould, S. 2022. Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, 20104–20112.), MUREN(Kim, S.; Jung, D.; and Cho, M. 2023. Relational context learning for human-object interaction detection.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition, 2925–2934.), DPAD (Gao, J.; Liang,K.; Wei, T.; Chen, W.; Ma, Z.; and Guo, J. 2024. Dual-Prior AugmentedDecoding Network for Long Tail Distribution in HOI Detection. In Proceedingsof the AAAI Conference on Artificial Intelligence, volume 38, 1806–1814.),CDN (Zhang, A.; Liao, Y.; Liu, S.; Lu, M.; Wang, Y.; Gao, C.; and Li, X.2021. Mining the benefits of two-stage and one-stage hoi detection. Advancesin Neural Information Processing Systems, 34: 17209–17220.), GEN-VLKT (Liao,Y.; Zhang, A.; Lu, M.; Wang, Y.; Li, X.; and Liu, S. 2022. Gen-vlkt: Simplifyassociation and enhance interaction understanding for hoi detection. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, 20123–20132.), RmLR (Cao, Y.; Tang, Q.; Yang, F.; Su, X.; You,S.; Lu, X.; and Xu, C. 2023. Re-mine, learn and reason: Exploring thecrossmodal semantic correlations for language-guided hoi detection.InProceedings of the IEEE / CVF International Conference on Computer Vision,23492–23503.), SCTC (Jiang, W.; Ren, W.; Tian, J.; Qu, L.; Wang, Z.; and Liu,H. 2024. Exploring Self-and Cross-Triplet Correlations for Human-ObjectInteraction Detection. In Proceedings of the AAAI Conference on ArtificialIntelligence, volume 38, 2543–2551.), FGAHOI (Ma, S.; Wang, Y.; Wang, S.; andWei, Y. 2023. Fgahoi: Fine-grained anchors for human-object interactiondetection. IEEE Transactions on Pattern Analysis and Machine Intelligence.),PViC (Zhang, F. Z.; Yuan, Y.; Campbell, D.; Zhong, Z.; and Gould, S. 2023.Exploring predicate visual context in detecting of human-object interactions.In Proceedings of the IEEE / CVF International Conference on Computer Vision,10411–10421.), MP-HOI (Yang, J.; Li, B.; Zeng, A.; Zhang, L.; and Zhang, R.2024a. Open-World Human-Object Interaction Detection via Multimodal Prompts.In Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, 1695416964.), HybHOI (Wu, EZ; Li, Y.; Wang, Y.; and Wang, S. 2024. Exploring Pose-Aware Human-Object Interaction via Hybrid Learning. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition, 17815–17825.). This invention uses mean average precision (mAP) as the evaluation metric and replaces the ResNet-50 model in the global feature encoding module with a ResNet-101 model and a Swin-Large model (Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 10012–10022.) to verify the scalability of the HOIMamba method of this invention.

[0242] Table 1

[0243]

[0244] Experimental results show that, under various settings on both datasets, the method of this invention outperforms other methods, thus proving the feasibility of the proposed method. In particular, it significantly surpasses previous methods in the rare category setting of the HICO-DET dataset, demonstrating the rationality of the designed decoupled progressive learning. Experiments demonstrate that the proposed method can effectively handle the recognition of rare and complex interaction categories, thereby extracting high-level interaction feature information and completing the human interaction detection task.

[0245] To verify the efficiency of the method of this invention, this example compares the proposed method HOIMamba with QPIC (Tamura, M.; Ohashi, H.; and Yoshinaga, T. 2021. QPIC: Query-based pairwise human-object interaction detection with image-wide contextual information. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 10410–10419.), AS-Net (Chen, M.; Liao, Y.; Liu, S.; Chen, Z.; Wang, F.; and Qian, C. 2021. Reformulating hoi detection as adaptive setprediction. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 9004–9013.), and MUREN (Kim, S.; Jung, D.; and Cho, M. 2023. Relational context learning for human-object interaction detection). A comparison of parameter count and computational complexity was conducted in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2925–2934. The experimental results are shown in Table 2.

[0246] Table 2

[0247]

[0248] Experimental results show that the method of this invention outperforms previous methods in terms of both parameter count and computational complexity, thus proving the feasibility of the proposed method. Experimental results also demonstrate that the proposed method can capture advanced and comprehensive visual semantics, efficiently and accurately completing the human interaction detection task.

Claims

1. A method for detecting human interaction based on the Mamba architecture, characterized in that, The procedure is as follows: Step 1: Obtain the image dataset for human interaction detection and perform preprocessing to obtain the preprocessed image dataset. ,in, Indicates the first Zhang represents the preprocessed human interaction detection image; N represents the total number of human interaction detection images; Indicates the length of the image used for detecting human interaction. This represents the width of the image used for detecting human interaction. This indicates the number of channels in the image used for detecting human interaction. Step 2: Construct a human interaction detection network based on the Mamba architecture, including: a global feature encoding module, a multi-view feature aggregation module, a human body branch, an object branch, and an interaction branch; wherein, the global feature encoding module includes: a ResNet-50 model, a feature mapping unit, and a Transformer encoder; the multi-view feature aggregation module includes: two CEM units; the human body branch and the object branch each include: two cascaded LoRA units, one Router unit for dynamic weighting, one detection context propagation module, and a fully connected layer; the interaction branch includes: two cascaded LoRA units, two Router units for dynamic weighting, and a fully connected layer; Step 3: The global feature encoding module... Processing is performed to obtain the first... A global visual representation ; Step 4: The multi-view feature aggregation module... Processing is performed to obtain the first... A shallow shared semantic and the Deep shared semantics These are then input together into the human body branch, object branch, and interaction branch for processing, resulting in the corresponding output of the human body branch. Human bounding box prediction results and its confidence level , object branch output Object category prediction results Object bounding box prediction results and its confidence level and the output of interactive branches Interaction category prediction results and its confidence level ; Step 5, based on , , And the actual results of human body bounding boxes The actual result of the object bounding box Actual results of object categories Interaction Category Real Results Construct the total loss function of the character interaction detection network. This was used to train the character interaction detection network until the total loss function was reached. The process continues until convergence, thus obtaining the optimal human interaction detection model; Step Six: During the reasoning process, the first... Zhang's image to be tested The input is processed by the optimal character interaction detection model to obtain... The first in Individual body bounding box prediction results and its detection confidence Object bounding box prediction results and its detection confidence Object category prediction results Interaction category prediction results and its confidence level ; , express The total number of predicted results; Will , , After addition, it becomes the first The confidence score of the prediction results of individual interactions is used to obtain... The top few prediction results with high confidence scores are used as... The final prediction result.

2. The method for detecting human interaction based on the Mamba architecture according to claim 1, characterized in that, Step three includes: Step 3.1, Inputting the data into a ResNet-50 model for processing yields the... Initial visual features , Indicates the length of visual features, Width representing visual features The number of channels representing visual features; Step 3.2: The feature mapping unit uses convolution operations to... The channel dimension is reduced to Then, the reduced visual features are unfolded using a flattening operator to obtain the first... Intermediate visual features ; Indicates the number of channels after the dimension; Step 3.3: Encode the location and After being superimposed, the result is input into a Transformer encoder for processing to obtain the [number of layers]. A global visual representation .

3. The method for detecting human interaction based on the Mamba architecture according to claim 2, characterized in that, Step four includes: Step 4.1: Randomly generate A query vector to be learned ; and will and The first CEM cell is input for processing to obtain the second cell. A shallow shared semantic ;in, Indicates the number of query vectors; Indicates the number of channels in the query vector; Step 4.2, The input is processed in the first LoRA unit of the human body branch to obtain the... Low-level detection features of human body ; Will The input is processed in the first LoRA unit of the object branch to obtain the... Low-level detection features of individual objects ; Step 4.3, and After superposition, with Together, they are input into the second CEM cell for processing to obtain the... Deep shared semantics ; Step 4.4, Feature enhancement processing is performed on the second LoRA unit of the input human body branch to obtain the... Intermediate detection features of individual human body ; Will Feature enhancement processing is performed in the second LoRA unit of the input object branch to obtain the... Intermediate detection features of individual objects ; Step 4.5, and Adaptive dynamic weighting is performed on the Router unit of the input human body branch to obtain the first... Advanced Personal Body Detection Features ; Will and Adaptive dynamic weighting is performed on the Router unit in the input object branch to obtain the first... Advanced detection features of individual objects ; Step 4.6, the first Router unit in the interaction branch , , Perform adaptive dynamic weighting to generate the first Low-level interaction features ; Step 4.7, the first LoRA unit pair in the interaction branch Perform feature enhancement to obtain the first Intermediate interaction features ; Step 4.8, the second Router unit in the interaction branch , , , Perform adaptive dynamic weighting to obtain the first... Advanced interactive features ; Step 4.9, the second LoRA unit pair in the interaction branch Perform feature enhancement to generate the first A comprehensive interactive feature ; Step 4.10, the detection context propagation module in the human body branch... and Processing yields the first... Personal comprehensive physical characteristics ; The object branch detection context propagation module for and Processing is performed to obtain the first... Comprehensive detection features of individual objects ; Step 4.11, the fully connected layers in the human body branches Processing is performed to obtain Human bounding box prediction results Detection confidence of human body bounding box , express Number of people; The fully connected layer pairs of the object branches Processing is performed to obtain Object bounding box prediction results Detection confidence of object bounding boxes and object category prediction results , This indicates the number of objects detected. This indicates the number of categories of the detected objects; The fully connected layer pairs in the interaction branch Processing is performed to obtain Interaction category prediction results and interaction category confidence ; Indicates the number of interactions detected; This indicates the number of categories of interactions detected.

4. The method for detecting human interaction based on the Mamba architecture according to claim 3, characterized in that, Each CEM unit includes: two self-Mamba blocks, two feature integration blocks, and two cross-Mamba blocks; the first CEM unit in step 4.1 is obtained according to the following process. A shallow shared semantic : Step 4.1.1, The feature enhancement is performed on a self-Mamba block input into the first CEM cell, and the second feature enhancement is obtained using equations (1)-(2). Step 1 global visual state quantity and the Step 1 global visual output Thus, the first Step 1 global visual output As the first One-stage global enhancement of visual features : (1) (2) In equations (1)-(2), Indicates the first The first-stage global visual state quantity of the step, when k=1, let = , Indicates the total number of steps; express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix; Step 4.1.2, The input is another Mamba block in the first CEM cell for feature enhancement, and the first Mamba block is obtained using equations (3)-(4). The first-stage query state quantity and the The first-stage query output volume Thus, the first The first-stage query output volume As a first-stage enhanced query vector : (3) (4) In equations (3)-(4), Indicates the first The first-stage query state variable of the step, when k=1, let = , express right The state transition matrix affected express for Influence matrix express for Influence matrix express for The influence matrix; Step 4.1.3: The first feature integration block is obtained using equations (5)-(6). A two-stage enhancement of global visual features and two-stage enhanced query vector : (5) (6) In equations (5)-(6), Norm represents a LayNorm layer; Step 4.1.4, and Input into a cross Mamba block in the first CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (7)-(8). Three-stage global visual state quantity and the The three-stage global visual output of the step Thus, the first The three-stage global visual output of the step As the first Three-stage enhancement of visual features : (7) (8) In equations (7)-(8), Indicates the first The three-stage global visual state variables of the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (9). express for The influence matrix is ​​obtained from equation (10). express for Influence matrix Indicates hyperparameters; (9) (10) In equations (9)-(10), Indicates a fully connected layer. Indicates hyperparameters; Step 4.1.5, and Input into another cross Mamba block in the first CEM cell, used for... Feature enhancement is performed, and the first feature is obtained using equations (11)-(12). The three-stage query state quantity of the step and the The output volume of the three-stage query Thus, the first The output volume of the three-stage query As a three-stage enhanced query vector : (11) (12) In equations (11)-(12), Indicates the first The three-phase query state variable of the step, when k=1, let = , express right The state transition matrix affected express for The influence matrix is ​​obtained from equation (13). express for The influence matrix is ​​obtained from equation (14). express for The influence matrix; (13) (14) Step 4.1.6: The second feature integration block uses equations (15)-(16) to obtain the first... Four-stage enhancement of global visual features and four-stage enhanced query vector : (15) (16) Step 4.1.7: Use equation (17) to obtain the first... A shallow shared semantic : (17) In equation (17), This indicates channel merging.

5. The method for detecting human interaction based on the Mamba architecture according to claim 4, characterized in that, Each detection context propagation module includes: two self-Mamba blocks and two cross-Mamba blocks. The detection context propagation module in the human branch in step 4.10 is obtained according to the following process: Personal comprehensive physical characteristics : Step 4.10.1, the detection of human branches, a self-Mamba block in the context propagation module, according to equations (1)-(2) Processing yields the first... The first phase strengthens human detection features ; Step 4.10.2, another Mamba block in the human branch detection context propagation module, according to equations (3)-(4), performs... Processing yields the first... Each stage enhances interactive features ; Step 4.10.3, the detection of human branches: a cross Mamba block in the context propagation module is used according to equations (7)-(8) to... and To process, in order to Perform adaptive feature enhancement to obtain the first Two-stage enhanced human detection features ; Step 4.10.4, another cross Mamba block in the human branch detection context propagation module, according to equations (11)-(12), performs... as well as and The product obtained by multiplication Interactive human detection features To process, in order to Perform adaptive feature enhancement to obtain the first Three-stage enhanced human detection features ; Step 4.10.5, and Add them together to obtain the comprehensive human body detection characteristics. .

6. The method for detecting human interaction based on the Mamba architecture according to claim 5, characterized in that, In step five, the total loss function is constructed using equation (16). : (16) In equation (16), , , and There are 4 hyperparameters. The GIOU loss represents the human body bounding box. Indicates the L1 loss of the human body bounding box; The GIOU loss represents the object bounding box. Represents the L1 loss of the object's bounding box; Represents the cross-entropy loss of object categories. This represents the Focal loss for the interaction category.

7. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing any of the character interaction detection methods according to claims 1-6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is run by the processor, it performs the steps of the character interaction detection method according to any one of claims 1-6.

Images