An Interactive Gesture Recognition Method Based on Feature Mining and Adaptive Occlusion Model

Abstract

The application relates to the field of image recognition, and discloses an interactive gesture recognition method based on feature mining and an adaptive shielding model, an adaptive shielding recognition module is designed, relevant information of a shielding area is accurately extracted, and the information is seamlessly spliced with information of a non-shielding area, so that real-time updating and optimization of to-be-recognized gesture feature information are realized, even if shielding exists, the model can still obtain relatively complete and accurate feature information, a deep feature mining module is also designed, information is efficiently screened and effectively spread, so that the flow of information in the model is more reasonable and orderly, threshold filtering, posture Transformer and action Transformer are used to sufficiently and deeply mine the feature information, and the application improves the accuracy of interactive gesture recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and in particular to an interactive gesture recognition method based on feature mining and adaptive occlusion model. Background Technology

[0002] With the rapid development of the computer vision industry, video recognition methods have been widely applied in various fields, especially interactive gesture recognition. This is because the hand is an important interactive organ, and gestures are a very frequent way of conveying and expressing information. Therefore, accurate gesture recognition is an essential part.

[0003] In existing technologies, hands are frequently obscured by various objects in daily interaction scenarios, or by others in multi-person interaction scenarios. The degree of obstruction can range from slight coverage to large-area obstruction, and the location of obstruction is highly random and diverse. This directly leads to the loss of many feature information of the hands. Once the feature information is incomplete, the model loses key basis when performing gesture recognition, thus easily falling into the dilemma of recognition error. Moreover, during the model training phase, due to the imperfection of feature extraction and processing mechanisms, feature information is often not fully utilized. Some potential effective information that is of great value to gesture category and posture recognition may be weakened or even lost in the complex training process. This hinders the accurate recognition of gesture category and posture, making it difficult for the model to accurately capture hand movements, thus leading to deviations in recognition results.

[0004] Therefore, how to design an interactive gesture recognition method to avoid hand occlusion and loss of effective information in order to improve recognition accuracy has become an urgent problem to be solved. Summary of the Invention

[0005] Based on this, the present invention proposes an interactive gesture recognition method based on feature mining and an adaptive occlusion model. By designing an adaptive occlusion recognition module, relevant information of the occluded area is accurately extracted and seamlessly stitched with information of the unoccluded area, thereby realizing real-time updating and optimization of the feature information of the gesture to be recognized. Even with occlusion, the model can still obtain relatively complete and accurate feature information. A deep feature mining module is also designed to efficiently filter and effectively propagate information, making the flow of information in the model more reasonable and orderly. Through threshold filtering, pose Transformer and action Transformer, feature information is fully and deeply mined. The present invention improves the accuracy of interactive gesture recognition.

[0006] This invention proposes an interactive gesture recognition method based on feature mining and adaptive occlusion models, comprising:

[0007] The interactive gesture video image sequence is acquired and basic feature extraction is performed to obtain initial features, which are based on the ResNet network.

[0008] The initial features are input into the adaptive occlusion recognition module to obtain adaptive occlusion enhancement features, which is based on a cross-attention mechanism;

[0009] The adaptive occlusion enhancement features are input into the deep feature mining module to obtain hand pose results and action type results. The deep feature mining module includes a threshold filtering submodule, a pose transformer submodule, and an action transformer submodule. The threshold filtering submodule filters redundant background information based on a dynamic threshold. The pose transformer submodule and the action transformer submodule construct a spatiotemporal correlation model to encode hand pose features and interaction action features, respectively.

[0010] In summary, based on the aforementioned interactive gesture recognition method using feature mining and an adaptive occlusion model, an adaptive occlusion recognition module is designed to accurately extract relevant information from the occluded area and seamlessly stitch it together with information from the unoccluded area. This enables real-time updating and optimization of the gesture feature information to be recognized. Even with occlusion, the model can still acquire relatively complete and accurate feature information. Furthermore, a deep feature mining module is designed to efficiently filter and effectively propagate information, making the flow of information within the model more rational and orderly. Through threshold filtering, pose Transformer, and action Transformer, feature information is fully and deeply mined. This invention improves the accuracy of interactive gesture recognition. Specifically, the process involves acquiring a sequence of interactive gesture video images and performing basic feature extraction to obtain initial features. This basic feature extraction is based on a ResNet network. The initial features are then input into an adaptive occlusion recognition module to obtain adaptive occlusion enhancement features. This adaptive occlusion recognition module, based on a cross-attention mechanism, accurately extracts relevant information from occluded areas and seamlessly stitches it with information from unoccluded areas, thereby achieving real-time updating and optimization of the gesture feature information to be recognized. Even with occlusion, the model can still obtain relatively complete and accurate feature information. The adaptive occlusion enhancement features are then input into a deep feature mining module to obtain hand pose and action type results. The data mining module includes a threshold filtering submodule, a pose transformer submodule, and an action transformer submodule. The threshold filtering submodule filters redundant background information based on a dynamic threshold. The pose transformer and action transformer submodules construct spatiotemporal correlation models to encode hand posture features and interactive action features respectively, enabling efficient information filtering and effective propagation. This makes the flow of information in the model more reasonable and orderly. Through threshold filtering, pose transformer, and action transformer, the feature information is fully and deeply mined, thus improving the accuracy of interactive gesture recognition.

[0011] Furthermore, the step of inputting the initial features into the adaptive occlusion recognition module to obtain adaptive occlusion enhancement features specifically includes:

[0012] The initial features are input into the adaptive occlusion recognition module, which then obtains a predicted mask based on the initial features. The specific algorithm for obtaining the predicted mask is as follows:

[0013] ,

[0014] in, Indicates the prediction mask. This represents the sigmoid function. This represents a basic convolutional block, which includes convolutional layers, BatchNorm layers, and ReLU layers. Indicates initial features;

[0015] The predicted mask and initial features are multiplied element-wise to obtain hand features and object features. The hand features are used for hand pose estimation, and the object features are used to assist in mining hand features. The specific algorithm for the element-wise multiplication process is as follows:

[0016] ,

[0017] in, This indicates the feature obtained through element-wise multiplication. ;

[0018] The key feature enhancement is performed on the hand and object features based on the cross-attention mechanism. The query features, key features, and value features of the hand and object features are extracted respectively. The transposes of the query features and key features of the hand and object features are then subjected to cross-matrix multiplication, followed by normalization and convolution processing to obtain the key enhanced features of the hand and the object respectively. The specific algorithm for key feature enhancement is as follows:

[0019] ,

[0020] ,

[0021] ,

[0022] in, , , These represent query features, key features, and value features, respectively. This represents the convolution operation. This indicates Softmax normalization. and These represent the query features and key features of the hand feature, respectively. and These represent the query features and key features of the object, respectively. Indicates transpose. and These represent key enhancement features of the hand and key enhancement features of the object, respectively.

[0023] Furthermore, the step of enhancing key features of the hand features and object features based on the cross-attention mechanism further includes:

[0024] The key enhancement features of the hand and the key enhancement features of the object are multiplied element-wise to obtain the fused key enhancement features. Then, the fused key enhancement features and the hand features are multiplied by a matrix and processed through a fully connected layer to obtain the fused key enhancement feature matrix. Finally, the fused key enhancement feature matrix is ​​residually concatenated with the hand features to obtain the output feature matrix of the adaptive occlusion recognition module. The specific algorithm for obtaining the output feature matrix of the adaptive occlusion recognition module is as follows:

[0025] ,

[0026] in, This indicates that the adaptive occlusion recognition module outputs a feature matrix. Indicates a fully connected layer. and These represent key enhancement features of the hand and key enhancement features of the object, respectively. Value features representing hand features Describing hand features, This represents element-wise multiplication. Represents matrix multiplication. This represents a residual connection.

[0027] Furthermore, the threshold filtering submodule specifically includes:

[0028] The initial features and the feature matrix output by the adaptive occlusion recognition module are input into the threshold filtering submodule. The feature dimensions of the initial features and the feature matrix output by the adaptive occlusion recognition module are adjusted to increase the feature dimensions. The initial features are then subjected to feature filtering processing according to a preset filtering threshold to filter redundant background features. The filtered initial features and the feature matrix output by the adaptive occlusion recognition module are then concatenated to obtain the threshold-filtered fused features. The specific algorithm for obtaining the threshold-filtered fused features is as follows:

[0029] ,

[0030] in, This indicates threshold filtering fusion features. This indicates calculating the average of a matrix. Indicates feature filtering processing, This indicates an increase in feature dimension. Indicates initial features, Represents the proportionality coefficient. Indicates matrix concatenation. This indicates that the adaptive occlusion recognition module outputs a feature matrix.

[0031] Furthermore, the step of inputting the adaptive occlusion enhancement features into the deep feature mining module to obtain hand pose results and action type results specifically includes:

[0032] The pose transformer submodule performs positional encoding on the threshold filtering fusion features to obtain pose tokens. Each pose token corresponds one-to-one with the number of interactive gesture video image sequences. The pose tokens are further processed to obtain hand object tokens. The pose tokens and hand object tokens are combined to obtain action tokens.

[0033] Two-dimensional image information of hand joints and one-dimensional depth information are obtained using a multilayer perceptron. The two-dimensional image information and the one-dimensional depth information are combined to obtain the hand pose result. The specific algorithm for obtaining the hand pose result is as follows:

[0034] ,

[0035] in, Indicates the result of hand posture. This represents a multilayer perceptron. Indicates Transformer encoding, Indicates position code, This indicates threshold filtering fusion features;

[0036] Then, based on the action transformer submodule, multilayer perceptron, and Softmax normalization processing, the action type result is obtained.

[0037] Furthermore, the pose transformer submodule and the action transformer submodule also include:

[0038] The pose transformer submodule and the action transformer submodule have the same internal structure. Both the pose transformer submodule and the action transformer submodule include a masked multi-head attention layer, a multi-head attention layer and a feedforward network layer. Each of the masked multi-head attention layer, multi-head attention layer and feedforward network layer is connected to an Add&Norm layer, which is a residual connection and layer normalization layer.

[0039] Furthermore, the step of inputting the adaptive occlusion enhancement features into the deep feature mining module to obtain hand pose results and action type results further includes:

[0040] Supervised learning is performed, which is based on a hand loss function, an object loss function, and an action loss function. The total loss function of the supervised learning is as follows:

[0041] ,

[0042] ,

[0043] ,

[0044] in, This represents the total loss in supervised learning. and These represent hand loss and object loss, respectively. Indicates the number of action categories. Indicates the ordinal number of the action category. Indicates the result of the action type. Indicates the actual action category label, Indicates the number of input frames. Indicates the ordinal number of the input frame. and Represents the proportionality coefficient. Indicates the number of joints in the hand. Indicates the ordinal number of the hand joints. Indicates the result of hand posture. This represents the actual label for each hand joint. Indicates the number of object categories. The actual label representing the object, The predicted label for an object.

[0045] This invention proposes an interactive gesture recognition system based on feature mining and adaptive occlusion models, comprising:

[0046] An initial feature extraction module is used to acquire interactive gesture video image sequences and perform basic feature extraction to obtain initial features, wherein the basic feature extraction is based on a ResNet network.

[0047] An adaptive occlusion recognition module is used to input the initial features into the adaptive occlusion recognition module to obtain adaptive occlusion enhancement features. The adaptive occlusion recognition module is based on a cross-attention mechanism.

[0048] The deep feature mining module is used to input the adaptive occlusion enhancement features into the deep feature mining module to obtain hand pose results and action type results. The deep feature mining module includes a threshold filtering submodule, a pose transformer submodule, and an action transformer submodule. The threshold filtering submodule filters redundant background information based on a dynamic threshold. The pose transformer submodule and the action transformer submodule construct a spatiotemporal correlation model to encode hand pose features and interaction action features, respectively.

[0049] The present invention also provides a storage medium that stores one or more programs, which, when executed by a processor, implement the interactive gesture recognition method based on feature mining and adaptive occlusion model as described above.

[0050] The present invention also provides a computer device, the computer device including a memory and a processor, wherein:

[0051] The memory is used to store computer programs;

[0052] When the processor executes the computer program stored in the memory, it implements the interactive gesture recognition method based on feature mining and adaptive occlusion model as described above. Attached Figure Description

[0053] Figure 1 This is a flowchart of the interactive gesture recognition method based on feature mining and adaptive occlusion model proposed in the first embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the interactive gesture recognition system based on feature mining and adaptive occlusion model proposed in the second embodiment of the present invention.

[0055] Figure 3 This is a general framework diagram of the interactive gesture recognition model of the first embodiment of the present invention;

[0056] Figure 4 This is a logic flowchart of the adaptive occlusion recognition module according to the first embodiment of the present invention;

[0057] Figure 5 This is a logic flowchart of the threshold filtering submodule in the first embodiment of the present invention;

[0058] Figure 6 This is a logic flowchart of the posture transformer submodule and the action transformer submodule of the first embodiment of the present invention.

[0059] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0060] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0061] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0063] Please see Figure 1 The diagram shows a flowchart of the interactive gesture recognition method based on feature mining and adaptive occlusion model proposed in the first embodiment of the present invention. This interactive gesture recognition method based on feature mining and adaptive occlusion model includes steps S01 to S03, wherein:

[0064] Step S01: Obtain the video image sequence of interactive gestures and perform basic feature extraction to obtain initial features;

[0065] It should be noted that in this embodiment, the basic feature extraction is based on a ResNet network. For the overall logical framework of the interactive gesture recognition model in this embodiment, please refer to [link to relevant documentation]. Figure 3 .

[0066] Step S02: Input the initial features into the adaptive occlusion recognition module to obtain adaptive occlusion enhancement features;

[0067] It should be noted that, for the logic flow of the adaptive occlusion recognition module in this embodiment, please refer to [link / reference needed]. Figure 4 The adaptive occlusion recognition module is based on a cross-attention mechanism. Initial features are input into the adaptive occlusion recognition module, which then obtains a predicted mask based on the initial features. The specific algorithm for obtaining the predicted mask is as follows:

[0068] ,

[0069] in, Indicates the prediction mask. This represents the sigmoid function. This represents a basic convolutional block, which includes convolutional layers, BatchNorm layers, and ReLU layers. Indicates initial features;

[0070] The predicted mask and initial features are multiplied element-wise to obtain hand features and object features. The hand features are used for hand pose estimation, and the object features are used to assist in mining hand features. The specific algorithm for the element-wise multiplication process is as follows:

[0071] ,

[0072] in, This indicates the feature obtained through element-wise multiplication. ;

[0073] The key feature enhancement is performed on the hand and object features based on the cross-attention mechanism. The query features, key features, and value features of the hand and object features are extracted respectively. The transposes of the query features and key features of the hand and object features are then subjected to cross-matrix multiplication, followed by normalization and convolution processing to obtain the key enhanced features of the hand and the object respectively. The specific algorithm for key feature enhancement is as follows:

[0074] ,

[0075] ,

[0076] ,

[0077] in, , , These represent query features, key features, and value features, respectively. This represents the convolution operation. This indicates Softmax normalization. and These represent the query features and key features of the hand feature, respectively. and These represent the query features and key features of the object, respectively. Indicates transpose. and These represent key enhancement features of the hand and key enhancement features of the object, respectively.

[0078] The key enhancement features of the hand and the key enhancement features of the object are multiplied element-wise to obtain the fused key enhancement features. Then, the fused key enhancement features and the hand features are multiplied by a matrix and processed through a fully connected layer to obtain the fused key enhancement feature matrix. Finally, the fused key enhancement feature matrix is ​​residually concatenated with the hand features to obtain the output feature matrix of the adaptive occlusion recognition module. The specific algorithm for obtaining the output feature matrix of the adaptive occlusion recognition module is as follows:

[0079] ,

[0080] in, This indicates that the adaptive occlusion recognition module outputs a feature matrix. Indicates a fully connected layer. and These represent key enhancement features of the hand and key enhancement features of the object, respectively. Value features representing hand features Describing hand features, This represents element-wise multiplication. Represents matrix multiplication. This represents a residual connection.

[0081] Step S03: Input the adaptive occlusion enhancement features into the deep feature mining module to obtain hand pose results and action type results;

[0082] It should be noted that, in this embodiment, the specific logic flow of the threshold filtering submodule, pose transformer submodule, and action transformer submodule can be found in the provided text. Figure 5 , Figure 6 The deep feature mining module includes a threshold filtering submodule, a pose transformer submodule, and an action transformer submodule. The threshold filtering submodule filters redundant background information based on a dynamic threshold. The pose transformer and action transformer submodules construct spatiotemporal correlation models to encode hand pose features and interaction action features, respectively. The initial features and the feature matrix output by the adaptive occlusion recognition module are input into the threshold filtering submodule. The feature dimension of the initial features and the feature matrix output by the adaptive occlusion recognition module is adjusted to increase the feature dimension. The initial features are then filtered according to a preset filtering threshold to filter redundant background features. The filtered initial features and the feature matrix output by the adaptive occlusion recognition module are concatenated to obtain the threshold-filtered fusion features. The specific algorithm for obtaining the threshold-filtered fusion features is as follows:

[0083] ,

[0084] in, This indicates threshold filtering fusion features. This indicates calculating the average of a matrix. Indicates feature filtering processing, This indicates an increase in feature dimension. Indicates initial features, Represents the proportionality coefficient. Indicates matrix concatenation. This indicates that the adaptive occlusion recognition module outputs a feature matrix;

[0085] The pose transformer submodule performs positional encoding on the threshold filtering fusion features to obtain pose tokens. Each pose token corresponds one-to-one with the number of interactive gesture video image sequences. The pose tokens are further processed to obtain hand object tokens. The pose tokens and hand object tokens are combined to obtain action tokens.

[0086] Two-dimensional image information of hand joints and one-dimensional depth information are obtained using a multilayer perceptron. The two-dimensional image information and the one-dimensional depth information are combined to obtain the hand pose result. The specific algorithm for obtaining the hand pose result is as follows:

[0087] ,

[0088] in, Indicates the result of hand posture. This represents a multilayer perceptron. Indicates Transformer encoding, Indicates position code, This indicates threshold filtering fusion features;

[0089] Then, based on the action transformer submodule, multilayer perceptron, and Softmax normalization processing, the action type result is obtained;

[0090] The pose transformer submodule and the action transformer submodule have the same internal structure. Both the pose transformer submodule and the action transformer submodule include a masked multi-head attention layer, a multi-head attention layer and a feedforward network layer. Each of the masked multi-head attention layer, multi-head attention layer and feedforward network layer is connected to an Add&Norm layer. The Add&Norm layer is a residual connection and layer normalization.

[0091] Supervised learning is performed, which is based on a hand loss function, an object loss function, and an action loss function. The total loss function of the supervised learning is as follows:

[0092] ,

[0093] ,

[0094] ,

[0095] in, This represents the total loss in supervised learning. and These represent hand loss and object loss, respectively. Indicates the number of action categories. Indicates the ordinal number of the action category. Indicates the result of the action type. Indicates the actual action category label, Indicates the number of input frames. Indicates the ordinal number of the input frame. and Represents the proportionality coefficient. Indicates the number of joints in the hand. Indicates the ordinal number of the hand joints. Indicates the result of hand posture. This represents the actual label for each hand joint. Indicates the number of object categories. The actual label representing the object, The predicted label for an object.

[0096] In summary, based on the aforementioned interactive gesture recognition method using feature mining and an adaptive occlusion model, an adaptive occlusion recognition module is designed to accurately extract relevant information from the occluded area and seamlessly stitch it together with information from the unoccluded area. This enables real-time updating and optimization of the gesture feature information to be recognized. Even with occlusion, the model can still acquire relatively complete and accurate feature information. Furthermore, a deep feature mining module is designed to efficiently filter and effectively propagate information, making the flow of information within the model more rational and orderly. Through threshold filtering, pose Transformer, and action Transformer, feature information is fully and deeply mined. This invention improves the accuracy of interactive gesture recognition. Specifically, the process involves acquiring a sequence of interactive gesture video images and performing basic feature extraction to obtain initial features. This basic feature extraction is based on a ResNet network. The initial features are then input into an adaptive occlusion recognition module to obtain adaptive occlusion enhancement features. This adaptive occlusion recognition module, based on a cross-attention mechanism, accurately extracts relevant information from occluded areas and seamlessly stitches it with information from unoccluded areas, thereby achieving real-time updating and optimization of the gesture feature information to be recognized. Even with occlusion, the model can still obtain relatively complete and accurate feature information. The adaptive occlusion enhancement features are then input into a deep feature mining module to obtain hand pose and action type results. The data mining module includes a threshold filtering submodule, a pose transformer submodule, and an action transformer submodule. The threshold filtering submodule filters redundant background information based on a dynamic threshold. The pose transformer and action transformer submodules construct spatiotemporal correlation models to encode hand posture features and interactive action features respectively, enabling efficient information filtering and effective propagation. This makes the flow of information in the model more reasonable and orderly. Through threshold filtering, pose transformer, and action transformer, the feature information is fully and deeply mined, thus improving the accuracy of interactive gesture recognition.

[0097] Please see Figure 2 The figure shows a schematic diagram of the interactive gesture recognition system based on feature mining and adaptive occlusion model proposed in the second embodiment of the present invention. The system includes:

[0098] The initial feature extraction module 10 is used to acquire the interactive gesture video image sequence and perform basic feature extraction to obtain initial features, wherein the basic feature extraction is based on the ResNet network;

[0099] An adaptive occlusion recognition module 20 is used to input the initial features into the adaptive occlusion recognition module to obtain adaptive occlusion enhancement features. The adaptive occlusion recognition module is based on a cross-attention mechanism.

[0100] The deep feature mining module 30 is used to input the adaptive occlusion enhancement features into the deep feature mining module to obtain hand pose results and action type results. The deep feature mining module includes a threshold filtering submodule, a pose transformer submodule, and an action transformer submodule. The threshold filtering submodule filters redundant background information based on a dynamic threshold. The pose transformer submodule and the action transformer submodule construct a spatiotemporal correlation model to encode hand pose features and interaction action features, respectively.

[0101] The present invention also proposes a computer storage medium storing one or more programs that, when executed by a processor, implement the above-described interactive gesture recognition method based on feature mining and adaptive occlusion model.

[0102] The present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to realize the above-mentioned interactive gesture recognition method based on feature mining and adaptive occlusion model.

[0103] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0104] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0105] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0106] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0107] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An interactive gesture recognition method based on feature mining and adaptive occlusion model, characterized in that, The method comprises the following steps: acquiring an interactive gesture video image sequence and performing basic feature extraction to obtain initial features, wherein the basic feature extraction is based on a ResNet network; inputting the initial features into an adaptive occlusion identification module to obtain adaptive occlusion enhanced features, wherein the adaptive occlusion identification module is based on a cross-attention mechanism; the step of inputting the initial features into the adaptive occlusion identification module to obtain adaptive occlusion enhanced features specifically comprises: inputting the initial features into the adaptive occlusion identification module, and the adaptive occlusion identification module acquires a predicted mask according to the initial features, wherein the specific algorithm for acquiring the predicted mask is as follows: ， wherein, denotes a prediction mask, denotes a sigmoid function, denotes a base convolution block comprising a convolution layer, a BatchNorm layer and a ReLU layer, denotes an initial feature; performing element-wise multiplication processing on the predicted mask and the initial features to obtain hand features and object features, wherein the hand features are used for hand pose estimation, and the object features are used for auxiliary mining of the hand features, and the specific algorithm for the element-wise multiplication processing is as follows: ， wherein, representing element-wise multiplication processing of the features obtained, ; performing key feature enhancement on the hand features and the object features according to the cross-attention mechanism, extracting query features, key features and value features of the hand features and the object features respectively, performing cross-matrix multiplication on the transposed query features and the key features of the hand features and the object features, and then performing normalization processing and convolution processing to obtain hand key enhanced features and object key enhanced features respectively, wherein the specific algorithm for the key feature enhancement is as follows: ， ， ， wherein, , , query feature, key feature and value feature, respectively, denotes a convolution operation, denotes a Softmax normalization, and denote query feature and key feature of hand feature, respectively, and denote query feature and key feature of object feature, respectively, denotes a transpose, and denote hand key enhanced feature and object key enhanced feature, respectively. inputting the adaptive occlusion enhanced features into a depth feature mining module to obtain hand pose results and action type results, wherein the depth feature mining module comprises a threshold filtering submodule, a pose transformer submodule and an action transformer submodule, the threshold filtering submodule filters redundant background information based on a dynamic threshold, and the pose transformer submodule and the action transformer submodule construct a space-time correlation model to respectively encode hand pose features and interactive action features. 2.The feature mining and adaptive occlusion model based interaction gesture recognition method of claim 1, wherein, the step of performing key feature enhancement on the hand features and the object features according to the cross-attention mechanism further comprises: performing element-wise multiplication on the hand key enhanced features and the object key enhanced features to obtain fused key enhanced features, performing matrix multiplication on the fused key enhanced features and the value features of the hand features, and then processing the fused key enhanced features through a fully connected layer to obtain a fused key enhanced feature matrix, performing residual connection on the fused key enhanced feature matrix and the hand features to obtain an adaptive occlusion identification module output feature matrix, and the specific algorithm for obtaining the adaptive occlusion identification module output feature matrix is as follows: ， wherein, represents an adaptive occlusion identification module output feature matrix, represents a fully connected layer, and respectively represent hand key enhancement features and object key enhancement features, represents a value feature of a hand feature, represents a hand feature, represents element-wise multiplication, represents matrix multiplication, represents a residual connection. 3.The feature mining and adaptive occlusion model based interaction gesture recognition method of claim 1, wherein, the threshold filtering submodule specifically comprises: The initial feature and the adaptive occlusion identification module output feature matrix are input into a threshold filtering sub-module, the initial feature and the adaptive occlusion identification module output feature matrix are subjected to feature dimension adjustment, the feature dimension adjustment is used to increase the feature dimension, the initial feature is subjected to feature filtering processing according to a preset filtering threshold, the feature filtering processing is used to filter redundant background features, and the filtered initial feature and adaptive occlusion identification module output feature matrix are spliced to obtain a threshold filtering fusion feature, and the specific algorithm for obtaining the threshold filtering fusion feature is as follows: ， wherein, denotes threshold filtering fusion features, denotes averaging a matrix, denotes feature filtering processing, denotes feature dimension lifting, denotes initial features, denotes a proportionality coefficient, denotes matrix splicing, denotes an adaptive occlusion identification module output feature matrix. 4.The feature mining and adaptive occlusion model based interaction gesture recognition method of claim 1, wherein, The step of inputting the adaptive occlusion enhanced feature into the depth feature mining module to obtain the hand posture result and the action type result specifically includes: The posture transformer sub-module encodes the position of the threshold filtering fusion feature to obtain posture Token, the posture Token corresponds to the number of interactive gesture video image sequences one by one, the posture Token is reprocessed to obtain hand-object Token, and the posture Token and the hand-object Token are combined to obtain action Token; According to the multi-layer perception, two-dimensional image information and one-dimensional depth information of the hand joint point are obtained, and the two-dimensional image information and the one-dimensional depth information are combined to obtain the hand posture result, and the specific algorithm for obtaining the hand posture result is as follows: ， wherein, represents a hand pose result, represents a multi-layer perception, represents a Transformer encoding, represents a position encoding, represents a threshold filter fusion feature; According to the action transformer sub-module, the multi-layer perception and the Softmax normalization processing, the action type result is obtained. 5.The feature mining and adaptive occlusion model based interaction gesture recognition method of claim 4, wherein, The posture transformer sub-module and the action transformer sub-module further include: The internal structures of the posture transformer sub-module and the action transformer sub-module are the same, and the posture transformer sub-module and the action transformer sub-module each include a mask multi-head attention layer, a multi-head attention layer and a feedforward network layer, an Add&Norm layer is connected after the mask multi-head attention layer, the multi-head attention layer and the feedforward network layer, and the Add&Norm layer is a residual connection and layer normalization. 6.The feature mining and adaptive occlusion model based interaction gesture recognition method of claim 1, wherein, The step of inputting the adaptive occlusion enhanced feature into the depth feature mining module to obtain the hand posture result and the action type result further includes: Supervised learning is performed, the supervised learning is based on a hand loss function, an object loss function and an action loss function, and the total loss function of the supervised learning is specifically as follows: ， ， ， wherein, represents the total loss of supervised learning, and respectively represent the hand loss and the object loss, represents the number of classes of actions, represents the ordinal number of actions, represents the action type result, represents the real action class label, represents the number of input frames, represents the ordinal number of input frames, and represents the proportionality coefficient, represents the number of hand joints, represents the ordinal number of hand joints, represents the hand pose result, represents the real label of each hand joint, represents the number of object classes, represents the real label of the object, represents the predicted label of the object.

7. An interactive gesture recognition system based on feature mining and adaptive occlusion model, characterized in that, It includes: An initial feature extraction module is configured to obtain an interactive gesture video image sequence and perform basic feature extraction to obtain an initial feature, and the basic feature extraction is based on a ResNet network; An adaptive occlusion identification module is configured to input the initial feature into the adaptive occlusion identification module to obtain an adaptive occlusion enhanced feature, and the adaptive occlusion identification module is based on a cross-attention mechanism; The step of inputting the initial feature into the adaptive occlusion identification module to obtain the adaptive occlusion enhanced feature specifically includes: The initial feature is input into the adaptive occlusion identification module, and the adaptive occlusion identification module obtains a predicted mask according to the initial feature, and the specific algorithm for obtaining the predicted mask is as follows: ， wherein, denotes a prediction mask, denotes a sigmoid function, denotes a base convolution block comprising a convolution layer, a BatchNorm layer and a ReLU layer, denotes initial features; The predicted mask and the initial feature are multiplied element by element to obtain hand features and object features, the hand features are used for hand pose estimation, and the object features are used for auxiliary mining of the hand features, and the specific algorithm for the element-by-element multiplication is as follows: ， wherein, representing element-wise multiplication processing of the features, ; According to the cross attention mechanism, the hand features and the object features are enhanced, the query features, the key features and the value features of the hand features and the object features are extracted, the transposition of the query features and the key features of the hand features and the object features are multiplied by the cross matrix, and then normalized and convoluted to obtain hand key enhanced features and object key enhanced features, and the specific algorithm for the key feature enhancement is as follows: ， ， ， wherein, , , query feature, key feature and value feature, respectively, denotes a convolution operation, denotes a Softmax normalization, and denote query feature and key feature of hand feature, respectively, and denote query feature and key feature of object feature, respectively, denotes a transpose, and denote hand key enhanced feature and object key enhanced feature, respectively; The depth feature mining module is used for inputting the adaptive occlusion enhanced features into the depth feature mining module to obtain hand pose results and action type results, the depth feature mining module includes a threshold filtering submodule, a pose transformer submodule and an action transformer submodule, the threshold filtering submodule filters redundant background information based on a dynamic threshold, and the pose transformer submodule and the action transformer submodule construct a space-time correlation model to encode hand pose features and interactive action features respectively.

8. A storage medium, characterized by The storage medium stores one or more programs, and the programs are executed by the processor to implement the interactive gesture recognition method based on feature mining and adaptive occlusion model according to any one of claims 1-6.

9. A computer device, comprising: The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer programs stored on the memory to implement the interactive gesture recognition method based on feature mining and adaptive occlusion model according to any one of claims 1-6. The processor is used to execute the computer programs stored on the memory to implement the interactive gesture recognition method based on feature mining and adaptive occlusion model according to any one of claims 1-6.

Images