A gesture classification method, apparatus, electronic device, computer readable storage medium and computer program product

Abstract

The application provides a gesture classification method and device, electronic equipment, computer program product and computer readable storage medium; the method comprises: performing feature extraction on gesture regions of a sample image and a to-be-identified image, to obtain first gesture features and second gesture features; the second gesture features and the first gesture features are respectively fused, and the feature components of each first fusion feature are determined; the first correlation degree of each feature component is calculated; based on the first correlation degree, the feature components are weighted and aggregated to obtain aggregated components, and the aggregated components are fused to obtain second fusion features; the second correlation degree of each second fusion feature is calculated, and based on the second correlation degree, the second fusion features are weighted and aggregated to obtain third fusion features; based on the third fusion features, the similarity between the to-be-identified image and the sample image is determined, and based on the similarity, the classification result of the to-be-identified image is determined. Through the application, the accuracy of gesture classification can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a gesture classification method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Gesture classification is widely used in human-computer interaction, artificial intelligence teaching, and other fields. The ability to quickly build classification models and the accuracy of recognition are crucial to the usability of interactive systems. Related technologies typically employ feature extraction and comparison based on deep networks to achieve gesture classification; however, this approach suffers from low accuracy. Summary of the Invention

[0003] This application provides a gesture classification method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of gesture classification.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a gesture classification method, the method comprising: Feature extraction is performed on the gesture regions of N sample images to obtain N first gesture features, and feature extraction is performed on the gesture regions of the image to be recognized to obtain second gesture features, where N is a positive integer; The second gesture feature is fused with N first gesture features respectively to obtain N first fused features, and M feature components of each first fused feature are determined, where M is a positive integer; Calculate the first correlation degree between each of the feature components and the M feature components; For each of the feature components, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component, and the M aggregated components are fused to obtain the second fused feature corresponding to each first fused feature; Calculate the second correlation degree between each second fusion feature and N second fusion features respectively, and for each second fusion feature, based on the N second correlation degrees, perform weighted aggregation on the N second fusion features to obtain the third fusion feature corresponding to each second fusion feature; Based on the N third fusion features, N similarities between the image to be identified and the N sample images are determined, and based on the N similarities, the classification result of the image to be identified is determined.

[0005] This application provides a gesture classification device, including: The feature extraction module is used to extract features from the gesture regions of N sample images to obtain N first gesture features, and to extract features from the gesture regions of the image to be recognized to obtain second gesture features, where N is a positive integer; The feature fusion module is used to fuse the second gesture feature with N first gesture features respectively to obtain N first fused features, and to determine M feature components of each first fused feature, where M is a positive integer; The first attention module is used to calculate the first correlation degree between each of the feature components and M of the feature components; for each feature component, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component, and the M aggregated components are fused to obtain the second fused feature corresponding to each first fused feature; The second attention module is used to calculate the second correlation degree between each second fusion feature and N second fusion features, and for each second fusion feature, based on the N second correlation degrees, to perform weighted aggregation on the N second fusion features to obtain the third fusion feature corresponding to each second fusion feature; The result determination module is used to determine N similarities between the image to be identified and the N sample images based on the N third fusion features, and to determine the classification result of the image to be identified based on the N similarities.

[0006] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the gesture classification method provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the gesture classification method provided in this application when executed by a processor.

[0008] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the gesture classification method provided in this application.

[0009] The embodiments of this application have the following beneficial effects: After fusing the second gesture features with N first gesture features to obtain N first fusion features and determining the M feature components of each first fusion feature, the first correlation degree between these feature components is calculated, and the M feature components are weighted and aggregated based on this. This allows for dynamic integration of local features according to the strength of mutual dependence within each feature component, thereby accurately capturing and strengthening the detailed correlation of the gesture locality, resulting in a second fusion feature with fine local correlation. Subsequently, the second correlation degree between each second fusion feature and the N second fusion features is used, and the N second fusion features are weighted and aggregated again using this second correlation degree. This allows different second fusion features to refer to and complement each other, fully extracting global interaction information between multiple samples, thereby obtaining a third fusion feature that combines fine local detailed correlation with rich global interaction. This makes the feature expression more accurate and comprehensive. Finally, based on the third fusion feature, the similarity is derived and the classification result of the image to be identified is finally determined, which can eliminate the comparison error caused by inaccurate feature expression, thereby improving the accuracy of gesture classification. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the gesture classification method provided in the embodiments of this application. Figure 1 ; Figure 2 This is a flowchart illustrating the gesture classification method provided in the embodiments of this application. Figure 2 ; Figure 3 This is a flowchart illustrating the gesture classification method provided in the embodiments of this application. Figure 3 ; Figure 4 This is a flowchart illustrating the processing of custom gesture classification for small samples provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hand gesture category similarity ranking and scoring process provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a CNN with shared weights provided in an embodiment of this application; Figure 7 This is a schematic diagram of the lightweight CNN structure provided in the embodiments of this application; Figure 8 This is a structural diagram of a gesture classification device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0014] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0015] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0016] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0017] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0018] 1) The gesture region refers to a local area of ​​interest (ROI) extracted from the original, complete field-of-view image using object detection or instance segmentation algorithms. This ROI contains only the gesture subject and its posture skeleton information. This region effectively eliminates redundant visual information from the environmental background, lighting interference, and irrelevant subject parts. For example, in a photograph containing a full-body image of a person against a complex indoor background, the palm and fingers are precisely selected using bounding boxes and normalized into a clean image block of fixed resolution; this image block is the gesture region.

[0019] 2) Gesture features refer to a high-dimensional mathematical tensor or vector representation obtained by inputting the two-dimensional pixel matrix of the original image into a deep learning network model, such as a convolutional neural network with residual modules, and then performing multiple layers of nonlinear mapping and dimensionality reduction compression. Essentially, gesture features transform intuitive visual forms into abstract semantic codes that can be understood and processed by the underlying computing architecture. This high-dimensional representation encompasses deep attributes such as the edge contours, texture depth, and spatial topology of the gesture.

[0020] 3) Relevance refers to a numerical metric used to quantitatively assess the strength of interdependence, information matching relevance, or contextual interaction importance of different features or different feature components in a deep abstract space.

[0021] 4) Similarity refers to an intuitive scalar score obtained by quantizing the high-dimensional comprehensive representation results through specific operational logic or mapping network. It is used to quantitatively measure the semantic closeness or matching probability between the gesture to be identified and the known reference gesture.

[0022] 5) Positional encoding features refer to auxiliary numerical identifier vectors introduced to compensate for the loss of original coordinate distribution information due to serialization operations when dividing a fusion processing matrix with two-dimensional or multi-dimensional physical dimensions into discrete one-dimensional sequence units. For example, an electronic device can construct a learnable parameter table that matches the length of the segmented sequence. Each row in the table stores a set of floating-point vectors of a specific length. The corresponding row vectors extracted from this parameter table based on the grid index at the initial segmentation of each unit constitute the identifier information providing geometric prior knowledge.

[0023] Gesture classification is widely used in fields such as human-computer interaction and artificial intelligence teaching. Its ability to quickly build classification models and its recognition accuracy are crucial to the usability of interactive systems.

[0024] In related technologies, gesture classification is typically achieved using feature extraction and comparison methods based on deep networks. Specifically, this method first extracts hand features from both the sample image and the image to be identified, then calculates the overall similarity between the features to be identified and the features of each sample, and outputs the predicted classification label based on this similarity score. However, the aforementioned method based on overall feature comparison has limitations in terms of the fineness of feature representation. Existing few-sample feature comparison methods usually only focus on the overall feature similarity between samples, ignoring the correlation features between local details within the gesture image, and also failing to extract and utilize the global interaction information between features from multiple different samples. Due to the lack of in-depth mining of local structural correlations and interaction information between multiple samples, the feature representation extracted by the related algorithms is not accurate enough when faced with complex background interference or subtle changes in gesture posture.

[0025] It is evident that in related technologies, the inability to fully capture the local features of gestures and the interaction information between multiple samples leads to inaccurate feature representation, resulting in low accuracy in gesture classification.

[0026] Furthermore, after obtaining similarity scores based on feature comparison, related technologies typically select the single category corresponding to the highest score as the final result. This decision-making logic lacks a fault tolerance mechanism and is easily affected by extreme abnormal samples or single feature extraction biases, thereby further impacting the accuracy of the classification results. In addition, during the extraction and fusion of the aforementioned gesture features, related technologies often only use single-layer channel splicing, failing to fully extract the deep information of the channel dimension. Moreover, the original spatial location information is lost during the conversion of the feature map into a sequence, resulting in a lack of spatial structure guidance for subsequent attention mechanism calculations, further limiting the accuracy of the classification results.

[0027] To address this issue, this application provides a gesture classification method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of gesture classification. This gesture classification method can be applied to gesture classification scenarios in fields such as human-computer interaction and artificial intelligence teaching. The method includes: extracting features from gesture regions of N sample images to obtain N first gesture features, and extracting features from gesture regions of the image to be recognized to obtain second gesture features, where N is a positive integer; fusing the second gesture features with the N first gesture features to obtain N first fused features, and determining M feature components for each first fused feature, where M is a positive integer; calculating a first correlation degree between each feature component and the M feature components; and for each feature component, based on the M feature components... The first correlation degree of the corresponding M features is used to weight and aggregate the M feature components to obtain the corresponding aggregated components. The M aggregated components are then fused to obtain the second fusion feature corresponding to each first fusion feature. The second correlation degree of each second fusion feature with N second fusion features is calculated. For each second fusion feature, based on the N second correlation degrees, the N second fusion features are weighted and aggregated to obtain the third fusion feature corresponding to each second fusion feature. Based on the N third fusion features, the N similarities between the image to be identified and the N sample images are determined. Based on the N similarities, the classification result of the image to be identified is determined. Thus, after fusing the second gesture features with N first gesture features to obtain N first fusion features, and determining the M feature components of each first fusion feature, the first correlation degree between these feature components is calculated. Based on this, the M feature components are weighted and aggregated. This allows for dynamic integration of local features according to the strength of interdependence within each feature component, thereby accurately capturing and strengthening the detailed correlation of the gesture's local area, resulting in a second fusion feature with fine local correlation. Subsequently, the second correlation degree between each second fusion feature and the N second fusion features is used to weight and aggregate the N second fusion features again. This allows different second fusion features to reference and complement each other, fully extracting global interaction information between multiple samples, thus obtaining a third fusion feature that combines fine local detailed correlation with rich global interaction. This makes the feature expression more accurate and comprehensive. Finally, based on this third fusion feature, the similarity is derived and the classification result of the image to be identified is determined. This eliminates the comparison error caused by inaccurate feature expression, thereby improving the accuracy of gesture classification.

[0028] The gesture classification method provided in the embodiments of this application will be described below.

[0029] See Figure 1 , Figure 1 This is a flowchart illustrating the gesture classification method provided in the embodiments of this application. Figure 1 , will combine Figure 1 The steps shown are explained below. Figure 1 The main component of the process is electronic equipment.

[0030] Step 101: Extract features from the gesture regions of N sample images to obtain N first gesture features, and extract features from the gesture regions of the image to be recognized to obtain second gesture features, where N is a positive integer.

[0031] It should be noted that, since the original images contain a lot of environmental noise, it is difficult to directly perform accurate gesture comparison and classification. Therefore, the electronic device first needs to extract the gesture regions of N sample images with known categories and the current image to be recognized. Then, using the feature extraction mechanism, the gesture regions of the N sample images with known categories and the gesture regions of the currently unknown image to be recognized are uniformly mapped to the same high-dimensional feature space to obtain N first gesture features corresponding to the N sample images and the second gesture features corresponding to the image to be recognized.

[0032] Here, a sample image refers to reference image data with known category labels, used as a comparison benchmark for gesture classification. For example, a sample image could be a photo of a user's custom-captured reference gesture such as scissors, rock, or paper using a camera. The image to be recognized refers to the target image data captured at the current moment, whose specific gesture category needs to be predicted. For example, the image to be recognized could be a frame of video captured in real-time during human-computer interaction.

[0033] A gesture region refers to a local region of interest (ROI) extracted from a complete field-of-view image, containing only the skeleton of the hand and fingers' pose. For example, a gesture region can be a 224×224 resolution image patch cropped from the center of the palm, after removing the environmental background. Gesture features can be high-dimensional feature maps of size C×H×W (i.e., number of channels × height × width, such as 512×7×7), or high-dimensional feature vectors, such as a 512-dimensional feature vector.

[0034] In some embodiments, the gesture region of the sample image and the gesture region of the image to be recognized can be obtained in the following way: a pre-trained target detector, such as the YOLO series detection model or the SSD model, is invoked to perform inference calculations on the global sample image and the global image to be recognized to predict the bounding box coordinates of the hand; then, based on the bounding box coordinates, the corresponding image sub-regions are cropped from the original sample image and the image to be recognized, and their size and pixel value are normalized to obtain the gesture region with redundant background information filtered out.

[0035] In some embodiments, feature extraction of the gesture region can be achieved as follows: a pre-defined convolutional neural network (CNN) is used as the backbone for feature extraction. Here, a Siamese network architecture with shared weights can be used for the extraction process of sample images and images to be recognized to ensure the consistency of feature mapping. For example, the cropped gesture region matrix can be input into a ResNet-34 network with residual connections. The image data is sequentially processed through a sliding window operation of multiple sets of convolutional kernels (Conv) within the network to extract spatial texture features. The feature distribution is stabilized by batch normalization (BN) layers, and the feature map is reduced in dimensionality and the receptive field is expanded by combining the ReLU nonlinear activation function and max pooling operation. After forward propagation of the network, N feature tensors corresponding to N sample images are output as N first gesture features, and a feature tensor corresponding to the image to be recognized is output as a second gesture feature.

[0036] Step 102: Fuse the second gesture feature with N first gesture features respectively to obtain N first fused features, and determine M feature components for each first fused feature, where M is a positive integer.

[0037] To more accurately capture gesture regions in the image to be recognized, and the relative relationships between gesture regions in different sample images, the electronic device can fuse the second gesture features with each of the first gesture features, allowing these two types of feature information to interact and obtain paired joint representations. Subsequently, the electronic device further divides the fused first features into multiple local feature components, refining the macroscopic feature tensor into fine-grained spatial sequences or channel sets. This provides a data foundation for subsequent attention calculations at the local detail level, such as the bending state of a specific finger or local texture.

[0038] Here, fusion refers to the process of combining and interactively aggregating feature tensors from two independent dimensions—the image to be identified and a specific sample image—along a preset dimension (such as spatial or channel dimension). For example, a specific fusion operation could be a concatenation operation along the channel dimension.

[0039] The first fusion feature refers to the feature tensor generated after the two gesture features are initially combined, representing the pairwise correlation and commonalities / differences between the two. For example, the first fusion feature can be a hybrid feature map containing the interaction semantics of the two images.

[0040] In some embodiments of this application, Figure 1The step 102, fusing the second gesture feature with N first gesture features to obtain N first fused features, can be achieved through the following process: concatenating the second gesture feature with N first gesture features along the feature channel dimension to obtain N third gesture features; and extracting features from each of the N third gesture features to obtain N first fused features.

[0041] It should be noted that the feature channel dimension refers to the dimension that characterizes the thickness attribute of the feature tensor of a deep network. Each channel independently records the feature representation of the image at a specific visual abstraction level (such as edge direction at a specific frequency or local texture activation response). The third gesture feature refers to an intermediate state of feature obtained by coarsely and directly physically splicing the image to be recognized with a single sample image, which has not yet undergone nonlinear semantic fusion.

[0042] In this embodiment, when concatenating the second gesture feature with N first gesture features along the feature channel dimension to obtain the third gesture feature, the electronic device first verifies whether the second gesture feature corresponding to the image to be recognized and the first gesture features corresponding to each sample image are at the same spatial resolution scale (i.e., height H and width W are aligned). While ensuring spatial dimension alignment, the electronic device sequentially appends the channel sequence of the second gesture feature directly to the end of the channel sequence of each first gesture feature. Thus, the electronic device obtains N sets of expanded feature map sequences containing two types of gesture information and identifies them as N third gesture features. This concatenation method along the feature channel dimension can preserve the original extracted deep semantic information to the greatest extent and without loss, providing complete underlying data support for subsequent correlation calculations.

[0043] For example, suppose that the first and second gesture features obtained after preprocessing are both normalized to a size of The feature tensor (i.e., containing) Each of the N first gesture features has a channel depth, and both its height and width are 14. During the fusion process, for any one of the N first gesture features, the electronic device directly concatenates and merges its 128-dimensional channels with the 128-dimensional channels of the second gesture feature; after the concatenation operation is completed, the original two tensors will be merged and output as a single feature tensor, whose corresponding size is expanded to [size missing]. ,Right now The feature map. These N expanded features have dimensions of... The sequence of feature maps constitutes N third gesture features.

[0044] In some embodiments, the above-described feature extraction for N third gesture features to obtain N first fusion features can be achieved by inputting the third gesture features into a preset lightweight convolutional aggregation network layer (e.g., a point-wise convolutional layer) for calculation. Specifically, for features with The channel depth of the first indivual( The third gesture feature is that electronic devices utilize a specific number of... The convolutional kernels of a certain size are linearly weighted across the channel dimensions at each spatial pixel of the tensor; then, non-linear transformations are introduced using activation functions such as ReLU to perform cross-channel deep semantic interaction on the original independent feature channels, resulting in a size that is compressed back to its original size. A tensor of dimension, which electronic devices determine as the first dimension. The first fusion feature.

[0045] In other embodiments, the above-described feature extraction for each of the N third gesture features to obtain N first fusion features can also be achieved by constructing a residual feature extraction unit that includes a channel attention mechanism (such as a Squeeze-and-Excitation, SE module). Here, the electronic device can first use a global average pooling layer to calculate the response weight allocation vector for each channel, adaptively recalibrate and scale each channel within the third gesture features, and then utilize... Dilated convolutional layers expand the receptive field and extract features to output the first fused features. This approach allows for the immediate introduction of advanced inter-channel context awareness during the feature fusion stage.

[0046] It is understood that, in this embodiment of the application, by concatenating the second gesture feature with N first gesture features along the feature channel dimension, the original expressive information of the second gesture feature and each of the first gesture features can be completely preserved, thereby improving the information richness and completeness of the obtained third gesture feature. Simultaneously, extracting features from the N third gesture features to obtain N first fusion features enables deeper cross-channel information interaction and semantic integration of the concatenated features, thereby enhancing the fusion depth between the second and first gesture features and effectively improving the expressive accuracy of the final obtained first fusion feature. This provides a higher quality data foundation for subsequent calculation of correlation and accurate inference of classification results.

[0047] In other embodiments of this application, Figure 1The step 102, which involves fusing the second gesture feature with N first gesture features to obtain N first fused features, can also be achieved through the following processing: For each first gesture feature, calculate its element-wise absolute difference matrix with the second gesture feature; calculate the element-wise product matrix of the two; and linearly superimpose the absolute difference matrix and the product matrix. The resulting mixed matrix is ​​then determined as the first fused feature.

[0048] Here, by calculating the absolute difference element by element, subtle deformation differences of the two gesture features at various spatial locations and semantic channels can be captured. Calculating the principal element product matrix can enhance the similar feature regions jointly activated by the two gestures; the absolute difference matrix and the product matrix can be superimposed according to preset weight coefficients to obtain a hybrid matrix.

[0049] Feature components refer to fine-grained feature units obtained by decomposing or mapping the overall first fused feature according to preset rules (such as spatial grid segmentation). They are used to characterize the attributes of the fused feature in a specific local region. For example, feature components can be M discrete one-dimensional token vectors obtained by flattening the fused feature map after it has been spatially segmented into multiple image patches.

[0050] In some embodiments of this application, Figure 1 The determination of the M feature components of each first fusion feature in step 102 can be achieved by the following process: dividing each first fusion feature into M initial components and determining the position coding feature for each initial component; fusing each initial component with the corresponding position coding feature and determining the fusion result as the feature component.

[0051] It should be noted that the initial components refer to a set of discrete, localized feature vector sequences obtained by spatially segmenting and flattening the overall two-dimensional high-dimensional feature map. These contain the local visual semantics of the image but lose the original two-dimensional spatial structure coordinates. Positional encoding, on the other hand, refers to a numerical vector used to represent the absolute coordinates or relative positions of each initial component in the original feature map. Its role is to supplement the discrete initial components with prior knowledge of spatial geometry.

[0052] In this embodiment, each first fusion feature is divided into M initial components, and a positional encoding feature is determined for each initial component. This can be achieved through spatial grid partitioning and querying the learnable parameter matrix. Here, the electronic device first uniformly crops the first fusion feature, which has height and width, into M non-overlapping image patches according to a preset grid size. Then, each image patch is flattened in the feature depth dimension to map it into an independent one-dimensional vector, thus obtaining the M serialized initial components. Next, the electronic device initializes a learnable parameter matrix in memory that perfectly matches the size of the M initial components. Each row in this matrix corresponds to a specific spatial location and is continuously updated through backpropagation during training. The electronic device extracts the corresponding row vectors from the learnable parameter matrix according to the original spatial arrangement order of each initial component and determines them as the positional encoding features of the corresponding initial component.

[0053] For example, if the first fusion feature is a size of Feature map (i.e., number of channels) ,high ,width If electronic equipment adopts If the spatial granularity is divided, it will be divided into the following on the spatial plane: A local area; at this time... The electronic device extracts 512-dimensional depth data from each local region and constructs it into 196 pixels with a size of [missing information]. These 196 one-dimensional vectors constitute the M initial components. Simultaneously, the electronic device can read a pre-constructed vector of size... The two-dimensional position encoding parameter table is used, and based on the original grid indices (from 1 to 196) of each of the 196 initial components, a lookup is performed in the parameter table to extract the corresponding values. The row vector (which can be obtained during training) is then determined as the corresponding positional encoding feature.

[0054] In some embodiments, fusing each initial component with its corresponding location-coded feature can be achieved through element-wise addition. Here, for any initial component, the electronic device directly adds each element in the initial component vector to the element at the same index position in the corresponding location-coded feature vector. Since both have the same feature dimension space, this addition operation allows the pure visual semantic features (i.e., the initial components) to be superimposed with specific spatial coordinate information frequency responses. The resulting M feature sequences with coordinate-aware capabilities are then determined as the M feature components used for subsequent correlation calculations.

[0055] In other embodiments, fusing each initial component with its corresponding location-coded feature can also be achieved through a concatenated dimensionality-reduced network structure. In this case, the electronic device can first concatenate the initial component with the location-coded feature, doubling the length of the original feature vector; then, the resulting feature is input into a dimensionality-reducing network consisting of a single fully connected linear layer for linear mapping, compressing the vector dimension back to its initial length, and the final output dimensionality-reduced vector is determined as the feature component.

[0056] Understandably, by dividing each first fused feature into M initial components, the overall first fused feature is transformed into a representation with multiple local dimensions, improving the granularity of the underlying feature processing. Simultaneously, by determining positional encoding features for each initial component and fusing each initial component with its corresponding positional encoding features, explicit spatial location-aware attributes are added to the local initial component features. This ensures that the final determined feature components not only contain the original basic feature information but also integrate corresponding spatial structure arrangement information. Thus, the expressive power of each feature component at the spatial topology level is enhanced.

[0057] In other embodiments of this application, Figure 1 The determination of the M feature components of each first fusion feature in step 102 can also be achieved by the following process: performing subspace segmentation and aggregation operations along the channel dimension of the first fusion feature to obtain the M feature components.

[0058] Here, if the first fusion feature is a three-dimensional tensor containing C channels (with a size of...) The electronic device can uniformly divide it into M non-overlapping channel groups along the channel dimension, each group containing The electronic device then uses global average pooling or global max pooling to eliminate spatial dimensional redundancy in the local three-dimensional tensor set corresponding to each channel group, compressing each channel group into a one-dimensional feature vector. These M one-dimensional feature vectors, generated by grouping and aggregating based on channel dimensions, constitute the M feature components, which respectively represent the interaction attributes of the first fused feature in different deep semantic subspaces such as edge deformation and local texture.

[0059] Step 103: Calculate the first correlation degree between each feature component and the M feature components.

[0060] Although each feature component independently represents the semantic attributes of the first fused feature in a specific local region or a specific channel subspace, there are objective dependencies between the various local details of a gesture, such as spatial linkage and posture constraints between adjacent finger joints. Therefore, by calculating the degree of correlation between each feature component, we can mine and quantify the global contextual dependencies and local spatial interactions within the first fused feature, thus providing a weighting basis for subsequently re-aggregating the local feature components into a fused feature with a global perspective.

[0061] Here, the first correlation refers to a numerical metric used to evaluate the strength of the interdependence, matching relevance, or information transmission importance between any two feature components within the first fused feature. For example, the first correlation can be the cosine similarity between two feature components based on spatial geometry, or the attention score between the query element and the key element determined by the self-attention mechanism in deep learning.

[0062] Here, taking a self-attention mechanism as an example, the calculation process of the first correlation degree is explained: For M feature components, two independent and learnable linear mapping matrices (i.e., and The M feature components are linearly transformed and projected to generate the corresponding query vector set (Query Matrix). ) and key vector set (Key Matrix, ); subsequently, regarding the first The query vector corresponding to each feature component ( ), calculate all key vectors corresponding to its M feature components. The matrix dot product is used to quantify the response excitation between feature components; to avoid the gradient vanishing due to excessively large dot product values, the calculation result can be divided by the square root of the feature depth. The results are then scaled and smoothed; finally, the Softmax activation function is used to normalize the probability of the output vector containing M scalar values, which is then determined as the nth vector. The first correlation degree between one feature component and M feature components. This calculation process can be expressed as equation (1):

[0063] Step 104: For each feature component, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component, and the M aggregated components are fused to obtain the second fused feature corresponding to each first fused feature.

[0064] Since the initial feature components only contain isolated information from specific subspaces or local receptive fields, lacking perception of complete gesture posture, this embodiment uses the pre-calculated first correlation degree as a weighting coefficient for feature reorganization. Based on this weighting coefficient, all local feature components are injected into the current feature component proportionally according to their correlation strength, thereby completing the dynamic interaction and correction of the contextual information of each feature component. Afterwards, the electronic device re-integrates the updated feature components into a unified feature, resulting in a high-order feature that can depict local details and possess global structural perception capabilities—the second fused feature.

[0065] Here, weighted aggregation refers to the process of using specific correlation as weighting coefficients to perform numerical scaling and linear superposition operations on multiple feature components to generate a single feature component containing comprehensive information. For example, weighted aggregation can be a weighted summation calculation at the output of a self-attention module.

[0066] An aggregated component refers to a new feature component output after a given original feature component has absorbed the correlation information of all M feature components and been reconstructed. For example, an aggregated component can be an output dense feature vector that incorporates the global response intensity.

[0067] The second fusion feature refers to the high-order feature obtained by integrating the M aggregated components that have completed context information updates in terms of spatial or channel dimensions. For example, the second fusion feature can be a three-dimensional feature map with specific height, width, and channel dimensions, which is restored from these M dense feature components after tensor splicing and residual mapping.

[0068] See Figure 2 , Figure 2 This is a flowchart illustrating the gesture classification method provided in the embodiments of this application. Figure 2 In some embodiments of this application, Figure 1 In step 104, for each feature component, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component. This can be achieved through the following processing: Step 1041: Using the M first correlation degrees, weight the M feature components respectively to obtain M weighted feature components.

[0069] It should be noted that weighting refers to the process of dynamically adjusting the numerical amplitude of feature components in a high-dimensional space by using correlation as a scaling factor, in order to highlight highly correlated features and suppress irrelevant noise interference.

[0070] In this embodiment, the M feature components are weighted using M first correlation degrees, which can be achieved through vector scalar multiplication. Here, the electronic device can iterate through the M feature components, extract the first correlation degree corresponding to any specific feature component, and then multiply the scalar value of the first correlation degree by each dimension element within the feature component to proportionally amplify or attenuate the amplitude of the local feature signal. After performing the above operations in parallel, M weighted feature components are obtained.

[0071] For example, if the feature component has a length of A one-dimensional vector, where the first feature component is Its corresponding first correlation (e.g., the attention score after normalization) is a scalar. Electronic devices utilize numerical values Performing element-wise scalar multiplication on the one-dimensional vector yields the weighted first eigencomponent. If the first correlation degree corresponding to the second feature component is only a low response... If the vector elements are weighted, their values ​​will approach 0, thus effectively attenuating and masking them in the feature space.

[0072] Step 1042: Superimpose the M weighted feature components to obtain the aggregated component corresponding to the feature component.

[0073] It should be noted that superposition refers to the process of summing multiple independent feature components along the same dimensional sequence in a common feature metric space, thereby fusing information distributed in different local receptive fields into a high-density vector.

[0074] In some embodiments, the M weighted feature components are superimposed to obtain the aggregated component, which can be achieved through element-wise matrix addition. Here, the electronic device can initialize an all-zero accumulator vector in video memory that is completely identical in dimension to that of a single feature component; then, the electronic device sequentially reads the M weighted feature components along the feature channel dimension and accumulates the element values ​​at their corresponding positions into the accumulator vector; after M complete accumulation calculations, the accumulator vector aggregates the weighted response information of all local components, and the electronic device determines it as the aggregated component corresponding to the current processing object (i.e., the specific feature component from the previous step).

[0075] For example, suppose the total number of weighted feature components that need to be superimposed after processing is... And the vector dimension is 3. If the first weighted feature component is... The second characteristic component is The third characteristic component is The electronic device sums the feature values ​​of these three vectors that have the same index position: the summation of the first dimension is... The summation of the second dimension is The summation of the third dimension is The final generated fusion vector This constitutes the aggregate component corresponding to the feature component.

[0076] In other embodiments, in addition to directly accumulating numerical values, the electronic device can also achieve this by using the superposition mechanism of residual connection. That is, on the result of superimposing M weighted feature components, the original unweighted feature components of the current processing stage are added to ensure that the original information is not lost and to prevent the gradient vanishing problem in deep network computation.

[0077] It is understood that, in this embodiment of the application, by utilizing M first correlation degrees to weight the M feature components respectively, it is possible to dynamically adjust the numerical proportions of different feature components in the calculation process according to the strength of the correlation between each feature component, effectively strengthening the core feature response with high correlation degree and improving the targeting and accuracy of feature processing; at the same time, by superimposing the M weighted feature components, efficient integration and reconstruction of multiple local features are achieved in the same feature dimension space, fully absorbing and integrating the advantageous information carried by each weighted feature component; thus, the resulting aggregated component can deeply integrate local correlation features, significantly enhancing the richness and accuracy of the underlying data feature representation.

[0078] In other embodiments of this application, Figure 1In step 104, for each feature component, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component. This can also be achieved through the following processing: using the M first correlation degrees as response activation coefficients, element-wise multiplication is performed on each of the M feature components to scale the local feature information proportionally, resulting in M ​​weighted feature components; all independent dimension information is preserved by concatenating and combining them rather than nonlinearly superimposing them, that is, the M weighted feature components are matrix concatenated along the preset feature channel dimensions to obtain a high-dimensional concatenated feature tensor containing all weighted information; finally, the high-dimensional concatenated feature tensor is input into a preset feature aggregation network, for example, a multilayer network with a nonlinear activation function. The convolutional module utilizes the cross-channel learning capability of the feature aggregation network to perform multi-dimensional information interaction and channel compression on the high-dimensional concatenated feature tensor. The dimensionality-reduced feature vector output by the feature aggregation network is then determined as the aggregated component corresponding to the feature component.

[0079] Step 105: Calculate the second correlation degree between each second fusion feature and N second fusion features respectively, and for each second fusion feature, based on the N second correlation degrees, perform weighted aggregation on the N second fusion features to obtain the third fusion feature corresponding to each second fusion feature.

[0080] Although the preceding processing has extracted deep feature responses within the image to be identified and individual sample images, in scenarios with very few samples, relying solely on isolated paired features is highly susceptible to interference from anomalous features or local image noise in a single sample image. Therefore, in this embodiment, the electronic device also needs to calculate the second correlation degree between each second fusion feature at the level of the entire sample set, i.e., N sample images, to establish a global contextual interaction between different sample images. Subsequently, the electronic device uses this second correlation degree to perform a secondary weighted aggregation, so that the final output third fusion feature can integrate the intra-class commonalities and inter-class differences within the sample set, thereby shielding the local interference caused by a single anomalous feature and improving the overall classification accuracy of various gestures.

[0081] Here, the second correlation refers to a metric used to quantitatively evaluate the strength of the mutual influence between the second fusion features corresponding to different sample images and their relationship with the global context. For example, the second correlation can be a probability distribution score matrix that characterizes the semantic similarity among multiple samples, output by a self-attention network.

[0082] The third fusion feature refers to the final high-order feature representation output by the feature recombination mechanism after each of the second fusion features has absorbed global sample interaction information. For example, the third fusion feature can be a 512-dimensional dense feature vector containing the topological structure information of the sample set.

[0083] It should be noted that the process of calculating the second correlation degree between each second fusion feature and N second fusion features is similar to the process of calculating the first correlation degree between each feature component and M feature components as described above, and will not be repeated here.

[0084] For each second fusion feature, the process of weighted aggregation of the N second fusion features based on the N second correlation degrees to obtain the third fusion feature corresponding to each second fusion feature is similar to the process described above of weighted aggregation of the M feature components based on the M first correlation degrees corresponding to the M feature components to obtain the corresponding aggregated component, and will not be repeated here.

[0085] Step 106: Based on N third fusion features, determine N similarities between the image to be identified and N sample images, and based on the N similarities, determine the classification result of the image to be identified.

[0086] Although the third fusion feature already possesses rich local details and global interaction semantics across samples, it still belongs to the abstract tensor in high-dimensional space in terms of data form. Therefore, electronic devices need to calculate the similarity between the image to be recognized and each sample image based on N third fusion features, and then perform the final gesture classification on the image to be recognized based on the N similarity scores corresponding one-to-one with the N sample images to obtain the classification result.

[0087] Here, similarity refers to a numerical metric used to characterize the semantic closeness or matching probability between the gesture in the image to be identified and the gesture in a specific sample image. For example, similarity can be a normalized score with a value between 0 and 1.

[0088] The classification result refers to the explicit gesture category label that the electronic device ultimately assigns to the image to be recognized after reasoning and decision-making. For example, the classification result can be a predefined discrete category enumeration value that the electronic device identifies the image to be recognized as scissors, rock, or paper.

[0089] See Figure 3 , Figure 3 This is a flowchart illustrating the gesture classification method provided in the embodiments of this application. Figure 3 In some embodiments of this application, Figure 1 Step 106, which determines the classification result of the image to be identified based on N similarities, can be achieved through the following processing: Step 1061: Filter the sample images corresponding to the S highest similarity scores among the N similarity scores to obtain S valid sample images, where S is an integer and S≥2.

[0090] Here, a valid sample image refers to the local neighborhood sample image that is closest to the image to be identified in the feature metric space and has the greatest classification reference value among all sample images.

[0091] In some embodiments, filtering the top S similarities and obtaining the corresponding sample images can be achieved by sorting and truncating the N similarities using a ranking statistical algorithm. For example, if the total number of sample images N=10, the 10 similarities obtained after prior calculation are as follows: If S=3, the electronic device can first use quick sort or heap sort to sort the above one-dimensional similarity vectors in descending order to obtain the sorted sequence. Subsequently, the electronic device performs a sequence truncation operation, extracts the similarity of the top 3 similarities, and retrieves the corresponding 3 sample images from memory as valid sample images based on the original index key values ​​(Key-Index) bound to these 3 similarities.

[0092] In other embodiments, the highest S similarity scores are filtered and the corresponding sample images are obtained. Alternatively, a cascaded method combining a hard similarity threshold and Top-S cutoff can be used for filtering. That is, firstly, all invalid samples with similarity scores below the threshold are removed using a preset threshold (e.g., 0.5) to narrow down the search space, and then the highest S similarity scores are selected from the remaining samples.

[0093] Step 1062: Based on the highest S similarity scores, assign weight coefficients to the S valid sample images.

[0094] Here, the weighting coefficient refers to the numerical value that quantifies and adjusts the contribution of each valid sample image in the final weighted voting decision.

[0095] In some embodiments, the assignment of weight coefficients to S effective sample images based on the highest S similarities described above can be achieved through the following process: summing S with a preset value to obtain a summation result; determining the difference between the summation result and i as the weight coefficient of the effective sample image corresponding to the i-th similarity among the highest S similarities, where i is an integer, 1≤i≤S.

[0096] It should be noted that the preset value refers to a constant offset set in advance to adjust the distribution benchmark of the weight coefficients, which is used to ensure that the last effective sample image in the S effective sample images can also be assigned a positive, non-zero base weight. The specific value of the preset value can be set according to actual needs, such as setting it to 1 or 2. i refers to the absolute rank of each effective sample image in the S highest similarity sequences after descending order. The smaller the value, the higher the similarity between the effective sample image and the image to be identified. If the preset value is 1, then the formula for calculating the weight coefficients can be expressed as equation (2):

[0097] For example, when S=3, the weighting coefficient for the valid sample image with the highest similarity is... For the second-ranked valid sample image, its weighting coefficient is... For the third-ranked sample image, its weighting coefficient is... .

[0098] Understandably, by summing S with a preset value, a summation result is obtained. The difference between the summation result and i is then used as the weight coefficient of the effective sample image corresponding to the i-th similarity among the highest S similarities. This establishes a mapping relationship between the weight allocation logic and the ranking position (i.e., i) of the effective sample image. Compared to directly using the absolute value of similarity for replication, this transforms the easily fluctuating absolute value of similarity into a stable relative ranking index with gradient differences. This improves the stability and robustness of the weight allocation mechanism in the face of extreme data disturbances, thereby helping to improve the accuracy of the final classification inference.

[0099] In other embodiments, the above-described assignment of weight coefficients to the S valid sample images based on the highest S similarities can also be achieved through the following processing: The absolute values ​​of the similarities are transformed into a probability distribution space using the Softmax function as the weight coefficients. For example, suppose S = 3 highest similarity values ​​are... Then the first The weight coefficients of each effective sample can be calculated using the formula for the Softmax function, i.e., equation (3):

[0100] in, This is a temperature hyperparameter used to dynamically control the smoothness of the weight distribution.

[0101] Step 1063: Based on the S weight coefficients and the gesture categories of the S valid sample images, determine the classification result of the image to be recognized.

[0102] Here, the process of determining the classification result refers to the decision process of mapping discrete scalar weights and class labels into a probability distribution tensor that reflects the global confidence of the model.

[0103] In some embodiments of this application, Figure 3 Step 1053, which is to determine the classification result of the image to be recognized based on the S weight coefficients and the gesture categories of the S valid sample images, can be achieved by the following processing: group the S valid sample images according to the gesture category to obtain T sample image groups, where T is a positive integer; sum the weight coefficients corresponding to the valid sample images in each sample image group to obtain the weight coefficient sum of each sample image group; and determine the gesture category corresponding to the sample image group with the largest sum of weight coefficients as the classification result.

[0104] It should be noted that a sample image group refers to a set of one or more valid sample images that have the same preset gesture category label, and T represents the total number of actual gesture categories covered in these S valid sample images.

[0105] In some embodiments, grouping S valid sample images according to gesture categories to obtain T sample image groups can be achieved through the following process: reading the metadata attributes of each valid sample image and extracting the known gesture category labels bound therein; then, initializing a key-value data structure (such as a hash table or dictionary) in memory space, using the gesture category label as the key, and appending the index identifiers of valid sample images with the same gesture category label to the list of corresponding keys, thereby completing the category isolation and aggregation of sample data and generating T sample image groups corresponding to different gesture categories.

[0106] For example, if S=5 valid sample images have been selected (labeled as sample 1 to sample 5 respectively), the electronic device reads these 5 sample images sequentially and extracts their gesture category label sequence as ["rock", "scissors", "rock", "paper", "scissors"]. The electronic device performs key-value classification based on the above labels and finally generates T=3 sample image groups: the key of the first sample image group is rock, containing elements [sample 1, sample 3]; the key of the second sample image group is scissors, containing elements [sample 2, sample 5]; and the key of the third sample image group is "paper", containing element [sample 4].

[0107] In other embodiments, grouping S valid sample images by gesture category to obtain T sample image groups in the above content may also be implemented through the following processing: using the SQL grouping query instruction (such as the GROUP BY statement) of a relational database to directly perform category dimension retrieval and set division on the associated entries of the valid sample images stored in the data table, so as to quickly generate T sample image groups.

[0108] It should be noted that the sum of weight coefficients refers to the scalar superposition result of all decision reference weights selected as valid samples under a specific gesture category, and its physical meaning is to quantify the overall confidence or comprehensive score that the specific category has in the current classification decision competition.

[0109] In some embodiments, accumulating the weight coefficients corresponding to the valid sample images in each sample image group to obtain the sum of weight coefficients of each sample image group in the above content may be implemented through the following processing: for any traversed sample image group, call the weight coefficients bound to each valid sample image contained in the group through a pointer; subsequently, perform a numerical linear addition operation on these discrete weight coefficients, and determine the single scalar sum value obtained by the addition as the sum of weight coefficients corresponding to the specific sample image group.

[0110] For example, following the above example, assuming that the weight coefficient sequence assigned by the electronic device to Sample 1 to Sample 5 is sequentially . For the first sample image group with the category key of "Rock", the electronic device extracts the weight coefficients corresponding to Sample 1 and Sample 3 in the group (i.e., 5 and 3), and performs accumulation calculation to obtain that the sum of weight coefficients of the Rock category is ; for the second sample image group with the category key of "Scissors", the electronic device extracts the weight coefficients corresponding to Sample 2 and Sample 5 (i.e., 4 and 1), and calculates that the sum of weight coefficients of the Scissors category is ; for the third sample image group with the category key of "Paper", since it only contains Sample 4, the sum of its weight coefficients is exactly the weight coefficient of Sample 4 .

[0111] In other embodiments, accumulating the weight coefficients corresponding to the valid sample images in each of the sample image groups to obtain the sum of weight coefficients of each sample image group in the above content may also be implemented through the following processing: constructing a category mask Matrix (where matrix elements use 1 or 0 to indicate whether a specific sample belongs to a specific category group) with a size of , and performing one forward matrix dot product multiplication operation on the weight vector with a size of and the category mask matrix, so as to obtain a sum-of-weight-coefficients vector containing T elements.

[0112] It is understood that in this embodiment, by grouping S valid sample images according to gesture categories to obtain T sample image groups, the originally independent sample images are integrated by attribute integration to obtain a category-oriented decision basis. On this basis, by accumulating the weight coefficients corresponding to the valid sample images in each sample image group, the evaluation index for a single valid sample image is objectively transformed into a comprehensive confidence level that measures the overall competitiveness of each gesture category, so that each gesture category can fully absorb the collaborative information of multiple valid sample images in its group. Finally, the gesture category corresponding to the sample image group with the largest sum of weight coefficients is determined as the classification result, so that the classification decision can comprehensively consider the overall matching advantage of samples of the same category, effectively reducing the impact of single matching bias or feature fluctuation on the overall inference process and improving the accuracy of the classification result of the image to be identified.

[0113] In other embodiments of this application, Figure 3 Step 1053, which determines the classification result of the image to be recognized based on S weight coefficients and the gesture categories of S valid sample images, can also be achieved through the following processing: For the S valid sample images, map their corresponding gesture categories to one-hot encoding vectors of a preset dimension; use the weight coefficients corresponding to each valid sample image to perform scalar multiplication on its corresponding one-hot encoding vector, i.e., feature scaling, to obtain S weighted category vectors; then, concatenate the S weighted category vectors in the preset dimension to construct a comprehensive decision tensor; finally, input the comprehensive decision tensor into a multilayer perceptron (MLP) composed of fully connected layers, perform matrix multiplication with each learnable weight matrix inside the MLP in sequence, and after superimposing the bias vector, use the Softmax activation function to perform probability normalization, output a probability distribution tensor containing the confidence of each candidate gesture category, and determine the gesture category corresponding to the maximum confidence in the probability distribution tensor as the classification result of the image to be recognized.

[0114] For example, if there are a total of 3 gesture categories, and S=2 valid sample images have been obtained, where the gesture category of the first valid sample image is category 1, the electronic device maps it to a basic one-hot encoded vector. And using its corresponding weight coefficient Perform scalar multiplication to obtain the weighted class vector. The gesture category of the second valid sample image is class 2, and the electronic device maps it to a basic one-hot encoded vector. Using its corresponding weight coefficient Perform scalar multiplication to obtain the weighted class vector. Subsequently, the electronic device concatenates these two weighted category vectors to construct a vector of size [size missing]. Comprehensive decision tensor Next, the electronic device inputs this integrated decision tensor into a multilayer perceptron containing hidden layers, via the internal forward propagation formula. After layer-by-layer feature extraction (where X is the comprehensive decision tensor, W is the weight matrix, and B is the bias term), the final output is a probability distribution tensor of length 3, as shown below. The comparison with electronic devices showed that the maximum confidence value of 0.85 was in the first dimension of the probability distribution tensor, thus classifying the first type of gesture as the classification result of the image to be recognized.

[0115] This completes the process of determining the classification result of the image to be identified based on N similarity scores.

[0116] It is understood that in this embodiment, the electronic device filters the sample images corresponding to the S highest similarities out of N similarities to obtain S valid sample images. By introducing multiple highly matched sample images as the reference basis for classification decisions, the information reference range of the judgment process is expanded, and the error tolerance of the feature comparison process is improved. At the same time, based on the S highest similarities, weight coefficients are assigned to the S valid sample images. This allows for the reasonable quantification of the contribution of each valid sample image in the final judgment according to the actual closeness between each valid sample image and the image to be identified. This objectively increases the decision weight of valid sample images with higher similarities. In this way, the accuracy and robustness of the classification results of the image to be identified can be effectively improved.

[0117] In other embodiments of this application, Figure 1 Step 105, which determines the classification result of the image to be identified based on N similarities, can also be achieved through the following process: iterate through and compare the numerical values ​​of the N similarities, select the highest similarity with the largest value, and extract the gesture category of the sample image corresponding to the highest similarity as the classification result of the image to be identified.

[0118] This completes the gesture recognition of the image to be recognized.

[0119] Understandably, in related technologies, the inaccuracy of feature representation due to the inability to fully capture local features of gestures and interaction information between multiple samples leads to low accuracy in gesture classification. In this embodiment, after fusing the second gesture feature with N first gesture features to obtain N first fused features, and determining the M feature components of each first fused feature, the first correlation degree between these feature components is calculated. Based on this, the M feature components are weighted and aggregated. This allows for dynamic integration of local features according to the strength of interdependence within each feature component, thereby accurately capturing and strengthening the detailed correlations of local gestures to obtain a more accurate classification. The second fusion feature is a finely detailed local correlation. Then, each second fusion feature is associated with the second correlation of N second fusion features, and the N second fusion features are weighted and aggregated again using the second correlation. This allows different second fusion features to refer to and complement each other, fully extracting global interaction information between multiple samples, thus obtaining a third fusion feature that combines fine local details and rich global interaction. This makes the feature expression more accurate and comprehensive. Finally, the similarity is derived based on the third fusion feature, and the classification result of the image to be identified is finally determined. This can eliminate the comparison error caused by inaccurate feature expression, thereby improving the accuracy of gesture classification.

[0120] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0121] The embodiments of this application are implemented in a custom gesture classification scenario based on a small sample size. In this scenario, a simple classifier can usually be trained by collecting only a very small number of image samples, which can then be used to classify and recognize the user's real-time actions.

[0122] Figure 4 This is a flowchart illustrating the processing of custom gesture classification for small samples provided in an embodiment of this application. See also... Figure 4 The process includes the following steps: Step 201: Obtain the gesture sample image provided by the user.

[0123] The classifier acquires sample images of gestures (called sample images) for several custom categories provided by the user. For example, the user predefines three gestures: scissors, rock, and paper, and records a small number of images to construct a classification sample set.

[0124] Step 202: Acquire the user's gesture image to be recognized in real time.

[0125] Step 203: Gesture detection and ROI extraction.

[0126] The classifier uses the same detection or segmentation model to process the currently captured real-time image of the gesture to be recognized (called the image to be recognized) and the gesture sample image, and extracts the region of interest of the hand. In this way, the gesture sample ROI and the gesture ROI to be recognized can be obtained.

[0127] Step 204: Score the similarity of gesture categories.

[0128] The classifier inputs the region of interest of the hand to be tested and all samples in the sample set into a preset deep neural network, performs in-depth feature comparison and ranking scoring, and outputs a score vector (called similarity) representing the comparison result.

[0129] Step 205: Dynamic weighted classification decision.

[0130] After obtaining the above-mentioned scoring vectors, the embodiments of this application introduce a dynamic weighting mechanism based on the ranking results. Here, the scoring vectors are arranged in descending order of similarity value from high to low, and the samples corresponding to the top S highest similarity values ​​are extracted as the effective set (called effective sample images).

[0131] Subsequently, the classifier assigns decreasing weight coefficients based on the ranking position of the valid samples. For example, the valid sample ranked 1st has a weight of S, and the valid sample ranked 1st has a weight of (K-i+1), where 1≤i≤S.

[0132] Next, the classifier scores each valid sample by weight according to the gesture category to which it belongs. The weight coefficients of all valid samples in the same category are summed (to obtain the weight coefficient sum). Finally, the gesture category with the highest total score is determined as the final recognition result of the gesture to be tested (called the classification result).

[0133] The following explains the process of ranking and scoring gesture categories based on similarity. Figure 5 This is a schematic diagram of the process for ranking and scoring gesture category similarity according to an embodiment of this application.

[0134] Step 2041: CNN feature extraction.

[0135] The classifier uses a weighted CNN (containing multiple Conv convolutional layers and a ResBlock residual block architecture derived from ResNet-34) to extract deep feature vectors for the gesture ROI to be recognized and for each sample gesture ROI (referred to as the second gesture features and N first gesture features, respectively). The classifier concatenates the extracted feature vectors along the channel dimension to obtain a fused feature tensor (referred to as the third gesture features). Figure 6 This is a schematic diagram of the shared-weights CNN structure provided in an embodiment of this application. See also... Figure 6 This CNN consists of two... 128-channel residual block, 1 A 128-channel convolutional layer, and one It consists of 64-channel convolutional layers connected together. The residual blocks used are ResNet-34 residual blocks.

[0136] Step 2042: Secondary feature fusion.

[0137] After completing the channel concatenation, the classifier uses a lightweight CNN network to perform secondary feature cross-mapping and purification on the feature tensor, thereby obtaining a preliminary pairwise mixed gesture feature representation (called the first fusion feature). Figure 7 This is a schematic diagram of the lightweight CNN structure provided in an embodiment of this application. The CNN consists of three sequentially... 512-channel residual block, 2 It is composed of 256 channels of residual blocks connected together.

[0138] To further extract global contextual information from the image, the classifier tokenizes the aforementioned feature layer, dividing the image features spatially into P discrete image patches (i.e., M initial components as described in this embodiment). Subsequently, a two-dimensional positional encoding vector (referred to as positional encoding feature) with matching feature dimensions added to each dimension is introduced, transforming it into a P×512 dimensional token sequence vector (referred to as feature component). This positional encoding is generated through trainable parameters and can adaptively learn the spatial structural features of the image.

[0139] Step 2043: Single-sample self-attention interaction.

[0140] The classifier applies the first multi-head self-attention module to the aforementioned P×512 dimensional feature vector. This module extracts global contextual features within the feature sequence. Subsequently, an average pooling layer is applied at the back end to compress the processed sequence output into a 1×512 dimensional feature vector (called the second fused feature). This feature vector integrates local details and global semantic relationships. The classifier performs the above process in parallel on all K samples in the sample set, obtaining K 1×512 dimensional feature vectors.

[0141] Step 2044: Cross-sample self-attention interaction at the sample set level.

[0142] To further compare the global contextual information between different sample gestures and between the gestures to be tested, the system inputs the aforementioned S feature vectors into a second multi-head self-attention mechanism module. This module extracts the interaction responses at the sample set level and outputs updated K 512-dimensional feature vectors. Introducing this module not only integrates global contextual information across samples but also facilitates faster convergence of network training.

[0143] Step 2045: Linear layer similarity mapping.

[0144] Following the second multi-head self-attention module, a learnable linear layer is added to the classifier. This linear layer maps each set of gesture features after dual interactions to the final similarity score, outputting an S-dimensional scalar vector (i.e., the similarity score described in this embodiment). Each element in this vector objectively represents the degree of similarity between the gesture to be tested and the corresponding sample gesture.

[0145] Step 2046: Output the similarity score. This means passing the similarity score to the aforementioned step 205 for weighted scoring.

[0146] In this way, the global context information of all samples can be obtained first and then scored, which allows for a more thorough comparison of different samples and the gestures to be recognized. At the same time, a weighted voting method based on similarity scores is used to determine the final category. By dynamically allocating ranking weights, the risk of insufficient robustness relying on a single best sample is avoided, while also strengthening the discriminative contribution of high-similarity samples and suppressing the interference of low-similarity samples, which significantly improves the accuracy and robustness of gesture classification.

[0147] It is understood that in the embodiments of this application, user information, such as images and other related data, is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0148] Based on the gesture classification method of the above embodiments, Figure 8 This is a structural diagram of a gesture classification device provided in an embodiment of this application. The gesture classification device 100 can be a device in an electronic device (e.g., a server). The gesture classification device can be implemented in software, which can be software in the form of programs and plug-ins, including the following software modules: feature extraction module 1001, feature fusion module 1002, first attention module 1003, second attention module 1004, and result determination module 1005. These modules are logically related, so they can be arbitrarily combined or further split according to the functions they implement.

[0149] The feature extraction module 1001 is used to extract features from the gesture regions of N sample images to obtain N first gesture features, and to extract features from the gesture regions of the image to be recognized to obtain second gesture features, where N is a positive integer. The feature fusion module 1002 is used to fuse the second gesture feature with N first gesture features respectively to obtain N first fused features, and to determine M feature components of each first fused feature, where M is a positive integer; The first attention module 1003 is used to calculate the first correlation degree between each feature component and M feature components respectively; for each feature component, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component, and the M aggregated components are fused to obtain the second fused feature corresponding to each first fused feature. The second attention module 1004 is used to calculate the second correlation degree between each second fusion feature and N second fusion features, and for each second fusion feature, based on the N second correlation degrees, to perform weighted aggregation on the N second fusion features to obtain the third fusion feature corresponding to each second fusion feature; The result determination module 1005 is used to determine N similarities between the image to be identified and the N sample images based on the N third fusion features, and to determine the classification result of the image to be identified based on the N similarities.

[0150] In the above scheme, the result determination module 1005 is further configured to filter the sample images corresponding to the highest S similarities among the N similarities to obtain S valid sample images, where S is an integer and S≥2; assign weight coefficients to the S valid sample images based on the highest S similarities; and determine the classification result of the image to be identified based on the S weight coefficients and the gesture categories of the S valid sample images.

[0151] In the above scheme, the result determination module 1005 is further configured to group the S valid sample images according to the gesture category to obtain T sample image groups, where T is a positive integer; accumulate the weight coefficients corresponding to the valid sample images in each sample image group to obtain the weight coefficient sum of each sample image group; and determine the gesture category corresponding to the sample image group with the largest weight coefficient sum as the classification result.

[0152] In the above scheme, the result determination module 1005 is further used to sum S with a preset value to obtain a summation result; the difference between the summation result and i is determined as the weight coefficient of the effective sample image corresponding to the i-th similarity among the highest S similarities, where i is an integer, 1≤i≤S.

[0153] In the above scheme, the feature fusion module 1002 is further configured to concatenate the second gesture feature with N first gesture features in the feature channel dimension to obtain N third gesture features; and to extract features from the N third gesture features to obtain N first fused features.

[0154] In the above scheme, the feature fusion module 1002 is further configured to divide each of the first fused features into M initial components, and determine a position coding feature for each initial component; fuse each initial component with the corresponding position coding feature, and determine the fusion result as the feature component.

[0155] In the above scheme, the first attention module 1003 is further configured to use M first correlation degrees to weight the M feature components respectively to obtain M weighted feature components; and to superimpose the M weighted feature components to obtain the aggregated component corresponding to the feature component.

[0156] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0157] This application provides an electronic device. Figure 9 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device 130 includes: at least one processor 131 ( Figure 7 (Only one is shown in the diagram), memory 132, and computer-executable instructions 133 stored in memory 132 and executable on at least one processor 131, wherein processor 131 executes computer-executable instructions 133 to implement the steps in any of the above-described trajectory planning method embodiments.

[0158] The electronic device may include, but is not limited to, a processor 131 and a memory 132. Those skilled in the art will understand that... Figure 7This is merely an example of electronic device 130 and does not constitute a limitation on electronic device 130. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0159] Processor 131 can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0160] In some embodiments, memory 132 may be an internal storage unit of electronic device 130, such as a hard disk or memory of electronic device 130. In other embodiments, memory 132 may be an external storage device of electronic device 130, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on electronic device 130. Furthermore, memory 132 may include both internal and external storage units of electronic device 130. Memory 132 is used to store operating system, application programs, boot loader, data, and other programs, such as program code of computer programs. Memory 132 may also be used to temporarily store data that has been output or will be output.

[0161] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the gesture classification method described above in this application.

[0162] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the gesture classification method provided in this application. For example, ... Figure 1 The gesture classification method is shown.

[0163] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0164] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0165] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0166] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0167] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A gesture classification method, characterized in that, The method includes: Feature extraction is performed on the gesture regions of N sample images to obtain N first gesture features, and feature extraction is performed on the gesture regions of the image to be recognized to obtain second gesture features, where N is a positive integer; The second gesture feature is fused with N first gesture features respectively to obtain N first fused features, and M feature components of each first fused feature are determined, where M is a positive integer; Calculate the first correlation degree between each of the feature components and the M feature components; For each of the feature components, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component, and the M aggregated components are fused to obtain the second fused feature corresponding to each first fused feature; Calculate the second correlation degree between each second fusion feature and N second fusion features respectively, and for each second fusion feature, based on the N second correlation degrees, perform weighted aggregation on the N second fusion features to obtain the third fusion feature corresponding to each second fusion feature; Based on the N third fusion features, N similarities between the image to be identified and the N sample images are determined, and based on the N similarities, the classification result of the image to be identified is determined.

2. The method according to claim 1, characterized in that, The process of determining the classification result of the image to be identified based on N similarities includes: The sample images corresponding to the S highest similarities among the N similarities are filtered to obtain S valid sample images, where S is an integer and S≥2; Based on the highest S similarities, weight coefficients are assigned to the S valid sample images. Based on the S weight coefficients and the gesture categories of the S valid sample images, the classification result of the image to be identified is determined.

3. The method according to claim 2, characterized in that, The step of determining the classification result of the image to be identified based on the S weight coefficients and the gesture categories of the S valid sample images includes: The S valid sample images are grouped according to the gesture category to obtain T sample image groups, where T is a positive integer; The weight coefficients corresponding to the effective sample images in each sample image group are summed to obtain the sum of the weight coefficients for each sample image group; The gesture category corresponding to the sample image group with the largest sum of the weight coefficients is determined as the classification result.

4. The method according to claim 2, characterized in that, The assignment of weight coefficients to the S valid sample images based on the highest S similarities includes: S is summed with the preset value to obtain the summation result; The difference between the summation result and i is determined as the weight coefficient of the effective sample image corresponding to the i-th similarity among the highest S similarities, where i is an integer, 1≤i≤S.

5. The method according to any one of claims 1 to 4, characterized in that, The step of fusing the second gesture feature with N first gesture features to obtain N first fused features includes: The second gesture feature is concatenated with N first gesture features along the feature channel dimension to obtain N third gesture features; Feature extraction is performed on each of the N third gesture features to obtain N first fusion features.

6. The method according to claim 1, characterized in that, Determining the M feature components of each of the first fused features includes: Each of the first fused features is divided into M initial components, and a position encoding feature is determined for each of the initial components; Each initial component is fused with the corresponding location coding feature, and the fusion result is determined as the feature component.

7. The method according to claim 1, characterized in that, For each of the feature components, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component, including: Using the M first correlation degrees, the M feature components are weighted respectively to obtain M weighted feature components; The M weighted feature components are superimposed to obtain the aggregated component corresponding to the feature component.

8. A gesture classification device, characterized in that, The device includes: The feature extraction module is used to extract features from the gesture regions of N sample images to obtain N first gesture features, and to extract features from the gesture regions of the image to be recognized to obtain second gesture features, where N is a positive integer; The feature fusion module is used to fuse the second gesture feature with N first gesture features respectively to obtain N first fused features, and to determine M feature components of each first fused feature, where M is a positive integer; The first attention module is used to calculate the first correlation degree between each of the feature components and M of the feature components; for each feature component, based on the M first correlation degrees corresponding to the M feature components, the M feature components are weighted and aggregated to obtain the corresponding aggregated component, and the M aggregated components are fused to obtain the second fused feature corresponding to each first fused feature; The second attention module is used to calculate the second correlation degree between each second fusion feature and N second fusion features, and for each second fusion feature, based on the N second correlation degrees, to perform weighted aggregation on the N second fusion features to obtain the third fusion feature corresponding to each second fusion feature; The result determination module is used to determine N similarities between the image to be identified and the N sample images based on the N third fusion features, and to determine the classification result of the image to be identified based on the N similarities.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 7.

11. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 7.

Images