Infrared gesture detection and segmentation method based on balance score fusion mechanism

Abstract

The invention discloses an infrared gesture detection and segmentation method based on a balance score fusion mechanism, and belongs to the field of gesture recognition, a model adopts the design of a double-flow network structure, and comprises a global motion network, a gesture posture network and a feature fusion module, the global motion network is composed of a plurality of two-dimensional convolutional neural networks, a recognition module, an association module, a one-dimensional convolutional neural network and a bidirectional long-short-term memory network in sequence so as to extract the overall spatial-temporal characteristics of gesture motion; the gesture attitude network is composed of a gesture attitude estimation network based on temperature sensing, a gesture attitude evolution body and an attitude feature extraction network in sequence so as to extract attitude change features of gestures, and the feature fusion module fuses the overall spatial-temporal features and the attitude change features by adopting a balance score; the model provided by the invention can realize accurate gesture boundary detection and segmentation functions, efficiently learn gesture transition features, and meet the gesture segmentation requirements of gesture recognition task automation.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of gesture recognition, and relates to an infrared gesture detection and segmentation method based on a balanced score fusion mechanism. BACKGROUND

[0002] Gesture has become an important means of daily communication by virtue of the characteristics of efficient transmission of semantic and emotional information. Gesture recognition, which integrates advanced sensor technology and pattern recognition methods, is an important part of human-computer interaction. The segmentation task of continuous gestures, as an upstream task of gesture recognition, the accuracy of gesture boundary segmentation will directly affect the accuracy of gesture recognition task, especially in high complexity and high similarity dynamic gesture recognition tasks. Gesture can be divided into static gesture and dynamic gesture. Static gesture is the posture of hand at a certain moment, and dynamic gesture is the change of posture and motion position of hand in a time interval. Dynamic gesture recognition task includes continuous dynamic gesture recognition and isolated dynamic gesture recognition. In continuous dynamic gesture recognition, the goal of gesture segmentation task is to detect the beginning and end of a complete gesture. Since the gesture segmentation boundary has fuzziness, how to confirm the boundary of gesture is a difficulty in continuous dynamic gesture research.

[0003] There are mainly three kinds of technologies for segmentation and boundary detection of continuous gestures, including traditional methods, motion-based methods and deep learning-based methods. Traditional methods include conditional random field, hidden Markov model, dynamic time warping, etc. Based on traditional methods, it is necessary to rely on manual feature design, the time sequence modeling capability is limited and it is difficult to fuse multiple modalities, which has limitations. The motion-based boundary detection method needs to design the motion according to the characteristics of the data set. The motion indicator is designed by calculating the pixel motion speed to segment the gesture, and the peak and valley values of the motion indicator correspond to the motion and rest state of the gesture; the stable hand position in the first few frames is used as a fixed height threshold, and the position of the gesture is compared with the height threshold to determine the boundary position of the gesture. The motion-based method has poor transferability, and the same motion has a large performance gap on different data sets. In the deep learning-based method, the boundary of the gesture is learned and detected by the neural network. The spatiotemporal features are learned by fusing a deep residual network and a bidirectional long short-term memory network; the gesture segmentation is performed by combining a probability method and a 3D convolutional neural network; a detection model based on a double-flow convolutional neural network is used, and then a recognition model is used for processing the depth mode and the visible light mode respectively; the continuous gesture sequence is segmented into isolated gesture segments by using a time dilation residual 3D network, and the features are extracted by using a convolutional neural network and a long short-term memory network to learn different spatiotemporal features. The existing methods are mainly based on visible light images, depth images or skeleton data, and there is a lack of research on gesture segmentation and recognition for infrared image characteristics. The formation of infrared image is directly related to temperature, and has characteristics that other types of data do not have. Directly transferring the research on visible light images will reduce the performance of the model.

[0004] By referring to the existing technologies and observing the characteristics of the gesture data set, the hand motion change and the hand posture change are included in the execution process of the gesture. It is found that the posture change of the gesture stops earlier than the motion change, which indicates that the posture change of the gesture can better reflect the boundary characteristics of the gesture than the motion change. Inspired by this, in order to solve the problem of performance decline of the existing research methods on infrared images, the present application proposes a double-flow network structure based on balanced score fusion mechanism (TS-BSFN) in infrared gesture segmentation. The TS-BSFN adopts a double-flow network structure, mainly including a global motion network, a gesture posture network and a feature fusion module. The global motion network is responsible for extracting the overall motion features, the gesture posture network is responsible for extracting the posture change features, and finally the feature fusion module performs score fusion to detect the start and end boundaries of the gesture. SUMMARY

[0005] The purpose of the application is to make up for the characteristics of less visual information of infrared images, fully decouple the skeleton motion and posture and integrate them into the feature extraction network, and provide an infrared gesture detection and segmentation method based on balanced score fusion mechanism.

[0006] Technical scheme: An infrared gesture detection and segmentation method based on a balanced score fusion mechanism, which adopts a double-flow network structure, including a global motion network, a gesture posture network and a feature fusion module; The global motion network is used for extracting the overall spatiotemporal features of gesture motion. The gesture posture network includes a gesture posture estimation network based on temperature perception, a gesture posture evolution body and a posture feature extraction network, and is used for extracting posture change features of gestures. The feature fusion module balances and fuses the overall spatiotemporal features and the posture change features. The method steps include: (1) Generating overall spatiotemporal features in the gesture motion process, the input original infrared video is input into the global motion network frame by frame, and the spatial feature extraction, short-term feature extraction and long-term feature extraction are sequentially performed, and the extracted features are output with the prediction score of the boundary through the activation function; (2) Generating posture change features in the gesture motion process, the video frame is converted into corresponding skeleton data through the gesture posture estimation network, the skeleton data is sequentially constructed into a gesture posture evolution body, the posture change features are extracted by using the posture feature network, and the boundary prediction score is generated; (3) Feature fusion, the two boundary prediction scores are fused by using the balanced score fusion method to generate the final gesture boundary score.

[0007] Further, the global motion network is modeled based on a feature extractor, and the spatial features in the gesture motion process are extracted. The feature extractor includes a two-dimensional convolutional neural network and parallel recognition modules and correlation modules, the spatial features of the image are extracted through the two-dimensional convolutional neural network, the correlation module performs point product similarity calculation on each pixel in adjacent frames and weights and converges the features of the current pixel and the related positions in the adjacent frames; the recognition module is decomposed into multiple parallel convolution kernels along the time dimension and the space dimension for feature extraction, the outputs of the correlation module and the recognition module are multiplied, and then one-dimensional convolutional neural network and bidirectional long short-term memory network are used for short-term and long-term time feature modeling, and finally the generated features are the overall spatiotemporal features in the gesture motion process. The spatiotemporal features are an overview of the global information of gesture motion, and the overall spatiotemporal features sequentially pass through a fully connected layer and an activation function layer, and output the prediction score Score of the i-th frame boundary through a Sigmoid function.

[0008] Further, the gesture posture network mainly performs the following three stages of operations: The first stage is IR image skeletonization, a gesture posture estimation model based on temperature structure perception generates skeleton data conforming to the temperature characteristics of a human hand, hand structure information is introduced into the network through the introduction of a graph convolution network, and a hand candidate frame most matched with the temperature distribution is selected as the final skeleton data for output; The second stage is to generate a hand posture evolution body, and the posture evolution in the gesture change process is decoupled to obtain the hand posture evolution body; The third stage is posture feature extraction, a gesture posture extraction network is used to extract the features of posture changes, which is used to strengthen the model to identify the transition features of gestures, local space-time features are captured through a convolution layer, high-level features of gesture changes are learned through a bottleneck layer, and a feature vector is output, the feature vector sequentially passes through a full connection layer and an activation function layer, and a prediction score Score of the i-th frame boundary is output through a Sigmoid function.

[0009] The gesture posture evolution body passes through a 7*3*3 three-dimensional convolution layer to capture local space-time features, and then passes through four bottleneck layers to gradually learn high-level features of gesture changes, the bottleneck layer uses a 3*3*3 depth separable convolution, and the output features of the last bottleneck layer are converted into a 512-channel feature vector through global average pooling.

[0010] Further, the feature fusion module fuses the overall space-time features and the posture change features, including considering the influence of balance, using and represent the proportion of non-boundary frames and boundary frames in the gesture sequence, the boundary proportion is multiplied by the prediction scores of the global motion network and the posture extraction network to balance the frame number gap, and the possibility score of each frame as a boundary is output , and a threshold method based on the probability score is used to detect the boundary.

[0011] The detection of the boundary needs to satisfy that the prediction score is greater than or equal to the threshold value 0.5.

[0012] Advantages: Compared with the prior art, the substantial features and significant progress of the present application include: 1. The double-flow network structure proposed in the present application, based on the characteristics of less visual information of infrared images, fully decouples the skeleton motion and the posture and integrates them into the feature extraction network, can effectively utilize the posture change information in gesture motion, and assists in accurately determining the gesture boundary.

[0013] 2. The balance score fusion mechanism proposed in the present application, based on the problem that the imbalance of the number of boundary frames and non-boundary frames leads to a decrease in segmentation accuracy, realizes feature fusion with small calculation overhead, optimizes the boundary recognition accuracy, and further improves the segmentation accuracy of gestures.

[0014] 3、The method for decoupling gesture based on the skeleton data construction gesture evolution body in the application, fully decouples gesture features from the overall motion features, and realizes the complementary advantages of skeleton data and image data by fusing hand gesture estimation technology. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is the double-flow network structure based on the balance score fusion mechanism constructed by the application; Figure 2 is a gesture boundary detection graph. Figure 3 is a hand structure graph. DETAILED DESCRIPTION

[0016] The application will be described in detail below in combination with the drawings and specific examples.

[0017] The application is an infrared gesture detection and segmentation method based on a balance score fusion mechanism. As shown in the network model, the specific implementation can be as follows. Figure 1

[0018] (1) Structure of global motion network

[0019] The global motion network (GMN) is used to extract the global features of motion in the input continuous infrared gesture video. The global features include the time features and spatial features of motion, and also include the motion features and gesture change features. The network first uses N feature extractors to extract features frame by frame. After each two-dimensional convolutional neural network of the feature extractor, a parallel recognition module and a correlation module are added for recognizing gesture trajectories between adjacent frames and realizing cross-frame target tracking. When the network extracts spatial features, one-dimensional convolutional neural networks and bidirectional long short-term memory networks are used for short-term and long-term time modeling. Finally, a feature vector containing all the spatio-temporal information in the gesture motion process is generated. The feature vector passes through a fully connected layer and an activation function layer, and the Sigmoid activation function is used as the activation function. The output interval is within (0, 1) boundary score prediction value.

[0020] ​The association module establishes the spatial correspondence of gesture parts in time sequence by analyzing the association of spatial features between adjacent frames, and then realizes accurate tracking of gesture trajectory, and significantly improves the semantic understanding ability of the model to dynamic gesture action. The core processing flow includes two key links: similarity calculation and trajectory aggregation. In the similarity calculation stage, the module uses dot product operation to measure the correlation of the pixel-level features of adjacent frames to generate the association feature map of the local region. In the trajectory aggregation stage, the features of the current frame pixels and the related positions of the adjacent frames are weighted and fused to construct the continuous motion trajectory at the pixel level. Considering that the contribution of different spatial regions to gesture representation is significantly different. The region containing the key gesture motion trajectory has a higher feature contribution, while the contribution of the background and other irrelevant regions is relatively limited. The recognition module can dynamically adjust the feature weights of each local feature block based on this feature. In order to optimize the calculation efficiency of the model, the scheme adopts a spatiotemporal multi-scale architecture. This architecture deploys multi-scale convolution kernels along the time and space dimensions in parallel, which significantly reduces the computational complexity while ensuring the ability to capture a wide range of spatiotemporal information.

[0021] (2) Structure of gesture posture network

[0022] The gesture posture network (GPN) mainly includes three stages. The first stage is IR image skeletonization. A hand posture estimation model based on temperature structure perception is used to generate skeleton data conforming to the temperature features of the human hand. The second stage generates a hand posture evolution body. The posture evolution in the gesture change process is decoupled to obtain a hand posture evolution body. The third stage is posture feature extraction. The gesture posture feature extraction network is used to extract the features of posture changes to strengthen the transition features of the model to recognize gestures and improve the segmentation ability of the model.

[0023] (2.1) Hand posture estimation network based on temperature structure perception

[0024] Existing hand posture estimation networks are mostly designed for visible light images and do not perform well in infrared images. The hand posture estimation model based on temperature structure perception (TBSA-Net) considers the feature that the temperature distribution of the human hand in the infrared image has significant regularity. During the training stage, when the temperature distribution of the predicted posture does not match the true temperature distribution, the model will give a higher penalty; otherwise, the penalty will be reduced. During the testing stage, the hand candidate box that best matches the temperature distribution is selected as the final output. In order to deal with the problem of insufficient visual information in infrared images, the hand structure information is integrated into the network by introducing a graph convolution network, thereby improving the performance of hand key point positioning. In order to decouple the motion features and posture change features of the gesture, the TBSA-Net is used to preprocess the input infrared video data, and accurate gesture skeleton data is generated from the infrared image. The construction of the gesture posture evolution body is based on the transformed gesture skeleton data.

[0025] (2.2) Gesture Pose Evolution Body Construction

[0026] To construct the gesture pose evolution body, the spatial structure of the hand skeleton needs to be encoded. The specific implementation process of the construction method is as follows: first, the gesture skeleton is modeled as a cubic structure, and the original skeleton coordinate system needs to be converted into a voxel coordinate system. The conversion process is divided into two steps: normalizing the original coordinates to the [-1, 1] interval; mapping the normalized coordinates to the corresponding voxel coordinate system. In the construction process of the gesture pose evolution body, only the gesture pose and its time evolution characteristics are focused on, and the hand movement information is deliberately excluded. To ensure modeling accuracy, each frame of skeleton data is placed at the center position of the voxel space, and the voxel space is maximally utilized to improve the resolution of the skeleton modeling. For this purpose, the maximum bounding box of all skeleton points in the training set is used as the normalization reference, and the voxel center is taken as the coordinate origin, so that all skeleton coordinates are normalized to the [-1, 1] interval. In order to facilitate the processing of the subsequent gesture pose feature extraction network, it is necessary to scale and discretize the normalized coordinates to integer values in the voxel coordinate system, and embed the skeleton in a cubic volume with a resolution of R × R × R In the modeling process of the gesture pose evolution body, the skeleton is placed in the voxel space, and the voxel values occupied by the skeleton are set to 1, and the values of the remaining voxels are set to 0. For a gesture sequence , where and represent the skeleton coordinates that have been converted into voxel coordinates, the gesture pose body of the skeleton frame can be represented as a vector V with a dimension of R, R, R . When modeling the gesture pose evolution body in the time dimension, all skeletons in the sequence are concatenated into a three-dimensional voxel space at a set interval θ . Finally, the gesture pose evolution body of the sequence is a vector with a dimension of R+ θ(T-1), R, R .

[0027] (2.3) Gesture Pose Feature Extraction Network

[0028] The gesture pose feature extraction network is used to learn the pose feature changes in the gesture evolution. In the gesture pose change feature extraction network, a three-dimensional convolution layer is first used to capture local spatial structure and time information at the same time. Then four bottleneck modules are stacked to gradually learn high-level features of gesture changes from the HPEV. Inside the bottleneck module, the input features are converted to 64 channels through a 1x1x1 convolution layer. Depth separable convolution is used in the bottleneck to replace the conventional convolution to reduce the model size. The output of the depth separable convolution is converted through a 1x1x1 convolution layer to match the output channel number. The output channels of the four bottleneck modules are 128, 128, 256 and 512 respectively, and the output features of the last bottleneck module are converted into a feature vector through global average pooling.

[0029] (3) Feature fusion module

[0030] Considering the huge imbalance between the number of boundary and non-boundary frames in the gesture sequence. If the boundary detection problem is simply regarded as a binary classification problem, the imbalance between the number of boundary and non-boundary frames will have a negative impact on the segmentation accuracy. The present application designs a novel boundary balance score fusion mechanism to solve the above problem, which considers the balance of the boundary in the feature fusion module to realize the fusion of the score, and represent the proportion of non-boundary frames and boundary frames in the gesture sequence. The proportion of non-boundary video frames is multiplied by the boundary prediction score of the gesture pose network, and the proportion of boundary video frames is multiplied by the boundary prediction score of the global motion network. The boundary balance score fusion mechanism enhances the contribution of gesture pose changes to gesture boundary recognition. In the feature fusion module, the boundary prediction scores output by the global motion network and the pose change network are fused, and the feature fusion module finally outputs the possibility score of each frame as a boundary. The score reflects the credibility of each frame being labeled as a boundary. Therefore, a threshold method based on probability score is used to detect the boundary.

[0031] For the specific implementation of the method of the present application, it can be combined with Figure 2 as shown below: Step 1: Generate the overall spatio-temporal features in the gesture motion process.

[0032] The input original gesture sequence is input frame by frame into the global motion network, and the gesture sequence is modeled and the spatial features in the gesture motion process are extracted through N feature extractors. The feature extractor is composed of a two-dimensional convolutional neural network and a parallel recognition module and a correlation module. The feature vector output by the 2D convolution has a size of C xH × W , C is the feature dimension, H and W is the height and width. In the association module, the local feature block of the t-th frame is multiplied with the local feature block of the adjacent t+1-th frame , and the similarity matrix is calculated. The similarity calculation method is as follows: ; The similarity matrix of the matrix is calculated, and is input into the Sigmoid function and subtracted by 0.5, so that the value range of the similarity matrix is (-0.5, 0.5). The positive value enhances the inter-frame information, and the negative value suppresses the inter-frame information. The calculation formula is as follows: ; The similarity matrix is multiplied with the features of the × K neighborhood of the adjacent frame K , and the calculation formula is as follows:

[0033] Finally, the features of the two frames before and after aggregation are obtained, and the bidirectional inter-frame information is calculated, and the calculation formula is as follows: .

[0034] In the recognition module, the large-scale 3D space-time convolution is decomposed into 4×3 parallel convolution kernels along the time and space dimensions, and the size of each convolution kernel is . The time expansion coefficient and the space expansion coefficient of different convolution kernels are increased between [1, 2, 3, 4] and [1, 2, 3] respectively. The outputs of the association module and the recognition module are multiplied, and then one-dimensional convolutional neural network and bidirectional long short-term memory network are used for short-term and long-term time feature modeling. The finally generated feature is the overall space-time feature in the gesture motion process, and the overall space-time feature passes through the full connection layer and the activation function layer in turn, and the prediction score of the i-th frame boundary is output through the Sigmoid function .

[0035] Step 2: Generate the posture change feature in the gesture motion process.

[0036] The first stage is the skeletonization of the IR image, and the gesture sequence The gesture posture estimation network (TBSA-Net) is converted into corresponding skeleton data. A graph convolution network is introduced to integrate hand structure information into the network, and the hand candidate box that best matches the temperature distribution is selected as the final skeleton data for output. Each frame of the generated skeleton data has 21 joint nodes, as shown in Figure 3 .

[0037] The two-stage hand gesture posture evolution body is generated. The spatial structure of the hand skeleton is encoded to construct the hand gesture posture evolution body. The construction method is as follows: 1) Calculate the maximum bounding box of the skeleton. For the nth gesture , the hand skeleton at the t-th frame is , and the side length of the bounding box of this frame skeleton is defined as , and , and the calculation formula is as follows: , wherein =21 represents the key nodes of the gesture, and the maximum side length of the bounding box of all skeletons is defined as , and , the calculation method of is the same, and the calculation formula is as follows: , wherein , is the number of skeleton sequence samples in the training set; 2) Skeleton normalization processing. Use , and as the scaling factor of each dimension to normalize the original coordinates of each skeleton to the [-1, 1] interval. For the x coordinate of each joint in each skeleton, the normalization formula is: , wherein and are the minimum and maximum x coordinate values of all joints in the skeleton; 3) Normalize and discretize the coordinates into integer values in the voxel coordinate system, and embed the skeleton into a cubic volume with a resolution of . The normalized x coordinates can be converted to volume coordinates by the following formula: ; and 4) The skeleton is placed in the voxel space, and the value of the voxel occupied by the skeleton is set to 1, and the value of the rest of the voxel is set to 0; the gesture sequence , wherein and indicates that the skeleton coordinates have been converted into voxel coordinates, and the gesture posture of the skeleton frame is represented as a vector with a dimension of V The element with index ( ) in the vector is represented by the following formula, which defines whether the voxel is occupied by the skeleton joint in space, wherein ; , 5) Model the change of the skeleton in the time dimension, splice all the skeletons in the sequence at a set interval into a three-dimensional voxel space; For a gesture sequence , wherein the skeleton of the t frame is , construct its corresponding gesture posture volume; Finally, the gesture posture evolution volume of the sequence is a vector with a dimension of , wherein θ represents the interval between adjacent skeletons, and the calculation formula is as follows: .

[0038] The three stages are posture feature extraction, and the gesture posture feature extraction network learns the posture feature changes in the gesture evolution volume. The structure of the gesture posture feature extraction network is as follows: first, a three-dimensional convolution layer with a kernel size of 7x3x3 is used to synchronously capture local spatial structure and temporal correlation information. Then, four bottleneck modules are cascaded to gradually extract high-order time sequence features from the gesture posture evolution volume (HPEV). The internal processing flow of each bottleneck module includes: mapping the input features to 64 channels through a 1x1x1 convolution layer; using a 3x3x3 depth separable convolution to replace the standard convolution operation, which significantly reduces the model parameter quantity; adjusting the output channel dimension through a 1x1x1 convolution layer The output channel numbers of the four bottleneck modules are 128, 128, 256, and 512, respectively. The output features of the final module are converted into a compact feature vector through a global average pooling layer, realizing the effective coding of the posture evolution features. The output feature vector passes through the full connection layer and the activation function layer in turn, and outputs the prediction score of the i frame boundary through the Sigmoid function.

[0039] Step 3: Perform feature fusion, and fuse the two boundary prediction scores using the balanced score fusion method to generate the final gesture boundary score.

[0040] There is a huge imbalance between the number of boundary and non-boundary frames in a gesture sequence. If the boundary detection problem is simply regarded as a binary classification problem, the imbalance between the number of boundary and non-boundary frames will have a negative impact on the segmentation accuracy. The present application designs a novel balanced score fusion mechanism to solve the above problem, and considers the balance of the boundary in the feature fusion module to realize the fusion of scores, and represent the proportion of non-boundary frames and boundary frames in the gesture sequence. The calculation formula of the balanced score fusion (Balanced Score Fusion) is as follows: ; The network outputs the possibility score of each frame as a boundary, and the score reflects the credibility of each frame being labeled as a boundary. Therefore, a threshold method based on probability score is used to detect the boundary. The judgment method is as follows: ; wherein, i represents the frame index in the sequence. The threshold used in the implementation is to distinguish between boundary and non-boundary.

[0041] The above describes in detail the dual-flow network structure based on the balanced score fusion mechanism in the infrared gesture segmentation provided by the present application. The principle and implementation mode of the present application are described, and the above description of the embodiments is a method for helping to understand the present application and its core idea. It should be pointed out that, for those skilled in the art, without departing from the principle of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. An infrared gesture detection and segmentation method based on a balanced score fusion mechanism, characterized in that, This method employs a two-stream network structure, including a global motion network, a gesture and pose network, and a feature fusion module; A global motion network is used to extract the overall spatiotemporal features of gesture movements; The gesture pose network includes a temperature-sensing-based gesture pose estimation network, a gesture pose evolutionary network, and a pose feature extraction network, which are used to extract the pose change features of gestures. The feature fusion module balances the score by fusing overall spatiotemporal features and attitude change features; The method includes the following steps: (1) Generate the overall spatiotemporal features of the gesture movement process. The original infrared video is input frame by frame into the global motion network, and spatial feature extraction, short-term feature extraction and long-term feature extraction are performed in sequence. The extracted features are used to output the prediction score of the boundary using the activation function. (2) Generate the posture change features during the gesture movement process. The video frame is converted into the corresponding skeleton data through the gesture posture estimation network. The gesture posture evolution body is constructed sequentially on the skeleton data. The posture change features are extracted using the posture feature network to generate the boundary prediction score. (3) Perform feature fusion and use the balanced score fusion method to fuse the two boundary prediction scores to generate the final gesture boundary score.

2. The infrared gesture detection and segmentation method based on the balanced score fusion mechanism according to claim 1, characterized in that, The global motion network is based on feature extractor modeling and extracts spatial features during gesture movement; The feature extractor includes a two-dimensional convolutional neural network and parallel recognition and association modules. The two-dimensional convolutional neural network extracts spatial features of the image. The association module calculates the dot product similarity of each pixel in adjacent frames and weights and aggregates the features of the current pixel with the features of related positions in adjacent frames. The recognition module decomposes into multiple parallel convolutional kernels along the temporal and spatial dimensions to extract features. The outputs of the association and recognition modules are multiplied, and then a one-dimensional convolutional neural network and a bidirectional long short-term memory network are used to model short-term and long-term temporal features, respectively. The final generated feature is the overall spatiotemporal feature of the gesture movement process. The spatiotemporal feature is an overview of the global information of the gesture movement. The overall spatiotemporal feature passes through a fully connected layer and an activation function layer in sequence, and the prediction score of the i-th frame boundary is output through the sigmoid function.

3. The infrared gesture detection and segmentation method based on the balanced score fusion mechanism according to claim 2, characterized in that, The original gesture sequence is represented as In the association module, for the local feature block of frame t... Local feature blocks in adjacent frame t+1 The similarity matrix is ​​calculated. : ； in, Indicates the pixel in the local feature block The row and column positions, Indicates the pixel in the local feature block The row and column positions, This represents the feature dimension in the feature vector; Further calculate the similarity matrix Similarity matrix The similarity matrix Input the Sigmoid function and subtract 0.5 to obtain the similarity matrix. The value range is (-0.5, 0.5). Positive values ​​in the range enhance inter-frame information, while negative values ​​suppress inter-frame information. Similarity matrix With adjacent frames of K × K The feature multiplication of the neighborhood is calculated as follows: Indicates the first frame of frame t+1 OK Column pixel values, Represents the characteristic product matrix; By aggregating the features of the two frames, bidirectional inter-frame information is obtained, calculated using the following formula: ； in , All of these represent learnable parameters. In the recognition module, the large-scale 3D spatiotemporal convolution is decomposed into 4×3 parallel convolution kernels along the temporal and spatial dimensions, with each kernel having a size of [missing information]. The time dilation coefficient of different convolution kernels and spatial expansion coefficient It increases between [1,2,3,4] and [1,2,3] respectively.

4. The infrared gesture detection and segmentation method based on the balanced score fusion mechanism according to claim 1, characterized in that, The gesture network mainly performs the following three stages of operations: The first stage is IR image skeletonization. Based on the hand gesture estimation model based on temperature structure perception, skeleton data that conforms to the temperature characteristics of human hands is generated. By introducing a graph convolutional network, hand structure information is incorporated into the network, and the candidate box of the hand that best matches the temperature distribution is selected as the final skeleton data for output. The second stage is to generate a hand posture evolution body, which decouples the posture evolution during the gesture change process to obtain the hand posture evolution body; The third stage is posture feature extraction. Based on the gesture posture extraction network, the features of posture changes are extracted to enhance the model's recognition of transitional features of gestures. The convolutional layer captures local spatiotemporal features, the bottleneck layer learns high-level features of gesture changes, and outputs a feature vector. The feature vector passes through a fully connected layer and an activation function layer in sequence, and the prediction score of the i-th frame boundary is output through the Sigmoid function.

5. The infrared gesture detection and segmentation method based on the balanced score fusion mechanism according to claim 4, characterized in that, The gesture pose evolution body captures local spatiotemporal features through a 7×3×3 three-dimensional convolutional layer, and then learns high-level features of gesture changes through four bottleneck layers. The bottleneck layers use 3×3×3 depthwise separable convolutions, and the output features of the final bottleneck layer are transformed into a feature vector with 512 channels through global average pooling.

6. The infrared gesture detection and segmentation method based on the balanced score fusion mechanism according to claim 4, characterized in that, The specific processing steps in the second phase include: 1) Calculate the maximum bounding box of the skeleton for the nth gesture. Its hand skeleton in frame t is The side length of the bounding box of the frame skeleton is defined as , and The calculation formula is as follows: ， in, The key nodes representing gestures, and the maximum side length of the bounding box of all skeletons are defined as follows: , and The calculation method and same, The calculation formula is as follows: ， in, , It is the number of skeleton sequence samples in the training set; 2) Skeleton normalization processing, using , and As a scaling factor for each dimension, the original coordinates of each skeleton are normalized to the interval [-1, 1], for each joint in each skeleton. x The coordinates, whose normalization formula is: ， in, and These are the minimum and maximum values ​​of all relevant nodes in the skeleton. x Coordinate values; 3) Normalize the coordinates, scale, and discretize them into integer values ​​in a voxel coordinate system, then embed the skeleton into a resolution of [resolution missing]. In the volume of the cube, the normalized... x Coordinates can Convert to volume coordinates using the following formula. The calculation formula is as follows: ； 4) The skeleton is placed in voxel space, with the voxel occupied by the skeleton set to 1, and the values ​​of the remaining voxels set to 0; Gesture sequence ,in and This indicates that the skeleton coordinates have been converted to voxel coordinates, and the gesture pose volume of the skeleton frame is represented as a single-dimensional... vector V ,vector The index is ( The elements of ) are represented by the following formula, which defines whether a voxel is occupied by skeletal joints in space, where ; ， 5) Model the changes in the skeleton over time, and arrange all skeletons in the sequence at set intervals. spliced ​​together into a three-dimensional voxel space; For a gesture sequence The skeleton of frame t is Construct its corresponding gesture body; Ultimately, the sequence of gestures evolved. It is a dimension The vector, where θ The interval between adjacent skeletons is represented by the following formula: 。 7. The infrared gesture detection and segmentation method based on the balanced score fusion mechanism according to claim 1, characterized in that, The feature fusion module performs score fusion on the overall spatiotemporal features and attitude change features, including considering the impact of balance, and uses... and This represents the proportion of non-boundary and boundary frames in the gesture sequence. The frame difference is balanced by multiplying the boundary proportion by the prediction scores of the global motion network and the pose extraction network, and the output is a probability score for each frame to be a boundary. A thresholding method based on probability scores is used to detect boundaries.

8. The infrared gesture detection and segmentation method based on the balanced score fusion mechanism according to claim 7, characterized in that, Boundary detection determines whether the i-th frame is a gesture boundary; the boundary must satisfy certain conditions. Greater than or equal to the threshold of 0.5.

Images