A static and dynamic combined-based fuzzy gesture recognition method, device and medium

Abstract

The application discloses a static and dynamic combined fuzzy gesture recognition method and device and medium. The method comprises the following steps: extracting gesture feature points of a gesture image to be recognized of a user at a current time, and taking the gesture feature points as first feature points; normalizing a first position difference function of the first feature points to generate a spatial domain feature matrix; obtaining a plurality of historical gesture images of the user within a preset historical time length; extracting gesture feature points of each historical gesture image, determining gesture feature points at the same position in adjacent historical gesture images, and taking the gesture feature points at the same position as second feature points; normalizing a second position difference function of the second feature points to generate a space-time sequence feature matrix; inputting the spatial domain feature matrix and the space-time sequence feature matrix into a pre-constructed deep convolutional neural network model respectively, and outputting a gesture posture and a gesture action category of the user at the current time, thereby improving the gesture recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device and medium for fuzzy gesture recognition based on a combination of static and dynamic gestures. Background Technology

[0002] With the development of technology, contactless human-computer interaction technology has become a hot application area. Among them, gesture touch control can effectively replace touch screen operation. For example, when using equipment such as hospital self-service machines and unmanned retail machines, it can effectively control the risk of cross-infection of viruses, becoming a hygienic and fashionable emerging technology.

[0003] Currently, using gestures to simulate mouse movement and clicks is also known as gesture mouse. Simply put, it involves using the mouse to perform actions to control software to complete certain operations. It is widely used in browser and desktop window management, such as drawing specific paths in blank areas to achieve pre-defined functions. Gesture mouse mainly includes two aspects: mouse position positioning and gesture command control. Mouse position positioning is mainly used for positioning screen controls or links during operations such as clicking or handwriting in the air. Gesture commands can simulate mouse clicks / double clicks or other specific instructions.

[0004] However, unlike optical mice or touchscreens placed on a regular desktop, gesture mice are prone to uncontrollable errors in position positioning and command classification. This is mainly because the gesture recognition and feature point positioning process is easily affected by factors such as ambient light, camera angle, slight tremors of the human body, and non-standard gesture postures, which can cause unwanted random jitter in the simulated mouse position. In particular, due to factors such as shooting angle or non-standard movements, the gesture posture may be ambiguous, causing the recognition result to repeatedly jump between two recognition categories, affecting the normal operation of the device and resulting in low gesture recognition accuracy. Summary of the Invention

[0005] This application provides a method, device, and medium for fuzzy gesture recognition based on a combination of static and dynamic gestures, which solves the problem of low gesture recognition accuracy when performing touch operations with a gesture mouse.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] On one hand, embodiments of this application provide a fuzzy gesture recognition method based on static and dynamic combination. The method includes: acquiring a user's gesture image to be recognized at the current moment; extracting gesture feature points from the gesture image to be recognized and using the gesture feature points as first feature points; determining a first position difference function for the first feature points; normalizing the first position difference function to generate a spatial domain feature matrix for the first feature points; acquiring multiple frames of historical gesture images of the user within a preset historical time period; wherein the preset historical time period ends at the current moment; extracting gesture feature points from each frame of historical gesture images; determining gesture feature points at the same position in adjacent frames of historical gesture images and using the gesture feature points at the same position as second feature points; determining a second position difference function for the second feature points; normalizing the second position difference function to generate a spatiotemporal sequence feature matrix for the second feature points; and inputting the spatial domain feature matrix and the spatiotemporal sequence feature matrix into a pre-constructed deep convolutional neural network model to output the user's gesture posture and gesture action category at the current moment.

[0008] In one example, the step of inputting the spatial domain feature matrix and the spatiotemporal sequence matrix into a pre-constructed deep convolutional neural network model to output the user's gesture posture and gesture category at the current moment specifically includes: extracting a first feature map corresponding to the spatial domain feature matrix through a first convolutional network; extracting a second feature map corresponding to the spatiotemporal sequence feature matrix through a second convolutional network; wherein the second feature map and the first feature map have the same scale; fusing the first feature map and the second feature map through a preset attention mechanism fusion expression to generate a fused feature; and inputting the fused feature into a third convolutional network to output the user's gesture posture and gesture category at the current moment.

[0009] In one example, the step of fusing the first feature map and the second feature map using a preset attention mechanism fusion expression to generate fused features specifically includes: merging the channels of the first feature map and the second feature map to obtain a concatenated feature map; performing a cross-multiplication on the features in the concatenated feature map to obtain cross-processed features; probabilizing the cross-processed features to obtain initial fused features; performing an inverse transformation on the initial fused features to generate features of the same scale as the concatenated feature map; and weighting the features with learnable parameters and adding them to the concatenated feature map to generate fused features.

[0010] In one example, the process of performing a cross-dot product on the features in the concatenated feature map to obtain cross-processed features specifically includes: in, Wherein, FF is the one-dimensional unfolded form feature corresponding to each feature map in the stitched feature map, and the scale of FF is C×D, where C is the total number of all feature maps in the stitched feature map, and D is the number of elements in the stitched feature map. mk It refers to the k-th element of feature map m, FF nk It refers to the k-th element of feature map number n, G mn This refers to the features of cross-processing; the probabilistic transformation of the cross-processed features to obtain the initial fused features specifically includes obtaining the initial fused features of dimension C using the following formula: Among them, TE mn For G mn Probabilistic initial fusion features; where TE mn For G mn The process involves: probabilistically transforming the initial fusion features to generate features at the same scale as the stitched feature map; weighting the features with a learnable parameter and adding them to the stitched feature map to generate fusion features. Specifically, this includes: combining C×C initial fusion features in determinant order to generate an initial fusion matrix of dimension C×C; and generating the fusion features using the following formula: FC = β·reshape(TE·FF) + FB, where FC is the fusion feature, TE is the initial fusion matrix, FB is the stitched feature map, reshape restores TE·FF to the same scale as FB, and β is a learnable parameter.

[0011] In one example, the first position difference function for determining the first feature point specifically includes: δ i =P i -P i-1 , Where, δ i Let P be the first position difference function value of the i-th first feature point in the gesture image. i P represents the position of the i-th first feature point in the gesture image. i-1 Let N be the position of the (i-1)th first feature point in the gesture image, where N is a positive integer greater than 1; the step of normalizing the first position difference function to determine the spatial domain feature matrix of the first feature point specifically includes: setting δ0 = 0, and the normalization function expression is as follows: Where, γ j δ refers to the normalized function value of the j-th first feature point in the gesture image. j δ refers to the first position difference function value of the j-th first feature point in the gesture image. iLet be the first position difference function value of the i-th first feature point in the gesture image; if the gesture image is a two-dimensional image, then the spatial domain feature matrix is ​​as follows: Where X1 is the spatial domain characteristic matrix, γ x，j γ refers to the normalized transformation value of the j-th first feature point in the x-direction. y，j It refers to the normalized transformation value of the j-th first feature point in the y-direction.

[0012] In one example, the second position difference function for determining the second feature point specifically includes: α t，i =P t，i -P t-1，i , Where T is a positive integer greater than 1, α t，i P refers to the second position difference function value of the i-th second feature point at time t. t，i This refers to the position of the i-th second feature point at time t; the normalization of the second position difference function to determine the spatiotemporal sequence feature matrix of the second feature point specifically includes: letting α 0，i If = 0, then the expression for the normalization function is as follows: Where, β t，k It refers to the normalized function value of the k-th second feature point at time t, α t，k It refers to the second position difference function value of the k-th second feature point at time t, α t，i This refers to the second position difference function value of the i-th second feature point at time t; if the historical gesture image is a two-dimensional image, then the spatiotemporal sequence feature matrix of a single second feature point is as follows: Among them, D k It refers to the spatiotemporal sequence feature matrix of the k-th feature point, β x，t，k β refers to the normalized function value of the k-th feature point at time t in the x-direction. y，t，k This refers to the normalized function value of the k-th feature point at time t in the y-direction; the spatiotemporal sequence feature matrix of all second feature points is as follows: X2 = [D0 D1 … D k … D N-1 ].

[0013] In one example, before inputting the spatial domain feature matrix and the spatiotemporal sequence feature matrix into a pre-constructed deep convolutional neural network model, the method further includes: acquiring a sample gesture image of a sample user at the current moment, and determining the sample spatial domain feature matrix of the sample gesture image; acquiring multiple frames of sample historical gesture images of the sample user within a preset historical time period, and determining the sample spatiotemporal sequence feature matrix of the multiple frames of sample historical gesture images; and performing supervised training on the first convolutional network, the second convolutional network, and the third convolutional network by labeling the gesture action tags and gesture posture tags in the sample gesture images, as well as the sample spatial domain feature matrix and the sample spatiotemporal sequence matrix.

[0014] In one example, outputting the user's gesture posture and gesture action category at the current moment specifically includes: outputting the user's gesture posture at the current moment through a first network output port; wherein the gesture posture includes at least one of palm open, palm closed, and fingers raised; outputting the user's gesture action category at the current moment through a second network output port; wherein the gesture action category includes gesture stillness and gesture movement; after outputting the user's gesture posture and gesture action category at the current moment, the method further includes: determining the loss of the gesture posture and the loss of the gesture action category; performing a weighted summation of the loss of the gesture posture and the loss of the gesture action category based on a preset weighting coefficient to determine a loss function; and adjusting the parameters of the deep convolutional neural network model using the loss function.

[0015] On the other hand, embodiments of this application provide a fuzzy gesture recognition device based on a combination of static and dynamic gestures, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: acquire a user's gesture image to be recognized at the current moment, extract gesture feature points from the gesture image to be recognized, and use the gesture feature points as first feature points; determine a first position difference function for the first feature points, normalize the first position difference function, and generate a spatial domain feature matrix for the first feature points; acquire the... The system describes multiple frames of historical gesture images of a user within a preset historical time period; wherein the preset historical time period ends at the current moment; gesture feature points are extracted from each frame of historical gesture images; in adjacent frames of historical gesture images, gesture feature points at the same position are identified and used as second feature points; a second position difference function of the second feature points is determined; the second position difference function is normalized to generate a spatiotemporal sequence feature matrix of the second feature points; the spatial domain feature matrix and the spatiotemporal sequence feature matrix are respectively input into a pre-constructed deep convolutional neural network model to output the user's gesture posture and gesture action category at the current moment.

[0016] On the other hand, embodiments of this application provide a non-volatile computer storage medium for fuzzy gesture recognition based on a combination of static and dynamic methods, storing computer-executable instructions. These computer-executable instructions are configured to: acquire a user's gesture image to be recognized at the current moment; extract gesture feature points from the gesture image to be recognized and use these gesture feature points as first feature points; determine a first position difference function for the first feature points; normalize the first position difference function to generate a spatial domain feature matrix for the first feature points; acquire multiple frames of historical gesture images of the user within a preset historical time period; wherein the preset historical time period ends at the current moment; extract gesture feature points from each frame of historical gesture images; determine gesture feature points at the same position in adjacent frames of historical gesture images and use these gesture feature points at the same position as second feature points; determine a second position difference function for the second feature points; normalize the second position difference function to generate a spatiotemporal sequence feature matrix for the second feature points; and input the spatial domain feature matrix and the spatiotemporal sequence feature matrix into a pre-constructed deep convolutional neural network model, respectively, to output the user's gesture posture and gesture action category at the current moment.

[0017] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0018] This application embodiment generates a spatial domain feature matrix of the gesture image to be recognized, which enables the correlation between gesture feature points, making it more conducive to the classification of the current gesture posture. It also generates a spatiotemporal sequence feature matrix, which enables the correlation of gesture feature points in the time dimension, making it more conducive to the classification of gesture actions. By preprocessing the gesture feature point data and then inputting it into a deep convolutional neural network model, the model uses an attention mechanism fusion method for feature fusion. The model outputs the user's gesture posture and gesture action category at the current moment. Since the static features and dynamic motion characteristics of the gesture are combined during the training and prediction process, the model is more robust to the influence of local conditions (such as light, angle, defocus) and other factors, improving the generalization recognition ability of gesture recognition, thereby improving the accuracy and efficiency of gesture recognition. Attached Figure Description

[0019] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which:

[0020] Figure 1 A flowchart illustrating a fuzzy gesture recognition method based on a combination of static and dynamic elements, provided in an embodiment of this application;

[0021] Figure 2 A schematic diagram of gesture feature points of a gesture image to be recognized, provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of the structure of a fuzzy gesture recognition device based on a combination of static and dynamic gestures, provided as an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a fuzzy gesture recognition method based on a combination of static and dynamic methods, provided as an embodiment of this application. This method can be applied to different business domains, and certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0026] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0027] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0028] Figure 1 The process may include the following steps:

[0029] S102: Obtain the image of the user's gesture to be recognized at the current moment, extract the gesture feature points of the gesture image to be recognized, and use the gesture feature points as the first feature points.

[0030] The gesture image to be recognized can be acquired by an image capturing device set on the screen device. When the image capturing device is a two-dimensional device, the gesture image to be recognized is a two-dimensional gesture image. When the image capturing device is a three-dimensional device, the gesture image to be recognized is a three-dimensional gesture image.

[0031] It should be noted that the method for extracting gesture feature points from the image of the gesture to be recognized can be restricted according to actual needs, and no specific limitations are made here. For example, gesture feature points can be extracted from the image of the gesture to be recognized through a pre-built image recognition model.

[0032] More intuitively, Figure 2 This is a schematic diagram of the gesture feature points in the image of the gesture to be recognized, such as... Figure 2 As shown, the gesture feature points include 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 and 20, for a total of 21 gesture feature points.

[0033] S104: Determine the first position difference function of the first feature point, normalize the first position difference function, and generate the spatial domain feature matrix of the first feature point.

[0034] In some embodiments of this application, if the positions of N gesture feature points in a two-dimensional or three-dimensional image are {P0, P1, ..., P...} N}, where N is a positive integer greater than 1. The first position difference function of the first feature point is determined by the following expression:

[0035]

[0036] Where, δ i Let P be the difference function value of the first position of the i-th first feature point in the gesture image. iLet P be the position of the i-th first feature point in the gesture image. i-1 Let be the position of the (i-1)th first feature point in the gesture image.

[0037] Furthermore, if the input uses relative displacement coordinates, then let δ0 = 0, and the normalization function expression is as follows:

[0038]

[0039] Where, γ j δ refers to the normalized function value of the j-th first feature point in the gesture image. j δ refers to the difference function value of the j-th first feature point in the gesture image, and δ refers to the difference function value of the i-th first feature point at the first position in the gesture image.

[0040] If the gesture image is a two-dimensional image, the spatial domain feature matrix after transformation in the x and y directions is as follows:

[0041]

[0042] Where X1 is the spatial domain characteristic matrix, γ x，j γ refers to the normalized transformation value of the j-th first feature point in the x-direction. y，j It refers to the normalized transformation value of the j-th first feature point in the y-direction.

[0043] It should be noted that gesture images can also be three-dimensional images, which correspond to a three-dimensional spatial domain feature matrix.

[0044] S106: Obtain multiple frames of the user's historical gesture images within a preset historical time period; wherein, the preset historical time period ends at the current moment. That is, it is equivalent to continuously sampling backward from the current moment, for example, continuously sampling 16 frames backward from the current moment, which may not necessarily correspond to real-time video frames.

[0045] S108: Extract gesture feature points from each frame of historical gesture images. In adjacent frames of historical gesture images, determine gesture feature points at the same position and use the gesture feature points at the same position as second feature points.

[0046] In other words, if there are multiple sets of adjacent frames, there will be multiple sets of gesture feature points with the same position. That is, gesture feature points with the same position at different times will be used as the second feature points.

[0047] S110: Determine the second position difference function of the second feature point, normalize the second position difference function, and generate the spatiotemporal sequence feature matrix of the second feature point.

[0048] In some embodiments of this application, based on the definition of the difference and normalization matrix of the same feature point positions in consecutive frames within a time window, the time window length of the video discrete sampling frame is taken as T, that is, T is the total number of samples within the time window. For example, if the number of samples is 8, it represents a time length of 8, not a physical time in minutes or seconds. T-1 represents the video frame number (the gesture image to be recognized) at the current moment, and 0 represents the video frame number (historical gesture image) at the farthest moment. Then, the difference function of the same feature point at different moments is:

[0049] α t，i =P t，i -P t-1，i ,in,

[0050] Where T is a positive integer greater than 1, and P represents the value of the second position difference function of the i-th second feature point at time t. t，i This refers to the position of the i-th second feature point at time t;

[0051] Let α 0，i If = 0, then the expression for the normalization function is as follows:

[0052]

[0053] Where, β t，k It refers to the normalized function value of the k-th second feature point at time t, α t，k It refers to the second position difference function value of the k-th second feature point at time t, α t，i It refers to the second position difference function value of the i-th second feature point at time t;

[0054] If the historical gesture image is a two-dimensional image, then the spatiotemporal sequence feature matrix of a single second feature point is as follows:

[0055]

[0056] Among them, D k It refers to the spatiotemporal sequence feature matrix of the k-th feature point, β x，t，k β refers to the normalized function value of the k-th feature point at time t in the x-direction. y，t，k It refers to the normalized function value of the k-th feature point at time t in the y-direction;

[0057] The spatiotemporal sequence feature matrices of all second feature points are as follows:

[0058] X2 = [D0 D1 … D] k … D N-1 ]

[0059] S112: Input the spatial domain feature matrix and the spatiotemporal sequence feature matrix into a pre-built deep convolutional neural network model, and output the user's gesture posture and gesture action category at the current moment.

[0060] Specifically, the user's current hand gesture is output through the first network output port. The hand gesture includes at least one of the following: palm open, palm closed, and fingers raised.

[0061] The user's gesture category at the current moment is output through the second network output port. The gesture category includes still gesture and moving gesture.

[0062] To verify the model's accuracy, the parameters of the deep convolutional neural network model are adjusted in reverse. After outputting the gesture pose through the first network output port, the loss of the gesture pose is calculated. After outputting the gesture action category through the second network output port, the loss of the gesture action category is calculated. Based on a preset weighting coefficient, the loss of the gesture pose and the loss of the gesture action category are weighted and summed to determine the loss function. The parameters of the deep convolutional neural network model are then adjusted using the loss function.

[0063] This application innovates in areas such as gesture feature point data preprocessing and the fusion of gesture action features and posture features. By generating a spatial domain feature matrix of the gesture image to be recognized, the application can realize the correlation between gesture feature points, which is more conducive to the classification of the current gesture posture. It also generates a spatiotemporal sequence feature matrix, which can realize the correlation of gesture feature points in the time dimension, which is more conducive to the classification of gesture actions. By preprocessing the gesture feature point data and then inputting it into a deep convolutional neural network model, the application improves the generalization recognition ability of gesture recognition, thereby improving the accuracy and efficiency of gesture recognition.

[0064] In some embodiments of this application, supervised learning is performed using a feature fusion network with an attention mechanism, enabling the network to enhance its ability to judge ambiguous gesture poses and actions based on motion features such as stillness and displacement. Clearly, matrix X1 represents the correlation between gesture feature points, which is more conducive to classifying the current gesture pose, while matrix X2 represents the correlation between gesture feature points in the time dimension, which is more conducive to classifying gesture actions.

[0065] Specifically, firstly, a first feature map corresponding to the spatial domain feature matrix is ​​extracted using a first convolutional network. Then, a second feature map corresponding to the spatiotemporal sequence feature matrix is ​​extracted using a second convolutional network. The second feature map and the first feature map have the same scale. Next, the first and second feature maps are fused using a preset attention mechanism fusion expression to generate a fused feature. Finally, the fused feature is input into a third convolutional network to output the user's gesture pose and gesture category at the current moment.

[0066] Furthermore, in generating the fused features, the first and second feature maps are first merged to obtain a concatenated feature map. Then, the features in the concatenated feature map are cross-multiplied to obtain cross-processed features. Next, the cross-processed features are probabilized to obtain the initial fused features. Finally, the initial fused features are inversely transformed to generate features of the same scale as the concatenated feature map. These features are then weighted using learnable parameters and added to the concatenated feature map to generate the fused features.

[0067] Furthermore, cross-multiplication is performed on the features in the concatenated feature map to obtain cross-processed features, specifically including:

[0068] in,

[0069] Here, FF represents the one-dimensional unfolded form of each feature map in the concatenated feature map. The scale of FF is C×D, where C is the total number of feature maps in the concatenated feature map, and D is the number of elements in the concatenated feature map. mk It refers to the k-th element of feature map m, FF nk It refers to the k-th element of feature map number n, G mn This refers to the characteristic of cross-processing.

[0070] Furthermore, the features of the cross-processing are probabilistically transformed to obtain initial fused features, specifically including:

[0071]

[0072] Among them, TE mn For G mn Probabilistic initial fusion features, G mj It's FF mk With FF jk The dot product between them.

[0073] Furthermore, by combining C×C initial fusion features in determinant order, an initial fusion matrix of dimension C×C is generated;

[0074] The fusion features are generated using the following formula:

[0075] FC = β·reshape(TE·FF) + FB

[0076] Where FC is the fused feature, TE is the initial fused matrix, FB is the stitched feature map, reshape is used to restore TE·FF to the same scale as FB, and β is a learnable parameter.

[0077] In other words, after undergoing convolution or concatenation operations again, the fully connected (FC) function enters the Softmax, Logistic, or similar functions for probability prediction, and outputs the pose and action category represented by the current gesture.

[0078] In some embodiments of this application, a deep convolutional neural network model needs to be pre-trained. First, the sample gesture image of the user to be identified at the current moment is obtained, and the sample spatial domain feature matrix of the sample gesture image is determined. Then, multiple frames of sample historical gesture images of the user within a preset historical time period are obtained, and the sample spatiotemporal sequence feature matrix of the multiple frames of sample historical gesture images is determined.

[0079] Finally, supervised training is performed on the first, second, and third convolutional networks by labeling the gesture action and gesture pose labels in the sample gesture images, as well as the sample spatial domain feature matrix and sample spatiotemporal sequence matrix.

[0080] Users expect their gestures to remain stable in position and posture when their actions are relatively still, so they can indicate, select, or consider the next action; and when their actions are relatively moving, they expect their gestures to move smoothly and their posture to switch seamlessly, so they can move the mouse to draw graphics, select the next control, or perform other operations. Therefore, when training a neural network, combining already labeled gesture action labels and posture labels with methods such as gesture feature point cross-preprocessing and internal feature fusion networks can effectively improve the generalization ability of gesture recognition.

[0081] The model employs an attention mechanism to fuse features. The model outputs the user's gesture posture and gesture category at the current moment. Because the static features and dynamic motion characteristics of the gesture are combined during training and prediction, the model is more robust to local conditions (such as lighting, angle, and defocus).

[0082] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S102 to S112 will be described sequentially, but this does not mean that steps S102 to S112 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1 The order in which steps S102 to S112 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S102 to S112 can be appropriately adjusted according to actual needs.

[0083] Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0084] Figure 3 A schematic diagram of a fuzzy gesture recognition device based on a combination of static and dynamic gestures, provided in an embodiment of this application, is shown. The device includes:

[0085] At least one processor; and,

[0086] A memory communicatively connected to the at least one processor; wherein,

[0087] The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to:

[0088] Obtain the image of the user's gesture to be recognized at the current moment, extract the gesture feature points of the gesture image to be recognized, and use the gesture feature points as the first feature points;

[0089] Determine the first position difference function of the first feature point, normalize the first position difference function, and generate the spatial domain feature matrix of the first feature point;

[0090] Acquire multiple frames of historical gesture images of the user within a preset historical time period; wherein the preset historical time period ends at the current moment;

[0091] Extract gesture feature points from each frame of historical gesture images. In adjacent frames of historical gesture images, determine gesture feature points at the same position and use these gesture feature points at the same position as second feature points.

[0092] Determine the second position difference function of the second feature point, normalize the second position difference function, and generate the spatiotemporal sequence feature matrix of the second feature point;

[0093] The spatial domain feature matrix and the spatiotemporal sequence feature matrix are respectively input into a pre-built deep convolutional neural network model, which outputs the user's gesture posture and gesture category at the current moment.

[0094] Some embodiments of this application provide a non-volatile computer storage medium for fuzzy gesture recognition based on a combination of static and dynamic gestures, storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0095] Obtain the image of the user's gesture to be recognized at the current moment, extract the gesture feature points of the gesture image to be recognized, and use the gesture feature points as the first feature points;

[0096] Determine the first position difference function of the first feature point, normalize the first position difference function, and generate the spatial domain feature matrix of the first feature point;

[0097] Acquire multiple frames of historical gesture images of the user within a preset historical time period; wherein the preset historical time period ends at the current moment;

[0098] Extract gesture feature points from each frame of historical gesture images. In adjacent frames of historical gesture images, determine gesture feature points at the same position and use these gesture feature points at the same position as second feature points.

[0099] Determine the second position difference function of the second feature point, normalize the second position difference function, and generate the spatiotemporal sequence feature matrix of the second feature point;

[0100] The spatial domain feature matrix and the spatiotemporal sequence feature matrix are respectively input into a pre-built deep convolutional neural network model, which outputs the user's gesture posture and gesture category at the current moment.

[0101] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0102] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0103] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0108] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0109] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A fuzzy gesture recognition method based on a combination of static and dynamic gestures, characterized in that, The method includes: Obtain the image of the user's gesture to be recognized at the current moment, extract the gesture feature points of the gesture image to be recognized, and use the gesture feature points as the first feature points; Determine the first position difference function of the first feature point, normalize the first position difference function, and generate the spatial domain feature matrix of the first feature point; Acquire multiple frames of historical gesture images of the user within a preset historical time period; wherein the preset historical time period ends at the current moment; Extract gesture feature points from each frame of historical gesture images. In adjacent frames of historical gesture images, determine gesture feature points at the same position and use these gesture feature points at the same position as second feature points. Determine the second position difference function of the second feature point, normalize the second position difference function, and generate the spatiotemporal sequence feature matrix of the second feature point; The spatial domain feature matrix and the spatiotemporal sequence feature matrix are respectively input into a pre-built deep convolutional neural network model, which outputs the user's gesture posture and gesture category at the current moment. The step of inputting the spatial domain feature matrix and the spatiotemporal sequence matrix into a pre-constructed deep convolutional neural network model to output the user's gesture posture and gesture category at the current moment specifically includes: The first feature map corresponding to the spatial domain feature matrix is ​​extracted through the first convolutional network; A second feature map corresponding to the spatiotemporal sequence feature matrix is ​​extracted using a second convolutional network; wherein the second feature map has the same scale as the first feature map; By using a preset attention mechanism fusion expression, the first feature map and the second feature map are fused to generate fused features; The fused features are input into the third convolutional network, which outputs the user's gesture posture and gesture category at the current moment. The step of fusing the first feature map and the second feature map using a preset attention mechanism fusion expression to generate fused features specifically includes: The first feature map and the second feature map are merged to obtain a spliced ​​feature map; Perform cross-dot product on the features in the spliced ​​feature map to obtain cross-processed features; The features of the cross-processing are probabilistically transformed to obtain initial fused features; The initial fused features are inversely transformed to generate features of the same scale as the stitched feature map. The features are then weighted using learnable parameters and added to the concatenated feature map to generate fused features; The step of performing a cross-dot product on the features in the concatenated feature map to obtain cross-processed features specifically includes: ,in, Wherein, FF is the one-dimensional unfolded form feature corresponding to each feature map in the concatenated feature map, and the scale of FF is... Then C is the total number of all feature maps in the concatenated feature map, and D is the number of elements in the concatenated feature map. It refers to the k-th element of feature map m. It refers to the k-th element of feature map numbered n. This refers to the characteristics of cross-processing; The step of probabilizing the cross-processed features to obtain initial fused features specifically includes: in, for Probabilistic initial fusion features; The initial fused features are inversely transformed to generate a feature map of the same scale as the stitched feature map. Features are generated by weighting the features using learnable parameters and then adding them to the concatenated feature map to generate fused features, specifically including: pass The initial fusion features are combined in deterministic order, generating a dimension of . The initial fusion matrix; The fusion features are generated using the following formula: Among them, FC represents the fusion feature. The initial fusion matrix, where FB is the concatenated feature map. Used to Restored to the same scale as Facebook. These are learnable parameters.

2. The method according to claim 1, characterized in that, The determination of the first position difference function of the first feature point specifically includes: in, Let the first position difference function value of the i-th first feature point in the gesture image be . The position of the i-th first feature point in the gesture image. For the first The position of a first feature point in the gesture image, where N is a positive integer greater than 1; The step of normalizing the first position difference function to determine the spatial domain feature matrix of the first feature point specifically includes: make The normalization function expression is as follows: in, It refers to the normalized function value of the j-th first feature point in the gesture image. It refers to the difference function value of the j-th first feature point in the gesture image. "It refers to the first position difference function value of the i-th first feature point in the gesture image;" If the gesture image is a two-dimensional image, then the spatial domain feature matrix is ​​as follows: in, For spatial domain characteristic matrix, It refers to the normalized transformation value of the j-th first feature point in the x-direction. It refers to the normalized transformation value of the j-th first feature point in the y-direction.

3. The method according to claim 1, characterized in that, The second position difference function for determining the second feature point specifically includes: ， ， Where T is a positive integer greater than 1. This refers to the second position difference function value of the i-th second feature point at time t. This refers to the position of the i-th second feature point at time t; The step of normalizing the second position difference function to determine the spatiotemporal sequence feature matrix of the second feature point specifically includes: make Then the expression for the normalization function is as follows: ， ， in, It refers to the normalized function value of the k-th second feature point at time t. This refers to the second position difference function value of the k-th second feature point at time t. It refers to the second position difference function value of the i-th second feature point at time t; If the historical gesture image is a two-dimensional image, then the spatiotemporal sequence feature matrix of a single second feature point is as follows: , in, It refers to the spatiotemporal sequence feature matrix of the k-th feature point. It refers to the normalized function value of the k-th feature point at time t in the x-direction. It refers to the normalized function value of the k-th feature point at time t in the y-direction; The spatiotemporal sequence feature matrices of all second feature points are as follows: 。 4. The method according to claim 1, characterized in that, Before inputting the spatial domain feature matrix and the spatiotemporal sequence feature matrix into the pre-constructed deep convolutional neural network model, the method further includes: Obtain the sample gesture image of the user to be identified at the current moment, and determine the sample spatial domain feature matrix of the sample gesture image; Obtain multiple frames of sample historical gesture images of the sample user within a preset historical time period, and determine the sample spatiotemporal sequence feature matrix of the multiple frames of sample historical gesture images; Supervised training is performed on the first convolutional network, the second convolutional network, and the third convolutional network by labeling the gesture action tags and gesture posture tags in the sample gesture images, as well as the sample spatial domain feature matrix and the sample spatiotemporal sequence matrix.

5. The method according to claim 1, characterized in that, The output of the user's gesture posture and gesture type at the current moment specifically includes: The user's gesture at the current moment is output through the first network output port; wherein the gesture includes at least one of palm open, palm closed, and fingers raised. The user's gesture category at the current moment is output through the second network output port; wherein, the gesture category includes static gesture and moving gesture. After outputting the user's gesture posture and gesture type at the current moment, the method further includes: Determine the loss of the gesture posture and the loss of the gesture action category; Based on preset weighting coefficients, the loss of the gesture posture and the loss of the gesture action category are weighted and summed to determine the loss function of the deep convolutional neural network. The parameters of the deep convolutional neural network model are adjusted using the loss function.

6. A fuzzy gesture recognition device based on a combination of static and dynamic gestures, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to: perform a fuzzy gesture recognition method based on static and dynamic combination as described in any one of claims 1-5.

7. A non-volatile computer storage medium for fuzzy gesture recognition based on a combination of static and dynamic methods, storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to execute the fuzzy gesture recognition method based on static and dynamic combination as described in any one of claims 1-5.

Images