A gesture recognition method, device, equipment and medium based on feature fusion

Abstract

The present invention discloses a method, device, equipment and medium for gesture recognition based on feature fusion. The method includes: obtaining a standardized video corresponding to a target vehicle, and processing the standardized video by regional cropping to generate a target video containing a hand area; extracting spatial features from the target video based on a spatial attention model to generate posture features corresponding to the target video, and extracting temporal features from the target video based on a temporal attention model to generate motion features corresponding to the target video combined with optical flow information; processing posture features and motion features by feature fusion to generate target fusion features corresponding to the target video; determining the gesture category corresponding to the target fusion feature based on a fully connected classification network as target gesture recognition information corresponding to the target video. Through the technical solution of the present invention, dynamic gestures can be accurately and real-time recognized, thereby improving the accuracy and robustness of gesture recognition.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a gesture recognition method, device, equipment and medium based on feature fusion. Background Art

[0002] With the development of modern human-computer interaction systems, dynamic gesture recognition technology has gradually played a vital role in human-computer interaction systems.

[0003] Traditional gesture recognition methods usually rely on simple spatial feature extraction or motion feature analysis based on a specific time window. However, in complex practical application scenarios, especially when the background is complex, the lighting conditions vary greatly, the gesture is executed quickly or there is occlusion, traditional gesture recognition methods cannot effectively capture the comprehensive features of the gesture, thus affecting the accuracy and robustness of the recognition results.

[0004] Therefore, how to accurately and in real time recognize dynamic gestures and improve the accuracy and robustness of gesture recognition is a problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for gesture recognition based on feature fusion, which can solve the problems of low accuracy and low robustness of gesture recognition.

[0006] According to one aspect of the present invention, a gesture recognition method based on feature fusion is provided, comprising:

[0007] Obtaining a standardized video corresponding to the target vehicle, and performing regional cropping processing on the standardized video to generate a target video containing a hand region;

[0008] Extracting spatial features of the target video based on the spatial attention model to generate posture features corresponding to the target video, and extracting temporal features of the target video based on the temporal attention model to generate motion features corresponding to the target video combined with optical flow information;

[0009] Feature fusion processes the posture feature and the motion feature to generate a target fusion feature corresponding to the target video;

[0010] The gesture category corresponding to the target fusion feature is determined based on a fully connected classification network as the target gesture recognition information corresponding to the target video.

[0011] According to another aspect of the present invention, there is provided a gesture recognition device based on feature fusion, comprising:

[0012] A data acquisition module, used to acquire a standardized video corresponding to a target vehicle, and perform regional cropping processing on the standardized video to generate a target video containing a hand area;

[0013] A feature extraction module is used to extract spatial features of the target video based on a spatial attention model to generate posture features corresponding to the target video, and to extract temporal features of the target video based on a temporal attention model to generate motion features corresponding to the target video combined with optical flow information;

[0014] A feature fusion module, used for feature fusion processing of the posture feature and the motion feature to generate a target fusion feature corresponding to the target video;

[0015] The gesture recognition module is used to determine the gesture category corresponding to the target fusion feature based on a fully connected classification network as the target gesture recognition information corresponding to the target video.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the gesture recognition method based on feature fusion described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the gesture recognition method based on feature fusion described in any embodiment of the present invention when executed.

[0021] The technical solution of the embodiment of the present invention processes the standardized video corresponding to the target vehicle through region cropping to generate a target video containing a hand region, then extracts spatial features of the target video based on a spatial attention model to generate posture features corresponding to the target video, and extracts temporal features of the target video based on a temporal attention model to generate motion features corresponding to the target video combined with optical flow information, further, performs feature fusion processing of posture features and motion features to generate target fusion features corresponding to the target video, and finally, determines the gesture category corresponding to the target fusion feature based on a fully connected classification network as the target gesture recognition information corresponding to the target video, thereby solving the problems of low accuracy and low robustness of gesture recognition, being able to accurately and real-timely recognize dynamic gestures, and improving the accuracy and robustness of gesture recognition.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a gesture recognition method based on feature fusion provided according to Embodiment 1 of the present invention;

[0025] Figure 2 is a flow chart of a gesture recognition method based on feature fusion provided according to Embodiment 2 of the present invention;

[0026] Figure 3 is a feature extraction flow chart based on a spatial attention model provided according to Embodiment 2 of the present invention;

[0027] Figure 4 is a feature extraction flow chart based on a temporal attention model provided according to the second embodiment of the present invention;

[0028] Figure 5 is a flowchart of an optional gesture recognition method based on feature fusion provided according to Embodiment 2 of the present invention;

[0029] Figure 6 is a structural diagram of a gesture recognition device based on feature fusion provided according to Embodiment 3 of the present invention;

[0030] Figure 7 It is a structural schematic diagram of an electronic device for implementing the gesture recognition method based on feature fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Embodiment 1

[0034] Figure 1 This is a flowchart of a method for gesture recognition based on feature fusion provided in the first embodiment of the present invention. This embodiment is applicable to the case of identifying gesture categories in real-time video inside a vehicle. The method can be performed by a gesture recognition device based on feature fusion. The gesture recognition device based on feature fusion can be implemented in the form of hardware and / or software. The gesture recognition device based on feature fusion can be configured in an electronic device. Figure 1 As shown, the method includes:

[0035] S110, obtaining a standardized video corresponding to the target vehicle, and performing regional cropping processing on the standardized video to generate a target video including a hand region.

[0036] The target vehicle may refer to a vehicle for which the gestures of the occupants need to be recognized. The standardized video may refer to a video generated after standardization processing.

[0037] In an optional embodiment, the step of obtaining a standardized video corresponding to the target vehicle may include: obtaining a real-time video inside the target vehicle; and standardizing the real-time video to generate a standardized video corresponding to the target vehicle. The real-time video may refer to a dynamic video captured in real time inside the target vehicle. For example, a high-resolution camera installed on the dashboard of the target vehicle or the connecting pillars (A-pilar, A-pillar) connecting the roof and the front cabin on the left and right front sides may be used to capture real-time video of the driver or passengers inside the target vehicle. Standardization processing may refer to related processing operations for optimizing and adjusting the real-time video. For example, it may be operations such as brightness adjustment, contrast optimization, and denoising. Thus, by performing standardized processing on the real-time video inside the target vehicle, the real-time video captured in real time may be optimized and adjusted to adapt to subsequent video analysis processing.

[0038] Among them, the hand area may refer to an area containing hand features in the standardized video. Exemplarily, the hand area may be an area containing wrist parts and fingers. The regional cropping process may refer to a processing operation of detecting and cropping the hand area in the standard video. Exemplarily, a target detection network model may be used to construct a data set for model training and testing, so as to select the hand area using a detection frame. The target video may refer to a video generated after the regional cropping process is performed on the hand area in the standardized video.

[0039] S120. Perform spatial feature extraction on the target video based on the spatial attention model to generate posture features corresponding to the target video, and perform temporal feature extraction on the target video based on the temporal attention model to generate motion features corresponding to the target video combined with optical flow information.

[0040] Among them, the spatial attention model may refer to a model for finding task-related areas in the network for processing. The spatial feature may refer to the spatial position or relative direction relationship between multiple hand areas segmented in each frame of the target video. The posture feature may refer to the hand posture feature information contained in the hand area in the target video. Usually, the posture feature may be the corresponding spatial feature.

[0041] Among them, the temporal attention model may refer to a model that assigns different importance or attention to information at different time steps when processing sequence data. The temporal feature may refer to the change characteristics of pixels in the target video in the time domain. The optical flow information may refer to the instantaneous speed of the pixel motion of a spatial moving object on the observation imaging plane. Exemplarily, it may be the instantaneous speed of the pixel motion in the hand area in the target video. The motion feature may refer to the pixel motion feature between two adjacent frames in the target video. Generally, the motion feature may be the corresponding temporal feature.

[0042] S130, feature fusion processing of the posture feature and the motion feature to generate a target fusion feature corresponding to the target video.

[0043] The feature fusion processing may refer to a processing operation of correspondingly fusing the posture features and motion features of all frames in the target video. Exemplarily, the feature fusion processing may be performed by adding the posture features and motion features of the corresponding frames. The target fusion feature may refer to the final feature representation after the feature fusion processing.

[0044] S140. Determine, based on a fully connected classification network, a gesture category corresponding to the target fusion feature as target gesture recognition information corresponding to the target video.

[0045] The fully connected classification network may refer to a neural network based on a multilayer perceptron (MLP). Generally, the fully connected classification network may include three layers: an input layer, a hidden layer, and an output layer.

[0046] The gesture category may refer to the posture category of the hand area in the target video. For example, the gesture category may be a one, a thumbs up, waving left and right, or waving up and down, etc. The target gesture recognition information may refer to the gesture category corresponding to the target video.

[0047] The technical solution of the embodiment of the present invention processes the standardized video corresponding to the target vehicle through region cropping to generate a target video containing a hand region, then extracts spatial features of the target video based on a spatial attention model to generate posture features corresponding to the target video, and extracts temporal features of the target video based on a temporal attention model to generate motion features corresponding to the target video combined with optical flow information, further, performs feature fusion processing of posture features and motion features to generate target fusion features corresponding to the target video, and finally, determines the gesture category corresponding to the target fusion feature based on a fully connected classification network as the target gesture recognition information corresponding to the target video, thereby solving the problems of low accuracy and low robustness of gesture recognition, being able to accurately and real-timely recognize dynamic gestures, and improving the accuracy and robustness of gesture recognition.

[0048] Embodiment 2

[0049] Figure 2 A flowchart of a gesture recognition method based on feature fusion is provided for the second embodiment of the present invention. This embodiment is refined based on the above embodiment. In this embodiment, the operation of extracting spatial features from the target video based on the spatial attention model to generate gesture features corresponding to the target video is specifically refined. Specifically, it may include: uniformly sampling and processing the target video to generate a sampled video frame sequence corresponding to the target video; wherein the sampled video frame sequence includes each sampled video frame; enhancing and processing each sampled video frame based on the spatial attention model to generate spatial features corresponding to each sampled video frame, and using the spatial features as gesture features corresponding to the target video. Also, the operation of extracting temporal features from the target video based on the temporal attention model to generate motion features combined with optical flow information corresponding to the target video is refined. Specifically, it may include: determining the optical flow information between adjacent frames in the target video based on the optical flow method; processing each optical flow information based on the temporal attention model feature allocation to generate motion features combined with optical flow information corresponding to the target video. Figure 2 As shown, the method includes:

[0050] S210, obtaining a standardized video corresponding to the target vehicle, and performing regional cropping processing on the standardized video to generate a target video including a hand region.

[0051] S220 , uniformly sample and process the target video to generate a sampled video frame sequence corresponding to the target video; wherein the sampled video frame sequence includes sampled video frames.

[0052] The uniform sampling process may refer to a sampling operation of acquiring a continuous time signal at a fixed sampling interval. For example, the fixed sampling interval may be set to 3 frames or 5 frames, etc. The sampled video frame may refer to each image frame obtained after uniform sampling of the target video. The sampled video frame sequence may refer to a set consisting of each sampled video frame corresponding to the target video.

[0053] Therefore, by uniformly sampling the target video, the amount of computation for subsequent processing can be reduced, and the consumption of computing resources can be reduced. It is worth noting that in order to retain the key action information in the target video, the sampling interval corresponding to the uniform sampling process needs to be optimized, and this embodiment does not limit this.

[0054] S230, enhancing and processing each sampled video frame based on the spatial attention model, generating a spatial feature corresponding to each sampled video frame, and using the spatial feature as a posture feature corresponding to the target video.

[0055] The enhanced processing may refer to the operation of the area related to the gesture recognition task.

[0056] In an optional embodiment, each sampled video frame is enhanced and processed based on a spatial attention model to generate spatial features corresponding to each sampled video frame, including: processing each sampled video frame by maximum pooling to generate a first pooling result, processing each sampled video frame by average pooling to generate a second pooling result, and splicing the first pooling result and the second pooling result to generate a spliced ​​feature map corresponding to each sampled video frame; processing each of the spliced ​​feature maps by a first convolution to generate a first convolution result corresponding to each sampled video frame, and calculating an enhanced probability value corresponding to each sampled video frame using a preset activation function; processing each sampled video frame and the enhanced probability value corresponding to each sampled video frame by a first multiplication to generate first feature information containing spatial attention corresponding to each sampled video frame; and calculating and processing the first feature information based on a convolutional neural network to generate spatial features corresponding to each sampled video frame.

[0057] Among them, the maximum pooling process may refer to the operation of taking the maximum value of the pixels in the pooling area. The first pooling result may refer to the data result generated after the sampled video frame is processed by the maximum pooling process. The average pooling process may refer to the operation of taking the average value of the pixels in the pooling area. The second pooling result may refer to the data result generated after the sampled video frame is processed by the average pooling process. The splicing process may refer to the operation of splicing the first pooling result corresponding to the sampled video frame with the second pooling result. The splicing feature map may refer to the feature map corresponding to the sampled video frame generated after the splicing process.

[0058] Among them, the first convolution processing may refer to a convolution calculation implemented by a convolution layer. The first convolution result may refer to a feature map obtained after the spliced ​​feature map is processed by the first convolution. The preset activation function may refer to a pre-selected function for completing the nonlinear transformation of data. The problem of insufficient expression and classification capabilities of the linear model can be solved by the preset activation function. Exemplarily, the preset activation function may be a sigmoid activation function. The enhanced probability value may refer to the probability that the hand area in the sampled video frame can be enhanced. The first multiplication processing may refer to a processing operation of correspondingly multiplying the sampled video frame and the matched enhanced probability value. The first feature information may refer to the feature information generated after the sampled video frame and the matched enhanced probability value are processed by the first multiplication.

[0059] Figure 3 The figure shows a feature extraction flow chart based on a spatial attention model provided by an embodiment of the present invention. Specifically, each sampled video frame is first subjected to maximum pooling processing and average pooling processing respectively to generate a first pooling result and a second pooling result, and the first pooling result and the second pooling result are spliced ​​to generate a spliced ​​feature map corresponding to each sampled video frame, and then, each spliced ​​feature map is processed by the first convolution of the convolution layer to generate a first convolution result corresponding to each sampled video frame, and further, the enhanced probability value corresponding to each sampled video frame is calculated by a preset activation function, and each enhanced probability value is subjected to a first multiplication processing with the corresponding sampled video frame to generate a first feature information containing spatial attention, and finally, the first feature information is calculated and processed by a convolutional neural network to extract the spatial features corresponding to each sampled video frame.

[0060] S240: Determine optical flow information between adjacent frames in the target video based on an optical flow method.

[0061] The optical flow method refers to a method of finding the corresponding relationship between the previous frame and the current frame by using the change of pixels in the time domain in the image sequence and the correlation between adjacent frames, so as to calculate the motion information of objects between adjacent frames.

[0062] S250, processing each optical flow information based on the temporal attention model feature allocation, and generating a motion feature corresponding to the target video combined with the optical flow information.

[0063] The feature allocation process may refer to a process of allocating weights to each feature channel.

[0064] In an optional embodiment, each optical flow information is processed based on the feature allocation of the temporal attention model to generate a motion feature combined with the optical flow information corresponding to the target video, including: a second convolution to process each optical flow information to generate a second convolution result corresponding to each optical flow information; a dimension removal operation is performed on the second convolution result to generate a compression feature corresponding to each optical flow information; a channel weight generation operation is performed on each compression feature to generate a channel weight corresponding to each optical flow information, and a weight feature corresponding to each optical flow information is generated based on the channel weight; a second multiplication processes the second convolution result and the weight feature to generate a motion feature corresponding to the target video.

[0065] Among them, the second convolution processing may refer to the convolution calculation implemented by the convolution layer. The second convolution result may refer to the feature map obtained after the optical flow information is processed by the second convolution. The dimension removal operation may refer to the operation of deleting the set dimension in the feature map. Exemplarily, it can be to delete all gradients with a dimension of 1. Specifically, the dimension removal operation can be implemented using the Squeeze function. The compressed feature may refer to the feature generated after the optical flow information is subjected to the dimension removal operation and the spatial dimension is compressed. Usually, a real number compression feature can be obtained by the dimension removal operation of the two-dimensional feature map. It is worth noting that the number of feature channels remains unchanged.

[0066] Among them, the channel weight generation operation may refer to an operation of generating weights for each feature channel. Exemplarily, the channel weight generation operation may be implemented by an Excitation function. The channel weight may refer to the weight matched for each feature channel after the channel weight generation operation. The weight feature may refer to the feature generated after the weight matched for each feature channel is applied to each feature channel. The second multiplication process may refer to a processing operation of multiplying the second convolution result and the matched weight feature.

[0067] Figure 4 The figure shows a feature extraction flow chart based on a temporal attention model provided by an embodiment of the present invention. Specifically, firstly, the second convolution operation is used to process each optical flow information to generate a second convolution result corresponding to each optical flow information, then, the second convolution result is subjected to a dimension removal operation to perform feature compression on the spatial dimension to generate a compressed feature corresponding to each optical flow information, further, a channel weight generation operation is performed on each compressed feature, and the channel weight corresponding to each feature channel in each optical flow information is generated by the parameter w, and the channel weight is applied to the corresponding feature channel to generate a weight feature corresponding to each optical flow information, and finally, the second convolution result and the matched weight feature are correspondingly multiplied to complete the second multiplication process to obtain the motion feature corresponding to the target video.

[0068] It is worth noting that the order of the feature extraction process based on the spatial attention model and the feature extraction process based on the temporal attention model is not limited in the embodiment of the present invention. Exemplarily, the feature extraction process based on the spatial attention model can be performed first, and then the feature extraction process based on the temporal attention model. It is also possible to perform the feature extraction process based on the temporal attention model first, and then the feature extraction process based on the spatial attention model. It is also possible to complete the feature extraction process based on the spatial attention model and the feature extraction process based on the temporal attention model in parallel.

[0069] S260, feature fusion processing of the posture feature and the motion feature to generate a target fusion feature corresponding to the target video.

[0070] Specifically, by performing feature fusion processing on the posture features and corresponding motion features of the hand area in the target video to generate target fusion features, the dynamic gestures of the hand area can be understood more comprehensively, thereby improving the accuracy of the recognition results.

[0071] S270. Determine, based on a fully connected classification network, a gesture category corresponding to the target fusion feature as target gesture recognition information corresponding to the target video.

[0072] Specifically, based on the fully connected classification network, the number of gesture categories corresponding to the target fusion feature can be obtained. For example, if the target fusion feature contains five gesture categories, the corresponding number of gesture categories is 5. Afterwards, the gesture category with the highest probability can be calculated by setting an activation function, such as a softmax function, and then the gesture category with the highest probability is used as the gesture category corresponding to the target fusion feature to achieve fully connected classification.

[0073] S280: Determine a target linkage function that matches the target gesture recognition information based on a preset function linkage rule, and trigger execution of the target linkage function.

[0074] The function linkage rule may refer to a pre-set gesture and function matching rule. For example, the function linkage rule may be left and right waving to match song switching and up and down waving to match volume adjustment. The target linkage function may refer to a function that matches the target gesture recognition information in the function linkage rule.

[0075] Specifically, if the target gesture recognition information is waving left and right, and the function linkage rule is waving left and right to match song switching, then the corresponding target linkage function is song switching, and thus the song switching function can be called.

[0076] The technical solution of the embodiment of the present invention processes the standardized video corresponding to the target vehicle by region cropping to generate a target video containing a hand region, then uniformly samples and processes the target video to generate a sampled video frame sequence corresponding to the target video, enhances and processes each sampled video frame based on a spatial attention model to generate a spatial feature corresponding to each sampled video frame, and uses the spatial feature as a posture feature corresponding to the target video, and determines the optical flow information between adjacent frames in the target video based on an optical flow method, and processes each optical flow information based on a temporal attention model feature allocation to generate a motion feature corresponding to the target video combined with the optical flow information, further, performs feature fusion processing of posture features and motion features to generate a target fusion feature corresponding to the target video, and determines the gesture category corresponding to the target fusion feature based on a fully connected classification network as the target gesture recognition information corresponding to the target video, and finally, determines the target linkage function matching the target gesture recognition information based on a pre-set function linkage rule, and triggers the execution of the target linkage function, thereby solving the problems of low accuracy and low robustness of gesture recognition, being able to accurately and real-timely recognize dynamic gestures, and improving the accuracy and robustness of gesture recognition.

[0077] Figure 5 The flowchart of an optional gesture recognition method based on feature fusion provided by an embodiment of the present invention is shown. Specifically, first, a real-time video inside the target vehicle is obtained, and the real-time video is standardized to generate a standardized video corresponding to the target vehicle. Then, the standardized video is processed by region cropping to generate a target video containing a hand region. Further, the target video is uniformly sampled to generate a sampled video frame sequence corresponding to the target video, and each sampled video frame is enhanced and processed based on a spatial attention model to generate a spatial feature corresponding to each sampled video frame, and the spatial feature is used as the gesture feature corresponding to the target video. In addition, the optical flow information between adjacent frames in the target video is determined based on the optical flow method, and each optical flow information is processed based on the feature allocation of the temporal attention model to generate a motion feature corresponding to the target video combined with the optical flow information. Further, the gesture feature and the motion feature are processed by feature fusion to generate a target fusion feature corresponding to the target video, and the gesture category corresponding to the target fusion feature is determined based on a fully connected classification network as the target gesture recognition information corresponding to the target video. Finally, the target linkage function that matches the target gesture recognition information in the pre-set function linkage rule is determined, and the target linkage function is triggered to execute. In this way, the recognition of dynamic gestures in the real-time video inside the target vehicle and the corresponding dynamic gesture linkage are completed.

[0078] Embodiment 3

[0079] Figure 6 This is a schematic diagram of the structure of a gesture recognition device based on feature fusion provided in the third embodiment of the present invention. Figure 6As shown, the device includes: a data acquisition module 310, a feature extraction module 320, a feature fusion module 330 and a gesture recognition module 340;

[0080] The data acquisition module 310 is used to acquire a standardized video corresponding to the target vehicle, and perform regional cropping processing on the standardized video to generate a target video containing a hand area;

[0081] A feature extraction module 320 is used to extract spatial features of the target video based on a spatial attention model to generate posture features corresponding to the target video, and to extract temporal features of the target video based on a temporal attention model to generate motion features corresponding to the target video combined with optical flow information;

[0082] A feature fusion module 330, used for feature fusion processing of the posture feature and the motion feature to generate a target fusion feature corresponding to the target video;

[0083] The gesture recognition module 340 is used to determine the gesture category corresponding to the target fusion feature based on a fully connected classification network as the target gesture recognition information corresponding to the target video.

[0084] The technical solution of the embodiment of the present invention processes the standardized video corresponding to the target vehicle through region cropping to generate a target video containing a hand region, then extracts spatial features of the target video based on a spatial attention model to generate posture features corresponding to the target video, and extracts temporal features of the target video based on a temporal attention model to generate motion features corresponding to the target video combined with optical flow information, further, performs feature fusion processing of posture features and motion features to generate target fusion features corresponding to the target video, and finally, determines the gesture category corresponding to the target fusion feature based on a fully connected classification network as the target gesture recognition information corresponding to the target video, thereby solving the problems of low accuracy and low robustness of gesture recognition, being able to accurately and real-timely recognize dynamic gestures, and improving the accuracy and robustness of gesture recognition.

[0085] Optionally, the data acquisition module 310 may be specifically used for:

[0086] Obtain real-time video of the interior of the target vehicle;

[0087] The real-time video is processed by standardization to generate a standardized video corresponding to the target vehicle.

[0088] Optionally, the feature extraction module 320 may specifically include: a uniform sampling unit and a spatial feature extraction unit;

[0089] The uniform sampling unit is used to uniformly sample and process the target video to generate a sampled video frame sequence corresponding to the target video; wherein the sampled video frame sequence includes each sampled video frame;

[0090] The spatial feature extraction unit is used to enhance the processing of each sampled video frame based on the spatial attention model, generate spatial features corresponding to each sampled video frame, and use the spatial features as posture features corresponding to the target video.

[0091] Optionally, the spatial feature extraction unit may be used to:

[0092] Processing each sampled video frame by maximum pooling to generate a first pooling result, processing each sampled video frame by average pooling to generate a second pooling result, and splicing the first pooling result and the second pooling result to generate a splicing feature map corresponding to each sampled video frame;

[0093] Performing a first convolution on each of the spliced ​​feature maps to generate a first convolution result corresponding to each sampled video frame, and calculating an enhancement probability value corresponding to each sampled video frame using a preset activation function;

[0094] The first multiplication processes each sampled video frame and the enhanced probability value corresponding to each sampled video frame to generate first feature information corresponding to each sampled video frame including spatial attention;

[0095] The first feature information is processed based on convolutional neural network calculations to generate spatial features corresponding to each sampled video frame.

[0096] Optionally, the feature extraction module 320 may specifically include: an optical flow calculation unit and a motion feature extraction unit;

[0097] The optical flow calculation unit is used to determine the optical flow information between adjacent frames in the target video based on the optical flow method;

[0098] The motion feature extraction unit is used to process each optical flow information based on the temporal attention model feature allocation to generate the motion features combined with the optical flow information corresponding to the target video.

[0099] Optionally, the motion feature extraction unit may be specifically used for:

[0100] The second convolution processes each optical flow information to generate a second convolution result corresponding to each optical flow information;

[0101] Performing a dimension removal operation on the second convolution result to generate a compression feature corresponding to each optical flow information;

[0102] Performing a channel weight generation operation on each of the compression features to generate a channel weight corresponding to each piece of optical flow information, and generating a weight feature corresponding to each piece of optical flow information based on the channel weight;

[0103] The second multiplication processes the second convolution result and the weight feature to generate a motion feature corresponding to the target video.

[0104] Optionally, the gesture recognition device based on feature fusion may further include: a function linkage module, which is used to determine the target linkage function that matches the target gesture recognition information based on a pre-set function linkage rule, and trigger the execution of the target linkage function after the gesture category corresponding to the target fusion feature is determined based on the fully connected classification network as the target gesture recognition information corresponding to the target video.

[0105] The gesture recognition device based on feature fusion provided in the embodiment of the present invention can execute the gesture recognition method based on feature fusion provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0106] Embodiment 4

[0107] Figure 7 A schematic diagram of the structure of an electronic device 410 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0108] like Figure 7 As shown, the electronic device 410 includes at least one processor 420, and a memory connected to the at least one processor 420 in communication, such as a read-only memory (ROM) 430, a random access memory (RAM) 440, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 420 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 430 or the computer program loaded from the storage unit 490 to the random access memory (RAM) 440. In the RAM 440, various programs and data required for the operation of the electronic device 410 can also be stored. The processor 420, the ROM 430, and the RAM 440 are connected to each other through the bus 450. The input / output (I / O) interface 460 is also connected to the bus 450.

[0109] Multiple components in the electronic device 410 are connected to the I / O interface 460, including: an input unit 470, such as a keyboard, a mouse, etc.; an output unit 480, such as various types of displays, speakers, etc.; a storage unit 490, such as a disk, an optical disk, etc.; and a communication unit 4100, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 4100 allows the electronic device 410 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The processor 420 may be a variety of general and / or dedicated processing components with processing and computing capabilities. Some examples of the processor 420 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 420 executes the various methods and processes described above, such as a gesture recognition method based on feature fusion.

[0111] The method includes:

[0112] Obtaining a standardized video corresponding to the target vehicle, and performing regional cropping processing on the standardized video to generate a target video containing a hand region;

[0113] Extracting spatial features of the target video based on the spatial attention model to generate posture features corresponding to the target video, and extracting temporal features of the target video based on the temporal attention model to generate motion features corresponding to the target video combined with optical flow information;

[0114] Feature fusion processes the posture feature and the motion feature to generate a target fusion feature corresponding to the target video;

[0115] The gesture category corresponding to the target fusion feature is determined based on a fully connected classification network as the target gesture recognition information corresponding to the target video.

[0116] In some embodiments, the gesture recognition method based on feature fusion may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 490. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 410 via the ROM 430 and / or the communication unit 4100. When the computer program is loaded into the RAM 440 and executed by the processor 420, one or more steps of the gesture recognition method based on feature fusion described above may be performed. Alternatively, in other embodiments, the processor 420 may be configured to perform the gesture recognition method based on feature fusion in any other appropriate manner (e.g., by means of firmware).

[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0121] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0122] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0123] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0124] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A gesture recognition method based on feature fusion, characterized in that: include: Obtaining a standardized video corresponding to the target vehicle, and performing regional cropping processing on the standardized video to generate a target video containing a hand region; wherein the standardized video refers to a real-time video generated after the standardized processing; and the regional cropping processing refers to a processing operation of detecting and cropping the hand region in the standardized video; Extracting spatial features of the target video based on the spatial attention model to generate posture features corresponding to the target video, and extracting temporal features of the target video based on the temporal attention model to generate motion features corresponding to the target video combined with optical flow information; Feature fusion processes the posture feature and the motion feature to generate a target fusion feature corresponding to the target video; Determine the gesture category corresponding to the target fusion feature based on a fully connected classification network as the target gesture recognition information corresponding to the target video; The step of extracting spatial features of the target video based on the spatial attention model to generate posture features corresponding to the target video includes: Uniformly sampling the target video to generate a sampled video frame sequence corresponding to the target video; wherein the sampled video frame sequence includes each sampled video frame; Each sampled video frame is enhanced and processed based on the spatial attention model to generate a spatial feature corresponding to each sampled video frame, and the spatial feature is used as a posture feature corresponding to the target video.

2. The method according to claim 1, characterized in that The step of obtaining a standardized video corresponding to the target vehicle includes: Obtain real-time video of the interior of the target vehicle; The real-time video is processed by standardization to generate a standardized video corresponding to the target vehicle.

3. The method according to claim 1, characterized in that The method of enhancing and processing each sampled video frame based on the spatial attention model to generate a spatial feature corresponding to each sampled video frame includes: Processing each sampled video frame by maximum pooling to generate a first pooling result, processing each sampled video frame by average pooling to generate a second pooling result, and splicing the first pooling result and the second pooling result to generate a splicing feature map corresponding to each sampled video frame; Performing a first convolution on each of the spliced ​​feature maps to generate a first convolution result corresponding to each sampled video frame, and calculating an enhancement probability value corresponding to each sampled video frame using a preset activation function; The first multiplication processes each sampled video frame and the enhanced probability value corresponding to each sampled video frame to generate first feature information corresponding to each sampled video frame including spatial attention; The first feature information is processed based on convolutional neural network calculations to generate spatial features corresponding to each sampled video frame.

4. The method according to claim 1, characterized in that: The step of extracting temporal features of the target video based on the temporal attention model to generate motion features corresponding to the target video combined with optical flow information includes: Determine the optical flow information between adjacent frames in the target video based on the optical flow method; Based on the temporal attention model feature allocation, each optical flow information is processed to generate a motion feature combined with the optical flow information corresponding to the target video.

5. The method according to claim 4, characterized in that The method of processing each optical flow information based on the temporal attention model feature allocation to generate a motion feature corresponding to the target video combined with the optical flow information includes: The second convolution processes each optical flow information to generate a second convolution result corresponding to each optical flow information; Performing a dimension removal operation on the second convolution result to generate a compression feature corresponding to each optical flow information; Performing a channel weight generation operation on each of the compression features to generate a channel weight corresponding to each piece of optical flow information, and generating a weight feature corresponding to each piece of optical flow information based on the channel weight; The second multiplication processes the second convolution result and the weight feature to generate a motion feature corresponding to the target video.

6. The method according to claim 1, characterized in that After determining the gesture category corresponding to the target fusion feature based on the fully connected classification network as the target gesture recognition information corresponding to the target video, the method further includes: Based on a preset function linkage rule, a target linkage function matching the target gesture recognition information is determined, and the target linkage function is triggered to be executed.

7. A gesture recognition device based on feature fusion, characterized in that: include: A data acquisition module is used to acquire a standardized video corresponding to a target vehicle, and perform regional cropping processing on the standardized video to generate a target video containing a hand area; wherein the standardized video refers to a real-time video generated after the standardized processing; and the regional cropping processing refers to a processing operation of detecting and cropping the hand area in the standardized video; A feature extraction module is used to extract spatial features of the target video based on a spatial attention model to generate posture features corresponding to the target video, and to extract temporal features of the target video based on a temporal attention model to generate motion features corresponding to the target video combined with optical flow information; A feature fusion module, used for feature fusion processing of the posture feature and the motion feature to generate a target fusion feature corresponding to the target video; A gesture recognition module, used to determine the gesture category corresponding to the target fusion feature based on a fully connected classification network as target gesture recognition information corresponding to the target video; Among them, the feature extraction module specifically includes: a uniform sampling unit and a spatial feature extraction unit; A uniform sampling unit, used for uniformly sampling the target video to generate a sampled video frame sequence corresponding to the target video; wherein the sampled video frame sequence includes each sampled video frame; The spatial feature extraction unit is used to enhance the processing of each sampled video frame based on the spatial attention model, generate spatial features corresponding to each sampled video frame, and use the spatial features as posture features corresponding to the target video.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the gesture recognition method based on feature fusion according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the gesture recognition method based on feature fusion according to any one of claims 1 to 6 when executed.

Images