Gesture recognition method and device based on single frame image

Abstract

The application provides a gesture recognition method and device based on a single frame image, and belongs to the field of image recognition, wherein the method comprises the following steps: acquiring a target image collected by a camera in real time; acquiring a historical time sequence feature of the target image from a storage module, wherein the historical time sequence feature is an image feature of a previous frame image of the target image; generating a current time sequence feature by fusing the image feature of the target image and the historical time sequence feature; and recognizing a gesture category according to the current time sequence feature. Through the embodiment of the application, the technical problem that a high computing power is required in the gesture recognition process of the related art is solved, the inference speed is relatively fast and the computing resource consumption is relatively small in the gesture recognition process, the car machine lag is reduced, and the gesture interaction smoothness and user experience are improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more specifically, to a gesture recognition method and apparatus based on a single-frame image. Background Technology

[0002] In related technologies, with the development of modern science and technology, automobiles are increasingly becoming an indispensable means of transportation in people's lives. The growing demand for automobiles has increased the necessity for vehicle comfort and intelligence. To enhance the in-car experience for drivers and passengers, automakers are implementing human-machine interaction functions through in-vehicle devices within the car cabin, improving the interaction between people and the car. Drivers and passengers can interact with in-vehicle devices using gestures to achieve different functions. For example, front and rear passengers can use gestures (such as swiping their entire hand to the left or right) to switch to the previous or next track of currently playing music, and can use gestures (heart gestures or thumbs-ups) to save music tracks.

[0003] In related technologies, gesture recognition technology is generally divided into two categories: static gesture recognition and dynamic gesture recognition. The former can realize fewer gesture categories and adapt to fewer scenarios, while the latter can realize more gesture categories and adapt to more scenarios. However, the latter requires a video sequence as input for prediction. Assuming that the video sequence contains 8 frames of images, the model needs to perform inference on 8 frames of images at a time, which requires high computing power, has a large inference delay, and is prone to causing vehicle system lag.

[0004] No efficient and accurate solution has yet been found to address the aforementioned issues in the relevant technologies. Summary of the Invention

[0005] This invention provides a gesture recognition method and apparatus based on a single-frame image to solve the technical problems in related technologies.

[0006] According to an embodiment of the present invention, a gesture recognition method based on a single-frame image is provided, comprising: acquiring a target image captured in real time by a camera; acquiring historical temporal features of the target image from a storage module, wherein the historical temporal features are image features of the previous frame image of the target image; fusing the image features of the target image and the historical temporal features to generate current temporal features; and identifying the gesture category based on the current temporal features.

[0007] Further, the process of fusing the image features of the target image and the historical time-series features to generate the current time-series features includes: extracting a first image feature of the target image using a feature extraction module; iteratively executing the following steps based on the first image feature until the next-level module of the current module is a classification module: determining whether the next-level module of the current module is a feature extraction module or a feature processing module; if the next-level module of the current module is a feature processing module, inputting the first image feature into the feature processing module, using the feature processing module to fuse the first image feature and the historical time-series features to obtain a fused feature, and inputting the fused feature into the next-level module; if the next-level module of the current module is a feature extraction module, inputting the first image feature into the feature extraction module, using the feature extraction module to extract a second image feature of the first image feature, and inputting the second image feature into the next-level module; and outputting the current time-series features after the iteration is completed.

[0008] Furthermore, the feature processing module is used to fuse the first image feature and the historical time-series feature to obtain the fused feature, which includes: splitting the first image feature into a first sub-feature of dimension m and a second sub-feature of dimension n, wherein the feature dimension of the historical image feature is n, and m and n are all positive integers greater than 0; fusing the first sub-feature and the historical time-series feature to obtain the fused feature of dimension m+n; and updating the historical time-series feature to the second sub-feature in the storage module of the feature processing module, wherein each feature processing module corresponds to a storage module.

[0009] Furthermore, identifying the gesture category based on the current temporal features includes: inputting the current temporal features into a pre-trained gesture recognition model, and outputting the current gesture category and current confidence level to which the current temporal features belong; obtaining the historical gesture categories and corresponding historical confidence levels of several historical images preceding the first image; and generating the final gesture category based on the current gesture category, the current confidence level, the historical gesture categories, and the historical confidence levels.

[0010] Furthermore, generating the final gesture category based on the current gesture category, the current confidence level, the historical gesture categories, and the historical confidence levels includes: taking a weighted average of the confidence levels of the current gesture category and all historical gesture categories to obtain the final confidence level; and selecting the final gesture category with the highest confidence level from the current gesture category and the historical gesture categories based on the final confidence level.

[0011] Furthermore, before inputting the current temporal features into the pre-trained gesture recognition model, the method further includes: acquiring image sequences of multiple gesture actions; for each gesture action image sequence, sequentially sampling from the image sequence to obtain a sample image group, wherein the sample image group includes multiple gesture images, all of which correspond to the same gesture category, to obtain training data including B sample image groups; processing the training data into T data of B*C*H*W dimensions, where T represents the number of image frames sampled from the image sequence, and C, H, and W represent the number of channels and the height and width of the image; inputting the B*C*H*W dimension data into the initial model for T inference training iterations to obtain T intermediate training results; calculating the T intermediate training results to obtain the final training result, and using the final training result to calculate the loss function to train the initial module to obtain the gesture recognition model.

[0012] Furthermore, after identifying the gesture category based on the current temporal features, the method further includes: determining the business scenario of the vehicle's infotainment controller where the camera is located; searching for a target control instruction matching the gesture category in a preset instruction library of the business scenario; and transmitting the target control instruction to the infotainment controller.

[0013] According to another embodiment of the present invention, a gesture recognition device based on a single-frame image is provided, comprising: a first acquisition module for acquiring a target image captured in real time by a camera; a second acquisition module for acquiring historical temporal features of the target image from a storage module, wherein the historical temporal features are image features of the previous frame image of the target image; a fusion module for fusing the image features of the target image and the historical temporal features to generate current temporal features; and a recognition module for recognizing gesture categories based on the current temporal features.

[0014] Furthermore, the fusion module includes: an extraction unit, used to extract a first image feature of the target image using a feature extraction module; an iteration unit, used to iteratively execute the following steps based on the first image feature until the subsequent module of the current module is a classification module: determining whether the subsequent module of the current module is a feature extraction module or a feature processing module; if the subsequent module of the current module is a feature processing module, inputting the first image feature into the feature processing module, using the feature processing module to perform fusion processing on the first image feature and the historical time-series feature to obtain a fused feature, and inputting the fused feature into the subsequent module; if the subsequent module of the current module is a feature extraction module, inputting the first image feature into the feature extraction module, using the feature extraction module to extract a second image feature of the first image feature, and inputting the second image feature into the subsequent module; and an output unit, used to output the current time-series feature after the iteration is completed.

[0015] Furthermore, the iterative unit includes: a splitting subunit, used to split the first image feature into a first sub-feature of m dimensions and a second sub-feature of n dimensions using the feature processing module, wherein the feature dimension of the historical image feature is n, and m and n are all positive integers greater than 0; and a fusion subunit, used to fuse the first sub-feature and the historical time-series feature to obtain a fused feature of m+n dimensions, and update the historical time-series feature to the second sub-feature in the storage module of the feature processing module, wherein each feature processing module corresponds to a storage module.

[0016] Furthermore, the recognition module includes: a processing unit, configured to input the current temporal feature into a pre-trained gesture recognition model and output the current gesture category and current confidence level to which the current temporal feature belongs; an acquisition unit, configured to acquire the historical gesture categories and corresponding historical confidence levels of several historical images preceding the first image; and a generation unit, configured to generate a final gesture category based on the current gesture category, the current confidence level, the historical gesture categories, and the historical confidence levels.

[0017] Furthermore, the generation unit includes: a calculation subunit, used to perform a weighted average of the confidence scores of the current gesture category and all historical gesture categories to obtain a final confidence score; and a selection subunit, used to select the final gesture category with the highest confidence score from the current gesture category and the historical gesture categories based on the final confidence score.

[0018] Furthermore, the device further includes: a third acquisition module, configured to acquire image sequences of multiple gesture actions before the recognition module inputs the current temporal features into the pre-trained gesture recognition model; a sampling module, configured to sequentially sample sample image groups from the image sequence for each gesture action, wherein the sample image group includes multiple gesture images, all of which correspond to the same gesture category, to obtain training data including B sample image groups; a processing module, configured to process the training data into T data of B*C*H*W dimensions, where T represents the number of image frames sampled from the image sequence, and C, H, and W represent the number of channels and the height and width of the image; a first training module, configured to input the B*C*H*W dimension data into the initial model for T inference training iterations to obtain T intermediate training results; and a second training module, configured to calculate the final training result from the T intermediate training results and use the final training result to calculate the loss function to train the initial module to obtain the gesture recognition model.

[0019] Furthermore, the device further includes: a determining module, used to determine the business scenario of the vehicle's infotainment controller after the recognition module identifies the gesture category based on the current temporal features; a searching module, used to search for a target control command matching the gesture category in a preset command library of the business scenario; and a transmitting module, used to transmit the target control command to the vehicle's infotainment controller.

[0020] According to another aspect of the embodiments of this application, a storage medium is also provided, the storage medium including a stored program that executes the above steps when the program is run.

[0021] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein: the memory is used to store computer programs; and the processor is used to execute the steps in the above method by running the programs stored in the memory.

[0022] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the above-described method.

[0023] Through the embodiments of the present invention, a first image captured in real time by a camera is obtained, and the historical temporal features of the target image are obtained from the storage module. The historical temporal features are the image features of the previous frame of the target image. The current temporal features are generated by fusing the image features of the target image and the historical temporal features. The gesture category is identified based on the current temporal features. After obtaining a single frame image, gesture recognition is performed by fusing it with the historical temporal features of the previous frame of the target image. This method can realize dynamic gesture recognition through a single frame image in a video stream. The model only needs to infer a single frame image to obtain the dynamic gesture category. This solves the technical problem of high computing power required in the gesture recognition process of related technologies. It achieves faster inference speed and lower computing resource consumption in the gesture recognition process, reduces vehicle system lag, and improves the smoothness of gesture interaction and user experience. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 This is a hardware structure block diagram of an in-vehicle terminal according to an embodiment of the present invention;

[0026] Figure 2 This is a flowchart of a gesture recognition method based on a single-frame image according to an embodiment of the present invention;

[0027] Figure 3 This is a flowchart of dynamic gesture recognition for a single frame image based on a video stream in an embodiment of the present invention;

[0028] Figure 4 This is a detailed flowchart of the single-frame image dynamic gesture recognition based on video stream according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram illustrating the execution of the feature processing module in an embodiment of the present invention;

[0030] Figure 6 This is a structural block diagram of a gesture recognition device based on a single-frame image according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, and not all of them. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present application can be combined with each other.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] The method embodiment provided in Embodiment 1 of this application can be executed in an in-vehicle terminal, an in-vehicle control module, an image interaction module, or a similar processing device. Taking its operation on an in-vehicle terminal as an example, Figure 1 This is a hardware structure block diagram of a vehicle-mounted terminal according to an embodiment of the present invention. Figure 1 As shown, the vehicle-mounted terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the vehicle-mounted terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned vehicle-mounted terminal. For example, the vehicle-mounted terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0035] The memory 104 can be used to store in-vehicle terminal programs, such as application software programs and modules, like the in-vehicle terminal program corresponding to a gesture recognition method based on a single frame image in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the in-vehicle terminal program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the in-vehicle terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the vehicle terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0037] This embodiment provides a gesture recognition method based on a single-frame image. Figure 2 This is a flowchart of a gesture recognition method based on a single-frame image according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0038] Step S202: Acquire the target image captured in real time by the camera;

[0039] In this embodiment, the target image is a single frame image in the video stream captured by the camera. The first frame image in which a finger first appears in the image is the first frame image for gesture recognition, and the first frame image after the finger disappears in the image is the last frame image for gesture recognition.

[0040] Step S204: Obtain the historical temporal features of the target image from the storage module, wherein the historical temporal features are the image features of the previous frame of the target image;

[0041] In this embodiment, the historical temporal features of the first frame image for gesture recognition are preset initialization features, and the historical temporal features of each subsequent frame image are the image features of the previous frame image.

[0042] Step S206: The current time-series features are generated by fusing the image features of the target image and the historical time-series features;

[0043] Optionally, before fusion, the target image can be input into a preprocessing module for preprocessing. The preprocessing module performs cropping on the gesture recognition region in the input image and normalizes the cropped image.

[0044] Step S208: Identify the gesture category based on the current temporal features.

[0045] Through the above steps, the first image captured in real time by the camera is obtained, and the historical temporal features of the target image are obtained from the storage module. The historical temporal features are the image features of the previous frame of the target image. The image features of the target image and the historical temporal features are fused to generate the current temporal features. The gesture category is identified based on the current temporal features. After obtaining a single frame image, gesture recognition is performed by fusing it with the historical temporal features of the previous frame of the target image. This method can realize dynamic gesture recognition through a single frame image in the video stream. The model only needs to infer a single frame image to obtain the dynamic gesture category. It solves the technical problem of high computing power required in the gesture recognition process of related technologies. It achieves faster inference speed and less computing resource consumption in the gesture recognition process, reduces vehicle system lag, and improves the smoothness of gesture interaction and user experience.

[0046] In one embodiment of this example, the process of generating current temporal features by fusing image features of the target image and historical temporal features includes:

[0047] S11, the feature extraction module is used to extract the first image features of the target image;

[0048] S12, based on the first image features, iteratively execute the following steps until the next module of the current module is a classification module: determine whether the next module of the current module is a feature extraction module or a feature processing module; if the next module of the current module is a feature processing module, input the first image features into the feature processing module, use the feature processing module to fuse the first image features and historical time-series features to obtain fused features, and input the fused features into the next module; if the next module of the current module is a feature extraction module, input the first image features into the feature extraction module, use the feature extraction module to extract the second image features of the first image features, and input the second image features into the next module;

[0049] The gesture recognition model in this embodiment includes: a feature extraction module, a feature processing module, a classification module, and a storage module. The feature extraction module extracts relevant features from the image and consists of convolutional layers, batch normalization (BN) layers, and activation functions. The feature processing module performs relevant processing on the extracted features, such as splitting, storing, and fusing features. The classification module classifies gesture actions in the image based on the extracted features and mainly consists of fully connected or convolutional layers and softmax layers. The storage module stores relevant image features, facilitating subsequent fusion of image features from historical frames and the current frame to obtain temporal features of the image, and classifying dynamic gestures using the classification module. The feature extraction module and feature processing module described above in this embodiment correspond to the feature extraction module and feature processing module of the gesture recognition model, respectively.

[0050] The gesture recognition module follows a pipeline mechanism, with multiple levels of feature extraction and feature processing modules. The final level module connects to the classification module. In one implementation scenario, the execution logic order is as follows:

[0051] Step S310: The feature extraction module extracts features from the input and outputs feature A;

[0052] Step S320: If the feature extraction module in step S310 is followed by the feature processing module, then proceed to step S330; otherwise, proceed to step S340.

[0053] Step S330: The feature processing module processes the feature A output in step S310 to output feature B, and updates the features stored in the storage module.

[0054] Step S340: If feature extraction and feature processing are complete, proceed to step S350; otherwise, return to step S310.

[0055] Step S350: The classification module classifies the gestures in the current image using the final acquired features.

[0056] In this embodiment, the feature extraction module and feature processing module of the gesture recognition model do not always appear in pairs. For example, some feature extraction modules are followed by feature processing modules, and some feature extraction modules are followed by feature extraction modules. Due to the presence of the storage module and feature processing module, this embodiment can achieve dynamic gesture recognition with inference time when the input is a single-frame image, which greatly improves the inference speed, and the recognition effect is similar to that of inference when the input is a multi-frame image.

[0057] In one example, a feature processing module is used to fuse the first image features and historical time-series features to obtain fused features. This fused feature includes: splitting the first image features into an m-dimensional first sub-feature and an n-dimensional second sub-feature, where the feature dimension of the historical image features is n, and m and n are both positive integers greater than 0; fusing the first sub-feature and the historical time-series features to obtain fused features of m+n dimensions; and updating the historical time-series features to the second sub-feature in the storage module of the feature processing module. Each feature processing module corresponds to one storage module.

[0058] In one implementation scenario, the execution logic sequence of the feature processing module is as follows: Step S331: Split the input feature (dimension C*H*W) into feature 1 and feature 2 (dimensions (CN)*H*W and N*H*W respectively); Step S332: Retrieve the required feature (dimension N*H*W) from the storage module; Step S333: Perform feature fusion on feature 1 (dimension (CN)*H*W) obtained in step S331 and feature (dimension N*H*W) obtained in step S332 to output a new feature (dimension C*H*W); Step S334: Update the feature in the storage module using feature 2 (dimension N*H*W) obtained in step S331.

[0059] S13, after the iteration is completed, outputs the current time series features.

[0060] In this embodiment, identifying the gesture category based on the current temporal features includes:

[0061] S21, Input the current temporal features into the pre-trained gesture recognition model, and output the current gesture category and current confidence level to which the current temporal features belong;

[0062] Optionally, before inputting the current temporal features into the pre-trained gesture recognition model, the method further includes: acquiring image sequences of multiple gesture actions; for each gesture action image sequence, sequentially sampling from the image sequence to obtain sample image groups, wherein each sample image group includes multiple gesture images, all corresponding to the same gesture category, resulting in training data including B sample image groups; processing the training data into T data of B*C*H*W dimensions, where T represents the number of image frames sampled from the image sequence, and C, H, and W represent the number of channels and the height and width of the image; inputting the B*C*H*W dimension data into the initial model for T inference training iterations to obtain T intermediate training results; calculating the T intermediate training results to obtain the final training result, and using the final training result to calculate the loss function to train the initial module, thus obtaining the gesture recognition model.

[0063] During model training, for an image sequence containing a complete hand gesture (an image sequence can only contain one gesture), T images are sampled as a set of training data while maintaining the original order (T is manually set, usually 8; the sampling method is manually set, usually equal-interval sampling). The label corresponding to this set of training data is the gesture category of the image sequence. Specific training data can be obtained through data acquisition, and the acquired image sequences include, but are not limited to, RGB images, grayscale images, depth maps, infrared images, etc. Assume there are B sets of training data in a single training iteration, with dimensions B*T*C*H*W, where T represents the number of image frames sampled from the image sequence, and C, H, and W represent the number of channels and the image's height and width. During training, the training data needs to be processed into T sets of data with dimensions B*C*H*W. The model performs T inferences using the training data, following the same inference process as previously described. During inference, the input data dimension is B*C*H*W, and T inferences yield T results. These T results are then calculated (averaged or weighted average) to obtain the final result. The final result is used to calculate a loss function (such as the cross-entropy loss function) to train the gesture recognition model. After this training, the model can be deployed and used for actual prediction. In actual prediction, the model only needs to input a single frame image and perform one inference to determine the dynamic gesture category of the current image, resulting in fast inference speed and low resource consumption.

[0064] S22, obtain the historical gesture categories and corresponding historical confidence levels of several historical images preceding the first image;

[0065] In the process of dynamic gesture recognition based on a single frame, each frame of the image will output the gesture category and the corresponding confidence score. If the current frame is the Nth frame, the historical gesture category and the corresponding historical confidence score include the gesture category and the corresponding confidence score of each frame from the 1st frame to the (N-1th frame). For example, if the current frame is the 3rd frame, the historical gesture category and the corresponding historical confidence score include the gesture category and the corresponding confidence score of the 1st frame and the gesture category and the corresponding confidence score of the 2nd frame.

[0066] S23, generate the final gesture category based on the current gesture category, current confidence level, historical gesture categories, and historical confidence level.

[0067] Optionally, generating the final gesture category based on the current gesture category, current confidence level, historical gesture categories, and historical confidence levels includes: taking a weighted average of the confidence levels of the current gesture category and all historical gesture categories to obtain the final confidence level; and selecting the final gesture category with the highest confidence level from the current gesture category and historical gesture categories based on the final confidence level.

[0068] In this embodiment, after identifying the gesture category based on the current timing features, the method further includes: determining the business scenario of the vehicle's infotainment controller where the camera is located; searching for a target control command that matches the gesture category in the preset command library of the business scenario; and transmitting the target control command to the infotainment controller.

[0069] Optionally, the target control commands can be human-computer interaction commands based on mobile phones, such as previous track, next track, pause, play, increase volume, decrease volume in music player control scenarios, or increase temperature, decrease temperature in air conditioner control scenarios.

[0070] Figure 3 This is a flowchart of dynamic gesture recognition based on a single frame image in an embodiment of the present invention, including the following steps:

[0071] Step S100: Acquire the image captured by the camera and input it into the preprocessing module.

[0072] Step S200: The preprocessing module processes the input image and inputs the processed data into the gesture recognition model.

[0073] Step S300: The gesture recognition model infers from the input data and obtains the output result.

[0074] Step S400: The post-processing module processes the model output to obtain the current gesture category.

[0075] Step S500: Combine gesture categories with corresponding scenarios to achieve corresponding functions. For example, swiping left with your hand switches songs, and making a heart shape with your hand enables the music collection function.

[0076] In this embodiment, step S300 includes two cases: one is that the current frame number is not the initial frame, i.e., T = N, N > 0, where T represents which frame in the video stream the current image is; the other is that the current frame number is the initial frame, i.e., T = 0. For the first case, the specific process is the same as... Figure 2 and Figure 3 The description is the same. For the second case, since the current frame is the initial frame, the storage module does not store the image features of the previous frame. Therefore, the gesture recognition model needs to initialize the features in the storage module for use in step S320 when performing feature processing. The rest of the process is the same as the first case.

[0077] Figure 4 This is a detailed flowchart of the single-frame image dynamic gesture recognition based on video stream according to an embodiment of the present invention. Step S300 includes the following steps:

[0078] Step S310: The feature extraction module extracts features from the input to obtain the output feature A.

[0079] In this step, the input to the feature extraction module is the output of the previous module, which has three possibilities: the feature processing module, the feature extraction module, and the original image.

[0080] Step S320: If the feature extraction module in step S310 is followed by the feature processing module, then proceed to step S330; otherwise, proceed to step S340.

[0081] Step S330: The feature processing module processes the feature A output in step S310 to output a new feature B, and updates the features stored in the storage module.

[0082] Step S340: If feature extraction and feature processing are complete, proceed to step S350; otherwise, return to step S310.

[0083] Step S350: The classification module classifies the gestures in the current image using the final acquired features.

[0084] In this step, since the features finally obtained by the classification module include temporal features (including image features at the current moment as well as image features at historical moments), the classification module can use these features to achieve dynamic gesture classification.

[0085] In the above steps, the feature extraction module and feature processing module of the gesture recognition model do not always appear in pairs. For example, some feature extraction modules are followed by feature processing modules, and some feature extraction modules are still followed by feature extraction modules. Due to the presence of the storage module and feature processing module, this invention can achieve dynamic gesture recognition with inference time when the input is a single-frame image, greatly improving the inference speed, and the recognition effect is similar to that when the input is a multi-frame image.

[0086] Figure 5 This is a schematic diagram of the execution of the feature processing module in an embodiment of the present invention. Step S330 includes the following steps:

[0087] Step S331: Split the input feature (dimension C*H*W) into feature 1 and feature 2 (dimensions (CN)*H*W and N*H*W respectively).

[0088] In this step, the input to the feature processing module is the output of the feature extraction module.

[0089] Step S332: Retrieve the required features (dimensions are N*H*W) from the storage module.

[0090] In this step, assuming the current time is T, when T = 0, the feature retrieved by the feature processing module is the feature initialized in the storage module, and when T > 0, the feature retrieved by the feature processing module is the feature stored in the storage module by the feature processing module at time T-1.

[0091] Step S333: Perform feature fusion on the feature 1 (dimension: (CN)*H*W) obtained in step S331 and the feature (dimension: N*H*W) obtained in step S332 to output a new feature (dimension: C*H*W).

[0092] In this step, assuming the current time is T, when T = 0, the features output by the feature processing module only include the features at time T = 0. When T > 0, the features output by the feature processing module include both the image features at time T and the image features at historical times.

[0093] Step S334: Update the features in the storage module using feature 2 (dimension: N*H*W) obtained in step S331.

[0094] Taking the current image frame time as time T as an example, the feature processing module can use feature fusion to fuse the image features of the current time T with the image features of time T-1 stored in the storage module. The fused features contain information from both the current frame and historical frames (i.e., sequential features). It should be further explained that the image features output by the feature processing module at time T include both the image information of the current time T and the image information of time T-1. When these image features flow into the next feature processing module after passing through the feature extraction module, the features stored in the storage module after executing steps 331 and 334 will have image features from both time T and time T-1. The features extracted by the feature processing module at time T+1 after executing step S332 will have image features from both time T and time T-1. The features fused after executing step 333 will have image features from times T-1, T, and T+1, and so on. Therefore, the gesture recognition model of this invention can simultaneously acquire image features from the current time and image features from historical times when the input is a single-frame image.

[0095] This embodiment provides a single-frame image dynamic gesture recognition scheme based on video streams. It can achieve dynamic gesture recognition even with a single-frame image as input. The model only needs to infer one frame at a time to obtain the dynamic gesture category. Compared to previous dynamic gesture recognition methods with multi-frame images as input, this invention has faster inference speed, lower resource consumption, and recognition accuracy comparable to methods with multi-frame images. The main idea of ​​this method is to input the image captured by the camera into a preprocessing module. The preprocessing module crops and normalizes the gesture recognition region in the image before inputting it into the gesture recognition model for inference. The post-processing module obtains the output of the gesture recognition model, performs relevant processing to obtain the corresponding gesture category, and then combines the output gesture category with the scene to realize the corresponding function. For example, swiping left or right switches songs, and a heart gesture enables music collection.

[0096] This embodiment presents a method for dynamic gesture recognition using a single frame image from a video stream. The model only needs to infer one frame image at a time to obtain the dynamic gesture category. This method has a fast inference speed and low computational resource consumption.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0098] Example 2

[0099] This embodiment also provides a gesture recognition device based on a single-frame image, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0100] Figure 6 This is a structural block diagram of a gesture recognition device based on a single-frame image according to an embodiment of the present invention, such as... Figure 6 As shown, the device includes:

[0101] The first acquisition module 60 is used to acquire the target image captured in real time by the camera;

[0102] The second acquisition module 62 is used to acquire the historical temporal features of the target image from the storage module, wherein the historical temporal features are the image features of the previous frame of the target image;

[0103] The fusion module 64 is used to fuse the image features of the target image and the historical time-series features to generate the current time-series features;

[0104] The recognition module 66 is used to identify the gesture category based on the current temporal features.

[0105] Optionally, the fusion module includes: an extraction unit, configured to extract a first image feature of the target image using a feature extraction module; an iteration unit, configured to iteratively execute the following steps based on the first image feature until the subsequent module of the current module is a classification module: determining whether the subsequent module of the current module is a feature extraction module or a feature processing module; if the subsequent module of the current module is a feature processing module, inputting the first image feature into the feature processing module, using the feature processing module to perform fusion processing on the first image feature and the historical time-series feature to obtain a fused feature, and inputting the fused feature into the subsequent module; if the subsequent module of the current module is a feature extraction module, inputting the first image feature into the feature extraction module, using the feature extraction module to extract a second image feature of the first image feature, and inputting the second image feature into the subsequent module; and an output unit, configured to output the current time-series feature after the iteration is completed.

[0106] Optionally, the iterative unit includes: a splitting subunit, used to split the first image feature into a first sub-feature of m dimensions and a second sub-feature of n dimensions using the feature processing module, wherein the feature dimension of the historical image feature is n, and m and n are all positive integers greater than 0; and a fusion subunit, used to fuse the first sub-feature and the historical time-series feature to obtain a fused feature of m+n dimensions, and update the historical time-series feature to the second sub-feature in the storage module of the feature processing module, wherein each feature processing module corresponds to a storage module.

[0107] Optionally, the recognition module includes: a processing unit, configured to input the current temporal feature into a pre-trained gesture recognition model and output the current gesture category and current confidence level to which the current temporal feature belongs; an acquisition unit, configured to acquire the historical gesture categories and corresponding historical confidence levels of several historical images preceding the first image; and a generation unit, configured to generate a final gesture category based on the current gesture category, the current confidence level, the historical gesture categories, and the historical confidence levels.

[0108] Optionally, the generation unit includes: a calculation subunit, used to calculate a weighted average of the confidence scores of the current gesture category and all historical gesture categories to obtain a final confidence score; and a selection subunit, used to select the final gesture category with the highest confidence score from the current gesture category and the historical gesture categories based on the final confidence score.

[0109] Optionally, the device further includes: a third acquisition module, configured to acquire image sequences of multiple gesture actions before the recognition module inputs the current temporal features into the pre-trained gesture recognition model; a sampling module, configured to sequentially sample sample image groups from the image sequence for each gesture action, wherein the sample image group includes multiple gesture images, all of which correspond to the same gesture category, to obtain training data including B sample image groups; a processing module, configured to process the training data into T data of B*C*H*W dimensions, where T represents the number of image frames sampled from the image sequence, and C, H, and W represent the number of channels and the height and width of the image; a first training module, configured to input the B*C*H*W dimension data into the initial model for T inference training iterations to obtain T intermediate training results; and a second training module, configured to calculate the final training result from the T intermediate training results and use the final training result to calculate the loss function to train the initial module to obtain the gesture recognition model.

[0110] Optionally, the device further includes: a determining module, configured to determine the business scenario of the vehicle's infotainment controller after the recognition module identifies the gesture category based on the current temporal features; a searching module, configured to search for a target control command matching the gesture category in a preset command library of the business scenario; and a transmitting module, configured to transmit the target control command to the vehicle's infotainment controller.

[0111] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0112] Example 3

[0113] Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0114] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0115] S1, acquire the target image captured in real time by the camera;

[0116] S2, obtain the historical temporal features of the target image from the storage module, wherein the historical temporal features are the image features of the previous frame of the target image;

[0117] S3, the current time-series features are generated by fusing the image features of the target image and the historical time-series features;

[0118] S4, identify the gesture category based on the current temporal features.

[0119] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0120] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0121] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0122] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0123] S1, acquire the target image captured in real time by the camera;

[0124] S2, obtain the historical temporal features of the target image from the storage module, wherein the historical temporal features are the image features of the previous frame of the target image;

[0125] S3, the current time-series features are generated by fusing the image features of the target image and the historical time-series features;

[0126] S4, identify the gesture category based on the current temporal features.

[0127] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0128] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0129] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0134] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A gesture recognition method based on a single-frame image, characterized in that, include: Acquire target images captured in real time by the camera; The historical temporal features of the target image are obtained from the storage module, wherein the historical temporal features are the image features of the previous frame of the target image; The current time-series features are generated by fusing the image features of the target image and the historical time-series features. The gesture category is identified based on the current temporal features; The process of identifying the gesture category based on the current temporal features includes: inputting the current temporal features into a pre-trained gesture recognition model, and outputting the current gesture category and current confidence level to which the current temporal features belong; obtaining the historical gesture categories and corresponding historical confidence levels of several historical images preceding the target image; generating a final gesture category based on the current gesture category, the current confidence level, the historical gesture categories, and the historical confidence levels; wherein generating the final gesture category based on the current gesture category, the current confidence level, the historical gesture categories, and the historical confidence levels includes: taking a weighted average of the confidence levels of the current gesture category and all historical gesture categories to obtain a final confidence level; and selecting the final gesture category with the highest confidence level from the current gesture category and the historical gesture categories based on the final confidence level. Before inputting the current temporal features into the pre-trained gesture recognition model, the method further includes: acquiring image sequences of multiple gesture actions; for each gesture action image sequence, sequentially sampling from the image sequence to obtain a sample image group, wherein the sample image group includes multiple gesture images, all of which correspond to the same gesture category, to obtain training data including B sample image groups; processing the training data into T data of B*C*H*W dimensions, where T represents the number of image frames sampled from the image sequence, and C, H, and W represent the number of channels and the height and width of the image, respectively; inputting the B*C*H*W dimension data into the initial model for T inference training iterations to obtain T intermediate training results; calculating the T intermediate training results to obtain the final training result, and using the final training result to calculate the loss function to train the initial model to obtain the gesture recognition model.

2. The method according to claim 1, characterized in that, The process of fusing the image features of the target image and the historical time-series features to generate the current time-series features includes: The first image feature of the target image is extracted using a feature extraction module; Based on the first image features, iteratively execute the following steps until the next module of the current module is a classification module: determine whether the next module of the current module is a feature extraction module or a feature processing module; if the next module of the current module is a feature processing module, input the first image features into the feature processing module, use the feature processing module to perform fusion processing on the first image features and the historical time series features to obtain fused features, and input the fused features into the next module; if the next module of the current module is a feature extraction module, input the first image features into the feature extraction module, use the feature extraction module to extract the second image features of the first image features, and input the second image features into the next module; After the iteration is completed, the current time series features are output.

3. The method according to claim 2, characterized in that, The feature processing module is used to fuse the first image features and the historical time-series features to obtain fused features, including: The feature processing module is used to split the first image feature into a first sub-feature with an m-dimensional dimension and a second sub-feature with an n-dimensional dimension, wherein the feature dimension of the historical image feature is n, and m and n are all positive integers greater than 0; The first sub-feature and the historical time-series feature are fused to obtain an m+n dimension fused feature. The historical time-series feature is then updated to the second sub-feature in the storage module of the feature processing module. Each feature processing module corresponds to one storage module.

4. The method according to claim 1, characterized in that, After identifying the gesture category based on the current temporal features, the method further includes: Determine the business scenario of the vehicle's infotainment controller where the camera is located; Search for a target control command that matches the gesture category in the preset command library of the business scenario; The target control command is transmitted to the vehicle controller.

5. A gesture recognition device based on a single-frame image, characterized in that, include: The first acquisition module is used to acquire target images captured in real time by the camera; The second acquisition module is used to acquire the historical temporal features of the target image from the storage module, wherein the historical temporal features are the image features of the previous frame of the target image; The fusion module is used to fuse the image features of the target image and the historical time-series features to generate the current time-series features; The recognition module is used to identify the gesture category based on the current temporal features; The recognition module includes: a processing unit, used to input the current temporal feature into a pre-trained gesture recognition model and output the current gesture category and current confidence level of the current temporal feature; an acquisition unit, used to acquire the historical gesture categories and corresponding historical confidence levels of several historical images preceding the target image; and a generation unit, used to generate a final gesture category based on the current gesture category, the current confidence level, the historical gesture categories, and the historical confidence levels. The generation unit includes: a calculation subunit, used to perform a weighted average of the confidence levels of the current gesture category and all historical gesture categories to obtain a final confidence level; and a selection subunit, used to select the final gesture category with the highest confidence level from the current gesture category and the historical gesture categories based on the final confidence level. The device further includes: a third acquisition module, configured to acquire image sequences of multiple gesture actions before the recognition module inputs the current temporal features into the pre-trained gesture recognition model; a sampling module, configured to sequentially sample sample image groups from the image sequence for each gesture action, wherein the sample image group includes multiple gesture images, all corresponding to the same gesture category, to obtain training data including B sample image groups; a processing module, configured to process the training data into T data of B*C*H*W dimensions, where T represents the number of image frames sampled from the image sequence, and C, H, and W represent the number of channels and the height and width of the image; a first training module, configured to input the B*C*H*W dimension data into the initial model for T inference training iterations to obtain T intermediate training results; and a second training module, configured to calculate the final training result from the T intermediate training results and use the final training result to calculate the loss function to train the initial model to obtain the gesture recognition model.

6. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 4 when it is run.

7. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method described in any one of claims 1 to 4.

Images