Real-time desktop gesture understanding method, system and device based on timing spatial features

Abstract

The present application belongs to the field of computer vision, and particularly relates to a real-time desktop gesture understanding method, system and device based on time sequence space features, aiming at solving the problem of poor gesture recognition accuracy of existing desktop gesture understanding methods. The method comprises: acquiring a desktop gesture RGB image to be classified and recognized in real time as an input image; converting the input image from an RGB space to an HSV space, segmenting a gesture region in the input image according to a depth value of a pixel point, performing binary processing after segmentation, and obtaining a gesture binary image; fusing gesture binary images in consecutive n frames of input images, and simulating a decay process of consecutive frame gestures by using an exponential decay model to construct a time sequence feature image containing space-time features; and inputting the time sequence feature image containing space-time features into a pre-constructed gesture classification model to obtain a gesture category recognition result corresponding to the input image. The present application improves gesture recognition accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a real-time desktop gesture understanding method, system, and device based on temporal and spatial features. Background Technology

[0002] Desktop gesture interaction is a common and simple human-computer interaction method with broad application prospects. For example, when a person interacts with a computer through gestures, they make a series of gestures in front of a camera, and the computer responds. To enable the computer to learn and initially understand the meaning of human gestures, it must first understand the meaning of each gesture. Common desktop gestures include "go," "take," "move," "return," "tap," "press," and "press." This is only a preliminary understanding of gestures; there are also more refined understandings of hand joints. However, accurately recognizing human gestures is a very complex problem. During human demonstrations to robots using gestures, the continuity of movements makes it crucial to enable machines to quickly and accurately recognize gestures during dynamic human operations. This becomes a pressing issue in transferring operational skills from humans to robots. Therefore, to improve gesture recognition accuracy, this invention proposes a real-time desktop gesture understanding method based on temporal and spatial features. Summary of the Invention

[0003] To address the aforementioned problems in the prior art, specifically the poor gesture recognition accuracy of existing desktop gesture understanding methods, the first aspect of this invention proposes a real-time desktop gesture understanding method based on temporal spatial features, comprising:

[0004] S100 acquires RGB images of desktop gestures to be classified and recognized in real time as input images;

[0005] S200, the input image is converted from RGB space to HSV space; in HSV space, the gesture region in the input image is segmented according to the depth value of the pixel, and after segmentation, it is binarized to obtain a gesture binary image;

[0006] S300 fuses binary images of gestures from n consecutive input images and uses an exponential decay model to simulate the decay process of gestures in consecutive frames, constructing a temporal feature image containing spatiotemporal features.

[0007] S400, the temporal feature image containing spatiotemporal features is input into a pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image; the gesture classification model is constructed based on an object detection network containing DarkNet.

[0008] In some preferred embodiments, the input image is converted from RGB space to HSV space by the following method:

[0009] The input image is converted from RGB space to HSV space using a pre-built skin color model:

[0010] V = max(R, G, B)

[0011]

[0012]

[0013] in,

[0014] In some preferred embodiments, the binary images of gestures from n consecutive input frames are fused, and an exponential decay model is used to simulate the decay process of gestures in consecutive frames to construct a temporal feature image containing spatiotemporal features. The method is as follows:

[0015] S301, In the temporal domain, the weights of each gesture binary image are decreased from near to far based on the time between the input images of the previous n-1 frames and the current frame.

[0016] S302, based on the attenuated weights, weighted fusion of the gesture binary images from the previous n-1 frames in the background region of the gesture binary image in the current frame:

[0017] S3021, Select the gesture binary image with the highest weight from the n frames of gesture binary images to be fused as the image M to be fused, and then select the gesture binary image with the highest weight from the remaining images as M1;

[0018] S3022, merge M and M1, and replace the original image M to be merged with the merged image;

[0019] S3023, select the gesture binary image with the largest weight from the remaining gesture binary images as the new M1, and then jump to S3022 until all gesture binary images are fused to obtain a temporal feature image containing the spatiotemporal features of gesture information fused from n consecutive frames.

[0020] In some preferred embodiments, the weights of each gesture binary image are attenuated according to the time elapsed between the current frame and the n-frame input image, as follows:

[0021] w x =e -βx (x≥0)

[0022] Among them, w x This represents the weight of the x-th frame preceding the current frame, and β represents the set attenuation value.

[0023] In some preferred embodiments, the fusion method of M and M1 at point P = (x, y) is as follows:

[0024]

[0025] Where M(P) represents the pixel value of point P in the image M to be fused, and M1(P) represents the pixel value of point P in M1.

[0026] In some preferred embodiments, the gesture classification model is constructed based on an object detection network including DarkNet, and then reconstructed and optimized using TensorRT. The reconstruction and optimization methods are as follows:

[0027] The object detection network containing DarkNet is reconstructed and optimized using TensorRT. The reconstruction and optimization method is as follows: TensorRT is used to fuse the three layers of the object detection network containing DarkNet, namely the conv layer, the BN layer, and the ReLU layer, into one layer. The object detection network containing DarkNet after fusion is used as the final gesture classification model.

[0028] In some preferred embodiments, the loss function corresponding to the gesture classification model is Loss WIOU for:

[0029]

[0030]

[0031] B = p 2 (b,b gt ) / c 2

[0032] C=αv

[0033] Among them, Loss IOU The loss values ​​are obtained from traditional IOU-based calculations, b and b gt Representing the predicted bounding box and the ground truth bounding box, ∩(b, b) gt ) represents b and b gt The size of the intersecting part, ∪(b, b) gt ) represents b and b gt The size of the combined area, p 2 (b,b gt ) represents b and b gt The Euclidean distance between the centers, where c represents the distance between b and b'. gt The maximum distance between the eight vertices, α is the weight function, and v is used to measure the similarity of aspect ratio. Its calculation formula is as follows:

[0034]

[0035] Among them, w, h and w gt h gt These represent the width and height of the predicted bounding box and the width and height of the ground truth bounding box, respectively.

[0036] In a second aspect, the present invention proposes a real-time desktop gesture understanding system based on temporal and spatial features, comprising: an image acquisition module, a binarization processing module, an image fusion module, and a gesture recognition module;

[0037] The image acquisition module is configured to acquire RGB images of desktop gestures to be classified and recognized in real time as input images;

[0038] The binarization processing module is configured to convert the input image from RGB space to HSV space; in HSV space, the gesture region in the input image is segmented according to the depth value of the pixel, and then binarized to obtain a gesture binary image;

[0039] The image fusion module is configured to fuse binary images of gestures from n consecutive input images and use an exponential decay model to simulate the decay process of gestures in consecutive frames, thereby constructing a temporal feature image containing spatiotemporal features.

[0040] The gesture recognition module is configured to input the temporal feature image containing spatiotemporal features into a pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image; the gesture classification model is constructed based on an object detection network including DarkNet.

[0041] A third aspect of the present invention provides an electronic device comprising: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor for implementing the above-described real-time desktop gesture understanding method based on temporal spatial features.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions for execution by the computer to implement the above-described real-time desktop gesture understanding method based on temporal spatial features.

[0043] The beneficial effects of this invention are:

[0044] This invention improves the accuracy of gesture recognition.

[0045] 1) This invention uses a skin color model combined with the depth information of the hand in the operation space to segment and binarize the gesture to obtain a binary image of the gesture; it fuses the depth and temporal information of multiple frames of binary gesture images to construct a temporal feature image containing spatiotemporal features, thus transforming the temporal feature image classification problem of desktop gestures into a temporal feature image detection problem for different desktop gestures, thereby improving the detection accuracy.

[0046] 2) This invention uses a deep learning neural network for training and TensorRT to accelerate the inference process, thereby improving the accuracy and speed of detection. Attached Figure Description

[0047] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0048] Figure 1 This is a flowchart illustrating a real-time desktop gesture understanding method based on temporal spatial features according to an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the framework of a real-time desktop gesture understanding system based on temporal and spatial features according to an embodiment of the present invention.

[0050] Figure 3 This is a simplified flowchart illustrating a real-time desktop gesture understanding method based on temporal spatial features according to an embodiment of the present invention.

[0051] Figure 4 This is a schematic flowchart illustrating the construction of a temporal feature image containing spatiotemporal features according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the model structure of a gesture classification model according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing electronic devices according to an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0055] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0057] The real-time desktop gesture understanding method based on temporal and spatial features of the present invention, such as... Figure 1 As shown, it includes the following steps:

[0058] S100 acquires RGB images of desktop gestures to be classified and recognized in real time as input images;

[0059] S200, the input image is converted from RGB space to HSV space; in HSV space, the gesture region in the input image is segmented according to the depth value of the pixel, and after segmentation, it is binarized to obtain a gesture binary image;

[0060] S300 fuses binary images of gestures from n consecutive input images and uses an exponential decay model to simulate the decay process of gestures in consecutive frames, constructing a temporal feature image containing spatiotemporal features.

[0061] S400, the temporal feature image containing spatiotemporal features is input into a pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image; the gesture classification model is constructed based on an object detection network containing DarkNet.

[0062] To more clearly explain the real-time desktop gesture understanding method based on temporal and spatial features of this invention, the following description is in conjunction with the appendix. Figure 3 , 4 The steps of one embodiment of the method of the present invention will be described in detail below.

[0063] S100 acquires RGB images of desktop gestures to be classified and recognized in real time as input images;

[0064] In this embodiment, the RGB images of the desktop gestures to be classified and recognized are first acquired in real time. Common desktop gestures include actions such as "go", "take", "move", "return", "tap", "press", and "press".

[0065] S200, the input image is converted from RGB space to HSV space; in HSV space, the gesture region in the input image is segmented according to the depth value of the pixel, and after segmentation, it is binarized to obtain a gesture binary image;

[0066] In this embodiment, the RGB image of the desktop gesture is preprocessed to obtain a binary image of the gesture. Specifically:

[0067] The input image is converted from RGB space to HSV space using a pre-built skin color model:

[0068] V = max(R, G, B) (1)

[0069]

[0070] (3)

[0071] in, The H channel determines the color tone information of an object. Experimental measurements have shown that the optimal H threshold for skin color segmentation is 6 to 48. Using this range, most gesture areas can be segmented, and it has excellent noise reduction performance.

[0072] Then, based on the gesture segmented from the skin color model, the depth value range near it is obtained. Pixels outside this range are removed to obtain a clear gesture image. That is, the gesture region in the input image is segmented based on the depth value of the pixels (e.g., ...). Figure 3 As shown in the figure, after segmentation, binarization processing is performed (i.e., binarization based on background and foreground distinction). Each frame after processing is a binarized image that only retains the shape of the gesture, i.e., a gesture binary image. Among them, the depth value of the depth image refers to the distance (i.e., a vertical distance) from the corresponding pixel point in space to the plane where the Kincet sensor is located.

[0073] S300 fuses binary images of gestures from n consecutive input images and uses an exponential decay model to simulate the decay process of gestures in consecutive frames, constructing a temporal feature image containing spatiotemporal features.

[0074] In this embodiment, the problem of classifying the temporal feature map of desktop gestures is transformed into the problem of detecting temporal feature images for different desktop gestures. Specifically, based on the order between different image frames, the binary images of gestures from n consecutive frames are fused, and an exponential decay model is used to simulate the decay process of gestures in consecutive frames, constructing a temporal feature image containing spatiotemporal features, such as... Figure 4 As shown, images with longer time intervals from the current frame have greater attenuation weights. Specifically:

[0075] S301, in the temporal domain, the weights of each gesture binary image are decayed from near to far based on the time between the input images of the previous n-1 frames and the current frame; in this invention, an exponential decay model is used to simulate the decay of historical gestures, as shown in formula (4):

[0076] w x =e -βx (x≥0) (4)

[0077] Among them, w x This represents the weight of the x-th frame preceding the current frame, and β represents the set attenuation value, which is preferably set to 0.5 in this invention.

[0078] S302, based on the attenuated weights, weighted fusion of the gesture binary images from the previous n-1 frames in the background region of the gesture binary image in the current frame:

[0079] S3021, select the gesture binary image with the highest weight from the n frames of gesture binary images to be fused as the image M to be fused, and then select the gesture binary image with the highest weight from the remaining images as M1; that is, when fusing these n frames of gesture binary images, the algorithm only weights the fusion in the background area, and first fuses the gesture binary image with the highest weight.

[0080] S3022, merge M and M1, and replace the original image M to be merged with the merged image; wherein, the merging method of M and M1 at point P = (x, y) is as follows:

[0081]

[0082] Where M(P) represents the pixel value of point P in the image M to be fused, and M1(P) represents the pixel value of point Pp in M1.

[0083] S3023, select the gesture binary image with the largest weight from the remaining gesture binary images as the new M1, and then jump to S3022 until all gesture binary images are fused to obtain a temporal feature image containing the spatiotemporal features of gesture information fused from n consecutive frames.

[0084] S400, the temporal feature image containing spatiotemporal features is input into a pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image; the gesture classification model is constructed based on an object detection network containing DarkNet.

[0085] In this embodiment, the gesture classification model is constructed based on an object detection network incorporating DarkNet. The object detection network is preferably the YOLO object detection network. In this invention, the YOLOv4 algorithm implemented using the CSPDarknet53 framework is preferably used to train the gesture classification model on fused images of gestures from different time periods (i.e., temporal feature images containing spatiotemporal features). In other embodiments, other YOLO series image detection algorithms or other object detection algorithms implemented using the DarkNet framework can also be used to construct the gesture classification model.

[0086] The overall structure of the YOLOv4 network is as follows: Figure 5As shown, the backbone network uses the CSPDarknet53 network. CSPDarknet53 has five more CSP modules than the YOLOv3 backbone network Darknet53. The CSP modules divide the feature maps of the base layers into two parts, and then merge them through a cross-stage hierarchical structure, reducing computation while maintaining accuracy. YOLOv4 only uses the Mish function in the backbone network; the remaining parts use the Leaky_ReLU activation function, and the Dropblock regularization method is used to randomly remove neurons, simplifying the network structure. The formula for the Mish function is as follows:

[0087] Mish=x×tanh(In(1+e x (6)

[0088] To achieve a larger receptive field for predicted bounding boxes and effectively separate salient features from the data, YOLOv4 adds an SPP module after the CSPDarknet53 structure. YOLOv4 employs an FPN+PAN module, with a bottom-up feature pyramid following the FPN, containing two PAN structures. The FPN layer conveys strong semantic features from top to bottom and strong localization features from bottom to top. This allows the improved FPN+PAN structure to be used for parameter aggregation at different levels of detectors, further enhancing feature extraction capabilities and thus improving target detection performance.

[0089] The binary images of continuous gesture motions fused with temporal features form fused images with significantly different attenuation in length, width, and distance along the gesture motion direction. To improve accuracy, for binary temporal fusion images, the IOU evaluation in the traditional DarkNet Loss function is modified to introduce an IOU evaluation method based on the aspect ratio of the detection box. The original DarkNet Loss function is: Where b and b gt Let ∩(b, b) represent the predicted bounding box (i.e., the gesture category recognition result output by the gesture classification model during training) and the ground truth bounding box (i.e., the ground truth label of the gesture category recognition result given in advance). gt ) represents b and b gt The size of the intersecting part, ∪(b, b) gt ) represents b and b gt The size of the combined area.

[0090] This invention introduces an IOU evaluation method based on the aspect ratio of the detection frame to calculate the loss. WIOU The method is as follows:

[0091]

[0092]

[0093] B = p 2 (b,b gt ) / c 2 (9)

[0094] C=αv (10)

[0095] Among them, Loss IOU The loss value is obtained from traditional IOU-based calculations, p 2 (b,b gt ) represents b and The Euclidean distance between the centers, where c represents the distance between b and b'. gt The maximum distance between the eight vertices, α is the weight function, and v is used to measure the similarity of aspect ratio. Its calculation formula is as follows:

[0096]

[0097] Among them, w, h and w gt h gt These represent the width and height of the predicted bounding box and the width and height of the ground truth bounding box, respectively.

[0098] In addition, to further improve the accuracy and speed of the gesture classification model for different desktop gestures, the gesture classification model was reconstructed and optimized using TensorRT, specifically as follows:

[0099] The three layers of the object detection network containing DarkNet—conv, BN, and ReLU—are fused into a single layer using the TensorRT structure (i.e., horizontally combining the network by fusing layers with the same input tensors and performing the same operations, which accelerates computation). The fused object detection network containing DarkNet is then used as the final gesture classification model.

[0100] Finally, the temporal feature image containing spatiotemporal features is input into the pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image, such as... Figure 5 The gesture category shown is 1... Gesture recognition is 6.

[0101] A real-time desktop gesture understanding system based on temporal and spatial features, as described in the second embodiment of the present invention, is as follows: Figure 2 As shown, it includes: an image acquisition module 100, a binarization processing module 200, an image fusion module 300, and a gesture recognition module 400;

[0102] The image acquisition module 100 is configured to acquire RGB images of desktop gestures to be classified and recognized in real time as input images;

[0103] The binarization processing module 200 is configured to convert the input image from RGB space to HSV space; in HSV space, the gesture region in the input image is segmented according to the depth value of the pixel, and then binarized to obtain a gesture binary image.

[0104] The image fusion module 300 is configured to fuse binary images of gestures from n consecutive input images and use an exponential decay model to simulate the decay process of gestures in consecutive frames to construct a temporal feature image containing spatiotemporal features.

[0105] The gesture recognition module 400 is configured to input the temporal feature image containing spatiotemporal features into a pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image; the gesture classification model is constructed based on an object detection network including DarkNet.

[0106] It should be noted that the real-time desktop gesture understanding system based on temporal and spatial features provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0107] An electronic device according to a third embodiment of the present invention includes: at least one processor; and a memory communicatively connected to at least one of the processors; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to implement the above-described real-time desktop gesture understanding method based on temporal spatial features.

[0108] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are executed by the computer to implement the above-described real-time desktop gesture understanding method based on temporal and spatial features.

[0109] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the electronic devices and computer-readable storage media described above can be referred to the corresponding processes in the foregoing method examples, and will not be repeated here.

[0110] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system suitable for using a server to implement the methods, systems, and apparatus embodiments of this application. Figure 6 The server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0111] like Figure 6 As shown, the computer system includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read Only Memory (ROM) 602 or programs loaded from storage section 608 into Random Access Memory (RAM) 603. RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0112] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0113] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by the central processing unit (CPU 601), it performs the functions defined in the method of this application. It should be noted that the computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above. In this application... In this context, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0114] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0117] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0118] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A real-time desktop gesture understanding method based on temporal and spatial features, characterized in that, The method includes the following steps: S100 acquires RGB images of desktop gestures to be classified and recognized in real time as input images; S200, the input image is converted from RGB space to HSV space; in HSV space, the gesture region in the input image is segmented according to the depth value of the pixel, and after segmentation, it is binarized to obtain a gesture binary image; The method for converting the input image from RGB space to HSV space is as follows: The input image is converted from RGB space to HSV space using a pre-built skin color model: ； ； ； in, , , ; S300 fuses binary images of gestures from n consecutive input images and uses an exponential decay model to simulate the decay process of gestures in consecutive frames, constructing a temporal feature image containing spatiotemporal features. S400, the temporal feature image containing spatiotemporal features is input into a pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image; the gesture classification model is constructed based on an object detection network containing DarkNet; By fusing binary gesture images from n consecutive input frames and employing an exponential decay model to simulate the decay process of gestures across consecutive frames, a temporal feature image containing spatiotemporal characteristics is constructed. The method is as follows: S301, In the temporal domain, the weights of each gesture binary image are decreased from near to far based on the time between the input images of the previous n-1 frames and the current frame. S302, based on the attenuated weights, weighted fusion of the gesture binary images from the previous n-1 frames in the background region of the gesture binary image in the current frame: S3021, Select the gesture binary image with the highest weight from the n frames of gesture binary images to be fused as the image M to be fused, and then select the gesture binary image with the highest weight from the remaining images as M1; S3022, merge M and M1, and replace the original image M to be merged with the merged image; S3023, select the gesture binary image with the largest weight from the remaining gesture binary images as the new M1, and then jump to S3022 until all gesture binary images are fused to obtain a temporal feature image containing the spatiotemporal features of gesture information fused from n consecutive frames.

2. The real-time desktop gesture understanding method based on temporal spatial features according to claim 1, characterized in that, The weights of each gesture binary image are attenuated from the nearest to the farthest time from the previous n-1 frames of the input image to the current frame. The method is as follows: ； in, Indicates the distance from the previous frame. x Frame weights, This indicates the set attenuation value.

3. The real-time desktop gesture understanding method based on temporal spatial features according to claim 1, characterized in that, M and M1 are at point The fusion method at the location is as follows: ； in, The midpoint of image M to be fused pixel values, Indicates the midpoint of M1 The pixel value.

4. The real-time desktop gesture understanding method based on temporal spatial features according to claim 1, characterized in that, The gesture classification model is constructed based on an object detection network including DarkNet, and then reconstructed and optimized using TensorRT. The reconstruction and optimization methods are as follows: The three layers of the object detection network containing DarkNet—conv, BN, and ReLU—are fused into one layer using the TensorRT structure. The fused object detection network containing DarkNet serves as the final gesture classification model.

5. The real-time desktop gesture understanding method based on temporal spatial features according to claim 1, characterized in that, The loss function corresponding to the gesture classification model for: ； ； ； ； in, The loss value is obtained through traditional IOU-based calculations. and Representing predicted bounding boxes and ground truth bounding boxes, Indicate b and The size of the area of ​​the intersecting part, Indicate b and The size of the combined area, represent and Euclidean distance between centers represent and The maximum distance between the eight vertices. It is a weighting function. The similarity measured by aspect ratio is calculated using the following formula: ； in, , and , These represent the width and height of the predicted bounding box and the width and height of the ground truth bounding box, respectively.

6. A real-time desktop gesture understanding system based on temporal and spatial features, characterized in that, The system includes: an image acquisition module, a binarization processing module, an image fusion module, and a gesture recognition module; The image acquisition module is configured to acquire RGB images of desktop gestures to be classified and recognized in real time as input images; The binarization processing module is configured to convert the input image from RGB space to HSV space; in HSV space, the gesture region in the input image is segmented according to the depth value of the pixel, and then binarized to obtain a gesture binary image; The method for converting the input image from RGB space to HSV space is as follows: The input image is converted from RGB space to HSV space using a pre-built skin color model: ； ； ； in, , , ; The image fusion module is configured to fuse binary images of gestures from n consecutive input images and use an exponential decay model to simulate the decay process of gestures in consecutive frames, thereby constructing a temporal feature image containing spatiotemporal features. The gesture recognition module is configured to input the temporal feature image containing spatiotemporal features into a pre-constructed gesture classification model to obtain the gesture category recognition result corresponding to the input image; the gesture classification model is constructed based on an object detection network including DarkNet; By fusing binary gesture images from n consecutive input frames and employing an exponential decay model to simulate the decay process of gestures across consecutive frames, a temporal feature image containing spatiotemporal characteristics is constructed. The method is as follows: S301, In the temporal domain, the weights of each gesture binary image are decreased from near to far based on the time between the input images of the previous n-1 frames and the current frame. S302, based on the attenuated weights, weighted fusion of the gesture binary images from the previous n-1 frames in the background region of the gesture binary image in the current frame: S3021, Select the gesture binary image with the highest weight from the n frames of gesture binary images to be fused as the image M to be fused, and then select the gesture binary image with the highest weight from the remaining images as M1; S3022, merge M and M1, and replace the original image M to be merged with the merged image; S3023, select the gesture binary image with the largest weight from the remaining gesture binary images as the new M1, and then jump to S3022 until all gesture binary images are fused to obtain a temporal feature image containing the spatiotemporal features of gesture information fused from n consecutive frames.

7. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement the real-time desktop gesture understanding method based on temporal spatial features as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are executed by the computer to implement the real-time desktop gesture understanding method based on temporal spatial features as described in any one of claims 1-5.

Images