Gesture-based real-time game object control system and method

Abstract

Disclosed are a gesture-based real-time game object control system and method. The system comprises: a data acquisition module, the data acquisition module being used for acquiring real-time video data of a user; a data preprocessing module, the data preprocessing module being used for preprocessing the real-time video data; a gesture feature extraction module, the gesture feature extraction module being used for extracting gesture features; a gesture feature recognition module, the gesture feature recognition module being used for recognizing hand actions; and a coordinate transformation module, the coordinate transformation module being used for performing coordinate transformations in a game on the basis of changes in the hand actions. The present invention provides a gesture-based real-time game object control system that uses video as input, matches and classifies different gestures via a deep learning network, identifies gesture types and action categories, and maps same to coordinates in the game to obtain coordinate transformations of game objects, thereby ensuring the accuracy and smoothness of game object control.

Description

Gesture-based game object real-time control system and method TECHNICAL FIELD

[0001] The present application belongs to the field of information technology, and particularly relates to a gesture-based game object real-time control system and method. BACKGROUND

[0002] Gesture posture estimation can achieve more efficient interaction by sensing and recognizing gesture actions and giving them specific meanings, thereby eliminating the need for most electronic control devices to be physically connected, and having great convenience.

[0003] In related technologies, such as a terminal gesture-based game control method and system disclosed in CN 107261498 A, a first picture of a scene is photographed, fixed objects of the scene are identified from the first picture; multiple pictures are periodically photographed, the distance between the hand and the fixed objects in the multiple pictures is obtained according to the photographing order; the specific first gesture is determined according to the change trend of the distance, and the game character is controlled to move according to the first gesture. However, the way of periodically photographing multiple pictures for recognition in related technologies has some deficiencies in accuracy, flow degree and anti-interference ability. SUMMARY

[0004] The purpose of the present application is to provide a gesture-based game object real-time control system and method, which solves the problem of the deficiency in accuracy, flow degree and anti-interference ability of the way of periodically photographing multiple pictures for recognition in the prior art.

[0005] To this end, the present application provides a gesture-based game object real-time control system, comprising:

[0006] A data acquisition module, configured to acquire real-time video data of a user;

[0007] A data preprocessing module, electrically connected with the data acquisition module, configured to preprocess the real-time video data;

[0008] A gesture feature extraction module, electrically connected with the data preprocessing module, configured to extract gesture features;

[0009] A gesture feature recognition module, electrically connected with the gesture feature extraction module, configured to recognize the action of the hand;

[0010] A coordinate conversion module, electrically connected with the gesture feature recognition module, configured to convert coordinates in the game according to the change in the action of the hand.

[0011] Preferably, the data preprocessing module decomposes the collected real-time video data into video frame data, pre-processes the video frame data through greying and binarization, and eliminates the video frame data not containing the hand. The pre-processing of the video frame data through greying and binarization can highlight the color similar to the human skin color, so as to better highlight the hand data information. Eliminating the video frame data not containing the hand can avoid performance waste and improve the accuracy of data processing.

[0012] Preferably, the gesture feature extraction module extracts the key points, contour and direction of the gesture from the pre-processed video frame data to obtain the gesture feature.

[0013] Preferably, the gesture feature recognition module is provided with a gesture feature recognition model, which is used for matching and classifying the gesture feature.

[0014] Preferably, the gesture feature recognition model matches the gesture feature to identify the type of the gesture.

[0015] Preferably, the gesture feature recognition model classifies the gesture feature to identify the action category of the gesture.

[0016] Preferably, it further comprises an initialization calibration module, which is electrically connected with the coordinate conversion module, and is used for initializing and calibrating the gesture and the object in the game.

[0017] Preferably, it further comprises a cycle module, which is electrically connected with the data acquisition module, the data preprocessing module, the gesture feature extraction module, the gesture feature recognition module and the coordinate conversion module, and is used for cyclically updating the coordinates of the object in the game in real time.

[0018] A game object real-time control method based on gesture, comprising the following steps:

[0019] Collecting real-time images of a user to obtain video data; pre-processing the video data to obtain pre-processed video data; extracting gesture features in the pre-processed video data; identifying the gesture features to obtain the posture of the corresponding gesture; converting the coordinates of the object in the game according to the posture of the gesture to obtain the coordinate mapping of the object in the game.

[0020] Preferably, the initialization calibration is performed before the real-time control of the game object. Beneficial effects:

[0021] 1.The application provides a gesture-based game object real-time control system, which uses video as input, matches and classifies different gestures through a deep learning network, identifies gesture types and action categories, and maps them through coordinates in the game to obtain coordinate transformation of objects in the game, ensuring the accuracy and smoothness of game object control.

[0022] 2.In the application, the color similar to human skin color is highlighted through pre-processing of video data, so as to better highlight hand data information and improve the accuracy and anti-interference ability of gesture feature extraction.

[0023] 3.In the application, users can control objects in the game without using traditional controllers, increasing the interest and interactivity of the game and improving the immersion and game experience of users. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0025] Fig. 1 is a system structure schematic diagram of embodiment 1 of a gesture-based game object real-time control system in the application;

[0026] Fig. 2 is a use method flow chart of embodiment 1 of a gesture-based game object real-time control system in the application; DETAILED DESCRIPTION

[0027] The content of the application can be more easily understood by referring to the following detailed description of the preferred embodiments of the application and the included embodiments. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the application belongs. When there is a conflict, the definition in the specification shall prevail.

[0028] Embodiment 1:

[0029] As shown in Fig. 1, a gesture-based game object real-time control system includes:

[0030] A data acquisition module for acquiring real-time video data of a user;

[0031] A data preprocessing module, which is electrically connected to the data acquisition module, for preprocessing real-time video data;

[0032] The gesture feature extraction module is electrically connected with the data preprocessing module, and is used for extracting gesture features.

[0033] The gesture feature recognition module is electrically connected with the gesture feature extraction module, and is used for recognizing the action of the hand.

[0034] The coordinate conversion module is electrically connected with the gesture feature recognition module, and is used for converting the coordinate in the game according to the action change of the hand.

[0035] The data preprocessing module decomposes the collected real-time video data into video frame data, and pre-processes the video frame data through greying and binarization, and eliminates the video frame data not containing the hand. The pre-processing of the video frame data through greying and binarization can highlight the color similar to the human skin color, so as to better highlight the hand data information. Eliminating the video frame data not containing the hand can avoid performance waste and improve the accuracy of data processing.

[0036] The gesture feature extraction module extracts the key points, contours and directions of the gesture from the pre-processed video frame data, and obtains gesture features.

[0037] The gesture feature recognition model is arranged in the gesture feature recognition module, and is used for matching and classifying the gesture features. The gesture feature recognition model uses the deep learning framework PyTorch to train the gesture feature recognition model through the image or video frame of the hand action by using the convolutional neural network (CNN).

[0038] The gesture feature recognition model matches the gesture features, and identifies the type of the gesture. In the embodiment, the type of the gesture includes gesture types such as lifting, pointing, holding and picking.

[0039] The gesture feature recognition model classifies the gesture features, and identifies the action category of the gesture. In the embodiment, the category of the gesture includes moving, rotating and scaling.

[0040] The initialization calibration module is electrically connected with the coordinate conversion module, and is used for initializing and calibrating the gesture and the object in the game.

[0041] The cycle module is electrically connected with the data acquisition module, the data preprocessing module, the gesture feature extraction module, the gesture feature recognition module and the coordinate conversion module, and is used for cyclically updating the coordinate of the object in the game in real time.

[0042] A gesture-based game object real-time control method includes the following steps:

[0043] S1, initialization calibration before real-time control of game objects.

[0044] The camera is used to capture the hand movement of the user, the hand is translated by a certain distance, and the mapping relationship between the translation distance of the object in the game and the translation distance of the hand is determined according to the set calibration distance. For example, in this embodiment, the hand is translated by any distance, and the calibration distance is 100 pixels, that is, it is determined that when the hand moves by this distance, the object in the game is translated by 100 pixels.

[0045] The camera is used to capture the hand movement of the user, the hand is rotated by a certain angle, and the mapping relationship between the rotation angle of the object in the game and the rotation angle of the hand is determined according to the set calibration angle. For example, in this embodiment, the hand is rotated by any angle, and the calibration angle is 45 degrees, that is, it is determined that when the hand is rotated by this distance, the object in the game is rotated by 45 degrees.

[0046] S2, collect real-time pictures of the user to obtain video data; the video data includes pictures containing hands and pictures not containing hands.

[0047] S3, pre-process the video data to obtain pre-processed video data;

[0048] The pre-processing includes decomposing the collected video data into video frames, using grayscale and binary processing, and deleting video frames not containing hand data. The color close to human skin color is protruded to better highlight the hand data information, avoid the influence of environmental color on hand recognition, and avoid performance waste.

[0049] S4, extract gesture features in the pre-processed video data;

[0050] The key points, contours and direction features of the gestures in the pre-processed video data are extracted for gesture feature recognition.

[0051] S5, recognize the gesture features to obtain the posture of the corresponding gesture;

[0052] The gesture features are recognized by the gesture feature recognition model, and the gesture features are matched and classified. In this embodiment, a large number of hand motion images and video frames of mobile phones are used, and a convolutional neural network is selected to train a gesture recognition model using a deep learning framework PyTorch.

[0053] The gesture feature recognition model matches the gesture features, and identifies the type of the gesture. The type of the gesture includes gesture types such as lifting, pointing, holding and lifting.

[0054] The gesture feature recognition model classifies the gesture features, and identifies the action category of the gesture. The category of the gesture includes movement, rotation and scaling.

[0055] The palm is pushed forward to represent the forward movement of the object in the game, the palm is pulled backward to represent the backward movement of the object in the game, and the palm is moved left and right to represent the lateral movement of the object in the game. The mapping relationship between the translation distance of the object in the game and the translation distance of the hand is determined according to the set calibration distance to obtain the pixel value of the movement of the object.

[0056] The palm is lifted and rotated by an angle to represent the rotation of the object in the game. The mapping relationship between the rotation angle of the object in the game and the rotation angle of the hand is determined according to the set calibration angle to obtain the rotation angle of the object.

[0057] S6, the coordinates of the object in the game are converted according to the posture of the gesture, and the coordinate mapping of the object in the game is obtained.

[0058] According to the pixel value of the movement of the object and the rotation angle of the object, the coordinates and the rotation angle of the object in the game are transformed.

[0059] S1-S6 are cycled, and the coordinate mapping of the object is updated in real time in the game, so that the gesture of the user is consistent with the movement or rotation of the object in the game.

Claims

1. A gesture-based game object real-time manipulation system, characterized in that, The application relates to a gesture control system for a game, which comprises the following parts: a data acquisition module for acquiring real-time video data of a user; a data preprocessing module electrically connected with the data acquisition module, which is used for preprocessing the real-time video data; a gesture feature extraction module electrically connected with the data preprocessing module, which is used for extracting gesture features; a gesture feature recognition module electrically connected with the gesture feature extraction module, which is used for recognizing the action of a hand; a coordinate conversion module electrically connected with the gesture feature recognition module, which is used for converting the action of the hand into coordinates in a game.

2. The gesture-based game object real-time manipulation system of claim 1, wherein, The data preprocessing module decomposes the acquired real-time video data into video frame data, and preprocesses the video frame data through greying and binarization, and eliminates the video frame data not containing a hand.

3. The gesture-based game object real-time manipulation system of claim 2, wherein, The gesture feature extraction module extracts key points, contours and directions of a gesture from the preprocessed video frame data, and obtains gesture features.

4. The gesture-based game object real-time manipulation system of claim 3, wherein, The gesture feature recognition module is provided with a gesture feature recognition model, which is used for matching and classifying gesture features.

5. The gesture-based game object real-time manipulation system of claim 4, wherein, The gesture feature recognition model matches gesture features, and identifies the type of the gesture.

6. The gesture-based game object real-time manipulation system of claim 4, wherein, The gesture feature recognition model classifies gesture features, and identifies the action category of the gesture.

7. The gesture-based game object real-time manipulation system of claim 1, wherein, The application further comprises an initialization calibration module electrically connected with the coordinate conversion module, which is used for initializing and calibrating a gesture and an object in a game. 8.The gesture-based game object real-time manipulation system of claim 1, wherein, The application further comprises a circulation module electrically connected with the data acquisition module, the data preprocessing module, the gesture feature extraction module, the gesture feature recognition module and the coordinate conversion module, which is used for circulating and real-time updating the coordinates of the object in the game.

9. A method for real-time manipulation of a game object based on a gesture, the method comprising: receiving a gesture input from a user; determining a gesture type of the gesture input; and manipulating the game object based on the gesture type. The application comprises the following steps: acquiring real-time images of a user to obtain video data, preprocessing the video data to obtain preprocessed video data, and extracting gesture features in the preprocessed video data; recognizing the gesture features to obtain the gesture posture, and converting the coordinates of an object in a game according to the gesture posture to obtain the coordinate mapping of the object in the game.

10. The gesture-based real-time game object manipulation method of claim 9, wherein, Initialization calibration is performed before real-time control of a game object.

Images