Gesture-based game object real-time control system and method

Through real-time video data processing and deep learning network recognition gestures, precise control of game objects is achieved, solving the problem of insufficient accuracy and fluency in gesture recognition in the prior art, and improving game experience and interactivity.

CN118987601APending Publication Date: 2024-11-22SHANGHAI CHENGRONG NETWORK TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411257616.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, the method of using periodic shooting of multiple pictures for gesture recognition is insufficient in terms of accuracy, fluency and anti-interference ability, making it difficult to achieve efficient game object control.

Method used

A real-time control system for game objects based on gestures is designed, including data acquisition, data preprocessing, gesture feature extraction, gesture feature recognition and coordinate conversion modules. Through real-time video data acquisition, preprocessing, gesture feature extraction and recognition, use depth The learning network performs gesture matching and classification to realize the precise coordinate mapping of game objects.

Benefits of technology

It improves the accuracy and smoothness of game objects control, enhances the accuracy and anti-interference ability of gesture feature extraction, allows users to efficiently manipulate objects in the game without the need for traditional controllers, and improves the fun and interactiveness of the game.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118987601A_ABST
    Figure CN118987601A_ABST
Patent Text Reader

Abstract

The invention discloses a game object real-time control system and method based on gestures, and the system comprises a data collection module which is used for collecting the real-time video data of a user; the data preprocessing module is used for preprocessing the real-time video data; the gesture feature extraction module is used for extracting gesture features; the gesture feature recognition module is used for recognizing the action of the hand; and the coordinate conversion module is used for converting coordinates in the game according to the action change of the hand. According to the gesture-based game object real-time control system provided by the invention, videos are taken as input, different gestures are matched and classified through a deep learning network, gesture types and action types are identified, mapping is carried out in a way of coordinates in a game, coordinate transformation of objects in the game is obtained, and the real-time control of the objects in the game is realized. And the accuracy and smoothness of game object control are ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a real-time control system and method for game objects based on gestures. Background Art

[0002] Gesture posture estimation can achieve more efficient interactivity by sensing and identifying gesture movements and assigning specific meanings, thereby eliminating the need for physical connections to most electronic control devices and providing great convenience.

[0003] In the related art, for example, CN 107261498 A discloses a terminal-based gesture game control method and system, which captures a first picture of a scene, identifies the first picture to obtain a fixed object in the scene; periodically captures multiple pictures, and the order of capture is the distance between the hand and the fixed object in the multiple pictures; determines a specific first gesture based on the change trend of the distance, and controls the movement of the game character based on the first gesture. However, the method of periodically capturing multiple pictures for recognition in the related art is insufficient in terms of accuracy, fluidity, and anti-interference capability. Summary of the invention

[0004] The purpose of the present invention is to propose a real-time game object manipulation system and method based on gestures, so as to solve the problem that the prior art method of periodically taking multiple pictures for recognition is insufficient in accuracy, fluidity and anti-interference ability.

[0005] To this end, the present invention provides a game object real-time control system based on gestures, comprising:

[0006] A data acquisition module, wherein the data acquisition module is used to collect real-time video data of the user;

[0007] A data preprocessing module, the data preprocessing module is electrically connected to the data acquisition module, and the data preprocessing module is used to preprocess the real-time video data;

[0008] A gesture feature extraction module, the gesture feature extraction module is electrically connected to the data preprocessing module, and the gesture feature extraction module is used to extract gesture features;

[0009] A gesture feature recognition module, the gesture feature recognition module is electrically connected to the gesture feature extraction module, and the gesture feature recognition module is used to recognize hand movements;

[0010] A coordinate conversion module is electrically connected to the gesture feature recognition module, and is used for converting coordinates in the game according to hand movements.

[0011] Preferably, the data preprocessing module decomposes the collected real-time video data into video frame data, preprocesses the video frame data by graying and binarizing, and removes the video frame data that does not contain hands. Preprocessing the video frame data by graying and binarizing can highlight colors similar to human skin color, so as to better highlight the hand data information. Removing the video frame data that does not contain hands can avoid performance waste and improve the accuracy of data processing.

[0012] Preferably, the gesture feature extraction module extracts key points, contours and directions of gestures from the preprocessed video frame data to obtain gesture features.

[0013] Preferably, a gesture feature recognition model is provided in the gesture feature recognition module, and the gesture feature recognition model is used to match and classify gesture features.

[0014] Preferably, the gesture feature recognition model matches gesture features to identify the type of gesture.

[0015] Preferably, the gesture feature recognition model classifies gesture features and identifies the action category of the gesture.

[0016] Preferably, an initialization calibration module is further included, wherein the initialization calibration module is electrically connected to the coordinate conversion module, and the initialization calibration module is used to perform initialization calibration on gestures and objects in the game.

[0017] Preferably, it also includes a circulation module, which is electrically connected to the data acquisition module, the data preprocessing module, the gesture feature extraction module, the gesture feature recognition module and the coordinate conversion module, and is used to cyclically update the coordinates of objects in the game in real time.

[0018] A method for real-time manipulation of game objects based on gestures, comprising the following steps:

[0019] Collect the user's real-time image to obtain video data; pre-process the video data to obtain pre-processed video data; extract gesture features in the pre-processed video data; recognize the gesture features to obtain the posture of the corresponding gesture; convert the coordinates of the object in the game according to the posture of the gesture to obtain the coordinate mapping of the object in the game.

[0020] Preferably, an initialization calibration is performed before real-time manipulation of the game object.

[0021] Beneficial effects:

[0022] 1. The present invention provides a real-time game object manipulation system based on gestures, which takes video as input, matches and classifies different gestures through a deep learning network, identifies gesture types and action categories, and maps them in the game in the form of coordinates to obtain the coordinate transformation of objects in the game, thereby ensuring the accuracy and smoothness of game object manipulation.

[0023] 2. In the present invention, by preprocessing the video data, colors similar to human skin color are highlighted, so as to better highlight the hand data information and improve the accuracy and anti-interference ability of gesture feature extraction.

[0024] 3. The present invention allows users to control in-game objects without using a traditional controller, which increases the fun and interactivity of the game and enhances the user's sense of immersion and gaming experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a schematic diagram of the system structure of Example 1 of a real-time control system for game objects based on gestures in the present invention;

[0027] Figure 2 A flowchart of a method for using a gesture-based real-time game object control system according to Embodiment 1 of the present invention; DETAILED DESCRIPTION

[0028] The content of the present invention can be more easily understood by selecting the following detailed description of the preferred implementation method of the present invention and the embodiments included. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present invention belongs. When there is a conflict, the definition in this specification shall prevail.

[0029] Embodiment 1:

[0030] like Figure 1 A gesture-based real-time game object manipulation system is shown, comprising:

[0031] A data acquisition module, wherein the data acquisition module is used to collect real-time video data of the user;

[0032] A data preprocessing module, the data preprocessing module is electrically connected to the data acquisition module, and the data preprocessing module is used to preprocess the real-time video data;

[0033] A gesture feature extraction module, the gesture feature extraction module is electrically connected to the data preprocessing module, and the gesture feature extraction module is used to extract gesture features;

[0034] A gesture feature recognition module, the gesture feature recognition module is electrically connected to the gesture feature extraction module, and the gesture feature recognition module is used to recognize hand movements;

[0035] The coordinate conversion module is electrically connected to the gesture feature recognition module, and is used to convert the coordinates in the game according to the hand movements.

[0036] The data preprocessing module decomposes the collected real-time video data into video frame data, preprocesses the video frame data through grayscale and binarization, and removes the video frame data that does not contain hands. Preprocessing the video frame data through grayscale and binarization can highlight colors similar to human skin color, so as to better highlight the hand data information. Removing the video frame data that does not contain hands can avoid performance waste and improve the accuracy of data processing.

[0037] The gesture feature extraction module extracts the key points, contours and directions of the gesture from the preprocessed video frame data to obtain the gesture features.

[0038] The gesture feature recognition module is provided with a gesture feature recognition model, which is used to match and classify gesture features. The gesture feature recognition model uses a convolutional neural network (CNN) to train the gesture feature recognition model using the deep learning framework PyTorch through images or video frames of hand movements.

[0039] The gesture feature recognition model matches the gesture features and identifies the type of gesture. In this embodiment, the types of gestures include lifting, pointing, holding, lifting, and other gesture types.

[0040] The gesture feature recognition model classifies the gesture features and identifies the action category of the gesture. In this embodiment, the gesture category includes moving, rotating, and scaling.

[0041] It also includes an initialization calibration module, which is electrically connected to the coordinate conversion module and is used to perform initialization calibration on gestures and objects in the game.

[0042] It also includes a circulation module, which is electrically connected to the data acquisition module, the data preprocessing module, the gesture feature extraction module, the gesture feature recognition module and the coordinate conversion module. The circulation module is used to cyclically update the coordinates of the objects in the game in real time.

[0043] A method for real-time manipulation of game objects based on gestures, comprising the following steps:

[0044] S1. Perform initialization calibration before real-time manipulation of game objects.

[0045] Use a camera to capture the user's hand movements. The hand is translated a certain distance, and the mapping relationship between the translation distance of the object in the game and the translation distance of the hand is determined according to the set calibration distance. For example, in this embodiment, the hand is translated an arbitrary distance, and the calibration distance is 100 pixels, that is, when the hand moves this distance, the object in the game is translated 100 pixels.

[0046] Use a camera to capture the user's hand movements, and rotate the hand to a certain angle. Determine the mapping relationship between the rotation angle of the object in the game and the hand rotation angle based on the set calibration angle. For example, in this embodiment, the hand rotates at any angle, and the calibration angle is 45 degrees, which means that when the hand rotates this distance, the object in the game rotates 45 degrees.

[0047] S2. Collect the real-time images of the user to obtain video data; the video data includes images containing hands and images not containing hands.

[0048] S3, preprocessing the video data to obtain preprocessed video data;

[0049] Preprocessing includes decomposing the collected video data into video frames, using grayscale and binarization processing to delete video frames that do not contain hand data. Colors similar to human skin color are highlighted to better highlight hand data information, avoid the influence of environmental colors on hand recognition, and avoid performance waste.

[0050] S4, extracting gesture features from the preprocessed video data;

[0051] Extract the key points, contours, and direction features of gestures in preprocessed video data for gesture feature recognition.

[0052] S5, identifying the gesture features to obtain the posture of the corresponding gesture;

[0053] The gesture features are identified by the gesture feature recognition model, and the gesture features are matched and classified. In this embodiment, a large number of hand motion images and video frames of the mobile phone are used, and a convolutional neural network is selected to train a gesture recognition model using the deep learning framework PyTorch.

[0054] The gesture feature recognition model matches the gesture features and identifies the type of gesture, which includes lifting, pointing, holding, and lifting.

[0055] The gesture feature recognition model classifies gesture features and identifies the action category of the gesture, which includes movement, rotation, and scaling.

[0056] Pushing the palm forward means the game object moves forward, pulling the palm backward means the game object moves backward, and moving left and right means the game object moves horizontally. The mapping relationship between the game object translation distance and the hand translation distance is determined based on the set calibration distance to obtain the pixel value of the object movement.

[0057] The rotation of the object in the game is represented by the palm of your hand holding it up and rotating it. The rotation angle of the object in the game is obtained by determining the mapping relationship between the rotation angle of the object and the rotation angle of the hand according to the set calibration angle.

[0058] S6. Convert the coordinates of the object in the game according to the gesture to obtain the coordinate mapping of the object in the game.

[0059] The pixel value of the object's movement and the angle of the object's rotation are transformed into the coordinates and rotation angle of the object in the game.

[0060] S1-S6 are performed in a loop to update the coordinate mapping of objects in the game in real time, so that the user's gestures are consistent with the movement or rotation of objects in the game.

Claims

1. A real-time game object manipulation system based on gestures, characterized in that: include: A data acquisition module, wherein the data acquisition module is used to collect real-time video data of the user; A data preprocessing module, the data preprocessing module is electrically connected to the data acquisition module, and the data preprocessing module is used to preprocess the real-time video data; A gesture feature extraction module, the gesture feature extraction module is electrically connected to the data preprocessing module, and the gesture feature extraction module is used to extract gesture features; A gesture feature recognition module, the gesture feature recognition module is electrically connected to the gesture feature extraction module, and the gesture feature recognition module is used to recognize hand movements; A coordinate conversion module is electrically connected to the gesture feature recognition module, and is used for converting coordinates in the game according to hand movements.

2. A gesture-based real-time game object manipulation system according to claim 1, characterized in that: The data preprocessing module decomposes the collected real-time video data into video frame data, preprocesses the video frame data by graying and binarizing, and removes the video frame data that does not contain the hand.

3. The real-time game object manipulation system based on gestures according to claim 2, characterized in that: The gesture feature extraction module extracts key points, contours and directions of gestures from the preprocessed video frame data to obtain gesture features.

4. The real-time game object manipulation system based on gestures according to claim 3, characterized in that: The gesture feature recognition module is provided with a gesture feature recognition model, and the gesture feature recognition model is used to match and classify gesture features.

5. The real-time game object manipulation system based on gestures according to claim 4, characterized in that: The gesture feature recognition model matches the gesture features and recognizes the type of gesture.

6. The real-time game object manipulation system based on gestures according to claim 4, characterized in that: The gesture feature recognition model classifies gesture features and identifies the action category of the gesture.

7. The real-time game object manipulation system based on gestures according to claim 1, characterized in that: It also includes an initialization calibration module, which is electrically connected to the coordinate conversion module and is used to perform initialization calibration on gestures and objects in the game.

8. The real-time game object manipulation system based on gestures according to claim 1, characterized in that: It also includes a circulation module, which is electrically connected to the data acquisition module, the data preprocessing module, the gesture feature extraction module, the gesture feature recognition module and the coordinate conversion module, and is used to cyclically update the coordinates of objects in the game in real time.

9. A method for real-time manipulation of game objects based on gestures, characterized in that: The steps include: Collecting the user's real-time picture to obtain video data; preprocessing the video data to obtain preprocessed video data; extracting gesture features from the preprocessed video data; The gesture features are identified to obtain the posture of the corresponding gesture; the coordinates of the object in the game are converted according to the posture of the gesture to obtain the coordinate mapping of the object in the game.

10. The method for real-time control of game objects based on gestures according to claim 9, characterized in that: Perform initial calibration before real-time manipulation of game objects.

Citation Information

Patent Citations

  • Game controlling method and system of terminal based on gesture

    CN107261498A

Cited By

  • Gesture-based real-time game object control system and method

    WO2026051469A1