Novel game control system

Abstract

The invention belongs to the field of electronic games, and relates to a novel electronic game control system. A camera is used for tracking the gesture of the hand of a player in real time, the gesture of the two hands of the game player is recognized by using the artificial intelligence technology, especially the finger landmark (finger key point) recognition technology, different gestures of the player are converted into different control instructions for the game, and the game player can control the game according to the control instructions. A novel game control mode and an implementation system thereof are designed. The device specifically comprises a finger landmark recognizer, a filter, a gesture recognizer and a sound collector. Through the connection between the mobile phone and the game host, the novel game control system can be expanded and applied to electronic games of a traditional television end.

Description

Technical Field

[0001] The invention belongs to the field of electronic games and relates to a novel electronic game control system. Background Art

[0002] Current electronic games, such as shooting games running on mobile phones, require users to use the touch screen of the mobile phone to operate. Mobile games using this operation method will display two virtual control buttons on the mobile phone screen, which control the direction of movement and shooting respectively. The user uses one hand to control the direction of travel by moving up, down, left, and right or clicking on the touch screen, and the other hand moves or clicks on the touch screen to control aiming and shooting. Electronic games running on TVs require users to use game remote controls for control. These operation methods are not intuitive, the learning cost of gamers is relatively high, and they may need to purchase additional physical equipment. Summary of the invention

[0003] The present invention uses a camera to track the player's hand gestures in real time, uses artificial intelligence technology, especially finger landmark (finger key point) recognition technology, recognizes the gestures of the game player's hands, converts different gestures of the player into different control instructions for the game, and designs a new game control method and its implementation system. Specifically, it includes: finger landmark identifier, filter, gesture identifier, and sound collector. According to the needs of the game, the user can control with one hand or two hands.

[0004] For mobile games, the specific workflow is shown in Figure 2. Finger landmark recognition technology is used to identify the three-dimensional coordinates of the player's hand key points, and then a filter algorithm is used to de-jitter the three-dimensional coordinates output by the finger landmark recognizer. A gesture recognizer is used to perform pattern recognition on the filtered three-dimensional coordinates of the finger key points to recognize the user's gesture, and the coordinates of the intersection of the extension line of the user's finger and the mobile phone screen are calculated as the target point. A sound collector is used to collect the voice commands issued by the user to trigger shooting actions or other actions on the target point.

[0005] Finger landmark recognition can use existing open source solutions, such as MediaPipe. MediaPipe first divides the continuous video signal collected from the camera into pictures, each of which is called a frame. It then analyzes the hand posture contained in each frame, marks the landmarks (key points of the hand) of each hand in each frame in order, and outputs an array from landmarks[0] to landmarks

[20] , where landmarks[i] represents the [i]th element in the array. For example, the wrist is point 0, which is the 0th element of the array, and the output is landmarks[0]; the tip of the thumb is point 4, which is output as landmarks[4]; the tip of the index finger is point 8, which is output as landmarks[8]. Each key point is marked as a normalized three-dimensional coordinate, where the x and y axis values ​​are the coordinates of the coordinate system established with the upper left corner of the current frame as the origin. The x and y axis coordinates of the upper left corner of the picture are (0, 0), and the x and y axis coordinates of the lower right corner of the picture are (1, 1). The x and y axis coordinates of the remaining points are between 0 and 1, so they are called normalized coordinates. The zero point of the z axis is the wrist, and the z axis coordinate value of the camera plane is -1. If the distance from a key point to the camera plane is greater than the distance from the zero point to the camera plane, the z axis value is positive, otherwise it is negative and between -1 and 0. In this way, the three-dimensional coordinates of the i-th landmark of the hand can be expressed as landmarks[i].x, landmarks[i].y, landmarks[i].z, which represent the coordinate values ​​of the x axis, y axis, and z axis of this landmark respectively. Therefore, for each frame of the video signal, MediaPipe outputs a 21 times 3 two-dimensional array for each hand. MediaPipe can be set to support one-handed working mode and two-handed working mode. In the two-handed working mode, when there are two hands in the picture, two 21 times 3 two-dimensional arrays are output. The coordinates of the key points of each hand in a continuous video signal are a series of 21 times 3 two-dimensional arrays.

[0006] Finger landmark recognition systems, such as MediaPipe, output 3D coordinates with jitter. Because the key point features of the hand are not as easy to distinguish as the key points of the face, the recognition software often outputs very jittery results. At this time, a filter needs to be used to filter the results to remove the jitter. Typical filter algorithms, such as one-euro, use min_cutoff, beta, derivative_cutoff and frequency as parameters. When the finger is static, the parameter min_cutoff controls the smoothness of the signal after filtering. When the finger moves, beta controls the response speed of the filtered signal. Derivative_cutoff is the cutoff frequency of the filter, and frequency is the sampling rate, that is, the number of frames per second in the video. When using MediaPipe to mark hand key points on a 720P resolution video image, the corresponding one-euro filter parameters are: min_cutoff is 1.0, beta is 0.8, derivative_cutoff is 1.2, and the typical sampling rate frequency is 30. This parameter setting allows the one-euro filter to have a significant smoothing effect on large jitters in the coordinates of the key points of the hand. In practical applications, the filter can be an independent software module or integrated into a finger landmark identifier, such as MediaPipe. It should be noted that the above filter parameters are the output results of MediaPipe's analysis of 720P resolution video images. If video images of other resolutions are used, the filter parameters need to be adjusted.

[0007] The three-dimensional coordinates of the finger landmarks after filtering are recognized by the gesture recognizer, and the intersection of the finger extension line and the mobile phone screen is calculated. For example, for shooting games, the most familiar gesture is that the index finger points to a target on the screen, the thumb is raised, and the middle finger, ring finger and pinky finger are curled together, as shown in the gesture in Figure 5(a). The gesture recognizer determines the gesture based on the three-dimensional coordinates of each finger key point. For example, to recognize the gesture in Figure 5(a), the vectors of each finger are first calculated. For example, the vector V_thumb of the thumb can be expressed as: ((landmarks[4].x-landmarks[2].x), (landmarks[4].y-landmarks[2].y), (landmarks[4].z-landmarks[2].z)), where landmarks[4] is the tip of the thumb and landmarks[2] is the base of the thumb. The above formula can be abbreviated as (V_thumb.x, V_thumb.y, V_thumb.z), that is, V_thumb.x, V_thumb.y, V_thumb.z correspond to the three formulas in the brackets above: (landmarks[4].x-landmarks[2].x), (landmarks[4].y-landmarks[2].y), and (landmarks[4].z-landmarks[2].z). The vector V_index of the index finger can be expressed as: ((landmarks[8].x-landmarks[5].x), (landmarks[8].y-landmarks[5].y), (landmarks[8].z-landmarks[5].z)), where landmarks[8] is the tip of the index finger and landmarks[5] is the base of the index finger, abbreviated as (V_index.x, V_index.y, V_index.z). Then, based on the vector of the index finger, determine whether the index finger is pointing to the screen. The calculation formula is: (V_index.z / sqrt(V_index.x * V_index.x + V_index.y * V_index.y + V_index.z * V_index.z))<ω. In the formula, sqrt is the square root operation, and ω is a simplified calculation parameter, which can be -0.6. If the inequality holds, it is determined that the index finger is pointing to the screen. After that, it is necessary to calculate the angle between the thumb and the index finger.The angle between two vectors V1 and V2 is calculated as follows: acos(dot / (mag1*mag2)) * 180 / π, where acos is the function for calculating the arccosine value, dot is calculated as follows: (V1.x * V2.x + V1.y * V2.y + V1.z * V2.z), mag1 is calculated as follows: sqrt(V1.x *V1.x + V1.y * V1.y + V1.z * V1.z), and mag2 is calculated as follows: sqrt(V2.x * V2.x + V2.y *V2.y + V2.z * V2.z). Substituting the thumb and index finger vectors into the above formula, the angle between the two can be calculated. In practical applications, if the angle between the thumb and index finger vectors calculated by the above formula is greater than 45 degrees, and the middle finger, ring finger, and little finger are not pointing to the screen, the gesture in Figure 5(a) can be considered to be established. The gesture recognizer calculates the intersection of the index finger extension line and the mobile phone screen, i.e. the projection point (, according to the three-dimensional coordinates of each key point of the index finger. Figure 4The two-dimensional coordinates of the projectedPoint in the figure are projectedPoint.x and projectedPoint.y. The calculation formula is shown in Figure 4: projectedPoint.x = landmarks[5].x - (dX / dZ) * scaleFactor, where dX = landmarks[8].x - landmarks[5]; dZ = landmarks[8].z – landmarks[5].z; landmarks[8] and landmarks[5] are the tip and base of the index finger, respectively. Similarly, the calculation formula for the y coordinate projectedPoint.y of the intersection of the finger extension line and the phone screen is landmarks[5].y - (dY / dZ) * scaleFactor, where dY = landmarks[8].y – landmarks[5].y. In practical applications, in order to further reduce the impact of jitter and improve computational efficiency, the calculation of dX and dY can introduce other key points on the index finger. For example, a formula for calculating dX can be dX = α * (landmarks[8].x - landmarks[5].x + landmarks[7].x - landmarks[5].x), and dY can be calculated as dY = γ * (landmarks[8].y - landmarks[5].y + landmarks[7].y - landmarks[5].y), where α and γ are parameters used to simplify the calculation and can be values ​​between 0.5 and 1.2. In addition to using the coordinates of the index finger base landmarks[5] and fingertip landmarks[8], this calculation method also introduces the coordinates of the second joint point of the index finger, that is, the coordinates of landmarks[7] in the array output by MediaPipe, which helps to eliminate the adverse effects of coordinate jitter of a single key point on the calculation results. dZ can be replaced by the distance of key points on the thumb to eliminate the influence of jitter on the z-axis, for example, dZ = θ *(landmarks[4].y -landmarks[2].y) + (landmarks[4].x - landmarks[2].x), which is the sum of the x-axis and y-axis distances from the thumb tip landmark[4] to the thumb base landmark[2] multiplied by the simplified calculation parameter θ, where θ can take a value between 0.5 and 1.5.In actual applications, the coordinates of the projection point can also be restricted. For example, if the calculated value of projectedPoint.x or projectedPoint.y is less than 0 or greater than 1, its value is directly set to 0 or 1, indicating that it reaches the edge of the image. When the image is displayed in full-screen mode, the edge of the image also represents the edge of the mobile phone screen. In actual applications, the projection point can be highlighted on the screen, such as a red dot. When the red dot stays somewhere on the edge of the screen, the player can clearly know that the direction of his finger has deviated from the screen and should move in the opposite direction. In addition to calculating the coordinates of the projection point of the finger extension line on the screen, the gesture recognizer can also perform different operations based on the recognized gesture. For example, the gesture displayed by the player is a thumb up, with the index finger and middle finger pointing to the screen at the same time. At this time, the player's weapon can be switched to a double-barreled shotgun or other weapons for aiming, as shown in Figure 5(b). The gesture displayed by the player is the index finger, middle finger, ring finger, and little finger pointing to the screen at the same time, indicating moving forward. The gesture displayed by the player is a fist, indicating stopping moving forward, as shown in Figure 5(f) and Figure 5(g). For example, in an airplane or tank control game, airplanes and tanks are controlled using a joystick. If the player's gesture is a thumb up and the other four fingers are curled together, it can indicate that the joystick is vertical, which means that the airplane or tank should go straight, as shown in Figure 5(c); the player's thumb is bent, as shown in Figure 5(d), indicating shooting; the gesture and the thumb rotated to the left or right indicate that the joystick turns to the left or right, as shown in Figure 5(e).

[0008] The sound collector collects the sounds made by the player and can use very simple logic to trigger shooting or other actions, such as triggering an action when the volume is above a certain threshold. For example, the user can simply make a "Bang" sound, and if the volume is above the threshold, the shooting is triggered. The sound collector can also use voice recognition technology to recognize the player's voice commands, such as "forward", "turn left", "shoot", etc. to complete the triggering of the action.

[0009] Finger landmark identifiers, such as MediaPipe, must accurately mark the coordinates of key finger points. There are certain requirements for light, the angle between the finger and the camera, and the distance. For example, when the phone is placed vertically, the jitter of MediaPipe finger landmark recognition will be greater. This is because the feature points of the finger are easier to distinguish when the camera is facing the palm, while when the camera is on the side of the palm, MediaPipe's ability to recognize the feature will decrease. In order to allow players to have a better gaming experience, the present invention distinguishes whether the player's main control hand (such as the hand responsible for aiming, which requires accurate calculation of coordinates) is the left hand or the right hand, and prompts the user to turn the phone 90 degrees to the right (if the user's main control hand is the left hand), as shown in Figure 1(b); or turn the phone 90 degrees to the left (if the user's main control hand is the right hand), as shown in Figure 1(a). The horizontal placement of the phone is consistent with the horizontal screen display of mainstream games on the one hand, and on the other hand, the phone camera is used to shoot at the angle facing the palm as much as possible, which greatly improves the accuracy of finger landmark recognition. In order to help players quickly complete the settings and find the best position of the hand relative to the phone, an intuitive user guide needs to be provided. The present invention uses the position of the calibration point of the player's hand to prompt the user to move the hand to the left, right, forward or backward relative to the mobile phone; it can also prompt the user to tilt the mobile phone forward or backward. For example, if the position of the fingertip of the thumb is used as the calibration point, for right-handed players, when the mobile phone is placed horizontally with a 90-degree left tilt, when the normalized x-axis coordinate of the calibration point on the plane of the mobile phone screen is greater than 0.53, the user is prompted to move the hand to the left, and when the normalized x-axis coordinate of the calibration point on the plane of the mobile phone screen is less than 0.47, the user is prompted to move the hand to the right. For right-handed players, when the normalized y-axis coordinate of the calibration point on the plane of the mobile phone screen is greater than 0.47, the user is prompted to tilt the mobile phone forward. For left-handed players, the normalized coordinates of the above axes should be corrected accordingly, so as to provide users with an intuitive and fast guide. It should be pointed out that the specific values ​​of the above calibration point coordinates depend on the setting of the coordinate system. For example, the above calibration point coordinates are the coordinates calculated based on the coordinate system with the upper left corner of the screen as the origin when the mobile phone is placed in a vertical direction. When a different coordinate system is used, the above coordinates need to be corrected. When the finger landmark identifier is set to support a single-handed working mode, if two hand images appear in the video signal collected by the camera, the present invention prompts the user to move the other hand out of the video collection area of ​​the camera through the main control hand selected by the user.

[0010] The new game control method designed by the present invention can be run as an independent software module, or integrated into other electronic games through the application programming interface API, so that other electronic games have similar control capabilities under the condition that the complexity and cost are controllable. The mobile phone running the new game control system implemented by the present invention can use Bluetooth, Wi-Fi or other connection methods to connect to the traditional electronic game console, and simulate the functions of the traditional electronic game remote control by implementing the communication protocol between the traditional electronic game remote control and the game console, converting the gesture into the remote control button signal and sending it to the game console. It can also convert the position of the player's finger projection point on the mobile phone into the button signal of the remote control and send it to the game console, so that the traditional electronic game displayed on the TV end can also be controlled by the player in a more intuitive way, as shown in Figure 3.

[0011] It should be pointed out that the key point coordinate representation method, key point coordinates, filter algorithms and parameters, gesture recognition calculation formula, the calculation formula of the intersection of the finger extension line and the mobile phone screen and the calculation method of each parameter in the formula, and the selection of hand calibration points used in this application are all examples to prove the feasibility of the present invention, and are not conditions that limit the application of the present invention. In other implementations, the use of different hand landmark recognition technologies, or the use of different key point coordinate representation methods, or the use of other filter algorithms, or the introduction of different hand key point coordinates for calculation in order to eliminate jitter, or the use of different methods to calculate the parameters in the formula, or the use of different simplified calculation coefficients, or the selection of different hand key points as calibration points to generate help or prompts for players can all be considered extensions of the present invention.

[0012] Unless otherwise specified, landmarks and key points in this application have the same meaning and can be interchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is an example of prompting the player to rotate the phone based on the player's primary control hand. If the player is right-handed, the prompt is to rotate the phone to the left; if the player is left-handed, the prompt is to rotate the phone to the right.

[0014] Figure 2 It is the main workflow of the new game control system.

[0015] Figure 3 This is an example diagram of a mobile phone running a new game control system and a traditional game console being connected via wired or wireless connection to simulate a traditional game control handle.

[0016] Figure 4 It is a schematic diagram and formula for calculating the intersection of the extended line of the finger and the mobile phone screen.

[0017] Figure 5It is a new set of game control gestures, including six side views of gestures and one front view of gesture.

[0018] Figure 6 It is the basic component and interaction method of the new game control system.

Claims

1. A new mobile game control system, including a camera, a finger landmark recognizer, a filter, a gesture recognizer, and a sound collector.

2. A new type of TV game control system, including a mobile phone running the new game control system, a wireless or wired connection, a game console, and a TV.

3. A new workflow of a game control system. It includes using a finger landmark identifier to identify the three-dimensional coordinates of the key points of the hand; using a filter to de-jitter the three-dimensional coordinates; Use the gesture recognizer to calculate the coordinates and then classify the gestures, and calculate the intersection of the finger extension line and the screen; Use the sound collector to collect the player's voice commands to trigger actions.

4. A method for calculating the intersection of the extended line of a finger and the screen, including a calculation method used to eliminate the influence of jitter and reduce the calculation complexity.

5. A set of new game control gestures, including aiming gestures, aiming gestures after changing weapons, forward and stop gestures, joystick driving gestures, joystick driving and shooting gestures, etc.

6. A new user prompt and guidance method. Including prompting the user to move the hand left, right, forward, or backward relative to the phone based on the coordinates of the calibration point of the player's hand, and prompting the user to tilt the phone forward or backward.

Images