Mobile terminal motion sensing game interaction method and device, equipment and medium

Abstract

The invention relates to a mobile terminal motion sensing game interaction method and device, equipment and a medium, and belongs to the field of motion sensing control. The method comprises the steps that a real-time real space image is captured, in combination with an IMU unit, space posture sensing parameters are obtained, and the user posture state is obtained, and the user posture state comprises the user space position and the user real-time dynamic state; obtaining an operation stability index based on the IMU unit, constructing an operation stability evaluation model, and obtaining a sensitivity adjustment parameter through the operation stability index; and generating a control instruction based on the user pose state and the sensitivity adjustment parameter and feeding back the control instruction to the game terminal. According to the invention, immersive experience of the FPS game is realized and is not limited by peripherals and space sites, so that the game experience cost of the user is greatly reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of somatosensory control, and particularly relates to a method, device, equipment and medium for mobile terminal somatosensory game interaction. Background Art

[0002] Traditional FPS games mainly rely on keyboards, mice or gamepads for operation. Mobile FPS games mainly rely on clicking on the mobile phone screen for operation, which has problems such as complex operations, a huge difference from the real offline wargame experience, and insufficient immersion. With the development of somatosensory technology, there have emerged solutions for controlling game characters using somatosensory devices (such as omnidirectional treadmills). However, most of the existing technologies require external devices to assist in completing somatosensory control, and it is impossible to realize the action of controlling game characters in mobile FPS somatosensory control with only one mobile phone, and there are certain requirements for the size of the real game space. Such solutions require bearing expensive peripheral costs and are difficult to meet the requirements of lightweight use anytime and anywhere, and cannot really benefit the public. Summary of the Invention

[0003] To solve the above problems existing in the prior art, the present invention provides a method, device, equipment and medium for mobile terminal somatosensory game interaction.

[0004] The object of the present invention can be achieved by the following technical solutions: A method for mobile terminal somatosensory game interaction, the implementation of the mobile terminal somatosensory game interaction method includes the following steps: By capturing real-time real-space images and combining with the IMU unit, obtain spatial attitude perception parameters and obtain the user pose state, where the user pose state includes the user's spatial position and the user's real-time dynamics; Based on the IMU unit, obtain an operation stability index and construct an operation stability evaluation model, and obtain a sensitivity adjustment parameter through the operation stability index; Generate a control instruction based on the user pose state and the sensitivity adjustment parameter and feedback it to the game terminal.

[0005] Preferably, the acquisition of the user pose state includes: Execute an initialization instruction, which is used to establish a mapping relationship between the real space and the virtual environment and determine the standard initial pose, and the standard initial pose includes the initial spatial position and the initial azimuth angle; Continuously collect the spatial attitude perception parameters through the IMU unit, the IMU unit includes an accelerometer, a gyroscope and a magnetometer, the spatial attitude perception parameters include three-axis linear acceleration and three-axis angular velocity, and based on the spatial attitude perception parameters, obtain the user pose state on the basis of the standard initial pose; Continuously capture the real-time reality space image, and correct the standard initial pose based on the initial space image.

[0006] Preferably, the correction of the standard initial pose includes: After the user completes a single / multiple operations, return to the standard initial pose, define the user's pose at this time as the corrected pose, and the corrected pose includes a corrected position and a corrected azimuth angle; Capture the real-time reality space image, select N feature points at the same position in the real-time reality space image and the initial space image respectively for feature point cloud matching. When the reprojection error exceeds a preset threshold, trigger a correction instruction, and use the corrected pose to replace the standard initial pose in the next operation, and continue this step until the game ends.

[0007] Preferably, the acquisition of the sensitivity adjustment parameter includes: Set N sample points, cache the three-axis linear acceleration, the three-axis angular velocity, the user's pose inclination, the user's facing direction, and the user's height within a preset time range through a sliding window, and calculate the time-domain jitter index and the frequency-domain stability coefficient; Perform a fast Fourier transform on the x-axis component of the three-axis linear acceleration vector to obtain the acceleration spectrum X( f ), and obtain a two-dimensional feature vector based on the time-domain jitter index and the frequency-domain stability coefficient. The mathematical description of the two-dimensional feature vector is , where F is the two-dimensional feature vector, J t is the time-domain jitter index, f is the frequency; Based on the two-dimensional feature vector, obtain an operation stability decision function. The mathematical description of the operation stability decision function is , where C is the operation stability decision function, N is the number of sample points, is the weight coefficient of the i-th sample point, F i is the two-dimensional feature vector of the i-th sample point; Calculate the operation stability index according to the time-domain jitter index, the frequency-domain stability coefficient and the operation stability decision function, and construct the operation stability evaluation model; Calculate the sensitivity adjustment parameter based on the operation stability index.

[0008] Preferably, the calculation formula of the time-domain jitter index is , where J t is the time-domain jitter index, a k is the three-axis linear acceleration vector of the k-th sample point, ω k is the three-axis angular velocity vector of the k-th sample point, μ ais the mean value of the three-axis linear acceleration within the sliding window, μ ω is the mean value of the three-axis angular velocity within the sliding window, σ a is the standard deviation of the three-axis linear acceleration within the sliding window, σ ω is the standard deviation of the three-axis angular velocity within the sliding window, and the calculation formula of the frequency-domain stability coefficient is , where J f is the frequency-domain stability coefficient, a x is the x-axis component of the three-axis linear acceleration vector, ω y is the y-axis component of the three-axis angular velocity vector, and FFT is the fast Fourier transform.

[0009] Preferably, the calculation formula of the operation stability index is , where T is the operation stability index.

[0010] Preferably, the calculation formula of the sensitivity adjustment parameter is , where S(t) is the sensitivity adjustment parameter at time t, S base is the base sensitivity, is the maximum adjustment amplitude, T(t) is the operation stability index at time t, is the time constant.

[0011] A mobile terminal somatosensory game interaction device for executing the above-mentioned mobile terminal somatosensory game interaction method, including a pose perception module, a stability evaluation module, and a control instruction generation module: The pose perception module is used to capture real-time real-space images and combine with the IMU unit to obtain spatial pose perception parameters and obtain the user pose state, and the user pose state includes the user's spatial position and the user's real-time dynamics; The stability evaluation module is used to obtain the operation stability index based on the IMU unit and construct an operation stability evaluation model, and obtain the sensitivity adjustment parameter through the operation stability index; The control instruction generation module is used to generate control instructions based on the user pose state and the sensitivity adjustment parameter and feedback them to the game terminal.

[0012] An electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor realizes the above-mentioned mobile terminal somatosensory game interaction method when executing the program.

[0013] A storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned mobile terminal somatosensory game interaction method when executed by a computer processor.

[0014] The beneficial effects of the present invention are: (1) No peripheral mapping: By accurately capturing the user's posture and movements through the IMU unit and the rear camera, the real posture and movements of the player are mapped to the characters in the game, greatly reducing the cost of the user's gaming experience and being unrestricted by the actual space size.

[0015] (2) Immersive experience: Through posture and movement mapping, the FPS gaming experience on the mobile device is highly similar to the real wargame experience, allowing players to move while playing the game and making the game healthier. At the same time, it can also make the wargame movement generate rich immersive experiences similar to those of FPS, making the movement more interesting.

[0016] (3) Low latency: By adopting efficient filtering algorithms and communication protocols, it is possible to reduce the latency of data transmission and processing and improve the real-time performance of game control.

[0017] (4) Strong compatibility: It supports a variety of game terminals and device platforms and can meet the needs of different users.

[0018] (5) High flexibility: It supports adjusting the sensitivity through the user operation stability control, and can flexibly change the game settings according to the user's nervousness, improving the user's gaming experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with the accompanying drawings.

[0020] Figure 1 It is a flowchart of the steps of a mobile terminal somatosensory game interaction method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0021] To further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following will describe in detail the specific embodiments, structures, features and their effects of the present invention in conjunction with the accompanying drawings and preferred embodiments.

[0022] Working principle and usage process of the present invention: Please refer to Figure 1 , a mobile terminal somatosensory game interaction method, including: S1: By capturing real-time real-space images and combining with the IMU unit, obtaining spatial posture perception parameters and obtaining the user pose state, where the user pose state includes the user's spatial position and the user's real-time dynamics; S2: Based on the IMU unit, obtaining an operation stability index and constructing an operation stability evaluation model, and obtaining a sensitivity adjustment parameter through the operation stability index; S3: Generate a control instruction based on the user's pose state and the sensitivity adjustment parameter and feedback it to the game terminal. Example: When it is detected that the user moves forward one step, a forward control instruction is generated according to a preset proportional relationship, so that the character in the game moves forward a certain distance at a certain speed. When it is detected that the user continuously runs forward, a forward control instruction is generated according to a preset proportional relationship, so that the character in the game continuously runs forward at a certain speed.

[0023] In this embodiment, obtaining the spatial attitude perception parameter and getting the user's pose state can be specifically implemented through the following steps: S101: Execute an initialization instruction, which is used to establish a mapping relationship between the real space and the virtual environment and determine the standard initial pose. The standard initial pose includes the initial spatial position and the initial azimuth angle. That is, the space where the user is located is mapped to the virtual space by means of scanning, etc. The user stands upright and holds a tactical interaction device (such as a gun) in the space, and the mobile terminal is mounted on the tactical interaction device. At this time, the position and azimuth angle of the mobile terminal are the initial spatial position and the initial azimuth angle; S102: Continuously collect the spatial attitude perception parameters through the IMU unit and multi-source sensors. The IMU unit includes an accelerometer, a gyroscope, and a magnetometer. The accelerometer is used to measure the three-axis linear acceleration and attitude inclination. The gyroscope is used to measure the three-axis angular velocity. The magnetometer is used to measure the three-axis magnetic field intensity, provide the absolute direction (heading angle), and compensate for gyro drift. The spatial attitude perception parameters include the user's lateral acceleration, longitudinal acceleration, attitude inclination, facing direction, height, etc. Based on the spatial attitude perception parameters, the user's pose state is obtained through a filtering algorithm on the basis of the standard initial attitude. Common filtering algorithms include Kalman filtering, extended Kalman filtering, and particle filtering, etc. These algorithms can combine the motion model and observation data (i.e., spatial attitude perception parameters) to estimate the user's state (position, velocity, attitude, etc.). Example: Detect the dynamics of the user relative to the initial spatial position and the initial azimuth angle. When it is detected that the user has a forward linear acceleration and the linear acceleration value is greater than a preset threshold, it is determined that the user is running forward continuously; if the linear acceleration value is less than or equal to the preset threshold, it is determined that the user is moving forward, and the moving speed and distance can be freely set according to the linear acceleration value; when the user's height suddenly decreases and a downward linear acceleration is detected, the user has a downward longitudinal movement (such as squatting, lying prone, etc.), and the specific longitudinal movement can be determined through the decrease in the user's height and the magnitude of the linear acceleration value; when the user's height increases and an upward linear acceleration is detected, the user has an upward longitudinal movement (such as jumping, standing up, etc.); the three-axis angular velocity is used to track the user's rotation (such as the angle change when turning around, squatting and standing up, etc.); the user's attitude inclination can be used to infer the user's sideward movement, etc.; if it is detected that the user has a downward longitudinal movement and a forward linear acceleration at the same time, the user may be in a state of crawling forward, etc.; S103: Continuously capture the real-time real-space image through the rear camera of the mobile terminal during the game process, and correct the standard initial attitude based on the initial space image; S104: When it is detected that the user has abnormal data during the game process, such as accelerating forward unreasonably, etc., automatically reset the user's position and azimuth angle; S105: The user can assist in correcting the spatial information relationship between the mobile phone and the player by waving the mobile phone casually instead of the dot network scanning method.

[0024] In this embodiment, correcting the standard initial attitude can be specifically implemented through the following steps: S103-1: After the user completes a single / multiple operations, the user returns to the standard initial posture, and the posture of the user at this time is defined as the corrected posture. The corrected posture includes a corrected position and a corrected azimuth angle, that is, it can support the user to play games in a narrow space. After the user completes a single / multiple operations, the user needs to return to the origin (the standard initial posture). However, due to human errors, the position where the user actually returns may be deviated, resulting in a decrease in the accuracy of subsequent posture judgment. Therefore, it is necessary to correct the posture of the user after returning. S103-2: Capture the real-time real-space image, select N feature points at the same position in the real-time real-space image and the initial space image respectively for feature point cloud matching. When the reprojection error exceeds the preset threshold, trigger a correction instruction, and use the corrected posture to replace the standard initial posture in the next operation, and continue this step until the user completes the game.

[0025] In this embodiment, the sensitivity adjustment parameter is obtained through the operation stability index, and it can be specifically implemented through the following steps: S201: Set N sample points, cache the triaxial linear acceleration and the triaxial angular velocity within a preset time range through a sliding window, and calculate the time-domain jitter index and the frequency-domain stability coefficient. The calculation formula of the time-domain jitter index is , where J t is the time-domain jitter index, N is the number of sample points, a k is the triaxial linear acceleration vector of the kth sample point, ω k is the triaxial angular velocity vector of the kth sample point, μ a is the mean value of the triaxial linear acceleration within the sliding window, μ ω is the mean value of the triaxial angular velocity within the sliding window, σ a is the standard deviation of the triaxial linear acceleration within the sliding window, σ ω is the standard deviation of the triaxial angular velocity within the sliding window. The calculation formula of the frequency-domain stability coefficient is , where J f is the frequency-domain stability coefficient, a x is the x-axis component of the triaxial linear acceleration vector, ω y is the y-axis component of the triaxial angular velocity vector, and FFT is the fast Fourier transform; S202: Perform 512-point FFT on a x to obtain the acceleration spectrum X( f ), and obtain a two-dimensional feature vector based on the time-domain jitter index and the frequency-domain stability coefficient. The mathematical description of the two-dimensional feature vector is , where F is the two-dimensional feature vector, f is the frequency; S203: Obtain an operation stability decision function based on the two-dimensional feature vector. The mathematical description of the operation stability decision function is , where C is the operation stability decision function, is the weight coefficient of the i-th sample point, and F i is the two-dimensional feature vector of the i-th sample point. When the value of C is 1, it indicates that the user moves normally. When the value of C is -1, the operation is unstable due to the user's nervous jitter; S204: Calculate the operation stability index based on the time-domain jitter index, the frequency-domain stability coefficient, and the operation stability decision function, and construct the operation stability evaluation model. The calculation formula is , where T is the operation stability index; S205: Calculate the sensitivity adjustment parameter based on the operation stability index. The calculation formula is , where S(t) is the sensitivity adjustment parameter at time t, and S base is the base sensitivity, is the maximum adjustment amplitude, with a default value of 0.4 to ensure that the minimum sensitivity is not lower than 60%. T(t) is the operation stability index at time t, is the time constant, with a default value of 0.6.

[0026] A mobile terminal somatosensory game interaction device includes a pose perception module, a stability evaluation module, and a control instruction generation module: The pose perception module is used to capture real-time real-space images and combine with the IMU unit to obtain spatial pose perception parameters and obtain the user's pose state. The user's pose state includes the user's spatial position and the user's real-time dynamics; The stability evaluation module is used to obtain the operation stability index based on the IMU unit and construct an operation stability evaluation model, and obtain the sensitivity adjustment parameter through the operation stability index; The control instruction generation module is used to generate control instructions based on the user's pose state and the sensitivity adjustment parameter and feedback them to the game terminal.

[0027] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable media can be computer-readable signal media or computer-readable storage media. The computer-readable storage media can, for example, but not be limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, the computer-readable storage media can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, device, or component.

[0028] The computer-readable signal media can include data signals propagated in a baseband or as part of a carrier wave, in which computer-readable program codes are carried. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal media can also be any computer-readable medium other than the computer-readable storage media, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, device, or component.

[0029] The program codes contained on the computer-readable media can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above. The computer program codes for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages - such as Java, Smalltalk, C++, and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program codes can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0030] The above are only the preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Although the present invention has been disclosed as above with the preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments by using the above-disclosed technical content within the scope of the technical solution of the present invention. However, as long as it does not depart from the content of the technical solution of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for interactive mobile terminal somatosensory games, characterized in that, The implementation of the mobile terminal somatosensory game interaction method includes the following steps: By capturing real-time real-space images and combining with the IMU unit, spatial attitude perception parameters are obtained and the user pose state is obtained. The user pose state includes the user's spatial position and the user's real-time dynamics; Based on the IMU unit, an operation stability index is obtained and an operation stability evaluation model is constructed, and a sensitivity adjustment parameter is obtained through the operation stability index; Based on the user pose state and the sensitivity adjustment parameter, a control instruction is generated and fed back to the game terminal.

2. The method for mobile terminal somatosensory game interaction according to claim 1, wherein The acquisition of the user pose state includes: Execute an initialization instruction, which is used to establish a mapping relationship between the real space and the virtual environment and determine the standard initial pose. The standard initial pose includes the initial spatial position and the initial azimuth angle; Continuously collect the spatial attitude perception parameters through the IMU unit. The IMU unit includes an accelerometer, a gyroscope, and a magnetometer. The spatial attitude perception parameters include three-axis linear acceleration, three-axis angular velocity, user pose inclination, user facing direction, and user height. Based on the spatial attitude perception parameters, the user pose state is obtained on the basis of the standard initial pose; Continuously capture the real-time real-space images, and correct the standard initial pose based on the initial space image.

3. The method for mobile terminal somatosensory game interaction according to claim 2, wherein, The correction of the standard initial pose includes: After the user completes a single / multiple operations, return to the standard initial pose, and define the user's pose at this time as the corrected pose. The corrected pose includes the corrected position and the corrected azimuth angle; Capture the real-time real-space images, select N feature points at the same position in the real-time real-space image and the initial space image for feature point cloud matching. When the reprojection error exceeds the preset threshold, trigger a correction instruction, and use the corrected pose to replace the standard initial pose in the next operation. Continue this step until the game ends.

4. The method for mobile terminal somatosensory game interaction according to claim 3, characterized in that, The acquisition of the sensitivity adjustment parameter includes: Set N sample points, cache the three-axis linear acceleration and the three-axis angular velocity within a preset time range through a sliding window, and calculate the time-domain jitter index and the frequency-domain stability coefficient; Performing a fast Fourier transform on the x-axis component of the three-axis linear acceleration vector to obtain the acceleration spectrum X( f ), and obtaining a two-dimensional feature vector based on the time-domain jitter index and the frequency-domain stability coefficient. The mathematical description of the two-dimensional feature vector is , where F is the two-dimensional feature vector, J t is the time-domain jitter index, f is the frequency; An operation stability decision function is obtained based on the two-dimensional feature vector, and the mathematical description of the operation stability decision function is , where C is the operation stability decision function, N is the number of sample points, is the weight coefficient of the i-th sample point, and F i is the two-dimensional feature vector of the i-th sample point; Calculate the operation stability index according to the time-domain jitter index, the frequency-domain stability coefficient, and the operation stability decision function, and construct the operation stability evaluation model; Calculate the sensitivity adjustment parameter based on the operation stability index.

5. The method for mobile terminal somatosensory game interaction according to claim 4, wherein The calculation formula for the time-domain jitter index is , where J t is the time-domain jitter index, a k is the three-axis linear acceleration vector of the k-th sample point, ω k is the three-axis angular velocity vector of the k-th sample point, μ a is the mean value of the three-axis linear acceleration within the sliding window, μ ω is the mean value of the three-axis angular velocity within the sliding window, σ a is the standard deviation of the three-axis linear acceleration within the sliding window, σ ω is the standard deviation of the three-axis angular velocity within the sliding window. The calculation formula for the frequency-domain stability coefficient is , where J f is the frequency-domain stability coefficient, a x is the x-axis component of the three-axis linear acceleration vector, ω y is the y-axis component of the three-axis angular velocity vector, and FFT is the fast Fourier transform.

6. The method for mobile terminal somatosensory game interaction according to claim 5, wherein, The calculation formula for the operation stability index is , where T is the operation stability index.

7. The method for mobile terminal somatosensory game interaction according to claim 6, wherein The calculation formula for the sensitivity adjustment parameter is , where S(t) is the sensitivity adjustment parameter at time t, and S base is the basic sensitivity, is the maximum adjustment range, T(t) is the operation stability index at time t, is the time constant.

8. A mobile terminal somatosensory game interaction device, characterized in that The device is applied to the mobile terminal somatosensory game interaction method according to any one of claims 1-7, and includes a pose perception module, a stability evaluation module, and a control instruction generation module: The pose perception module is used to obtain spatial attitude perception parameters and obtain the user pose state by capturing real-time real-space images and combining with the IMU unit. The user pose state includes the user's spatial position and the user's real-time dynamics; The stability evaluation module is used to obtain an operation stability index based on the IMU unit, construct an operation stability evaluation model, and obtain a sensitivity adjustment parameter through the operation stability index; The control instruction generation module is configured to generate control instructions based on the user pose state and the sensitivity adjustment parameter and feedback them to the game terminal.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the mobile terminal somatosensory game interaction method according to any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, When executed by a computer processor, the computer-executable instructions are used to execute the mobile terminal somatosensory game interaction method according to any one of claims 1-7.

Images