A gaming experience measurement method based on PPG signals

Abstract

The present invention relates to a method for measuring gaming experience based on PPG signals, which can detect a player's cognitive load in real time and with minimal intrusion during gameplay. The method includes: stimulating different levels of cognitive load through a classic n-back task similar to instantaneous memory, while simultaneously recording the vascular volume pulse wave signal (PPG(t)) and the corresponding subjective evaluation value of cognitive load (C(t)); establishing, training, and verifying a cognitive load prediction model based on an LSTM neural network, and evaluating the model's prediction accuracy based on relevant indicators; and implementing a strategy for controlling the game's difficulty based on the player's cognitive load experience during gameplay.

Description

Technical Field

[0001] The present invention relates to the field of game technology, and in particular to a game experience measurement method based on physiological signals. Background Art

[0002] Currently, the measurement of gaming experience is mostly done in the form of questionnaires. However, this method is a retrospective measurement and cannot restore the experience during the game in real time, nor can it obtain the user experience during the game. It is also difficult to improve the design of game details in this way.

[0003] Game experience measurement based on physiological signals can obtain user data of game players in real time, but most physiological signals are highly invasive, and the process of collecting physiological signals has already affected the user's feelings, so it cannot accurately reflect the experience during the game.

[0004] For example, in the game "NEVERMIND", players need to wear a Bluetooth bracelet with a heart rate detection function to play the game. This additional equipment will make players pay attention to the bracelet itself and cannot concentrate on the game. There is currently no good solution for how to collect physiological signals without leaving any trace. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of existing methods and provide a real-time, objective and accurate measurement method and device for the gaming experience of gamers during the game process based on PPG signals.

[0006] A method for measuring gaming experience based on PPG signals, characterized by comprising the following steps:

[0007] Step 1: Organize human-computer interaction experiments on game players, collect relevant physiological data and subjective evaluation data, and form a database;

[0008] Step 2: Establish, train, and verify a cognitive load prediction model based on PPG signals;

[0009] Step 3: Establish a control strategy based on the player's cognitive load experience during the game;

[0010] Specifically:

[0011] Step 1: Organize cognitive load physiological experiments and form a database;

[0012] The classic n-back task, similar to short-term memory, was used to stimulate different levels of cognitive load. During the experiment, subjects received a series of stimuli, which could be sounds, numbers, images, etc., and were required to indicate whether the current stimulus was the same as the stimulus one step earlier. This allowed them to make a decision: if the stimulus was the same, they pressed button 1; if it was not, they pressed button 2.

[0013] During the entire physiological experiment, the subject's vascular volume pulse wave signal PPG(t) and the corresponding subjective evaluation value of cognitive load C(t) were collected. The subjective cognitive load evaluation value was set on a scale of 1-10, with the subject selecting one. The final subjective evaluation value C(t) = the selected level / 10. Experimental data from at least 100 different subjects were required.

[0014] Step 2: Establish, train, and verify the cognitive load prediction model based on LSTM neural network;

[0015] Data preprocessing involves first segmenting the vascular volume pulse wave signal (PPG(t)) based on the C(t) value. C(t) ≤ 0.2 is classified as low cognitive load, 0.3 ≤ C(t) ≤ 0.7 as moderate cognitive load, and C(t) ≥ 0.8 as high cognitive load. After classification, the first-order derivative (PPG'(t)) and second-order derivative (PPG"(t)) of PPG(t) are calculated for each category.

[0016] Establish a cognitive load prediction model based on an LSTM neural network. Based on database data, use the original signal, first-order derivative, and second-order derivative of the vascular volume pulse wave as the input of the LSTM neural network-based cognitive load prediction model. Use moderate, too low, and too high cognitive load as the output of the LSTM neural network-based cognitive load prediction model. Train the LSTM neural network-based cognitive load prediction model.

[0017] The AUC was calculated using ten-fold cross-validation. When AUC > 0.7, the LSTM neural network-based cognitive load prediction model was considered accurate. The AUC is the area under the ROC curve of the prediction result. The ROC is a two-dimensional curve whose horizontal and vertical coordinates are both defined in the range [0, 1]. Its horizontal and vertical coordinates are the FPR false positive rate and the TPR true positive rate, respectively. These two components are calculated using the following formulas:

[0018]

[0019] Taking binary classification as an example, its meanings are: the number of true positive samples divided by the number of all true positive samples and the number of false positive samples divided by the number of all true negative samples;

[0020] Step 3: Establish a control strategy based on the player's cognitive load experience during the game;

[0021] When the player's cognitive load shows that the challenge of this game is too easy for the player to gain a sense of accomplishment, the system will increase the difficulty of the current game; conversely, when the player's cognitive load is too high, it indicates that the player's ability is not up to the current game difficulty. The system will appropriately reduce the game difficulty so that the player can regain a sense of accomplishment in the next challenge.

[0022] The present invention uses the real-time acquired PPG signal and LSTM neural network model to make real-time predictions on the cognitive load of game players during the game. When the prediction result shows that it is lower than the set threshold, the game process is recorded, which can provide great help in the improvement design of game details.

[0023] Specifically, this can be reflected in further enhancing the interactivity between players and games: in the past, players might have needed to express their current state through specific actions or text. However, if human physiological characteristics can be processed and used as game feedback, players' emotions can be expressed naturally. The game can present different scenes and interfaces to players according to specified programs. This makes the interaction between game and player a two-way feedback loop, greatly improving the one-way nature of previous games, which only unilaterally conveyed emotions to players. Furthermore, due to the high degree of component integration, players will not feel disconnected from the gameplay process due to cognitive load measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The overall flow chart of the measurement method of the present invention is

[0025] Figure 2 Input diagram for the LSTM model structure diagram

[0026] Figure 3 Schematic diagram of the connection of the lower computer hardware device DETAILED DESCRIPTION

[0027] First, you need to combine Figure 3 Detailed description of the process of collecting, processing, and sending the player's PPG signal to the host computer:

[0028] Step 1: Collect PPG signals

[0029] Regarding the method of collecting PPG signals, the present invention uses a non-invasive fingertip infrared pulse sensor, which consists of a light-emitting diode and an optical sensor that measures the return signal of the light-emitting diode. Its major advantage is that it can return a complete PPG signal by simply pressing lightly on the sensor, and at the same time, it will not generate any other noise, light, etc. to interfere with the player's gaming process.

[0030] The main difficulty in measuring PPG signals is the interference of motion artifacts and ambient light on the signal:

[0031] DC errors caused by sunlight are relatively easy to remove, but fluorescent tubes or the light emitted by the screen during gaming may cause AC errors. For AC errors, a separate sensor is used to collect ambient light (while ensuring that the distance between the sensor and the PPG sensor is very close to prevent the difference in their positions from losing the role of the ambient light sensor as a control group). It can then be used to subtract it from the PPG signal to obtain a pure PPG signal.

[0032] Motion artifacts can be collected by placing the sensor in a position where the fingers cannot easily move: for example, when using a handle controller, the player's ring finger needs to hold the handle tightly at all times, so the sensor can be placed there to minimize motion artifacts.

[0033] Step 2: Data processing

[0034] After a light signal is emitted by a light-emitting diode (LED), it is reflected by the tissue within the finger, causing a certain amount of loss. The optical sensor then receives the light signal, generating a photocurrent. This photocurrent is amplified by a transconductance amplifier (TIA) and then transmitted to a microcontroller through analog-to-digital conversion. Let this signal at time t be f(t). Similarly, a sensor that collects ambient light, such as sunlight, also transmits this signal to the microcontroller as previously described. Let this signal at time t be g(t). The desired PPG signal, PPG(t), is equal to f(t) - g(t).

[0035] Step 3: Send PPG signal to the host computer

[0036] Considering that a wired connection may affect the portability of the game, this invention chooses to use Bluetooth connection to transmit data. Therefore, the host computer also uses a laptop with a Bluetooth module. After correctly pairing with the host computer, the PPG data will be used for learning and prediction.

[0037] A method for measuring a game player's experience during a game of the present invention is described in detail below with reference to the accompanying drawings.

[0038] Step 1: Organize cognitive load physiological experiments and form a database;

[0039] The classic n-back task, a short-term memory task, was used to stimulate different levels of cognitive load. The n-back task is a short-term memory task that is often used as an important experimental method to assess cognitive load in psychology and cognitive neuroscience. For example, in the 1-back experiment, subjects receive a series of stimuli, such as sounds, numbers, and images, and are asked to indicate whether the current stimulus is the same as the previous stimulus. This allows them to make appropriate decisions (if the stimulus is the same, press button 1; if it is different, press button 2).

[0040] Throughout the physiological experiment, the device described in this invention will collect the subject's vascular volume pulse wave signal (PPG(t)) and the corresponding subjective cognitive load evaluation value (C(t)). The subjective cognitive load evaluation value is set on a scale of 1-10, with the subject selecting one of these values. The final subjective evaluation value (C(t)) is calculated as the selected value / 10. Experimental data from at least 100 different subjects must be collected.

[0041] Step 2: Establish, train, and verify the cognitive load prediction model based on LSTM neural network;

[0042] Data preprocessing begins by segmenting the vascular volume pulse wave signal (PPG(t)) based on its C(t) value. C(t) values ​​≤ 0.2 are classified as low cognitive load, 0.3 ≤ C(t) ≤ 0.7 as moderate cognitive load, and C(t) ≥ 0.8 as high cognitive load. After classification, the first-order derivative (PPG'(t)) and second-order derivative (PPG"(t)) of PPG(t) are calculated for each category.

[0043] A cognitive load prediction model based on LSTM neural network was established. Based on database data, the original signal, first-order derivative and second-order derivative of the vascular volume pulse wave were used as the input of the cognitive load prediction model based on LSTM neural network. Moderate, too low and too high cognitive load were used as the output of the cognitive load prediction model based on LSTM neural network. The cognitive load prediction model based on LSTM neural network was trained.

[0044] AUC was calculated using ten-fold cross-validation. When AUC > 0.7, the LSTM neural network-based cognitive load prediction model is considered accurate. The so-called AUC is the area under the ROC curve of the prediction result. ROC is a curve with a two-dimensional coordinate system whose horizontal and vertical coordinates are both defined in the range [0,1]. Its horizontal and vertical coordinates are FPR (false positive rate) and TPR (true positive rate), respectively. These two components are calculated by the following formulas:

[0045]

[0046] Taking binary classification as an example, their meanings are: the number of true positive samples divided by the number of all true positive samples and the number of false positive samples divided by the number of all true negative samples.

[0047] Step 3: Establish a control strategy based on the player's cognitive load experience during the game;

[0048] The key to this strategy is how to dynamically adjust the difficulty level so that the player's cognitive load always remains at a balance between challenge and enjoyment. Simply put, when the player's cognitive load indicates that the game's challenges are too easy for them to achieve a sense of accomplishment, the system will increase the difficulty of the current game. Conversely, when the player's cognitive load is too high, indicating that the player's abilities are insufficient for the current game difficulty, the system will appropriately reduce the difficulty level so that the player can regain a sense of accomplishment on the next attempt.

[0049] Specifically, since the present invention makes predictions based on the LSTM model learned from the data in the database, after the PPG signal is transmitted to the neural network for prediction, if the obtained C(t)≤0.2 is considered to be too low a cognitive load, the system will increase the current difficulty of the game; when the obtained C(t)≤0.7 and C(t)≥0.3, the system will not make any changes to the difficulty of the game; finally, when C(t)≥0.8, it will be considered that the current player's cognitive load is too high, and the system will appropriately reduce the difficulty of the game.

Claims

1. A gaming experience measurement method based on PPG signals, characterized in that The following steps are involved: Step 1: Organize human-computer interaction experiments on game players, collect relevant physiological data and subjective evaluation data, and form a database; Step 2: Establish, train, and verify a cognitive load prediction model based on PPG signals; Step 3: Establish a control strategy based on the player's cognitive load experience during the game; Specifically: Step 1: Organize cognitive load physiological experiments and form a database; The classic n-back task, similar to short-term memory, was used to stimulate different levels of cognitive load. During the experiment, subjects received a series of stimuli, including sounds, numbers, and images, and were required to indicate whether the current stimulus was the same as the stimulus one step earlier. This led to a decision: if the stimulus was the same, they pressed button 1; if it was different, they pressed button 2. During the entire physiological experiment, the subject's vascular volume pulse wave signal PPG(t) and the corresponding subjective evaluation value of cognitive load C(t) were collected. The subjective cognitive load evaluation value was set on a scale of 1-10, with the subject selecting one. The final subjective evaluation value C(t) = the selected level / 10. Experimental data from at least 100 different subjects were required. Step 2: Establish, train, and verify the cognitive load prediction model based on LSTM neural network; For data preprocessing, the vascular volume pulse wave signal PPG(t) was first segmented according to the C(t) value. C(t) ≤ 0.2 was classified as low cognitive load, 0.3 ≤ C(t) ≤ 0.7 was classified as moderate cognitive load, and C(t) ≥ 0.8 was classified as high cognitive load. After classification, the first-order derivative PPG'(t) and the second-order derivative PPG"(t) of PPG(t) in each class are obtained; Establish a cognitive load prediction model based on an LSTM neural network. Based on database data, use the original signal, first-order derivative, and second-order derivative of the vascular volume pulse wave as the input of the LSTM neural network-based cognitive load prediction model. Use moderate, too low, and too high cognitive load as the output of the LSTM neural network-based cognitive load prediction model. Train the LSTM neural network-based cognitive load prediction model. The AUC was calculated using ten-fold cross-validation. When AUC > 0.7, the LSTM neural network-based cognitive load prediction model was considered accurate. The AUC is the area under the ROC curve of the prediction result. The ROC is a two-dimensional curve whose horizontal and vertical coordinates are both defined in the range [0, 1]. Its horizontal and vertical coordinates are the FPR false positive rate and the TPR true positive rate, respectively. These two components are calculated using the following formulas: Taking binary classification as an example, their meanings are: the number of true positive samples TP divided by the number of all true positive samples TP+FN; and the number of false positive samples FP divided by the number of all true negative samples FP+TN; Step 3: Establish a control strategy based on the player's cognitive load experience during the game; When the player's cognitive load shows that the challenge of this game is too easy for the player to gain a sense of accomplishment, the system will increase the difficulty of the current game; conversely, when the player's cognitive load is too high, it indicates that the player's ability is not up to the current game difficulty. The system will appropriately reduce the game difficulty so that the player can regain a sense of accomplishment in the next challenge.

2. The method for measuring gaming experience based on PPG signals according to claim 1, wherein: Regarding the method of collecting PPG signals, a non-invasive fingertip infrared pulse sensor is used. For AC error, a separate sensor is used to collect ambient light, which can then be subtracted from the PPG signal to obtain a pure PPG signal. Motion artifacts are captured by placing the sensor in a position where the finger cannot easily move.

3. The method for measuring gaming experience based on PPG signals according to claim 2, wherein: After the light-emitting diode emits a light signal, it will be reflected by the tissue inside the finger, resulting in a certain amount of consumption. The optical sensor then receives the light signal and generates a photocurrent. The photocurrent is amplified by the transconductance amplifier and then transmitted to the microcontroller through analog-to-digital conversion. Let the signal at time t be f(t); the sensor that separately collects ambient light is transmitted to the microcontroller. Let the signal at time t be g(t); the required PPG signal PPG(t) is equal to f(t)-g(t).

4. The method for measuring gaming experience based on PPG signals according to claim 3, wherein: We choose to use Bluetooth connection to transmit data; therefore, the host computer also uses a laptop with a Bluetooth module; after correctly pairing with the host computer, we will use PPG data for learning and prediction.

Images