Game assisting method, device, electronic device, storage medium and program product

Abstract

The present application discloses a game assistance method, device, electronic device, storage medium and program product, which relate to the field of game technology. The method includes: in the process of running a game application, obtaining multimodal data of a game player, extracting features from the multimodal data, obtaining feature vectors corresponding to each of the multimodal data, calculating the correlation between the feature vectors corresponding to each of the multimodal data using a cross-modal self-attention mechanism, obtaining a self-attention weight matrix, performing feature fusion on the feature vectors corresponding to each of the multimodal data based on the self-attention weight matrix, obtaining fusion features, determining the emotions of the game player based on the fusion features, and determining and outputting game assistance information for the game player based on the emotions. The present application achieves efficient fusion of data of different modalities by introducing a cross-modal self-attention mechanism, thereby improving the accuracy of emotion prediction and the effect of game assistance.

Description

Technical Field

[0001] The present application relates to the field of game technology, and more specifically, to a game assistance method, device, electronic device, storage medium and program product. Background Art

[0002] The game industry is developing rapidly, and various games have become one of the ways for people to pass the time after work. In the process of playing games through electronic devices (such as smartphones), game players often expect the electronic devices to provide some game auxiliary information to improve the game winning rate. However, the game auxiliary information that electronic devices can provide is currently limited, and the auxiliary effect on the game is not good. Summary of the invention

[0003] In view of the above problems, the present application proposes a game assistance method, device, electronic device, storage medium and program product to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a game assistance method, the method comprising: obtaining multimodal data of a game player during the running of a game application; performing feature extraction on the multimodal data to obtain feature vectors corresponding to each of the multimodal data; using a cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data to obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data; performing feature fusion on the feature vectors corresponding to each of the multimodal data based on the self-attention weight matrix to obtain fusion features; determining the emotion of the game player based on the fusion features, and determining and outputting game assistance information for the game player based on the emotion.

[0005] In a second aspect, an embodiment of the present application provides a game assistance device, comprising: a multimodal data acquisition module, used to acquire multimodal data of a game player during the operation of a game application; a feature extraction module, used to perform feature extraction on the multimodal data to obtain feature vectors corresponding to each of the multimodal data; a weight determination module, used to calculate the correlation between the feature vectors corresponding to each of the multimodal data using a cross-modal self-attention mechanism, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data; a feature fusion module, used to perform feature fusion on the feature vectors corresponding to each of the multimodal data based on the self-attention weight matrix to obtain fusion features; and a game assistance information output module, used to determine the emotion of the game player based on the fusion features, and determine and output game assistance information for the game player based on the emotion.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and the computer program implements the above method when executed by a processor.

[0009] The game assistance method, device, electronic device, storage medium and program product provided in the embodiments of the present application first obtain the multimodal data of the game player in the process of running the game application, perform feature extraction on the multimodal data, and obtain the feature vectors corresponding to each of the multimodal data. Then, a cross-modal self-attention mechanism is used to calculate the correlation between the feature vectors corresponding to each of the multimodal data to obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data. Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features. Finally, the game player's emotions are determined based on the fused features, and game assistance information for the game player is determined and output based on the emotions, thereby achieving efficient fusion of different modal data by introducing a cross-modal self-attention mechanism, thereby improving the accuracy of emotion prediction and the effect of game assistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram showing a flow chart of a game assisting method provided by an embodiment of the present application is shown;

[0012] Figure 2 A schematic diagram showing a flow chart of a game assisting method provided by an embodiment of the present application is shown;

[0013] Figure 3 A schematic diagram showing a flow chart of a game assisting method provided by an embodiment of the present application is shown;

[0014] Figure 4 A flowchart of a game assisting method provided by an embodiment of the present application is shown;

[0015] Figure 5 A schematic diagram showing a flow chart of a game assisting method provided by an embodiment of the present application is shown;

[0016] Figure 6 A flowchart of a game assisting method provided by an embodiment of the present application is shown;

[0017] Figure 7 A module block diagram of a game auxiliary device provided by an embodiment of the present application is shown;

[0018] Figure 8 A block diagram of an electronic device for executing a game assistance method according to an embodiment of the present application is shown;

[0019] Fig. 9 A storage unit for storing or carrying program codes for implementing a game assistance method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0021] At present, in order to provide some auxiliary game information, electronic devices generally collect various sensory data of game players through a series of sensors, such as audio, vision, physiological signals, etc., and use their learning algorithms to analyze these sensory data to predict the emotional state of game players. According to the emotional prediction results, the static configuration information of the game, such as game difficulty and environment settings, is dynamically adjusted to enhance the gaming experience. Specifically, it combines a variety of sensory data with machine learning models to predict the emotions of game players, and dynamically adjusts game settings based on these predictions; through big data analysis and model training, it can not only predict the emotions of game players, but also analyze the churn rate of game players, and adjust game features accordingly to optimize game player retention. It is understandable that the current solution has at least the following shortcomings:

[0022] First, there is a lack of efficient integration strategies for multimodal data

[0023] Existing technologies seriously lack effective fusion strategies when processing multimodal data of gamers, resulting in isolated analysis of audio, visual and physiological signals. This processing method ignores the intrinsic connection between each modal data, thereby weakening the accuracy and depth of emotion recognition. For example, although audio analysis can capture subtle changes in tone, it cannot see into the facial expressions of gamers; although visual analysis can recognize smiles, it cannot perceive the physiological fluctuations of gamers. The defect of this isolated data processing not only limits the comprehensive grasp of gamers' emotions, but is also likely to cause significant distortion of information when emotions fluctuate violently, thereby damaging the personalized experience of the game and the overall feelings of gamers.

[0024] Second, the limitations of multimodal data processing and analysis solutions

[0025] When analyzing the emotions of game players, existing technologies rely on traditional machine learning models to process each modal data in isolation and then simply combine them through static weights. This approach ignores the dynamic interaction and real-time changes between data modalities and cannot accurately capture the complexity and dynamics of game players' emotions. Existing technologies have obvious deficiencies in accuracy and predictive ability and cannot meet the needs of accurately capturing and responding to game players' emotions.

[0026] Third, the timing synchronization and real-time performance of data collection are insufficient

[0027] Existing technologies have obvious shortcomings in the timing synchronization and real-time performance of multimodal data collection. The key problem is that different data collection devices lack a unified timestamp standard, which results in the inability to accurately align multimodal data such as audio, visual and physiological signals on the timeline. This time mismatch seriously weakens the accuracy of sentiment analysis, especially in dynamic gaming environments, where emotions change rapidly and time deviations may lead to completely wrong interpretations. In addition, existing technologies usually store multimodal data independently in a physical database rather than processing it in real time, which not only increases the cost of data storage and maintenance, but also slows down data processing due to high I / O read costs. In gaming scenarios that require instant response, this delayed processing method cannot meet the real-time requirements, affecting the game's rapid adaptation and response to the emotions of gamers, thereby reducing the quality of the gamer experience.

[0028] Fourth, the dynamic adjustment of game strategies is simple and crude, and does not meet the personalized needs of players.

[0029] Existing technologies usually adopt a unified adjustment strategy in game design, such as fixed game difficulty and NPC types. This one-size-fits-all adjustment strategy requires extremely high background permissions and poor scalability. Secondly, it ignores the personalized preferences of game players and is likely to cause the game experience to be inconsistent with player expectations. Therefore, existing technologies urgently need a more flexible and personalized game adjustment mechanism to enhance player experience and support applicability in different games.

[0030] Fifth, the dynamic adjustment of game strategies does not meet the personalized needs of players.

[0031] Existing technologies only make adjustments to the static configuration of the game and fail to fully adapt to the personalized needs of gamers when adjusting game strategies. Emotion classification is limited to likes and dislikes, and the emotional result processing strategy is too simple, such as adjusting the brightness or simply reducing the disgust elements. This makes it difficult to accurately improve the satisfaction and engagement of gamers in various game environments.

[0032] In view of the above problems, the inventors have discovered through long-term research and proposed the game assistance method, device, electronic device, storage medium and program product provided in the embodiments of the present application, which realizes the efficient fusion of different modal data by introducing a cross-modal self-attention mechanism, improves the accuracy of emotion prediction and the effect of game assistance. The specific game assistance method is described in detail in the subsequent embodiments.

[0033] See also Figure 1 , Figure 1 The flowchart of the game assistance method provided by an embodiment of the present application is shown. The method is used to achieve efficient fusion of different modal data by introducing a cross-modal self-attention mechanism, thereby improving the accuracy of emotion prediction and the effect of game assistance. In a specific embodiment, the game assistance method is applied to Figure 7 The game assisting device 200 and the electronic device 100 equipped with the game assisting device 200 are shown in FIG. Figure 8 ). It is understandable that the electronic device may include a smart phone, a tablet computer, a wearable electronic device, etc., which is not limited here. Figure 1 The process shown is described in detail, and the game assistance method may specifically include the following steps:

[0034] Step S110: Acquire multimodal data of game players during the running of the game application.

[0035] Optionally, the electronic device may run a game application, wherein the game application may be a local game application that has been downloaded and installed on the electronic device, or the game application may be an online game application that has not been downloaded and installed on the electronic device, which is not limited here.

[0036] Optionally, the game application can be a multiplayer online tactical competitive game, in which case the game players are usually divided into two or more teams to compete on a well-designed map, each game player controls a game character, and these game characters usually have unique skills, attributes and positioning (such as warriors, mages, assassins, etc.), and the goal of the game is to destroy the opponent's base or achieve other specific victory conditions through teamwork. In this embodiment, the game characters included in the game application can be divided into one's own game characters and the opponent's game characters according to the battle party. It can be understood that the one's own game character is the game character controlled by the game player, and the opponent's game character is the game character that fights against the one's own game character. Of course, in addition to the one's own game character and the opponent's game character, other teammate game characters belonging to the same team as the one's own game character can also be included, wherein the number of the one's own game characters can be one or more, the number of the opponent's game characters can be one or more, and the number of other teammate game characters can be zero, one or more, which is not limited here.

[0037] In this embodiment, during the running of the game application, multimodal data of the game player can be obtained.

[0038] Optionally, the multimodal data of the game player can be collected and obtained through a device running the game application (such as a smart phone), or part of the modal data of the game player can be collected through multiple devices (such as smart phones, smart watches, cameras, microphones, etc.) respectively, and then sent to the device running the game application. Then, the device running the game application can obtain the multimodal data of the game player, which is not limited here.

[0039] Optionally, the multimodal data may include at least two of physiological data, image data, voice data, and game data. Among them, physiological data may include, for example, the heart rate and skin galvanic response of the game player; image data may include, for example, the facial expression of the game player; voice data may include, for example, the speech content of the game player; game data may include, for example, the operation frequency, click interval, in-game text (such as dialog box content) of the game player, etc., which are not limited here.

[0040] Step S120: extracting features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0041] In this embodiment, when the multimodal data of the game player is obtained, feature extraction can be performed on the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0042] In some embodiments, a feature extraction method corresponding to different modal data may be pre-set and stored. When multimodal data of a game player is obtained, feature extraction may be performed on the multimodal data respectively using feature extraction methods corresponding to different modal data to obtain feature vectors corresponding to each multimodal data.

[0043] As an implementable approach, when multimodal data includes physiological data, such as heart rate, blood pressure, etc., feature extraction usually involves statistical analysis and signal processing techniques. The features that need to be extracted may include: mean: calculate the average value of the heart rate over a period of time; standard deviation: measure the degree of heart rate change; heart rate variability: evaluate by calculating the difference between adjacent heartbeat intervals; spectrum analysis: convert the heart rate signal into the frequency domain and analyze the intensity of different frequency components. In addition, for more complex physiological signals (such as ECG), more advanced signal processing techniques such as wavelet transform or Kalman filtering can be used to extract features.

[0044] As an implementable approach, when the multimodal data includes image data, feature extraction usually involves image processing techniques and computer vision algorithms. Features that need to be extracted may include: facial feature point coordinates: such as the positions of the 68 facial feature points mentioned earlier; texture features: such as image texture described by the gray-level co-occurrence matrix (GLCM); shape features: such as facial contour, width, height, etc.; color features: such as RGB color histograms or statistics in HSV color space. In addition, deep learning techniques (such as convolutional neural networks (CNNs)) can also be used to automatically extract high-level features from image data.

[0045] As an implementable approach, when the multimodal data includes speech data features, feature extraction usually involves speech signal processing techniques. Features that need to be extracted may include: MFCC (Mel Frequency Cepstral Coefficients): As mentioned above, it is a commonly used feature in speech emotion recognition; Spectral Envelope: describes the spectral shape of the speech signal.; Fundamental Frequency (F0): The frequency of the fundamental tone in the speech signal, which is related to the speaker's pitch; Formant: The resonant frequency in the speech signal, which is related to the shape of the vocal tract; Energy and Duration: The energy and duration of the speech segment are also useful features. Similarly, deep learning techniques (such as recurrent neural networks RNN or long short-term memory networks LSTM) can also be used to automatically extract high-level features from speech data.

[0046] Among them, after extracting the features of each modality, they can be combined into a feature vector, and then the feature vectors corresponding to each multimodal data can be obtained. For multimodal data, it also involves splicing or fusing the feature vectors of different modalities, that is, the scale and correlation between different modal features need to be considered. Therefore, the multimodal data can be standardized. Assuming that each modal data is a vector X, the standardization formula is: Among them, μ is the mean and σ is the standard deviation. Through this step, the scale of different modal data can be guaranteed to be unified.

[0047] In some embodiments, when the multimodal data of the game player is obtained, the multimodal data can be subjected to noise filtering to obtain the multimodal data after noise filtering, and feature extraction can be performed on the multimodal data after noise filtering to obtain the feature vectors corresponding to each of the multimodal data after noise filtering. Optionally, when the multimodal data includes physiological data, the noise in the collected physiological data can be removed by Kalman Filter to ensure the accuracy of physiological data such as heart rate; when the multimodal data includes image data, the face of the game player can be located by a face detection algorithm (such as a model based on Haar features or DNN), irrelevant background information can be removed, and then the expression data can be marked with feature points, such as using Active Shape Model (ASM) to obtain 68 facial feature points; when the multimodal data includes voice data, the voice spectrum can be extracted by short-time Fourier transform (STFT), and combined with Mel frequency cepstral coefficients (MFCC), the emotional features in the voice data can be further extracted.

[0048] Step S130: Utilize the cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data.

[0049] Among them, considering that the current prediction of game players' emotions through multimodal data such as audio, vision and physiological signals often analyzes audio, visual and physiological signals in isolation, ignoring the inherent connection between each modal data, thereby limiting the accuracy and comprehensiveness of emotion recognition. In this regard, this embodiment solves the problem of multimodal data fusion through a cross-modal self-attention mechanism, in which the self-attention mechanism effectively associates audio, visual and physiological data with each other by dynamically adjusting the weights of different modal data. This fusion strategy not only improves the accuracy of emotion recognition, but also avoids the information loss that may be caused when each modal data is processed in isolation. Therefore, the emotional state of game players is more comprehensively and accurately grasped, further enhancing the personalized experience of the game and the timeliness of emotional feedback.

[0050] In this embodiment, when the feature vectors corresponding to the multimodal data are obtained, the correlation between the feature vectors corresponding to the multimodal data can be calculated using the cross-modal self-attention mechanism to obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data;

[0051] It can be understood that in the cross-modal self-attention mechanism, some parameters need to be initialized, such as query, key and value matrices. These parameters are usually obtained through training and used in subsequent calculation processes. Optionally, the query, key and value matrices can also be directly converted from the feature vectors corresponding to the multimodal data, which is not limited here.

[0052] Next, the correlation between the feature vectors of different modal data can be calculated. For example, it can be calculated through dot product attention (directly calculating the dot product of two feature vectors to obtain the attention score) or scaled dot product attention (in order to avoid the gradient vanishing problem caused by the dot product result being too large, the dot product result can be scaled, that is, divided by the square root of the feature vector dimension). For any two sets of feature vectors of modal data, the similarity matrix (or attention score matrix) between them can be calculated. Each element in this matrix represents the similarity (or correlation) between the two feature vectors.

[0053] Afterwards, the attention score matrix is ​​normalized by the Softmax function to obtain the self-attention weight matrix, where the Softmax function can convert the attention score into a probability distribution form so that each self-attention weight is between 0 and 1, and the sum of all weights is 1.

[0054] It can be understood that each element in the self-attention weight matrix (i.e., self-attention weight) represents the degree of attention of one modal data to other modal data. The higher the weight, the higher the attention of the modal data to other modal data in the current situation. This weight distribution is dynamic and can change according to the input data. Furthermore, the association of multimodal data can be obtained based on the cross-modal self-attention mechanism, and it can be used for subsequent game player emotion prediction.

[0055] Step S140: Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features.

[0056] In this embodiment, when the self-attention weight matrix is ​​obtained, feature fusion can be performed on the feature vectors corresponding to each of the multimodal data based on the self-attention weight matrix to obtain fused features.

[0057] In some embodiments, when the self-attention weight matrix is ​​obtained, the self-attention weights corresponding to each of the multimodal data can be determined based on the self-attention weight matrix, and the fusion features can be obtained based on the self-attention weights corresponding to each of the multimodal data and the feature vectors corresponding to each of the multimodal data. As an implementable method, the product of the feature vector corresponding to each of the multimodal data and the corresponding self-attention weight can be calculated respectively to obtain the weighted feature vector corresponding to each of the multimodal data, and the weighted feature vectors corresponding to each of the multimodal data can be summed to obtain the fusion features.

[0058] As an example, assume that the feature vectors corresponding to the multimodal data include: image feature vector F img , speech feature vector F audio , physiological feature vector F bio , then, the fusion feature F fusion The expression can be:

[0059] F fusion =W img ·F img +W audio ·F audio +W bio ·F bio

[0060] Among them, W img is the self-attention weight corresponding to the image feature vector, W audio is the self-attention weight corresponding to the speech feature vector, W bio The self-attention weights corresponding to the physiological feature vectors can be adaptively learned and adjusted through model training. That is, the model can adaptively extract and fuse the deep features of multimodal data, and can better identify and predict subtle changes in emotions. The advantage of the large model lies in its powerful learning ability and wide parameter space, which can accurately capture the emotional changes of gamers, especially when the emotions fluctuate violently, providing more accurate predictions and feedback, and enhancing the immersion and interactivity of the gamer experience.

[0061] In addition, Graph Neural Network (GNN) can be introduced as an extended alternative. GNN can represent data nodes of different modalities as a graph structure, and capture the interdependence between modal data through the connection relationship between nodes in the graph, thereby achieving efficient fusion of multimodal data. Although the effect of this method may not be better than the self-attention mechanism in some scenarios, it can also perform effective information processing when the data complexity is high, as an alternative solution for multimodal data fusion.

[0062] Step S150: determining the emotion of the game player based on the fusion feature, and determining and outputting game auxiliary information for the game player based on the emotion.

[0063] In this embodiment, when a fusion feature is obtained, the emotion of the game player can be determined based on the fusion feature, and game assistance information for the game player can be determined and output based on the emotion of the game player, thereby improving the accuracy of emotion prediction and the effect of game assistance.

[0064] Optionally, the game player's emotions may include: happiness, anxiety, sadness, loss, joy, etc., which are not limited here.

[0065] Optionally, the game auxiliary information may include: game strategies, text for encouragement or praise, audio for encouragement or praise, etc., which are not limited here.

[0066] In some embodiments, when a fusion feature is obtained, the fusion feature can be input into a trained emotion prediction model, wherein the trained emotion prediction model is obtained through machine learning. Specifically, a first training data set is first collected, wherein the attributes or characteristics of one type of data in the first training data set are different from those of another type of data, and then a first neural network is trained and modeled according to a preset algorithm using the collected first training data set, thereby summarizing the rules based on the first training data set to obtain a trained emotion prediction model.

[0067] Optionally, the trained emotion prediction model can be pre-trained and stored locally in the electronic device. Based on this, when the fusion feature is obtained, the trained emotion prediction model can be directly called locally. For example, an instruction can be directly sent to the emotion prediction model to instruct the trained emotion prediction model to read the fusion feature in the target storage area, or the electronic device can directly input the fusion feature into the trained emotion prediction model stored locally, thereby effectively avoiding the influence of network factors that reduce the speed of inputting the fusion feature into the trained emotion prediction model, thereby increasing the speed of the trained emotion prediction model acquiring the fusion feature and improving the user experience.

[0068] Optionally, the trained emotion prediction model can also be pre-trained and stored in a server that is in communication with the electronic device. Based on this, when the fusion feature is obtained, an instruction can be sent through the network to the trained emotion prediction model stored in the server to instruct the trained emotion prediction model to read the fusion feature determined by the electronic device through the network, or the electronic device can send the fusion feature to the trained emotion prediction model stored in the server through the network, thereby reducing the storage space occupied by the electronic device and reducing the impact on the normal operation of the electronic device by storing the trained emotion prediction model in the server.

[0069] As an implementable method, the emotion prediction model can adopt a multimodal fusion model architecture and combine it with self-supervised learning technology, so that the emotion prediction model can be efficiently trained and inferred based on multimodal data. It can be understood that the input of the emotion prediction model is the fusion feature, and the emotion classification result is calculated layer by layer through the multi-layer perceptron (MLP) and the nonlinear activation function (ReLU). The output of the emotion classification result is: y = softmax (W·F fusion +b), where W is the weight matrix and b is the bias term.

[0070] In some embodiments, the emotion prediction model can be combined with the ability to dynamically capture real-time emotion changes and use a time series prediction algorithm to model continuous emotion changes. Through sliding window technology, the emotion prediction model can predict current emotions from previous emotional states, so that emotion changes can be captured in advance and fed back into game auxiliary information.

[0071] In addition, traditional support vector machines (SVM) or random forest algorithms can be used as an alternative as an extension of multimodal emotion recognition. These traditional machine learning models can be used for emotion classification tasks in simple scenarios, especially when the amount of data is small or computing resources are limited. SVM or random forests can still provide relatively stable emotion prediction results. Although the prediction accuracy may be lower than that of deep learning models, it is still feasible in certain specific scenarios and can avoid the deep learning model's reliance on large amounts of data.

[0072] In some implementations, a mapping relationship may be pre-set and stored, and the mapping relationship may include a correspondence relationship between multiple emotions and multiple game auxiliary information, wherein the correspondence relationship between multiple emotions and multiple game auxiliary information may include: one emotion corresponds to one game auxiliary information, or several emotions correspond to one game auxiliary information, etc., which is not limited here. In this embodiment, when the game player's emotions are obtained, the game auxiliary information that has a corresponding relationship with the game player's emotions may be determined based on the mapping relationship, that is, it is determined as the game auxiliary information for the game player and output.

[0073] For example, if the game player is in a happy mood, the first game assistance information for the game player can be determined and output; if the game player is in a sad mood, the second game assistance information for the game player can be determined and output, wherein the first game assistance information and the second game assistance information are different.

[0074] In some embodiments, the electronic device may be provided with a game assist switch, which is used to control the opening or closing of the game assist function of the electronic device. It is understood that if the game assist switch is in an on-off state, it indicates that the game assist function of the electronic device is turned off, and at this time, the electronic device no longer provides game assist information to the game player; if the game assist switch is in an on state, it indicates that the game assist function of the electronic device is turned on, and at this time, the electronic device can provide game assist information to the game player.

[0075] As an implementable method, the conditions for turning on or off the game auxiliary switch can be preset and stored, and the game auxiliary switch can be controlled to turn on or off according to the conditions. Based on this, it can be judged whether the electronic device or the game player meets the conditions. If the conditions are met, the game auxiliary switch can be controlled to turn on, or, if the conditions are not met, the game auxiliary switch can be controlled to turn off.

[0076] Optionally, the condition may include a preset emotion, wherein the preset emotion may include a sad emotion, a lost emotion, or the like, which indicates that the user finds it difficult to win the game. Based on this, when the game player's emotion is obtained, the game player's emotion may be compared with the preset emotion to determine whether the game player's emotion satisfies the preset emotion, that is, to determine whether the game player's emotion is a sad emotion or a lost emotion. If it is determined that the game player's emotion satisfies the preset emotion, it may indicate that the user needs the electronic device to provide game auxiliary information to improve the game winning rate, and therefore, the game auxiliary function may be turned on, and game auxiliary information for the game player may be determined and output based on the game player's emotion. If it is determined that the game player's emotion does not meet the preset emotion, it may indicate that the user does not need the electronic device to provide game auxiliary information to the game player, and therefore, the game auxiliary function may not be turned on.

[0077] A game assistance method provided in an embodiment of the present application, during the process of running a game application, first obtains multimodal data of a game player, performs feature extraction on the multimodal data, obtains feature vectors corresponding to each of the multimodal data, and then uses a cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data, and obtains a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data, and based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features, and finally the game player's emotions are determined based on the fused features, and game assistance information for the game player is determined and output based on the emotions, thereby achieving efficient fusion of data of different modalities by introducing a cross-modal self-attention mechanism, thereby improving the accuracy of emotion prediction and the effect of game assistance.

[0078] See also Figure 2 , Figure 2 The flowchart of the game assisting method provided by an embodiment of the present application is shown in FIG. Figure 2 The process shown is described in detail, and the game assistance method may specifically include the following steps:

[0079] Step S210: Acquire multimodal data of game players during the running of the game application.

[0080] Step S220: extracting features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0081] Step S230: Utilize the cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data.

[0082] Step S240: Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features.

[0083] Step S250: Determine the game player's emotion based on the fused features.

[0084] For the detailed description of steps S210 to S250 , please refer to steps S110 to S150 , which will not be repeated here.

[0085] Step S260: Based on the emotion, determine a game strategy for the game player, wherein the game strategy includes one or a combination of game difficulty adjustment of the game application, equipment suggestions for the game character corresponding to the game player in the game application, and skill adjustment of the game character corresponding to the game player in the game application.

[0086] Optionally, the game auxiliary information may include a game strategy, wherein the game strategy may include one or a combination of game difficulty adjustment for the game application, equipment suggestions for the game character corresponding to the game player in the game application, and skill adjustment for the game character corresponding to the game player in the game application.

[0087] In this embodiment, when the emotions of the game player are obtained, the game strategy for the game player can be determined based on the game player's emotions. Then, through the emotion-driven dynamic game strategy adjustment, a more personalized game strategy is generated in combination with the real-time emotional state, game situation and personalized preferences of the game player. For example, according to the emotional fluctuations of the game player, real-time equipment suggestions, skill adjustments or game difficulty adjustments are provided. Through this solution, the game experience of the game player is more personalized, the adaptability and interactivity of the game are improved, and the game strategy can be intelligently adjusted according to the different emotional states of each game player, which enhances the immersion and fun of the game.

[0088] Among them, game difficulty is one of the key factors that affect the gamer experience. Too high a difficulty may cause gamers to feel frustrated and give up the game, while too low a difficulty may make gamers feel that it lacks challenge and lose interest. Based on this, the game difficulty can be dynamically changed according to the gamer's emotions, such as the strength of the other game character, the design of the level, and the time limit, to adapt to the skill levels and preferences of different gamers. For example, if the gamer's emotions show sadness or loss, the game difficulty can be lowered to enhance the gamer's gaming experience. Of course, the game difficulty adjustment can also be intelligently adjusted according to the gamer's progress and achievements to ensure that the game is both challenging and interesting.

[0089] Among them, outfit suggestions refer to suggestions for game players to choose equipment during the game. Outfit suggestions can help game players use resources more effectively, improve the combat capabilities of their own game characters, and thus achieve better results in the game. Advanced outfit suggestions may also involve matching and upgrading strategies between equipment to increase the diversity and tactical depth of game characters. In this embodiment, the outfit suggestions for the game character corresponding to the game player can be determined according to the game player's emotions. For example, if the game player's emotions show sadness or loss, the outfit suggestions for the game character can be made according to the game situation or the other party's game character. Of course, these outfit suggestions can also be combined with factors such as the current game state, the type of enemy, and the game character, which are not limited here.

[0090] Among them, skill adjustment refers to configuring and optimizing the skills of the game character to adapt to different combat scenarios and enemy types. In this embodiment, specific skills can be selectively upgraded or activated according to the emotions of the game player to improve the combat efficiency and survivability of the character. For example, if the game player's emotions show sadness or loss, skills that help improve the game winning rate can be upgraded or activated to improve the combat effectiveness of the game character and the game winning rate. Of course, skill adjustment can also include the combination and linkage between skills to create more powerful combos and tactical effects.

[0091] In some implementations, a rule-based engine alternative can be introduced to trigger dynamic adjustments to game strategies through predefined rule sets and the emotions of game players. This approach can simplify the complex analysis of emotional changes and perform basic game strategy adjustments through some simple "if-then" rules. Although this solution is not as intelligent as a deep learning model, in some games that do not require high real-time performance and have fewer scene changes, the rule engine is a low-cost alternative to achieve emotion-driven strategy adjustments.

[0092] Step S270: Display the game strategy.

[0093] In this embodiment, when a game strategy for a game player is determined, the game strategy may be displayed. For example, one or a combination of game difficulty adjustment, equipment suggestion, and skill adjustment may be displayed so that the game player can perform corresponding operations according to the displayed game strategy.

[0094] As an implementable method, game difficulty adjustment, equipment suggestions and skill adjustments can be displayed. Subsequently, game players can choose whether to adjust the game difficulty based on the displayed game difficulty adjustment, equip their own game characters according to the displayed equipment suggestions, and choose whether to adjust skills based on the displayed skill adjustment, etc., without limitation here.

[0095] As another feasible method, only equipment suggestions may be displayed, and subsequently, the game player may equip his own game character according to the equipment suggestions displayed, and the electronic device may directly adjust the difficulty of the game application according to the game difficulty adjustment, and directly adjust the skills of the game character corresponding to the game player in the game application according to the skill adjustment.

[0096] In some implementations, the game strategy can be displayed in the game interface of the game application while the game player is controlling his own game character. It is understandable that the game strategy is displayed in real time, that is, when the game strategy is determined, the game strategy is immediately displayed in the game interface, which can ensure the real-time display of the game strategy and enhance the auxiliary effect of the game strategy.

[0097] In some embodiments, the game strategy can be displayed in the game interface of the game application when the game character of the own party dies and waits for resurrection. It is understandable that when the game strategy is displayed when the game character of the own party dies and waits for resurrection, the game player does not need to control the game character of the own party, and the display of the game strategy will not affect the control experience of the game player, which can improve the game experience of the game player.

[0098] As an implementable method, in the early stage of the game (such as less than 10 minutes), considering that the resurrection time of one's own character is short, if too many game strategies are displayed at this time, it may cause the game players to be unable to read them completely. Therefore, only the equipment suggestions of one's own game character can be displayed in the game interface, so that the players can have a preliminary understanding of the equipment of their own game character in the early stage of the game; in the middle stage of the game (such as more than 10 minutes and less than 20 minutes), considering that the resurrection time of one's own game character is increased compared with the early stage of the game, at this time, the content of the displayed game strategy can be increased accordingly, therefore, the equipment suggestions and skill adjustments can be displayed in the display interface at the same time, so that the game players can improve the combat effectiveness of their own game characters in the middle stage of the game; in the late stage of the game (such as more than 20 minutes), considering that the resurrection time of one's own game character is long, at this time, the content of the displayed game strategy can be further increased, therefore, the game difficulty adjustment, equipment suggestions and skill adjustments can be displayed in the game interface at the same time, so that the game players can improve the game winning rate in the late stage of the game.

[0099] As another feasible method, the display mode of game difficulty adjustment, outfit suggestion and skill adjustment for one's own character can be determined according to the situation information in the game. As an example, if the situation information indicates that one's own game character has fewer deaths and the other's game character has more deaths (such as the number of deaths of one's own game character has not reached the death number threshold and the number of deaths of the other's game character has reached the death number threshold), it can be considered that the game player is familiar with the outfit, and then only the game difficulty adjustment and skill adjustment can be displayed in the game interface; if the situation information indicates that one's own game character has more deaths and the other's game character has fewer deaths (such as the number of deaths of one's own game character has reached the death number threshold and the number of deaths of the other's game character has not reached the death number threshold), it can be considered that the player is not familiar with the outfit, and then the game difficulty adjustment, outfit suggestion and skill adjustment can be displayed at the same time, so that the game player can improve the game winning rate in the later stage of the game.

[0100] In some embodiments, a switching control may be provided in the game interface, and the user may select the information to be displayed through the switching control. For example, the electronic device may receive a switching operation from the user, and the game interface may switch from displaying skill adjustment to displaying game difficulty adjustment, or the game interface may switch from displaying outfit recommendations to displaying actual game difficulty adjustment, and so on.

[0101] Compared with the game assisting method provided in one embodiment of the present application, Figure 1 The game assistance method shown in this embodiment also determines a game strategy for the game player based on the game player's emotions, wherein the game strategy includes one or a combination of game difficulty adjustment of the game application, equipment suggestions for the game character corresponding to the game player in the game application, and skill adjustment of the game character corresponding to the game player in the game application, and displays the game strategy, thereby generating a personalized game strategy in combination with the game player's emotions, which can improve the game performance of the game player to enhance the game experience of the game player.

[0102] See also Figure 3 , Figure 3 The flowchart of the game assisting method provided by an embodiment of the present application is shown in FIG. Figure 3 The process shown is described in detail, and the game assistance method may specifically include the following steps:

[0103] Step S310: Acquire multimodal data of game players during the running of the game application.

[0104] Step S320: extracting features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0105] Step S330: Utilize the cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data.

[0106] Step S340: Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features.

[0107] Step S350: Determine the game player's emotion based on the fused features.

[0108] For the detailed description of steps S310 to S350 , please refer to steps S110 to S150 , which will not be repeated here.

[0109] Step S360: Generate a target text based on the emotion, wherein the target text includes text content for encouraging or praising the game player.

[0110] Optionally, the game auxiliary information may include target text for encouraging or praising the game player.

[0111] In this embodiment, when the emotions of the game player are obtained, a target text including text content for encouraging or praising the game player may be generated based on the emotions of the game player.

[0112] In some implementations, when the emotions of the game players are obtained, the prompt word generation technology based on the large language model (such as GPT) can be used to dynamically generate the emotional support and interactive content required by the game players, that is, combining the emotion prediction and the game situation, a target text with emotional color can be generated. Optionally, the prompt word generation formula can be: emotion =LLM(E player ,C context ), where T emotion is the generated emotional prompt text, E player For the emotions of game players, C context For context information.

[0113] As an example, if the game player's emotions show sadness or loss, a target text including text content for encouraging the game player, such as "Keep doing it next time" can be generated; if the game player's emotions show happiness or joy, a target text including text content for praising the game player, such as "Great job", can be generated.

[0114] Step S370: Display the target text.

[0115] In this embodiment, when the target text is obtained, the target text may be displayed.

[0116] In some embodiments, when the target text is obtained, the target text can be displayed in the game interface. For example, the target text can be displayed in the game interface in the form of a floating window, in the game interface in the form of a bullet screen, etc., which is not limited here.

[0117] Step S380: using speech synthesis technology, converting the target text into target audio corresponding to the voice of the game character corresponding to the game player in the game application.

[0118] In some embodiments, when the target text is obtained, the target text can be converted into a target audio corresponding to the voice of the game character corresponding to the game player in the game application through speech synthesis technology, so as to encourage or praise the game player through audio.

[0119] Among them, in game applications, each game character usually has unique voice features, including pitch, speaking speed, timbre, dialect or accent, etc. Based on this, the voice features of one's own game character can be determined, and according to the voice features of one's own game character, a suitable speech synthesis technology can be selected, such as a synthesis system based on text to speech (TTS), such as splicing synthesis, parameter synthesis or waveform synthesis, etc. The target text is preprocessed, including word segmentation, part of speech tagging, prosody tagging, etc., so as to better adapt to the processing flow of the speech synthesis system. The preprocessed text is input into the speech synthesis system, and the system generates a speech waveform corresponding to the target text, that is, the target audio, according to the selected technology and the voice features of one's own game character.

[0120] Step S390: Play the target audio.

[0121] In this embodiment, when the target audio is obtained, the target audio can be played to encourage or praise the game player through the voice of the game character, thereby enhancing the game player's sense of immersion.

[0122] Of course, in some implementations, it is also possible to use a preset voice library as an alternative, and directly use a pre-recorded voice file to call according to the emotional state without relying on real-time generation. This alternative can greatly reduce the system's computing resource requirements, especially for scenarios that do not require a large number of voice changes or have low requirements for language models. Although the flexibility of the preset voice is poor, its implementation cost is low and it is suitable for some games that require simple emotional comfort.

[0123] Compared with the game assisting method provided in one embodiment of the present application, Figure 1In the game assisting method shown in the embodiment, the target text is generated based on the emotions of the game player, wherein the target text includes text content for encouraging or praising the game player, and the target text is displayed. Alternatively, the target text is converted into a target audio corresponding to the voice of the game character corresponding to the game player in the game application through speech synthesis technology, and the target audio is played, so that natural and personalized text and / or voice feedback can be generated according to the game player's emotional prediction, thereby enhancing the game player's immersion and game interactivity.

[0124] See also Figure 4 , Figure 4 The flowchart of the game assisting method provided by an embodiment of the present application is shown in FIG. Figure 4 The process shown is described in detail, and the game assistance method may specifically include the following steps:

[0125] Step S410: Acquire multimodal data of game players during the running of the game application.

[0126] Step S420: extracting features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0127] Step S430: Utilize the cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data.

[0128] Step S440: Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features.

[0129] Step S450: Determine the game player's emotion based on the fused features.

[0130] For the detailed description of step S410 to step S450, please refer to step S110 to step S150, which will not be repeated here.

[0131] Step S460: Determine reference information associated with the game player, wherein the reference information includes the current game situation of the game application and / or the historical game behavior of the game player.

[0132] In this embodiment, reference information associated with the game player may be determined, wherein the reference information may include the current game situation of the game application and / or the historical game behavior of the game player.

[0133] As an implementable method, the current game situation of the game application can be monitored in real time, including but not limited to the location of the game character, game progress, task completion, distribution of hostile forces, resource distribution, etc. This information can be obtained through the API inside the game engine. In addition, key events in the game can be detected and recorded, such as the start and end of the battle, the triggering and completion of the task, the acquisition of special items, etc. Based on the collected game status information and event detection, the overall situation of the game can be further analyzed, such as judging whether the current stage is a peaceful exploration stage, a tense battle stage, or a dangerous escape stage, etc., and then the current game situation of the game application can be determined.

[0134] As an implementable method, the historical game behavior of game players can be recorded, including but not limited to the game player's movement path, combat record, item use, skill release, interaction with NPC, etc. These behaviors can be collected through game logs or player behavior analysis systems. The collected historical game behaviors are deeply analyzed to understand the player's game style, preferences, strategies, etc. For example, analyze the player's decision-making in combat, the efficiency of resource management, and the familiarity with the rules of the game. Based on the results of behavioral analysis, the game system can identify the player's behavior pattern, such as aggressive, conservative, strategic, etc., and then determine the historical game behavior of the game player.

[0135] Step S470: Based on the emotion and the reference information, determine and output game auxiliary information for the game player.

[0136] In this embodiment, when the emotions of the game players and the reference information associated with the game players are obtained, the game auxiliary information for the game players can be determined and output based on the emotions and the reference information, so that the accuracy of the output game auxiliary information can be improved. That is, this embodiment can deeply analyze the emotional changes of the game players, and provide more complex and accurate game auxiliary information in combination with the historical behavior and real-time emotional state of the game players. This is not just a simple reduction of disgusting elements or an increase of favorite elements, but also the atmosphere of the entire game environment, the interaction method of NPCs, etc. can be adjusted according to the changes in emotions, ensuring that the game players can always maintain a positive gaming experience in different game environments. This solution not only improves the interactivity of the game, but also increases the lasting appeal of the game.

[0137] As an implementable method, the generation logic of game auxiliary information can be: game =f(E player ,G state ,H history ), where E player For the emotions of game players, E playerFor the current game situation, H history For historical game behavior, function f uses emotions, situations, and behaviors to calculate and generate game auxiliary information.

[0138] Compared with the game assisting method provided in one embodiment of the present application, Figure 1 The game assistance method shown in this embodiment also determines reference information associated with the game player, wherein the reference information includes the current game situation of the game application and / or the historical game behavior of the game player, and based on the game player's emotions and the reference information, determines and outputs game assistance information for the game player, so that the determined game assistance information is more in line with the game player and the game situation, thereby enhancing the game player's gaming experience.

[0139] See also Figure 5 , Figure 5 The flowchart of the game assisting method provided by an embodiment of the present application is shown in FIG. Figure 5 The process shown is described in detail. In this embodiment, multimodal data is collected by multiple devices respectively. The game assistance method may specifically include the following steps:

[0140] Step S510: During the running of the game application, time error correction is performed on the multiple devices.

[0141] Optionally, the multimodal data is collected and obtained by multiple devices respectively. For example, the multiple devices may include IOT devices such as smart phones, smart watches, cameras, microphones, etc., and the multimodal data may include physiological data, image data, voice data, game data, etc. Among them, the physiological data of game players, such as heart rate and skin galvanic response, can be monitored and collected by smart watches; the image data of game players, such as facial expressions, can be captured by cameras; the voice signals of game players can be collected by microphones; the game data of game players can be collected by smart phones, etc., which are not limited here.

[0142] Since each modal data comes from different devices and has a collection time deviation, this embodiment can perform time error correction on multiple devices during the running of the game application to ensure that the timestamps of different devices are synchronized.

[0143] In some implementations, a distributed network time protocol (NTP) can be used to synchronize different devices. All acquisition devices correct time errors through the NTP protocol to ensure that timestamps are synchronized at the millisecond level. During data acquisition, the data stream of each device carries a unified timestamp T_s, and synchronization correction is achieved through the following formula:

[0144] T s =T d +ΔT, where Td is the device clock, and ΔT is the offset corrected by NTP.

[0145] In some implementations, a hardware clock synchronization solution can be used to achieve time synchronization directly on the data acquisition device side through hardware calibration. Each device is independently calibrated using a high-precision hardware clock, and the central server performs the final timestamp integration. Although this solution requires additional hardware support and increases the complexity of the system, it can provide a more accurate time synchronization mechanism in a non-network environment and is suitable for scenarios with less reliance on the network.

[0146] Step S520: Acquire the multimodal data collected by multiple devices after time error correction, wherein the multimodal data corresponds to the same collection timestamp.

[0147] In this embodiment, when time error correction is performed on multiple devices, multimodal data collected by multiple devices after time error correction can be obtained. It can be understood that the multimodal data at this time corresponds to the same acquisition timestamp, which can ensure the time synchronization of the multimodal data. That is, this embodiment adopts a unified timestamp management mechanism to ensure that the timestamps of different devices are accurately synchronized to the millisecond level. Through this technology, multimodal data such as audio, visual and physiological signals can be aligned on the same timeline, thereby improving the accuracy of emotion recognition.

[0148] In addition, this embodiment can also adopt a real-time processing architecture, through an efficient I / O processing flow, to process data immediately after collection, avoiding the delay problem in traditional technologies. This solution enables the system to respond to emotional changes in the game instantly and adjust the game strategy in time, significantly improving the real-time performance and the smoothness of the game player experience.

[0149] In addition, edge computing can be introduced to use electronic devices to process some data and perform sentiment analysis locally, reducing reliance on central servers. Electronic devices first perform preliminary processing on the data, and then transmit important feature data back to the server for further in-depth analysis. Although edge computing has weaker capabilities and may not be as good as server solutions in processing complex emotions, it is an effective supplement or alternative for scenarios that require reducing latency and network load (such as gaming experience in low bandwidth or offline states).

[0150] Step S530: extracting features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0151] Step S540: Utilize the cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data.

[0152] Step S550: Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features.

[0153] Step S560: determining the emotion of the game player based on the fusion feature, and determining and outputting game auxiliary information for the game player based on the emotion.

[0154] For the detailed description of step S530 to step S560, please refer to step S120 to step S150, which will not be repeated here.

[0155] Compared with the game assisting method provided in one embodiment of the present application, Figure 1 The game assistance method shown in this embodiment also performs time error correction on multiple devices for the scenario where multimodal data is collected by multiple devices, and obtains multimodal data collected by multiple devices after time error correction, wherein the multimodal data corresponds to the same collection timestamp to ensure that the timestamps of different devices are synchronized, so that different devices collect modal data at the same time, thereby improving the accuracy of emotion recognition of game players.

[0156] See also Figure 6 , Figure 6 The flowchart of the game assisting method provided by an embodiment of the present application is shown in FIG. Figure 6 The process shown is described in detail, and the game assistance method may specifically include the following steps:

[0157] Step S610: Acquire multimodal data of game players during the running of the game application.

[0158] Step S620: extracting features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0159] Step S630: Utilize the cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each of the multimodal data, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data.

[0160] Step S640: Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features.

[0161] For the detailed description of step S610 to step S640, please refer to step S110 to step S140, which will not be repeated here.

[0162] Step S650: Determine the game level of the game player, and determine the emotion of the game player based on the fusion feature.

[0163] In this embodiment, the game level of the game player can be determined.

[0164] In some embodiments, the game data of the game player can be obtained, and the game level of the game player can be determined based on the game data of the game player. Optionally, the game data can include game performance data, game duration data, and game achievement data, wherein the game performance data collects the performance data of the game player in the game through the game engine or server, such as the number of enemies defeated, the number of tasks completed, the scores obtained, the number of skills used, etc., the game duration data can record the total time of the player in the game and the duration of each session, and the game achievement data can record whether the player has completed a specific achievement or milestone, such as obtaining a rare item, reaching a specific score, etc.

[0165] Step S660: Determine and output game assistance information for the game player based on the emotion and the game level.

[0166] In this embodiment, when the game player's emotions and the game player's game level are obtained, game auxiliary information for the game player can be determined and output based on the game player's emotions and the game player's game level, so that the output game auxiliary information is consistent with the game player's emotions and game level, thereby improving the accuracy of the game auxiliary information.

[0167] Compared with the game assisting method provided in one embodiment of the present application, Figure 1 The game assistance method shown in this embodiment also determines the game level of the game player, and determines and outputs game assistance information for the game player based on the game player's emotions and game level, so that the output game assistance information is more in line with the actual situation of the game player, thereby improving the game assistance effect.

[0168] See also Figure 7 , Figure 7 The module block diagram of the game auxiliary device provided by an embodiment of the present application is shown below. Figure 7 As shown in the block diagram, the game auxiliary device 200 includes: a multimodal data acquisition module 210, a feature extraction module 220, a weight determination module 230, a feature fusion module 240 and a game auxiliary information output module 250, wherein:

[0169] The multimodal data acquisition module 210 is used to acquire multimodal data of game players during the running of the game application.

[0170] Furthermore, the multimodal data is collected and obtained by a plurality of devices respectively, and the multimodal data acquisition module 210 includes: a time error correction submodule and a multimodal data acquisition submodule, wherein:

[0171] The time error correction submodule is used to perform time error correction on the multiple devices.

[0172] The multimodal data acquisition submodule is used to acquire the multimodal data collected by multiple devices after time error correction, wherein the multimodal data corresponds to the same collection timestamp.

[0173] The feature extraction module 220 is used to extract features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data.

[0174] The weight determination module 230 is used to calculate the correlation between the feature vectors corresponding to each of the multimodal data using the cross-modal self-attention mechanism to obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data.

[0175] The feature fusion module 240 is used to perform feature fusion on the feature vectors corresponding to each of the multimodal data based on the self-attention weight matrix to obtain fused features.

[0176] The game auxiliary information output module 250 is used to determine the emotion of the game player based on the fusion feature, and determine and output game auxiliary information for the game player based on the emotion.

[0177] Furthermore, the game auxiliary information output module 250 includes: a game strategy determination submodule and a game strategy display submodule, wherein:

[0178] A game strategy determination submodule is used to determine a game strategy for the game player based on the emotion, wherein the game strategy includes one or a combination of game difficulty adjustment of the game application, equipment suggestions for the game character corresponding to the game player in the game application, and skill adjustment of the game character corresponding to the game player in the game application.

[0179] The game strategy display submodule is used to display the game strategy.

[0180] Furthermore, the game strategy display submodule includes: a first game strategy display unit and a second game strategy display unit, wherein:

[0181] The first game strategy display unit is used to display the game strategy in the game interface of the game application during the process of the game player controlling the game character.

[0182] The second game strategy display unit is used to display the game strategy in the game interface of the game application when the game character of the own side is killed and waiting for resurrection.

[0183] Furthermore, the game auxiliary information output module 250 includes: a target text generation submodule and a target text display submodule, wherein:

[0184] The target text generation submodule is used to generate a target text based on the emotion, wherein the target text includes text content for encouraging or praising the game player.

[0185] The target text display submodule is used to display the target text.

[0186] Furthermore, the game auxiliary information output module 250 includes: a target audio acquisition submodule and a target audio playback submodule, wherein:

[0187] The target audio acquisition submodule is used to convert the target text into a target audio corresponding to the voice of the game character corresponding to the game player in the game application through speech synthesis technology.

[0188] The target audio playing submodule is used to play the target audio.

[0189] Furthermore, the game auxiliary information output module 250 includes: a reference information determination submodule and a first game auxiliary information output submodule, wherein:

[0190] The reference information determination submodule is used to determine reference information associated with the game player, wherein the reference information includes the current game situation of the game application and / or the historical game behavior of the game player.

[0191] The first game auxiliary information output submodule is used to determine and output game auxiliary information for the game player based on the emotion and the reference information.

[0192] Furthermore, the game auxiliary information output module 250 includes: an emotion determination submodule and a game auxiliary function activation submodule, wherein:

[0193] The emotion determination submodule is used to determine the emotion of the game player based on the fused features.

[0194] The game assistance function activation submodule is used to activate the game assistance function if the emotion meets the preset emotion, and determine and output game assistance information for the game player based on the emotion, wherein the preset emotion is used to represent that the user thinks it is difficult to win the game.

[0195] Furthermore, the game auxiliary information output module 250 includes: a game level determination submodule and a second game auxiliary information output submodule, wherein:

[0196] The game level determination submodule is used to determine the game level of the game player and determine the emotion of the game player based on the fusion feature.

[0197] The second game auxiliary information output submodule is used to determine and output game auxiliary information for the game player based on the emotion and the game level.

[0198] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0199] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0200] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0201] See also Figure 8 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0202] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0203] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0204] In some embodiments, the electronic device 100 may further include a touch screen, wherein the touch screen is used to display information input by a user, information provided to a user, and various graphical user interfaces of the electronic device 100, and these graphical user interfaces may be composed of graphics, text, icons, numbers, videos, and any combination thereof. In one example, the touch screen may be a liquid crystal display (LCD) or an organic light-emitting diode (OLED), which is not limited here.

[0205] In some implementations, the electronic device 100 may further include a camera for collecting the user's motion data. Optionally, the camera may include a front camera, a rear camera, a telescopic camera, a rotating camera, etc., which are not limited here.

[0206] In some embodiments, the electronic device 100 may further include a sensor, wherein the sensor may include a light sensor, which may be used to turn off the display of the touch screen when an object approaches the touch screen, for example, when the main body of the electronic device moves to the ear. The sensor may include a pressure sensor, which may be used to detect the pressure generated by pressing on the electronic device, that is, the pressure sensor may detect the pressure generated by the contact or pressing between the user and the electronic device, such as the pressure generated by the contact or pressing between the user's ear or finger and the electronic device. The sensor may include a gravity sensor, which is used to detect the magnitude of acceleration in each direction (generally three axes), and the magnitude and direction of gravity can be detected when stationary, which can be used for applications that identify the posture of the electronic device (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. In addition, the electronic device may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, etc., which are not limited here.

[0207] See also Fig. 9 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0208] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0209] In some implementations, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the game assistance method in the embodiment of the present application is implemented.

[0210] In summary, the game assistance method, device, electronic device, storage medium and program product provided by the embodiments of the present application, in the process of running the game application, first obtain the multimodal data of the game player, perform feature extraction on the multimodal data, and obtain the feature vectors corresponding to each multimodal data, and then use the cross-modal self-attention mechanism to calculate the correlation between the feature vectors corresponding to each multimodal data, and obtain the self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data, and based on the self-attention weight matrix, the feature vectors corresponding to each multimodal data are feature fused to obtain fused features, and finally the game player's emotions are determined based on the fused features, and game assistance information for the game player is determined and output based on the emotions, thereby achieving efficient fusion of different modal data by introducing the cross-modal self-attention mechanism, thereby improving the accuracy of emotion prediction and the effect of game assistance.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A game assisting method, characterized in that: The method comprises: In the process of running the game application, obtain multimodal data of game players; Performing feature extraction on the multimodal data to obtain feature vectors corresponding to each of the multimodal data; Calculate the correlation between the feature vectors corresponding to each of the multimodal data using a cross-modal self-attention mechanism to obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data; Based on the self-attention weight matrix, feature fusion is performed on the feature vectors corresponding to each of the multimodal data to obtain fused features; The emotion of the game player is determined based on the fused feature, and game assistance information for the game player is determined and output based on the emotion.

2. The method according to claim 1, characterized in that The determining and outputting game auxiliary information for the game player based on the emotion comprises: Based on the emotion, determining a game strategy for the game player, wherein the game strategy includes one or a combination of game difficulty adjustment of the game application, outfit suggestions for the game character corresponding to the game player in the game application, and skill adjustment of the game character corresponding to the game player in the game application; Displays the game strategy.

3. The method according to claim 2, characterized in that The displaying of the game strategy comprises: During the process of the game player controlling the game character, displaying the game strategy in the game interface of the game application; or When the game character of the own side is killed and waiting for resurrection, the game strategy is displayed in the game interface of the game application.

4. The method according to claim 1, characterized in that The determining and outputting game auxiliary information for the game player based on the emotion comprises: generating a target text based on the emotion, wherein the target text includes text content for encouraging or praising the game player; The target text is displayed.

5. The method according to claim 4, characterized in that After generating the target text based on the emotion, the method further includes: By using speech synthesis technology, the target text is converted into a target audio corresponding to the voice of the game character corresponding to the game player in the game application; Play the target audio.

6. The method according to any one of claims 1 to 5, characterized in that: The determining and outputting game auxiliary information for the game player based on the emotion comprises: Determining reference information associated with the game player, wherein the reference information includes a current game situation of the game application and / or a historical game behavior of the game player; Based on the emotion and the reference information, game assistance information for the game player is determined and output.

7. The method according to any one of claims 1 to 5, characterized in that: The multimodal data is collected by a plurality of devices respectively, and the obtaining of the multimodal data of the game player includes: performing time error correction on the plurality of devices; The multimodal data collected by the multiple devices after time error correction is obtained, wherein the multimodal data corresponds to the same collection timestamp.

8. The method according to any one of claims 1 to 5, characterized in that: The determining the emotion of the game player based on the fusion feature, and determining and outputting game auxiliary information for the game player based on the emotion, comprises: determining the game player's emotion based on the fused feature; If the emotion satisfies a preset emotion, a game assistance function is activated, and game assistance information for the game player is determined and output based on the emotion, wherein the preset emotion is used to represent that the user believes it is difficult to win the game.

9. The method according to any one of claims 1 to 5, characterized in that: The determining the emotion of the game player based on the fusion feature, and determining and outputting game auxiliary information for the game player based on the emotion, comprises: determining a game level of the game player, and determining an emotion of the game player based on the fused feature; Game assistance information for the game player is determined and output based on the emotion and the game level.

10. A game assisting device, characterized in that: The device comprises: A multimodal data acquisition module is used to acquire multimodal data of game players during the running of the game application; A feature extraction module, used to extract features from the multimodal data to obtain feature vectors corresponding to each of the multimodal data; A weight determination module, configured to calculate the correlation between the feature vectors corresponding to each of the multimodal data using a cross-modal self-attention mechanism, and obtain a self-attention weight matrix, wherein each self-attention weight in the self-attention weight matrix represents the degree of attention of one modal data to other modal data; A feature fusion module, used for performing feature fusion on the feature vectors corresponding to each of the multimodal data based on the self-attention weight matrix to obtain fused features; The game auxiliary information output module is used to determine the emotion of the game player based on the fusion feature, and determine and output game auxiliary information for the game player based on the emotion.

11. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images