Emotion recognition method and system based on multimodal perception data

Abstract

The present invention provides an emotion recognition method and system based on multimodal perception data, which relates to the field of virtual reality technology. The multimodal VR perception data is finely processed through a VR interactive emotion perception model, and the summary and detail emotion expression vectors are effectively extracted and simplified, which significantly reduces the complexity of data processing while retaining key emotion information. By using the pre-learned summary and detail to guide the emotion expression vector, and combining the feature self-focusing strategy, the present invention can accurately output the expression vector that integrates the self-focusing feature, thereby improving the pertinence and accuracy of emotion recognition. In addition, through the feature interactive focusing strategy, the present invention further explores the deep correlation between the self-focusing expression vector and the summary and detail emotion expression vector sets, thereby more comprehensively predicting the user emotion perception results in VR interaction. As a result, the present invention significantly improves the efficiency and accuracy of emotion recognition in multimodal situations.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to an emotion recognition method and system based on multimodal perception data. Background Art

[0002] With the rapid development of virtual reality (VR) technology, VR interactive experience has been widely used in entertainment, education, medical and other fields. In the process of VR interaction, the user's emotional state is crucial to evaluate the experience effect, optimize the interactive design and improve the user experience. However, it is not easy to accurately identify the user's emotions in VR interaction, because emotional expression involves information from multiple modalities, including physiological reactions (such as heart rate changes, skin galvanic response), behavioral movements and facial expressions.

[0003] Traditional emotion recognition methods often rely on single-modal data, such as inferring emotional states only through facial expression recognition or physiological signal analysis. This method often has difficulty in obtaining accurate and reliable recognition results in complex and changeable VR interactive scenarios. In order to more comprehensively capture the user's emotional information, multimodal emotion recognition methods have gradually become a research hotspot in recent years. This type of method attempts to fuse multimodal data from different sensors to provide richer and more accurate basis for emotion recognition.

[0004] However, the fusion and processing of multimodal data faces many challenges. First, data of different modalities may have different feature dimensions and spatiotemporal scales. How to effectively extract and integrate these features is a key issue. Secondly, emotional expressions in VR interactive scenes are often complex and diverse. How to accurately capture and distinguish different emotional states is also a major problem. In addition, with the continuous advancement of VR technology, users' requirements for experience are increasing, and traditional emotion recognition methods can no longer meet actual needs in terms of processing efficiency and accuracy. Summary of the invention

[0005] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides an emotion recognition method based on multimodal perception data, the method comprising:

[0006] According to the VR interactive emotion perception model, feature extraction is performed on each path space perception segment in the target multimodal VR perception data to generate corresponding summary emotion expression vectors and detail emotion expression vectors, and according to the vector correlation, the generated summary emotion expression vector distribution and detail emotion expression vector distribution are simplified to generate summary emotion expression vector sets and detail emotion expression vector sets of set feature scales, wherein the target multimodal VR perception data includes user feature data perceived by a heart rate monitor, a galvanic skin response sensor, a motion capture device, and a facial expression recognition camera in a VR interactive scene;

[0007] After integrating the pre-learned summary-guided emotion expression vector and the detail-guided emotion expression vector, the corresponding self-focusing expression vector is output through the feature self-focusing strategy; the summary-guided emotion expression vector and the detail-guided emotion expression vector are obtained by learning the corresponding sample-guided emotion expression vector based on the template multimodal VR perception data; the self-focusing expression vector combines the self-focusing feature parts in the summary-guided emotion expression vector and the detail-guided emotion expression vector;

[0008] Through the feature interaction focusing strategy, respectively extracting the first feature interaction focusing vector between the self-focusing expression vector and each summary emotion expression vector in the summary emotion expression vector set, and respectively extracting the second feature interaction focusing vector between the self-focusing expression vector and each detail emotion expression vector in the detail emotion expression vector set;

[0009] Based on the generated first feature interaction focus vectors and second feature interaction focus vectors, a VR interactive emotion perception result of the target multimodal VR perception data is predicted.

[0010] On the other hand, an embodiment of the present invention also provides an emotion recognition system based on multimodal perception data, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0011] Based on the above aspects, the embodiment of the present application uses a VR interactive emotion perception model to perform fine processing on multimodal VR perception data, effectively extracts and simplifies summary and detail emotion expression vectors, significantly reduces the complexity of data processing while retaining key emotion information. By utilizing the pre-learned summary and detail guidance emotion expression vectors, and combining the feature self-focusing strategy, the present invention can accurately output expression vectors that integrate self-focusing features, thereby improving the pertinence and accuracy of emotion recognition. In addition, through the feature interaction focusing strategy, the present invention further explores the deep correlation between the self-focusing expression vector and the summary and detail emotion expression vector sets, thereby more comprehensively predicting the user emotion perception results in VR interaction. As a result, the present invention significantly improves the efficiency and accuracy of emotion recognition in multimodal situations, thereby facilitating the emotional evaluation and optimization of subsequent VR interactive experiences. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of the execution flow of the emotion recognition method based on multimodal perception data provided by an embodiment of the present invention.

[0013] Figure 2 It is a schematic diagram of the hardware architecture of an emotion recognition system based on multimodal perception data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 1 is a flow chart of an emotion recognition method based on multimodal perception data provided by an embodiment of the present invention. The emotion recognition method based on multimodal perception data is introduced in detail below.

[0015] Step S110, based on the VR interactive emotion perception model, feature extraction is performed on each path space perception segment in the target multimodal VR perception data, and corresponding summary emotion expression vectors and detailed emotion expression vectors are generated. Based on the vector correlation, the generated summary emotion expression vector distribution and detailed emotion expression vector distribution are simplified to generate a summary emotion expression vector set and a detailed emotion expression vector set of set feature scales, wherein the target multimodal VR perception data includes user feature data perceived by a heart rate monitor, a galvanic skin response sensor, a motion capture device, and a facial expression recognition camera in a VR interactive scene.

[0016] In this embodiment, for a VR game experience center, taking a user who is experiencing an adventure VR game as an example, the user's VR interactive scene is equipped with a heart rate monitor, a skin galvanic response sensor, a motion capture device, and a facial expression recognition camera. These devices continuously collect the user's feature data to form target multimodal VR perception data.

[0017] When the server receives the target multimodal VR perception data of the user, it starts processing according to the VR interactive emotion perception model. Assuming that there are multiple different areas in this VR game scene, such as the mysterious forest area, the dangerous valley area, etc., the user experience process in each area can be regarded as a path space perception segment.

[0018] For the spatial perception segment of the mysterious forest area, the server uses the feature extraction module in the VR interactive emotion perception model to analyze the data from the heart rate monitor. For example, when a user suddenly hears a mysterious sound in the mysterious forest, the heart rate may rise briefly, and the model will extract features from the data of this heart rate change, which may be related to nervousness. At the same time, the data of the galvanic skin response sensor is also analyzed. When the user feels excited or nervous, the skin conductivity may change, and the model extracts corresponding features from this change. The motion capture device records the user's movements and postures. For example, the user may squat slightly and turn his head more frequently because of vigilance. The model extracts features from these movements and postures. The facial expression recognition camera captures subtle changes in the user's facial expressions, which may be eyes wide open, eyebrows slightly raised, etc., which are also extracted as features.

[0019] Combining these features from different devices, the server generates a summary emotion expression vector corresponding to this spatial perception segment of the trail. This summary emotion expression vector can be regarded as an abstract expression of the user's overall emotional state in the entire mysterious forest area. For example, this vector may indicate that the user is mainly in an emotional state of both curiosity and slight tension in this area.

[0020] At the same time, the server will also generate a detailed emotion expression vector. For the mysterious forest area, according to certain division rules, it may be further subdivided into several perception areas, such as the entrance area, the central area where the mysterious sound comes from, etc. A detailed emotion expression vector will be generated for each perception area. For example, in the central area where the mysterious sound comes from, the user's heart rate may increase more significantly and the facial expression may be more tense. The detailed emotion expression vector generated from these more detailed data can more accurately reflect the user's emotional state in this small area.

[0021] In the process of generating these summary emotion expression vectors and detailed emotion expression vectors, since the amount of data may be very large, simplification will be performed based on the vector correlation. Assume that the feature scale set by the server is 100. When processing the distribution of summary emotion expression vectors, if 200 summary emotion expression vectors are generated within a period of time, the server will simplify them based on the vector correlation. For example, if there are multiple summary emotion expression vectors that all indicate that the user is in a moderate degree of tension, and the path space perception fragments corresponding to these vectors have certain similarities in time or scene, then the server will fuse these vectors and finally generate a set of summary emotion expression vectors with a set feature scale of 100. Similarly, for the distribution of detailed emotion expression vectors, simplification will also be performed in this way to generate a set of detailed emotion expression vectors with a set feature scale.

[0022] Step S120, after integrating the pre-learned summary-guided emotion expression vector and the detail-guided emotion expression vector, the corresponding self-focusing expression vector is output through the feature self-focusing strategy. The summary-guided emotion expression vector and the detail-guided emotion expression vector are obtained by learning the corresponding sample-guided emotion expression vector based on the template multimodal VR perception data. The self-focusing expression vector combines the self-focusing feature parts in the summary-guided emotion expression vector and the detail-guided emotion expression vector.

[0023] In this embodiment, the learning process has been carried out using template multimodal VR perception data. It is assumed that the template multimodal VR perception data is data collected from multiple test users when they experience the same VR game or similar VR scenes. For these template multimodal VR perception data, there are pre-set sample guide emotion expression vectors.

[0024] For example, for some specific scene templates, such as the battle scene template in the game, there are corresponding sample-guided emotional expression vectors, which may represent some typical emotional characteristics that users should show in the battle scene, such as high tension, excitement, etc. Similarly, for some exploration scene templates, there are also corresponding sample-guided emotional expression vectors, which represent the emotional characteristics that users may have when exploring, such as curiosity and relaxation.

[0025] The server uses a specific learning algorithm to learn the corresponding sample guided emotion expression vectors based on these template multimodal VR perception data, thereby obtaining the summary guided emotion expression vector and the detail guided emotion expression vector.

[0026] Taking the adventure VR game mentioned above as an example, the summary guidance emotion expression vector may be a guiding emotion expression for the entire game scene. For example, during the entire game, the user's overall emotion should gradually change from curiosity and expectation to tension and excitement as the game plot progresses. The detail guidance emotion expression vector focuses more on the emotion guidance corresponding to the specific scene elements in the game. For example, in the puzzle solving scene in the game, the user should have an emotional tendency to focus and think.

[0027] After the server obtains these summary-guided emotion expression vectors and detail-guided emotion expression vectors, it will integrate them. This integration operation may be to integrate the information in the summary-guided emotion expression vector and the detail-guided emotion expression vector according to a certain weight distribution. For example, for some important scene elements, the weight of the detail-guided emotion expression vector may be higher, while for the control of the overall game process, the weight of the summary-guided emotion expression vector is relatively high.

[0028] Then, the corresponding self-focus expression vector is output through the feature self-focus strategy. This self-focus expression vector combines the self-focus feature parts in the summary-guided emotional expression vector and the detail-guided emotional expression vector. For example, in this adventure VR game, the self-focus expression vector may highlight those features that have a greater impact on the user's emotions during the game, such as the key plots in the game, high-difficulty challenges, and other corresponding emotional features. These features may be scattered in the summary-guided emotional expression vector and the detail-guided emotional expression vector, but are integrated into the self-focus expression vector through the self-focus strategy.

[0029] Step S130, through the feature interaction focusing strategy, respectively extract the first feature interaction focusing vector between the self-focusing expression vector and each summary emotion expression vector in the summary emotion expression vector set, and respectively extract the second feature interaction focusing vector between the self-focusing expression vector and each detail emotion expression vector in the detail emotion expression vector set.

[0030] In this embodiment, it is assumed that the self-focus expression vector contains feature information that has a key impact on the user's emotions during the entire VR game process, such as the emotional characteristics of high tension and excitement that the user should achieve in the climax of the game (such as the final BOSS battle), and the emotional characteristics of concentration and curiosity that the user should have in the exploration and puzzle-solving parts of the game.

[0031] For each summary emotion expression vector in the summary emotion expression vector set, for example, one of the summary emotion expression vectors represents the overall emotional state of the user in a certain chapter of the game (such as the chapter of exploring the mysterious ruins). This state may be a relatively complex mixed emotion, including curiosity, slight tension, etc. The server uses the feature interaction focusing strategy to extract the first feature interaction focusing vector between the self-focused expression vector and the summary emotion expression vector. This process is like looking for the associated part between the two vectors. For example, in this example, it may focus on the impact of some key elements in the process of exploring the mysterious ruins on the user's emotions, such as how the mysterious symbols or hidden passages in the ruins interact with the curiosity feature in the self-focused expression vector, thereby generating the first feature interaction focusing vector.

[0032] Similarly, for each detail emotion expression vector in the detail emotion expression vector set, assume that one of the detail emotion expression vectors is about the user's emotional state in a specific room (such as the treasure room) in the mysterious ruins, which may be highly excited and curious. The server extracts the second feature interaction focus vector between the self-focus expression vector and the detail emotion expression vector through the feature interaction focus strategy. This process focuses on how specific elements in the treasure room, such as precious treasures or complex mechanisms, are related to the excitement and curiosity features in the self-focus expression vector, thereby generating a second feature interaction focus vector.

[0033] Step S140, predicting the VR interactive emotion perception result of the target multimodal VR perception data according to the generated first feature interaction focus vectors and second feature interaction focus vectors.

[0034] For example, in the adventure VR game scenario mentioned above, the numerous first feature interaction focus vectors and second feature interaction focus vectors generated by the previous steps contain a large amount of emotional feature interaction information about the user in various aspects of the game.

[0035] For the first feature interactive focus vector, it may include the interaction between the user's overall emotions in different chapters and scenes of the game and the key emotional features in the self-focus expression vector. For example, in the early exploration chapter of the game, the first feature interactive focus vector shows that the user's curiosity emotion is highly matched with the expected curiosity emotion feature in the self-focus expression vector, but in the mid-game battle chapter, the user's nervousness is lower than the expected nervousness in the self-focus expression vector.

[0036] For the second feature interactive focus vector, for example, in each specific scene room in the game, such as the treasure room, trap room, etc., the second feature interactive focus vector shows the interactive relationship between the user's emotions in these small areas and the relevant emotional features in the self-focus expression vector. In the treasure room, the user's excitement is highly matched with the expected excitement in the self-focus expression vector, but in the trap room, the user's fear is slightly lower than the expected fear in the self-focus expression vector.

[0037] Combining the information of these first feature interaction focus vectors and second feature interaction focus vectors, the server can predict the VR interactive emotional perception results of the target multimodal VR perception data. In this example, the server may deduce the changes in the user's emotional state during the entire VR game experience, such as the emotional state in the early exploration stage of the game is basically in line with expectations, but the tension is insufficient in the battle part, the excitement in scenes such as the treasure room is in line with expectations, and the fear in scenes such as the trap room is slightly weaker. This result can be used for subsequent analysis, such as adjusting the game difficulty, optimizing the game scene design, etc., to better meet the user's emotional experience needs.

[0038] Based on the above steps, the embodiment of the present application uses a VR interactive emotion perception model to perform fine processing on multimodal VR perception data, effectively extracts and simplifies summary and detail emotion expression vectors, significantly reduces the complexity of data processing while retaining key emotion information. By utilizing the pre-learned summary and detail guidance emotion expression vectors, and combining the feature self-focusing strategy, the present invention can accurately output expression vectors that integrate self-focusing features, thereby improving the pertinence and accuracy of emotion recognition. In addition, through the feature interaction focusing strategy, the present invention further explores the deep correlation between the self-focusing expression vector and the summary and detail emotion expression vector sets, thereby more comprehensively predicting the user emotion perception results in VR interaction. As a result, the present invention significantly improves the efficiency and accuracy of emotion recognition in multimodal situations, thereby facilitating the emotional evaluation and optimization of subsequent VR interactive experiences.

[0039] In a possible implementation, the VR interactive emotion perception model includes an emotion feature focus sub-model. Then step S130 includes:

[0040] Step S131, splitting the self-focus expression vector into a first self-focus expression vector corresponding to the summary emotion expression vector and a second self-focus expression vector corresponding to the detail emotion expression vector.

[0041] Step S132, using the first feature interaction focusing unit in the emotional feature focusing submodel, according to the feature interaction focusing strategy, respectively extract the first self-focusing expression vector and the first feature interaction focusing vector between each summary emotional expression vector in the summary emotional expression vector set.

[0042] Step S133, utilizing the second feature interaction focusing unit in the emotion feature focusing submodel, and extracting the second self-focusing expression vector and the second feature interaction focusing vector between each detail emotion expression vector in the detail emotion expression vector set according to the feature interaction focusing strategy.

[0043] In a possible implementation, the emotion feature focusing sub-model includes a plurality of cascaded emotion parsing units. For the first emotion parsing unit, the unit input part includes the summary emotion expression vector set and the detail emotion expression vector set. For other emotion parsing units, the unit input part includes each first feature interaction focusing vector and each second feature interaction focusing vector generated by the previous emotion parsing unit.

[0044] Step S140 includes: predicting the VR interactive emotion perception result of the target multimodal VR perception data according to the first feature interaction focus vectors and the second feature interaction focus vectors generated by the last emotion analysis unit.

[0045] In this embodiment, for the process of extracting relevant vectors separately through the feature interaction focusing strategy, the self-focus expression vector is first split. Taking the adventure VR game mentioned above as an example, the self-focus expression vector contains the key feature information of the user's emotions throughout the game. When splitting, the part corresponding to the summary emotion expression vector is split into the first self-focus expression vector. This first self-focus expression vector focuses on the part related to the summary emotion in the big scene of the game. For example, in the whole adventure game, from a macro perspective, the user's emotional development trend is viewed, such as the overall emotional trend related features from the curious expectation at the beginning of the game to the nervous excitement of the climax of the game. The part corresponding to the detailed emotion expression vector is split into the second self-focus expression vector, which pays more attention to the emotional characteristics of each specific scene element in the game, such as the characteristics related to the focused curiosity emotion that the user should have in a specific puzzle scene.

[0046] Next, the server uses the first feature interaction focusing unit in the emotional feature focusing sub-model to operate according to the feature interaction focusing strategy. For each summary emotional expression vector in the first self-focusing expression vector and the summary emotional expression vector set, it is assumed that there is a summary emotional expression vector in the adventure game that is about the overall emotional state of a certain exploration chapter in the game, which contains the user's curiosity, slight tension and other emotional information in this chapter. The first feature interaction focusing unit will search for the feature interaction part related to this summary emotional expression vector in the first self-focusing expression vector according to the feature interaction focusing strategy. For example, the curious emotional feature in the first self-focusing expression vector about the overall game exploration process is interactively analyzed with the curious emotional part in the summary emotional expression vector of this exploration chapter, and the first feature interaction focusing vector between them is extracted. This vector can reflect the correlation between the curiosity emotional trend of the overall game and the specific curiosity emotional expression of this chapter in this exploration chapter, such as which specific elements in this chapter (such as special landforms, mysterious creatures, etc.) have what impact on the overall curiosity emotional trend.

[0047] At the same time, the server uses the second feature interactive focusing unit in the emotional feature focusing sub-model to operate the second self-focused expression vector and each detail emotional expression vector in the set of detail emotional expression vectors according to the feature interactive focusing strategy. For example, in the game, there is a detail emotional expression vector about the emotional state of the user in a mysterious treasure room in the game, which includes high excitement and curiosity. The second feature interactive focusing unit will analyze the interactive relationship between the emotional features related to the treasure room in the second self-focused expression vector (such as the excitement and curiosity emotional features that should be present in a special scene such as encountering treasure) and this detail emotional expression vector according to the feature interactive focusing strategy, thereby extracting the second feature interactive focusing vector. This vector can reflect the connection between the overall game's emotional expectations for the treasure scene and the actual emotional state of the room in this treasure room scene, such as how the rarity of the treasure, the layout of the room and other elements affect this connection.

[0048] In the case where the emotional feature focusing sub-model contains multiple cascaded emotion parsing units, for the first emotion parsing unit, its input part includes a set of summary emotion expression vectors and a set of detailed emotion expression vectors. Continuing with the example of adventure games, the set of summary emotion expression vectors contains the overall emotion state vectors for each chapter and each large scene of the game, such as the summary emotion expression vectors for the game opening scene, exploration scene, and battle scene. The set of detailed emotion expression vectors contains the emotion state vectors for each specific small scene in the game (such as each room, each special location, etc.). The first emotion parsing unit will perform a preliminary analysis of these input vectors, such as analyzing the relationship between the overall curiosity and expectation emotion in the game opening scene and the detailed emotions corresponding to some special elements in the opening scene (such as the initial mysterious guide), laying the foundation for subsequent emotion analysis.

[0049] For other emotion analysis units, their input part is each first feature interaction focus vector and each second feature interaction focus vector generated by the previous emotion analysis unit. For example, the second emotion analysis unit receives the first feature interaction focus vector and the second feature interaction focus vector output by the first emotion analysis unit. Assume that the first feature interaction focus vector output by the first emotion analysis unit contains information about the overall curiosity emotion in the game opening exploration phase and the emotion interaction of specific elements in the opening scene, and the second feature interaction focus vector output by the second emotion analysis unit contains the emotion interaction information of a special small area in the opening scene (such as the area where the initial guide is located). The second emotion analysis unit will conduct a more in-depth analysis on this basis, and may analyze how this opening emotional interaction affects the emotional development in subsequent plots during the advancement of the game plot, such as how the opening curiosity emotion guides the player into the next exploration scene and affects the emotional state in the next scene.

[0050] When predicting the VR interactive emotion perception result of the target multimodal VR perception data based on the generated first feature interaction focus vectors and second feature interaction focus vectors, the server operates based on the first feature interaction focus vectors and second feature interaction focus vectors generated by the last emotion analysis unit. For example, in an adventure game, if there are three cascaded emotion analysis units, the first feature interaction focus vector generated by the third (last) emotion analysis unit may include the final overall emotion interaction relationship of each major chapter in the game (such as the exploration chapter, the battle chapter, etc.) after the previous emotion analysis units have analyzed it layer by layer, such as the actual nervous excitement of the player in the battle chapter and the final correlation relationship of the game's overall emotional expectations for the battle chapter after the previous analysis. Similarly, the second feature interaction focus vector generated by the third emotion analysis unit includes the final emotion interaction relationship of each specific small scene in the game (such as each key room, key battle location, etc.) after layer by layer analysis, such as the final correlation relationship between the player's high nervousness and the game's overall emotional expectations for this scene in the room of the final BOSS battle. The server combines the first feature interaction focus vector and the second feature interaction focus vector generated by the last emotion analysis unit to predict the VR interactive emotion perception result of the target multimodal VR perception data. This result can accurately reflect the emotional state of the player throughout the adventure game, such as in which scenes the player's emotions meet the game expectations and in which scenes the emotions deviate, and can provide a basis for game improvements (such as adjusting the scene difficulty, optimizing scene elements, etc.) to better meet the player's emotional experience needs.

[0051] In a possible implementation, the VR interactive emotion perception model includes an emotion understanding sub-model. Step S140 may also include:

[0052] Step S141: After integrating the first feature interaction focus vectors and the second feature interaction focus vectors, an emotional knowledge space matching the VR scene semantic dimension of the target multimodal VR perception data is mapped and output.

[0053] Step S142, taking the VR scene vector and the interactive task vector of the target multimodal VR perception data as emotion perception guidance information, and loading them into the emotion understanding sub-model in combination with the emotion knowledge space, so as to generate a VR interactive emotion perception result of the target multimodal VR perception data.

[0054] In this embodiment, first, each first feature interaction focus vector and each second feature interaction focus vector obtained in the previous step are integrated. Taking the adventure VR game mentioned above as an example, the first feature interaction focus vector contains information about the expected interaction relationship between user emotions and overall emotions in each chapter and large scene of the game, such as the interaction relationship between the user's overall curiosity emotion and the expected curiosity emotion of the chapter in the exploration chapter of the game, including which elements affect this interaction during the exploration process, such as the influence of special terrain, mysterious creatures and other elements on the interaction of curiosity emotions. The second feature interaction focus vector contains information about the interaction relationship between user emotions and expected emotions in each specific small scene of the game, such as the interaction relationship between the user's excitement and curiosity emotion and the expected excitement and curiosity emotion of the scene in the treasure room in the game, involving the influence of factors such as the value of the treasure and the layout of the room on this interaction. After integrating these first feature interaction focus vectors and second feature interaction focus vectors, the server maps and outputs the emotional knowledge space according to the VR scene semantic dimension of the target multimodal VR perception data. Under the semantic dimension of the adventure game scene, this emotional knowledge space covers the knowledge representation of various scene elements, plot development and user emotional interaction throughout the game from the beginning to the end. For example, at the beginning of the game, based on the semantics of the opening scene (such as the initial mysterious atmosphere, simple exploration guidance, etc.), the emotional knowledge space shows the curiosity and expectation that the user should have, as well as the relationship between this emotion and the elements of the opening scene; in the battle scene of the game, based on the semantics of the battle scene (such as the strength of the enemy, the terrain of the battle scene, etc.), the emotional knowledge space shows the nervousness and excitement that the user should have, as well as the relationship between this emotion and the elements of the battle scene.

[0055] Then, the server obtains the VR scene vector and interactive task vector of the target multimodal VR perception data as the emotion perception guidance information. In adventure games, the VR scene vector contains various attribute information of the game scene, such as the type of scene (exploration scene, battle scene or puzzle scene, etc.), the complexity of the scene (the complexity of the terrain, the difficulty of the puzzle, etc.), and the distribution of elements in the scene (such as the location of the treasure, the distribution of the enemy, etc.). The interactive task vector represents the user's task goal information in the game. For example, in an exploration chapter, the task goal may be to find a specific item or explore a specific area; in a battle scene, the task goal may be to defeat a specific enemy or complete a specific battle task, etc. The server uses these VR scene vectors and interactive task vectors as emotion perception guidance information, and loads them into the emotion understanding sub-model in combination with the previously generated emotional knowledge space. The emotion understanding sub-model processes these input information to generate VR interactive emotion perception results of the target multimodal VR perception data. For example, in a mixed scene containing puzzle solving and combat, the VR scene vector indicates that the scene is highly complex, with complex puzzles and powerful enemies, and the interactive task vector indicates that the user needs to solve the puzzle first and then defeat the enemy. The emotional knowledge space shows that in this scenario, the user should have high tension and concentration, and successful puzzle solving and battle victory will bring different degrees of excitement. The emotional understanding sub-model combines this information and comprehensively analyzes the user's actual emotional state in this scenario. For example, the user may feel highly nervous and focused during the puzzle solving process due to the high difficulty of the puzzle, and feel excited after successfully solving the puzzle, but the tension is further increased when facing a powerful enemy, etc. This is the final generated VR interactive emotional perception result. This result can provide a basis for game optimization, such as adjusting the scene difficulty and optimizing the task objectives according to the user's actual emotional state, so as to enhance the user's gaming experience.

[0056] In a possible implementation, each detail emotion expression vector corresponds to a perception area in the perception segment of the path space. Step S110 includes:

[0057] Step S111, performing feature extraction on each path space perception segment, adding the summary emotion expression vector of the path space perception segment to the summary cache module, and adding the detail emotion expression vector of the path space perception segment to the detail cache module.

[0058] Step S112, when the number of vectors cached in the summary cache module reaches the set feature scale, the summary emotion expression vectors of multiple path space perception fragments in the summary cache module are merged according to vector association, and the summary emotion expression vector to be cached is added to the summary cache module.

[0059] And, step S113, for each perception area, when the number of corresponding detail emotion expression vectors cached in the detail cache module reaches the set feature scale, based on the vector correlation, the corresponding detail emotion expression vectors of multiple path space perception fragments in the detail cache module are fused, and the detail emotion expression vectors to be cached are added to the detail cache module.

[0060] In this embodiment, the server performs feature extraction for each path space perception segment. In this adventure game, the path space perception segment can be different game process parts of the player in the game, such as the exploration process starting from entering the initial area of ​​the game, the battle process when encountering the first enemy, the process of entering the mysterious ruins to solve puzzles, etc. When feature extraction is performed on these path space perception segments, corresponding summary emotion expression vectors and detailed emotion expression vectors are generated.

[0061] Taking the exploration process of the player entering the initial area of ​​the game as an example, the server obtains the player's heart rate from the heart rate monitor, which is relatively stable. The skin galvanic response sensor shows that the skin conductivity is in the normal range. The motion capture device records the player's movements as walking and looking around in a relatively relaxed manner. The facial expression recognition camera captures the player's facial expression as a relatively relaxed state. Combining the data of these devices, the server generates a summary emotion expression vector for this path space perception segment. This vector reflects that the player is in a relaxed and curious emotional state as a whole in this initial exploration stage. At the same time, this path space perception segment is further subdivided according to the perception area, such as dividing the initial area into the entrance area and the surrounding environment viewing area. For the perception area of ​​the entrance area, the server generates the corresponding detailed emotion expression vector, which more accurately reflects the player's emotional state in the entrance area. For example, the unique decoration at the entrance may make the player's curiosity stronger. Then, the server adds the summary emotion expression vector of this path space perception segment to the summary cache module and the detailed emotion expression vector to the detail cache module.

[0062] As the player progresses in the game, more path space perception fragments will be processed and their corresponding vectors will be added to the cache module. When the number of vectors cached in the summary cache module reaches the set feature scale, the server will fuse the summary emotion expression vectors of multiple path space perception fragments in the summary cache module according to the vector relevance and add new vectors. Assuming that the feature scale is set to 100, when 100 summary emotion expression vectors have been cached in the summary cache module, for example, some of the summary emotion expression vectors come from the emotional state vectors of the player after different exploration chapters and different small battle scenes. There may be correlations between these vectors. For example, in several consecutive exploration chapters, the player's emotional state is mainly curiosity, but the degree of curiosity and other accompanying emotions are slightly different. The server will fuse these summary emotion expression vectors representing similar emotional states based on vector relevance. Specifically, for the part representing the curious emotion, the server will integrate the intensity and duration of the curious emotion in each emotion expression vector, and fuse it into a summary emotion expression vector that better represents the overall curious emotional state of these similar path space perception fragments. Then, when there is a summary emotion expression vector of a new path space perception segment that needs to be cached, it is added to the summary cache module.

[0063] Similarly, for the detail cache module, for each perception area, when the number of cached corresponding detail emotion expression vectors reaches the set feature scale, a similar fusion operation will be performed. For example, in the game, different small rooms in multiple mysterious ruins are regarded as different perception areas. When the number of detail emotion expression vectors corresponding to these small rooms reaches the set scale, take one of the perception areas as an example, assuming that it is the treasure room perception area in the mysterious ruins. In different game processes, players may enter this treasure room many times, and the detail emotion expression vectors at each entry may be different, but there is a certain correlation. The server fuses these detail emotion expression vectors of different entries into the treasure room based on the vector correlation. It is possible that during one entry, the player's excitement is very high because a special treasure has appeared in the treasure room; while during another entry, although there is no new treasure, the player's curiosity has increased because the arrangement of the previous treasure has changed. The server will combine these factors and merge the relevant parts of the detailed emotion expression vectors of multiple entries into the treasure room about emotions such as excitement and curiosity to generate a detailed emotion expression vector that can better represent the overall emotional state of the treasure room perception area, and then add the new detailed emotion expression vector to be cached to the detail cache module.

[0064] This fusion and caching operation based on vector relevance can effectively reduce the amount of data while retaining key emotional information. Throughout the adventure game, as players continue to play the game, the server continues to process the summary emotional expression vector and the detail emotional expression vector in this way, ensuring that the vector sets in the summary cache module and the detail cache module can accurately reflect the player's emotional state and provide an effective data basis for subsequent operations such as the prediction of VR interactive emotional perception results within a controllable data scale. For example, in the later stages of the game, when further emotional analysis is required based on these cached vector sets, these simplified and fused vector sets can quickly and accurately provide players with emotional state information at various stages and areas of the game, without inefficient processing due to excessive data volume or loss of key emotional information due to oversimplification.

[0065] In a possible implementation manner, before step S130, the method further includes:

[0066] Step A110, when it is detected that the set perception node is currently reached, extracting the summary emotion expression vector set cached in the summary cache module, and extracting the detail emotion expression vector set cached in the detail cache module.

[0067] And, the step of fusing the target emotion expression vectors of multiple path space perception segments comprises:

[0068] According to the vector relevance, the target emotion expression vectors of the most matching associated path space perception fragments in the cache module are merged.

[0069] According to the vector association, the target emotion expression vector to be cached and the target emotion expression vector cached in the cache module are clustered, and the target emotion expression vectors of each path space perception segment that have completed the clustering are evenly converged. Wherein, the target emotion expression vector is a summary emotion expression vector or a detail emotion expression vector, and correspondingly, the cache module is a summary cache module or a detail cache module.

[0070] In a possible implementation, if there are multiple most matching associated path space perception segments, merging the target emotion expression vectors of the most matching associated path space perception segments in the cache module includes:

[0071] The target emotion expression vectors of the associated path space perception segments with the longest cache duration and the best match in the cache module are merged.

[0072] The target emotion expression vectors that best match each group of associated path space perception segments in the cache module are balanced and converged respectively.

[0073] In this embodiment, when the server detects that the set perception node is currently reached, it starts to extract the set of summary emotion expression vectors cached in the summary cache module and the set of detail emotion expression vectors cached in the detail cache module. For example, in this adventure game scene, the set perception node can be a specific plot development point in the game, such as when the player completes a large task, reaches a specific game area, or passes a certain game time. Take the completion of a large puzzle task in the game as an example of setting the perception node, and the server starts the operation at this time. The set of summary emotion expression vectors cached in the summary cache module contains the summary emotion expression vectors of each path space perception fragment from the start of the game to the completion of the puzzle task. For example, in the exploration stage at the beginning of the game, the player's summary emotion expression vector may show that the player is in an overall state of curiosity and expectation; when encountering some small battle scenes, the summary emotion expression vector reflects that the player has a certain degree of tension and excitement; in the process of approaching this large puzzle task, the summary emotion expression vector reflects the player's focused and thinking emotional state. The server extracts the summary emotion expression vectors of these different path space perception fragments from the summary cache module.

[0074] At the same time, the server extracts a set of detail emotion expression vectors from the detail cache module. The detail emotion expression vectors in the detail cache module correspond to the perception areas in each path space perception segment. For example, in the entrance area of ​​the game's opening exploration phase, the detail emotion expression vector may indicate that the player has a stronger curiosity due to the unique visual effect of the entrance; in a specific combat area in a small combat scene, the detail emotion expression vector reflects the player's high tension in the area due to the enemy's special attack method, etc.

[0075] Then, there is a step of fusing the target emotion expression vectors of the multiple path space perception segments. The target emotion expression vector may be a summary emotion expression vector or a detail emotion expression vector, and the corresponding cache module is a summary cache module or a detail cache module.

[0076] For the operation of merging the target emotion expression vectors of the most matching associated path space perception fragments in the cache module according to vector relevance, take the summary emotion expression vector as an example. In an adventure game, it is assumed that there are some associated path space perception fragments, such as several consecutive exploration chapters in the game, which are similar in terms of game scene type, task objectives, player's emotional trends, etc. The server determines that these are the most matching associated path space perception fragments based on vector relevance. If there are multiple such associated path space perception fragments, in one of the ways, the server will merge the target emotion expression vectors of the associated path space perception fragments with the longest cache duration and the most matching in the cache module. For example, among these consecutive exploration chapters, there is a chapter in which the player explores the longest time, and the summary emotion expression vector in this chapter contains the player's emotional state information over a long period of time, such as the duration of curiosity, intensity changes, etc. The server merges the summary emotion expression vector of this chapter with the summary emotion expression vectors of other similar exploration chapters. During the merging process, for the curiosity emotion part of the emotional state, the curiosity emotion intensity, change trend and other factors in each vector will be integrated to generate a summary emotional expression vector that can better represent the overall curiosity emotion state of these related path space perception fragments.

[0077] Or in another way, the target emotional expression vectors that best match each group of associated path space perception fragments in the cache module are balanced and converged. For example, there are two different groups of associated path space perception fragments in the game, one group is several consecutive small battle scenes, and the other group is several consecutive puzzle scenes. For the group of small battle scenes, the server will balance the summary emotional expression vectors of each small battle scene in this group. In the process of convergence, the intensity and duration of the player's nervous and excited emotions in each battle scene are considered, and balanced processing is performed through a certain algorithm, so that the final converged summary emotional expression vector can accurately reflect the overall emotional state of this group of small battle scenes. The same operation is performed for the group of puzzle scenes, and the summary emotional expression vectors of each puzzle scene are balanced and converged to generate a summary emotional expression vector that can represent the overall emotional state of this group of puzzle scenes.

[0078] At the same time, the target emotion expression vectors to be cached and the target emotion expression vectors cached in the cache module are clustered according to the vector relevance, and the target emotion expression vectors of each path space perception segment that has completed the clustering are evenly converged. Taking the detail emotion expression vector as an example, in the game, it is assumed that some new detail emotion expression vectors to be cached are generated, for example, the player enters a new game area, this area contains multiple perception areas, and each perception area has a corresponding new detail emotion expression vector. The server clusters these new detail emotion expression vectors to be cached and the detail emotion expression vectors cached in the detail cache module according to the vector relevance. For example, the detail emotion expression vectors related to the exploration perception area are divided into one cluster, and the detail emotion expression vectors related to the combat perception area are divided into another cluster. For the cluster of the exploration perception area, the detail emotion expression vectors of each path space perception segment (here, each exploration perception area) that has completed the clustering are evenly converged. In this process, the intensity and duration of the emotions such as curiosity and concentration of the player in each exploration perception area are comprehensively considered, and the algorithm is used for balancing processing, so that the final converged detail emotion expression vector can accurately reflect the overall emotional state of this cluster of exploration perception areas. The same operation is performed for the cluster of combat perception areas, thereby completing the fusion operation of the detailed emotion expression vector. This fusion operation is similar to the process for the summary emotion expression vector, except that it corresponds to the vector in the summary cache module.

[0079] Through these operations, the cached emotional expression vectors can be effectively integrated and optimized before performing operations related to the feature interaction focus strategy, laying the foundation for the subsequent accurate extraction of the first feature interaction focus vector and the second feature interaction focus vector, thereby more accurately predicting the VR interactive emotional perception results and providing more effective data support for optimizing the game experience. For example, in the subsequent analysis of the emotional development trend of players throughout the game, these fused and optimized vectors can more accurately reflect the emotional state of players, allowing game developers to adjust game plots, difficulty and other elements based on these results to better meet the emotional needs of players.

[0080] In a possible implementation, the method further includes:

[0081] Step S101, based on the initialized VR interactive emotion perception model, feature extraction is performed on each path space perception segment in the template multimodal VR perception data to generate corresponding summary emotion expression vectors and detailed emotion expression vectors, and based on the vector correlation, the generated summary emotion expression vector distribution and detailed emotion expression vector distribution are simplified to generate a summary emotion expression vector set and a detailed emotion expression vector set of a set feature scale.

[0082] Step S102: After integrating the initialized first sample guided emotion expression vector and the second sample guided emotion expression vector, a corresponding self-focus expression vector is output through a feature self-focus strategy. The self-focus expression vector combines the self-focus feature parts in the first sample guided emotion expression vector and the second sample guided emotion expression vector.

[0083] Step S103, through the feature interaction focusing strategy, respectively extracting the first feature interaction focusing vector between the self-focusing expression vector and each summary emotion expression vector in the summary emotion expression vector set, and the second feature interaction focusing vector between the self-focusing expression vector and each detail emotion expression vector in the detail emotion expression vector set.

[0084] Step S104, based on the VR interactive emotion perception results and emotional semantic expression information of the template multimodal VR perception data predicted according to the generated first feature interaction focus vectors and second feature interaction focus vectors, and the error between the corresponding training supervision data, the VR interactive emotion perception model is trained, and the trained VR interactive emotion perception model is used to obtain the VR interactive emotion perception results of the target multimodal VR perception data.

[0085] In a possible implementation, the VR interactive emotion perception model includes an emotion understanding sub-model and an emotion feature focusing sub-model implemented according to a feature self-focusing strategy and a feature interaction focusing strategy, and the emotion feature focusing sub-model is used to extract the first feature interaction focusing vector and the second feature interaction focusing vector. Then, based on the generated first feature interaction focusing vectors and second feature interaction focusing vectors, the VR interactive emotion perception results and emotion semantic expression information of the template multimodal VR perception data are predicted, including:

[0086] At the first model learning node, after integrating the first feature interaction focus vectors and the second feature interaction focus vectors, an emotional knowledge space matching the VR scene semantic dimension of the template multimodal VR perception data is mapped and output. The first interactive task vector is used as emotional perception guidance information and loaded into the emotional understanding sub-model in combination with the emotional knowledge space to generate emotional semantic expression information of the template multimodal VR perception data.

[0087] At the second model learning node, after integrating the first feature interaction focus vector and the second feature interaction focus vector, an emotional knowledge space matching the VR scene semantic dimension of the template multimodal VR perception data is mapped and output.

[0088] The VR scene vector and the second interactive task vector are used as emotion perception guidance information, combined with the emotion knowledge space and loaded into the emotion understanding sub-model to generate a VR interactive emotion perception result of the template multimodal VR perception data.

[0089] Wherein, step S104 includes:

[0090] Step S1041: At the first model learning node, the emotion feature focusing sub-model is trained based on the error between the predicted emotion semantic expression information and the emotion semantic expression information in the training supervision data.

[0091] Step S1042: At the second model learning node, the emotion feature focusing sub-model and the emotion understanding sub-model are trained based on the error between the predicted VR interaction emotion perception result and the VR interaction emotion perception result in the training supervision data.

[0092] In this embodiment, for example, in this adventure game, the template multimodal VR perception data can come from the data of multiple test players when experiencing this game. For example, the test players perceive each path space segment in the game, such as the exploration process of entering the initial area of ​​the game, the battle process of encountering different enemies, the puzzle-solving process of exploring mysterious ruins, etc. For each path space perception segment, the server extracts features from the data perceived by devices including heart rate monitors, galvanic skin response sensors, motion capture devices, facial expression recognition cameras, etc.

[0093] Taking the exploration process of entering the initial area of ​​the game as an example, the server analyzes the heart rate monitor data. If the player's heart rate is stable, this may indicate that the player is in a relaxed state; the skin galvanic response sensor data shows normal skin conductivity, which also implies a relatively calm mood; the motion capture device records the player's relaxed walking and looking around; the facial expression recognition camera captures the player's relaxed facial expression. Combining this information, the server generates a summary emotion expression vector for this path space perception segment, which reflects that the player is in a relaxed and curious emotional state as a whole in this initial exploration stage. At the same time, this path space perception segment is further subdivided according to the perception area, such as the entrance area, the surrounding environment viewing area, etc. For the perception area of ​​the entrance area, the server generates the corresponding detailed emotion expression vector, which more accurately reflects the player's curious emotional state when in the entrance area, which may be because the unique decoration at the entrance makes the curiosity stronger.

[0094] As each path space perception segment is processed, a large number of summary emotion expression vectors and detailed emotion expression vectors will be generated. Since the amount of data may be large, it is necessary to simplify the summary emotion expression vector distribution and the detailed emotion expression vector distribution according to the vector correlation, and generate a summary emotion expression vector set and a detailed emotion expression vector set with a set feature scale. Assuming that the feature scale is set to 100, when processing the summary emotion expression vector distribution, if more than 100 summary emotion expression vectors are generated, for example, a large number of summary emotion expression vectors are generated in the exploration chapters, battle scenes and other path space perception segments in different game processes of multiple test players. The server simplifies according to the vector correlation and merges those summary emotion expression vectors that represent similar emotional states. For example, in the initial exploration stage of multiple players, they are all in a relaxed and curious emotional state. Although there are slight differences in the degree of curiosity and other accompanying emotions, the server combines these factors and merges these summary emotion expression vectors into vectors that can represent the overall emotional state, and finally generates a summary emotion expression vector set with a set feature scale of 100. A similar operation is performed for the distribution of detail emotion expression vectors. For each perception area, the relevant detail emotion expression vectors are fused and simplified to generate a set of detail emotion expression vectors with a set feature scale.

[0095] Next, the server integrates the initialized first sample guided emotion expression vector and the second sample guided emotion expression vector. In the adventure game scenario, the first sample guided emotion expression vector may be a guided emotion expression for the overall process of the game. For example, from the beginning to the end of the game, the overall emotion of the player should develop according to a certain trend, such as the vector related to the emotional development path from curiosity and expectation to tension and excitement and finally satisfaction. The second sample guided emotion expression vector focuses more on the emotional guidance corresponding to the specific scene elements in the game, such as the vector related to the emotional tendency of players to focus and think in a specific puzzle scene. The server integrates the two sample guided emotion expression vectors according to a certain weight distribution and other integration methods, and then outputs the corresponding self-focus expression vector through the feature self-focus strategy. This self-focus expression vector combines the self-focus feature parts in the first sample guided emotion expression vector and the second sample guided emotion expression vector. For example, the features that have a greater impact on the player's emotions in scenes such as key plots and high-difficulty challenges in the game may be scattered in the first sample guided emotion expression vector and the second sample guided emotion expression vector, but are integrated into the self-focus expression vector through integration and self-focus strategies.

[0096] Afterwards, the server extracts the first feature interactive focusing vector between the self-focused expression vector and each summary emotion expression vector in the summary emotion expression vector set, and the second feature interactive focusing vector between the self-focused expression vector and each detail emotion expression vector in the detail emotion expression vector set through the feature interactive focusing strategy. Taking the first feature interactive focusing vector as an example, for a summary emotion expression vector in the summary emotion expression vector set, for example, a vector about the overall emotional state of the player in a battle scene in the game, this vector contains the emotional information such as tension and excitement of the player in the battle. The self-focused expression vector of the server contains the emotional features corresponding to the key plots under the overall process of the game, such as the features related to the emotional expectations of the overall game for this battle scene in this battle scene. Through the feature interactive focusing strategy, the correlation between the two vectors is found, such as which specific elements in the battle scene (such as the strength of the enemy, the terrain of the battle scene, etc.) affect the interaction between the overall emotional expectations and the actual emotional state, thereby extracting the first feature interactive focusing vector. For the second feature interactive focus vector, for example, the detailed emotional expression vector in a mysterious treasure room in the game contains the player's excitement and curiosity. The emotional expectation features in the treasure scene in the self-focus expression vector are interactively analyzed with this detailed emotional expression vector to extract the second feature interactive focus vector in this treasure room scene, reflecting how elements such as the rarity of the treasure and the layout of the room affect this interactive relationship.

[0097] Then, the VR interactive emotion perception results and emotional semantic expression information of the template multimodal VR perception data are predicted based on the generated first feature interaction focus vectors and second feature interaction focus vectors. At the first model learning node, the server integrates the first feature interaction focus vectors and the second feature interaction focus vectors, and then maps and outputs the emotional knowledge space that matches the VR scene semantic dimension of the template multimodal VR perception data. In adventure games, the VR scene semantic dimension contains the semantic information of various scene elements, plot development and user emotional interaction throughout the game from the beginning to the end. For example, at the beginning of the game, based on the semantics of the opening scene (such as the initial mysterious atmosphere, simple exploration guidance, etc.), the emotional knowledge space expresses the curious expectation emotion that the player should have and the relationship between this emotion and the elements of the opening scene; in the battle scene of the game, based on the semantics of the battle scene (such as the strength of the enemy, the terrain of the battle scene, etc.), the emotional knowledge space reflects the nervous excitement that the player should have and the relationship between this emotion and the elements of the battle scene. Then, the first interactive task vector is used as the guidance information for emotion perception. The first interactive task vector represents the task goal information of the player in the game during this model learning phase. For example, in a certain exploration chapter, the task goal may be to find a specific item or explore a specific area. Combined with the emotional knowledge space, it is loaded into the emotion understanding sub-model to generate the emotional semantic expression information of the template multimodal VR perception data.

[0098] At the second model learning node, after integrating the first feature interaction focus vector and the second feature interaction focus vector, the emotional knowledge space that matches the VR scene semantic dimension of the template multimodal VR perception data is also mapped and output. Then the VR scene vector and the second interactive task vector are used as emotional perception guidance information. The VR scene vector contains various attribute information of the game scene, such as the type of scene (exploration scene, battle scene or puzzle scene, etc.), the complexity of the scene (the complexity of the terrain, the difficulty of the puzzle, etc.), and the distribution of elements in the scene (such as the location of the treasure, the distribution of the enemy, etc.). The second interactive task vector represents the task goal information of the player in the game at this learning stage. For example, in the battle scene, the task goal may be to defeat a specific enemy or complete a specific combat task. Combined with the emotional knowledge space, it is loaded into the emotional understanding sub-model to generate the VR interactive emotional perception results of the template multimodal VR perception data.

[0099] Finally, based on the VR interactive emotion perception results and emotional semantic expression information of the template multimodal VR perception data predicted based on the generated first feature interaction focus vectors and second feature interaction focus vectors, and the error between the corresponding training supervision data, the VR interactive emotion perception model is trained. At the first model learning node, the emotional feature focusing sub-model is trained based on the error between the predicted emotional semantic expression information and the emotional semantic expression information in the training supervision data. For example, if the player's curiosity intensity in a certain exploration scene in the predicted emotional semantic expression information is different from that in the training supervision data, the emotional feature focusing sub-model will adjust its own parameters according to this error to improve the prediction accuracy of the emotional semantic expression information. At the second model learning node, the emotional feature focusing sub-model and the emotional understanding sub-model are trained based on the error between the predicted VR interactive emotion perception results and the VR interactive emotion perception results in the training supervision data. For example, if the predicted level of tension and excitement of the player in a certain battle scene is different from that in the training supervision data, the emotional feature focus sub-model and the emotion understanding sub-model will adjust the parameters according to this error. After such a training process, the trained VR interactive emotion perception model can be used to obtain the VR interactive emotion perception results of the target multimodal VR perception data, thereby accurately analyzing the emotional state of the player in the adventure game in practical applications, and providing a basis for game optimization, etc.

[0100] Figure 2 The hardware structure of the emotion recognition system 100 based on multimodal perception data for implementing the above-mentioned emotion recognition method based on multimodal perception data provided by an embodiment of the present invention is shown as follows: Figure 2 As shown, the emotion recognition system 100 based on multimodal perception data may include a processor 110 , a machine-readable storage medium 120 , a bus 130 , and a communication unit 140 .

[0101] The machine-readable storage medium 120 may store data and / or instructions. In some embodiments, the machine-readable storage medium 120 may store data acquired from an external terminal. In some embodiments, the machine-readable storage medium 120 may store data and / or instructions that the emotion recognition system 100 based on multimodal perception data uses to execute or use to complete the exemplary method described in the present invention.

[0102] During the specific implementation process, one or more processors 110 execute computer executable instructions stored in the machine-readable storage medium 120, so that the processor 110 can execute the emotion recognition method based on multimodal perception data in the above method embodiment. The processor 110, the machine-readable storage medium 120 and the communication unit 140 are connected through the bus 130, and the processor 110 can be used to control the sending and receiving actions of the communication unit 140.

[0103] The specific implementation process of the processor 110 can refer to the various method embodiments executed by the above-mentioned emotion recognition system 100 based on multimodal perception data. The implementation principles and technical effects are similar and will not be repeated in this embodiment.

[0104] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer executable instructions are preset. When a processor executes the computer executable instructions, the above-mentioned emotion recognition method based on multimodal perception data is implemented.

[0105] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. An emotion recognition method based on multimodal perception data, characterized in that: The method comprises: According to the VR interactive emotion perception model, feature extraction is performed on each path space perception segment in the target multimodal VR perception data to generate corresponding summary emotion expression vectors and detail emotion expression vectors, and according to the vector correlation, the generated summary emotion expression vector distribution and detail emotion expression vector distribution are simplified to generate summary emotion expression vector sets and detail emotion expression vector sets of set feature scales, wherein the target multimodal VR perception data includes user feature data perceived by a heart rate monitor, a skin galvanic response sensor, a motion capture device, and a facial expression recognition camera in a VR interactive scene, and the VR interactive emotion perception model includes an emotion feature focusing sub-model and an emotion understanding sub-model, and the emotion feature focusing sub-model includes a plurality of cascaded emotion parsing units; After integrating the pre-learned summary-guided emotion expression vector and the detail-guided emotion expression vector, the corresponding self-focusing expression vector is output through the feature self-focusing strategy; the summary-guided emotion expression vector and the detail-guided emotion expression vector are obtained by learning the corresponding sample-guided emotion expression vector based on the template multimodal VR perception data; the self-focusing expression vector combines the self-focusing feature parts in the summary-guided emotion expression vector and the detail-guided emotion expression vector; Through the feature interaction focusing strategy, respectively extracting the first feature interaction focusing vector between the self-focusing expression vector and each summary emotion expression vector in the summary emotion expression vector set, and respectively extracting the second feature interaction focusing vector between the self-focusing expression vector and each detail emotion expression vector in the detail emotion expression vector set; Predicting a VR interactive emotion perception result of the target multimodal VR perception data according to each generated first feature interactive focus vector and each second feature interactive focus vector; Each detail emotion expression vector corresponds to a perception area in the perception segment of the path space; the generated summary emotion expression vector distribution and detail emotion expression vector distribution are simplified according to the vector correlation, and a summary emotion expression vector set and a detail emotion expression vector set of a set feature scale are generated, including: Each time feature extraction is performed on a path space perception segment, a summary emotion expression vector of the path space perception segment is added to a summary cache module, and a detail emotion expression vector of the path space perception segment is added to a detail cache module; When the number of vectors cached in the summary cache module reaches the set feature scale, the summary emotion expression vectors of multiple path space perception segments in the summary cache module are merged according to vector association, and the summary emotion expression vectors to be cached are added to the summary cache module; And, for each perception area, when the number of corresponding detail emotion expression vectors cached in the detail cache module reaches the set feature scale, the corresponding detail emotion expression vectors of multiple path space perception fragments in the detail cache module are fused according to the vector correlation, and the detail emotion expression vectors to be cached are added to the detail cache module.

2. The emotion recognition method based on multimodal perception data according to claim 1, characterized in that: The method of extracting a first feature interactive focusing vector between the self-focusing expression vector and each summary emotion expression vector in the summary emotion expression vector set and extracting a second feature interactive focusing vector between the self-focusing expression vector and each detail emotion expression vector in the detail emotion expression vector set respectively through a feature interactive focusing strategy comprises: Splitting the self-focus expression vector into a first self-focus expression vector corresponding to the summary emotion expression vector and a second self-focus expression vector corresponding to the detail emotion expression vector; Utilizing the first feature interaction focusing unit in the emotion feature focusing sub-model, and according to the feature interaction focusing strategy, respectively extracting the first self-focusing expression vector and the first feature interaction focusing vector between each summary emotion expression vector in the summary emotion expression vector set; By utilizing the second feature interaction focusing unit in the emotion feature focusing sub-model, the second self-focusing expression vector and the second feature interaction focusing vector between each detail emotion expression vector in the detail emotion expression vector set are extracted respectively according to the feature interaction focusing strategy.

3. The emotion recognition method based on multimodal perception data according to claim 2 is characterized in that: For the first emotion analysis unit, the unit input part includes the summary emotion expression vector set and the detail emotion expression vector set; for other emotion analysis units, the unit input part includes each first feature interaction focus vector and each second feature interaction focus vector generated by the previous emotion analysis unit; The method of predicting the VR interactive emotion perception result of the target multimodal VR perception data based on the generated first feature interaction focus vectors and the generated second feature interaction focus vectors includes: According to the first feature interaction focus vectors and the second feature interaction focus vectors generated by the last emotion analysis unit, the VR interactive emotion perception result of the target multimodal VR perception data is predicted.

4. The emotion recognition method based on multimodal perception data according to claim 1, characterized in that: The method of predicting the VR interactive emotion perception result of the target multimodal VR perception data based on the generated first feature interaction focus vectors and the generated second feature interaction focus vectors includes: After integrating the first feature interaction focus vectors and the second feature interaction focus vectors, mapping and outputting an emotional knowledge space that matches the VR scene semantic dimension of the target multimodal VR perception data; The VR scene vector and the interactive task vector of the target multimodal VR perception data are used as emotion perception guidance information, combined with the emotion knowledge space and loaded into the emotion understanding sub-model to generate VR interactive emotion perception results of the target multimodal VR perception data.

5. The emotion recognition method based on multimodal perception data according to claim 1, characterized in that: Before respectively extracting the first feature interaction focusing vector between the self-focusing expression vector and each summary emotion expression vector in the summary emotion expression vector set through the feature interaction focusing strategy, and respectively extracting the second feature interaction focusing vector between the self-focusing expression vector and each detail emotion expression vector in the detail emotion expression vector set, the method further includes: When it is detected that the set perception node is currently reached, extracting the summary emotion expression vector set cached in the summary cache module, and extracting the detail emotion expression vector set cached in the detail cache module; And, the step of fusing the target emotion expression vectors of multiple path space perception segments comprises: According to the vector relevance, the target emotion expression vectors of the most matching associated path space perception segments in the cache module are merged; According to the vector correlation, the target emotion expression vector to be cached and the target emotion expression vector cached in the cache module are clustered, and the target emotion expression vectors of each path space perception segment that have completed the clustering are evenly converged; wherein, the target emotion expression vector is a summary emotion expression vector or a detail emotion expression vector, and correspondingly, the cache module is a summary cache module or a detail cache module.

6. The emotion recognition method based on multimodal perception data according to claim 5 is characterized in that: If there are multiple most matching associated path space perception segments, merging the target emotion expression vectors of the most matching associated path space perception segments in the cache module comprises: Merging the target emotion expression vectors of the associated path space perception segments with the longest cache duration and the best match in the cache module; or The target emotion expression vectors that best match each group of associated path space perception segments in the cache module are balanced and converged respectively.

7. The emotion recognition method based on multimodal perception data according to claim 1, characterized in that: The method further comprises: According to the initialized VR interactive emotion perception model, feature extraction is performed on each path space perception fragment in the template multimodal VR perception data to generate the corresponding summary emotion expression vector and detail emotion expression vector. Based on the vector correlation, the generated summary emotion expression vector distribution and detail emotion expression vector distribution are simplified to generate summary emotion expression vector set and detail emotion expression vector set with set feature scale. After integrating the initialized first sample guided emotion expression vector and the second sample guided emotion expression vector, a corresponding self-focusing expression vector is output through a feature self-focusing strategy; the self-focusing expression vector combines the self-focusing feature parts in the first sample guided emotion expression vector and the second sample guided emotion expression vector; Through a feature interaction focusing strategy, extracting a first feature interaction focusing vector between the self-focusing expression vector and each summary emotion expression vector in the summary emotion expression vector set, and a second feature interaction focusing vector between the self-focusing expression vector and each detail emotion expression vector in the detail emotion expression vector set; Based on the VR interactive emotion perception results and emotional semantic expression information of the template multimodal VR perception data predicted according to the generated first feature interaction focus vectors and second feature interaction focus vectors, and the error between the corresponding training supervision data, the VR interactive emotion perception model is trained, and the trained VR interactive emotion perception model is used to obtain the VR interactive emotion perception results of the target multimodal VR perception data.

8. The emotion recognition method based on multimodal perception data according to claim 7, characterized in that: The VR interactive emotion perception model includes an emotion understanding sub-model and an emotion feature focusing sub-model implemented according to a feature self-focusing strategy and a feature interaction focusing strategy, wherein the emotion feature focusing sub-model is used to extract the first feature interaction focusing vector and the second feature interaction focusing vector; and according to each generated first feature interaction focusing vector and each second feature interaction focusing vector, the VR interactive emotion perception result and emotion semantic expression information of the template multimodal VR perception data are predicted, including: At the first model learning node, after integrating the first feature interaction focus vectors and the second feature interaction focus vectors, an emotional knowledge space matching the VR scene semantic dimension of the template multimodal VR perception data is mapped and output; the first interactive task vector is used as emotional perception guidance information, and is loaded into the emotional understanding sub-model in combination with the emotional knowledge space to generate emotional semantic expression information of the template multimodal VR perception data; At the second model learning node, after integrating the first feature interaction focus vector and the second feature interaction focus vector, mapping and outputting an emotional knowledge space that matches the VR scene semantic dimension of the template multimodal VR perception data; The VR scene vector and the second interactive task vector are used as emotion perception guidance information, and are loaded into the emotion understanding sub-model in combination with the emotion knowledge space to generate a VR interactive emotion perception result of the template multimodal VR perception data; The VR interactive emotion perception model is trained based on the error between the VR interactive emotion perception result and the emotional semantic expression information of the template multimodal VR perception data predicted according to each generated first feature interactive focus vector and each second feature interactive focus vector and the corresponding training supervision data, including: At the first model learning node, the emotion feature focusing sub-model is trained based on the error between the predicted emotion semantic expression information and the emotion semantic expression information in the training supervision data; At the second model learning node, the emotion feature focusing sub-model and the emotion understanding sub-model are trained based on the error between the predicted VR interaction emotion perception result and the VR interaction emotion perception result in the training supervision data.

9. An emotion recognition system based on multimodal perception data, characterized in that: The emotion recognition system based on multimodal perception data includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the emotion recognition method based on multimodal perception data described in any one of claims 1 to 8.

Images