Intelligent collaborative recommendation method and device for multi-modal immersive content

Abstract

The invention provides an intelligent collaborative recommendation method and device oriented to multi-modal immersive content, and relates to the technical field of immersive interaction.The method comprises the steps that multi-modal data are collected, and the attention weight of interaction between the multi-modal data and other modalities is calculated for each modality; and obtaining a corresponding fusion feature through the calculated attention weight and the representation weight after self-modal internal attention processing, and determining a recommended action by using a predefined multi-agent according to the obtained fusion feature. According to the method and the device, the problem that the recommended immersive content cannot meet the actual demand of the user due to the fact that multiple modal data for understanding the user state in the virtual environment and the interaction among the modal data are not comprehensively considered in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of immersive interaction technologies, and in particular, to an intelligent collaborative recommendation method and device for multi-modal immersive content. Background Art

[0002] With the rapid development of technology, virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies have brought revolutionary changes to the gaming and entertainment industries, and immersive experiences have gradually become the industry standard. However, in terms of personalized content recommendation, for example, in the field of cultural tourism, the application process of these technologies is relatively slow, and there are still many problems to be solved.

[0003] Currently, when different industries such as the cultural tourism industry provide users with immersive experiences, the degree of personalization is seriously insufficient. Content recommendation systems such as cultural tourism recommendation systems usually rely too much on simple user portraits and historical behavior data, and it is difficult to deeply explore users' real interests and needs. This situation not only makes the integration of immersive content and real cultural knowledge superficial, but also causes the system to respond severely lag to the changing dynamic interests of users. In immersive scenarios such as immersive cultural tourism scenarios, rich multi-modal interaction data, such as users' voice commands, gesture actions, gaze focus, physiological reactions, etc., contains a large amount of information that can reflect users' real-time interests, but traditional recommendation systems cannot make full use of multi-modal data resources, resulting in a large gap between the recommended results and users' expectations. Summary of the Invention

[0004] Embodiments of the present invention provide an intelligent collaborative recommendation method and device for multi-modal immersive content, aiming to solve the problem that the existing technology does not comprehensively consider multiple modal data used to understand the user state in a virtual environment and their interaction, resulting in the recommended immersive content not meeting the actual needs of users.

[0005] To achieve the above object, in a first aspect, the present invention provides an intelligent collaborative recommendation method for multi-modal immersive content, including the following steps: Collect multi-modal data associated with multiple modalities of a user in a virtual environment, where the multiple modalities include: user behavior, physiological reaction, and display feedback; For each modality, calculate the attention weight of the interaction between the modality itself and other modalities, and obtain the corresponding fusion feature by weighting the representation processed by the internal attention of the modality itself with the calculated attention weight, where the fusion feature corresponding to the user behavior is the user interest feature, the fusion feature corresponding to the physiological reaction is the emotional state feature, and the fusion feature corresponding to the display feedback is the group interaction tendency feature; Based on the obtained fused features, use a predefined multi-agent to determine the recommended actions, where the multi-agent includes: A content recommendation agent, which is used to calculate the first matching degree between the user interest feature and the candidate cultural and tourism content, and when the first matching degree meets a predetermined first threshold, recommend personalized content based on the user's interest; An experience adjustment agent, which is used to calculate the second matching degree between the emotional state feature and the preset ideal emotional state, and adjust the visual effect and / or volume of the recommended content display according to the relationship between the second matching degree and a predetermined second threshold; An exploration guidance agent, which is used to calculate the third matching degree between the user interest feature and the un-recommended content when the number of times of the current recommended content exceeds a preset threshold, and when the third matching degree meets a third threshold, recommend the un-recommended content; A social connection agent, which is used to recommend groups with the same interests according to the group interaction tendency feature.

[0006] Furthermore, the data of the user behavior includes: a movement trajectory represented by three-dimensional space coordinates, movement speed, and stay time; a line of sight focus represented by fixation point coordinates, fixation duration, and saccade path; and an operation record represented by click position and operation frequency; The data of the physiological reaction includes a heart rate signal represented by sequential heart rate values and heart rate variability, and α waves, β waves, and θ an electroencephalogram signal represented by the wave power spectrum density, and an electromyogram signal represented by the muscle activation degree; The data of the display feedback includes voice commands, satisfaction scores, and text evaluations.

[0007] Furthermore, after collecting the multi-modal data, it also includes steps of feature extraction and normalization processing for the modal data, including: For the non-sequential data in the multi-modal data, feature extraction is performed through the following formula: , In the formula, represents the mapped feature of the th modal data ; tanh is an activation function; is a feature extraction weight matrix, which is used to linearly transform the original modal data to a high-dimensional feature space; is a feature bias term; is used to generate the input of the sigmoid activation function; is a gating bias term; For the time-series data in the multimodal data, feature extraction is performed through the following formula: , In the formula ，X time is the original time-series data of the modality X raw is the time-series feature; FFT is the Fast Fourier Transform; Wavelet is the wavelet transform; α is the fusion weight coefficient; Among them, it also includes noise reduction and filtering of the original time-series data using the following formula: , In the formula, X filtered is the original time-series data after noise reduction; Mask is the noise recognition function based on the threshold ; represents the Hadamard product.

[0008] Furthermore, the attention weight is calculated through the following formula: , In the formula, represents the attention weight of the interaction between the -th modality and the -th modality; is the specific Query projection matrix between the -th modality and the -th modality; is the specific Key projection matrix between the -th modality and the -th modality; represents the correlation matrix between the -th modality and the -th modality; and are the representations after internal attention processing of the -th modality and the -th modality respectively, and are the representations after encoding the input data of the -th modality and , which is understood as abstracting the input data into a high-dimensional space; d K is the dimension of the Key projection matrix.

[0009] Furthermore, the fusion feature corresponding to one modality is calculated through the following formula: , , In the formula,H fused It is the fused feature vector representation for all modalities; v interest It is the user interest feature; v emotion It is the emotional state feature; v social It is the group interaction tendency feature; It represents the fused feature of the th modality. When the th modality is the user behavior, it is the user interest feature; when the th modality is the physiological response, it is the emotional state feature; when the th modality is the display feedback, it is the group interaction tendency feature; is the dynamically calculated modality weight, which is activated by the sigmoid activation function for the cosine similarity between the th modality and the th modality; is the projection parameter.

[0010] Furthermore, the multi - agents also optimize the recommended content through a collaboration mechanism, and the collaboration mechanism includes: a dynamic weight communication network, an interdependent strategy evolution method, and / or a hierarchical differential reward architecture. Among them, the dynamic weight communication network calculates the communication dynamic weights between agents according to the task correlation degree and determines the communication path with the highest communication score; the interdependent strategy evolution method optimizes the recommendation strategy by evaluating the influence of the recommended content updates among the multi - agents; the hierarchical differential reward architecture evaluates the performance of the multi - agents in the virtual environment through a multi - level reward mechanism.

[0011] Furthermore, the communication score is calculated by the following formula: , , , , , , In the formula, M j is the communication score of the j th agent; is the communication weight; is the The message sent by one agent to the j th agent after non - linear transformation; Denote the recommended action of the th agent; Is the activation function; Is the weight matrix for adjusting the original message u ; Is the bias term; Is the feature extracted by the agent according to the fused feature and context content , which is the state representation of the agent; Denote connecting the state and recommended action of the th agent, so that the transmitted information contains the complete decision - making context; Is the agent - specific non - linear mapping function, including M attention heads; Is the Kronecker product; Is the Hadamard product; GELU and Swish are activation functions; Is the output projection matrix; Is the Query projection matrix of the t th attention head; Is the Key projection matrix of the t th attention head; Is the Value projection matrix of the t th attention head; And Are the weight and bias respectively; d K Is the dimension of the Key projection matrix; Denote the task correlation degree between the th agent and the j th agent; ELU is the exponential linear unit; Denote the cosine similarity of the state representations between the th agent and the j th agent; v And Are learnable parameters, v Used to measure the importance of the state features between agents to the communication weight, Is the state projection matrix; T is the transpose.

[0012] It should be noted that, Denote accumulating all the results of whose value is not j ;

[0013] Furthermore, for the recommended strategy, the strategy objective function is: , wherein, represents the average reward that the agent expects to obtain under the distribution when all agents operate according to the joint strategy s and the joint action a follow ; represents the joint strategy of all agents; represents the joint strategy of all agents except the -th agent; is the global environmental state of the agent; s represents the recommended action of the agent except the -th agent; is the state-action distribution; is a parameter for adjusting the constraint strength; β is the KL divergence that restricts the difference between the current strategy and the reference strategy ; represents the immediate reward obtained when the -th agent takes the action in the global state s while other agents take the joint action ; The gradient formula adopted for strategy update is: , , , , wherein, is the gradient function; is the communication score of the -th agent; is the policy parameter of the -th agent; is the interdependent advantage function of the -th agent considering other agents except itself; represents the state-recommended action value function of the -th agent; j represents the state-recommended action value function of the -th agent; is the state value function of the -th agent; j ​​The state value function of an agent; For the th agent and the j th agent, the interdependence weight between them; MLP is a multi - layer perceptron; Measure the average gradient impact of the policy change of the th agent on the value function of the j th agent; For the th agent and the j th agent, the correlation between their value functions.

[0014] It should be noted that Indicates the sum of all results where the value of j is not .

[0015] Furthermore, the multi - level reward mechanism includes: professional task reward, interaction and cooperation reward, and / or overall experience reward. Among them, The professional task reward is used to evaluate the performance score of an agent in its professional field. The calculation formula is as follows: , , , In the formula, Represents the professional task reward; Is the correlation between the recommended content and the user vector , Is the corresponding balance parameter; The diversity of the recommended content relative to the user's historical vector , Is the corresponding balance parameter; Represents the user vector, which contains the user's interests and historical behavior information; Represents the user vector of past history, h Represents each record in the The interaction and cooperation reward is used to evaluate the performance score of an agent under the influence of the recommended actions of other agents. The calculation formula is as follows: , In the formula, Is the interaction and cooperation reward; For the th agent and the j th agent, the cooperation coefficient between them; Represents in the global states Under the condition that the j th agent takes the recommended action, while the other agents except the a i th agent take the joint action under the action of a -i , the value function; represents that under the global state s , the j th agent takes the benchmark action , while the other agents except the th agent take the joint action under the action of , the value function; is the benchmark action; is the recommended action for agents except the th agent; , In the formula, is the overall experience reward, f global is a non-linear mapping function that integrates multiple metrics into a single global score; satisfaction is the user satisfaction; duration is the user's stay time in the current recommended content; revisit_intention represents the intention to revisit; The comprehensive reward of the agent is: , In the formula, SUM i is the comprehensive reward function of the th agent; , and are adaptive weights, dynamically adjusted according to the system state and learning stage, and the adjustment calculation formula is: , In the formula, is the learning rate; is the gradient function; is the weight optimization objective function; t represents time, t -1 is the previous time state.

[0016] In a second aspect, the present invention provides an electronic device, characterized in that it includes a memory and a processor, the memory stores at least one program, and the at least one program is executed by the processor to implement the intelligent collaborative recommendation method for multi-modal immersive content as described above.

[0017] The above technical solution has the following technical effects: The technical solution of the embodiment of the present invention collects multi-modal data and calculates the interaction attention between itself and other modalities for each modality. The corresponding fusion features are obtained by weighting the calculated attention weights with the representations processed by the internal attention of its own modality. According to the obtained fusion features, predefined multi-agents are used to determine the recommendation actions. The present invention solves the problem that the prior art does not comprehensively consider multiple modality data for understanding the user state in a virtual environment and their interaction, resulting in the recommended immersive content not meeting the actual needs of users.

[0018] In a further embodiment, by constructing a dynamic weight communication network between multi-agents, calculating the communication dynamic weights between agents according to the task correlation degree, and determining the communication path with the highest communication score, the agents can preferentially receive the most relevant information, improving the adaptability and collaboration efficiency of the multi-agent system in the recommendation scenario.

[0019] In a further embodiment, by evaluating the influence of the recommended content updates among multi-agents on each other, and updating the strategy for each agent with the strategies of other agents as constraints, it is ensured that the strategy update can not only improve the performance but also will not deviate too far and cause system instability.

[0020] In a further embodiment, by comprehensively measuring the performance of agents in a multi-objective environment from multiple levels of professional field, interactive collaboration, and overall experience, accurately reflecting the collaborative contributions between agents, and realizing the accurate evaluation and guidance of the performance of agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flowchart of an intelligent collaborative recommendation method for multi-modal immersive content according to an embodiment of the present invention; Figure 2 It is a schematic flowchart of immersive content recommendation after establishing a preset collaboration mechanism when using the intelligent collaborative recommendation method for multi-modal immersive content according to an embodiment of the present invention; Figure 3 It is a schematic structural diagram of an electronic device for immersive content recommendation based on multi-modal data interaction according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] To further illustrate the embodiments, the present invention provides accompanying drawings. These drawings are a part of the disclosure of the present invention, mainly used to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the relevant descriptions in the specification. With reference to these contents, those of ordinary skill in the art should be able to understand other possible implementation manners and the advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are usually used to represent similar components.

[0023] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation manners.

[0024] Embodiment 1: Figure 1 The following is a schematic flowchart of an intelligent collaborative recommendation method for multi-modal immersive content according to an embodiment of the present invention. The method of this embodiment includes the following steps: Collect multi-modal data associated with multiple modalities of the user in a virtual environment. The multiple modalities include: user behavior, physiological response, and display feedback. In a specific implementation, the data of user behavior includes: a movement trajectory represented by three-dimensional space coordinates, movement speed, and residence time; a line-of-sight focus represented by fixation point coordinates, fixation duration, and saccade path; and an operation record represented by click position and operation frequency. The data of physiological response includes a heart rate signal represented by sequential heart rate values and heart rate variability, and an electroencephalogram signal represented by the power spectral density of α waves, β waves, and θ waves, as well as an electromyogram signal represented by muscle activation degree. The data of display feedback includes voice commands, satisfaction scores, and text evaluations. For each modality, calculate the attention weights of the interaction between the modality itself and other modalities, and obtain the corresponding fusion features by weighting the representation after the internal attention processing of the modality itself with the calculated attention weights. Among them, the fusion feature corresponding to user behavior is the user interest feature, the fusion feature corresponding to physiological reaction is the emotional state feature, and the fusion feature corresponding to display feedback is the group interaction tendency feature. In a specific implementation, the user interest feature is mainly obtained through the synergistic effect of user behavior data and physiological reaction data. The core lies in the temporal correspondence and complementarity of the two modalities of data: the user behavior data records the specific actions of the user during the experience and reflects which content or activities the user shows attention to, while the physiological reaction data captures the emotional reactions of the user when performing these behaviors. Since the physiological reaction data corresponds one by one with the user behavior data in time, the system can infer the user's preference degree for relevant scenarios or recommended content by analyzing the physiological changes of the user when performing a certain action at a certain moment. The emotional state feature is mainly based on the physiological reaction data, but through cross-fusion with the user behavior data and explicit feedback data. The user behavior data can provide context for the physiological reaction data. For example, if the user shows an accelerated heart rate in a certain horror scene and the user behavior data also shows that the user quickly leaves the area, it may be inferred that the user is in a state of fear. The explicit feedback data provides the user's subjective expression of emotions. For example, the user says "This is so interesting" or gives a five-star rating, indicating a positive emotion. Combining it with the physiological reaction data can verify and strengthen the inference of the emotional state. The group interaction tendency feature is mainly based on the explicit feedback data, through cross-fusion with the user behavior data and physiological reaction data. The explicit feedback data directly reflects the user's attitude towards social activities. The user behavior data provides the user's actual performance in the social environment. For example, whether the user actively approaches the crowded area or participates in multi-person interaction tasks reflects objective evidence of the user's social behavior. The physiological reaction data reveals the user's emotional reactions in the social situation. By clustering users based on the comprehensive three types of data, the system can judge the degree of gregariousness or isolation of the user among the same group of people.

[0025] According to the obtained fusion features, use a pre-defined multi-agent to determine the recommended actions. Among them, the multi-agent includes: A content recommendation agent, which is used to calculate the first matching degree between the user interest feature and the candidate cultural and tourism content, and when the first matching degree meets a predetermined first threshold, recommend personalized content based on the user's interest; An experience adjustment agent, which is used to calculate the second matching degree between the emotional state feature and the preset ideal emotional state, and adjust the visual effect and / or volume of the recommended content display according to the relationship between the second matching degree and a predetermined second threshold; An exploration and guidance agent is used to calculate the third matching degree between the user interest feature and the un-recommended content when the number of times of the current recommended content exceeds a preset threshold, and recommend the un-recommended content when the third matching degree meets the third threshold; A social connection agent is used to recommend groups with the same interests according to the group interaction tendency feature.

[0026] Further, after collecting the multi-modal data, it also includes the steps of feature extraction and standardization processing of the modal data, including: For the non-temporal data in the multi-modal data, the feature extraction is carried out through the following formula: , In the formula, represents the th modal data mapping feature; tanh is the activation function; is the feature extraction weight matrix, which is used to linearly transform the original modal data to the high-dimensional feature space; is the feature bias term; is used to generate the input of the sigmoid activation function; is the gating bias term; For the temporal data in the multi-modal data, the feature extraction is carried out through the following formula: , In the formula ，X time is the temporal feature of the modal original temporal data X raw ; FFT is the fast Fourier transform; Wavelet is the wavelet transform; is the fusion weight coefficient; Among them, it also includes the noise reduction filtering of the original temporal data using the following formula: , In the formula, X filtered is the original temporal data after noise reduction; Mask is the noise recognition function based on the threshold ; represents the Hadamard product.

[0027] Further, the attention weight is calculated through the following formula: , In the formula, represents the attention weight of the interaction between the th modal and the th modal; is the specific Query projection matrix between the th modality and the th modality; is the specific Key projection matrix between the th modality and the th modality; represents the correlation matrix between the th modality and the th modality; and are the representations after internal attention processing of the d K th modality and the

[0028] th modality respectively; Further, the fused feature corresponding to one modality is calculated by the following formula: , wherein, H fused is the fused feature vector representation of all modalities; v interest is the user interest feature; v emotion is the emotional state feature; v social is the group interaction tendency feature; represents the fused feature of the th modality. When the th modality is user behavior, is the user interest feature; when the th modality is physiological reaction, is the emotional state feature; when the th modality is display feedback, is the group interaction tendency feature; is the dynamically calculated modality weight, which is activated by the sigmoid activation function on the cosine similarity between the th modality and the th modality; is the projection parameter.

[0029] It should be noted that means to accumulate all the results where the value of is not .

[0030] Furthermore, the multi - agents also optimize the recommended content through a collaboration mechanism, which includes: a dynamic weight communication network, an interdependent strategy evolution method, and / or a hierarchical differential reward architecture. Among them, The dynamic weight communication network attempts to solve existing problems, that is, traditional multi - agent systems often adopt fixed - structure or simple - average communication methods and cannot adapt to the dynamically changing relationship importance among agents in different scenarios. Agents in the dynamic weight communication network not only need to decide what information to transmit, but also need to clarify to whom to transmit the information and how to weigh information from different sources. The th agent sends a message to the j th agent, which is generated through a special non - linear transformation. The dynamic weight communication network calculates the communication dynamic weights among agents according to the task correlation degree and determines the communication path with the highest communication score; Furthermore, the communication score is calculated through the following formula: , , , , , , In the formula, M j is the communication score of the j th agent; is the communication weight; is the non - linearly transformed message sent by the th agent to the j th agent; represents the recommended action of the th agent; is the activation function; is the weight matrix for adjusting the original message ; is the bias term; is the feature extracted by the agent according to the fused feature and context content , which is the state representation of the agent; represents connecting the state and recommended action of the th agent, so that the transmitted information contains the complete decision - making context; is the agent - specific non - linear mapping function, which includes M attention heads; is the Kronecker product; is the Hadamard product;GELU and Swish is the activation function; is the output projection matrix; is the Query projection matrix of the t -th attention head; is the t -th attention head's Key projection matrix; is the t -th attention head's Value projection matrix; and are the weight and bias respectively; d K is the dimension of the Key projection matrix; represents the task correlation degree between the -th agent and the j -th agent; ELU is the exponential linear unit; represents the cosine similarity of the state representations between the -th agent and the j -th agent; and are learnable parameters, used to measure the importance of the state features between agents to the communication weight, is the state projection matrix; T is the transpose.

[0031] The unique value of the dynamic weight communication network lies in its ability to automatically form an optimal communication path according to the task progress. For example, in the content exploration stage, the information of the exploration guiding agent may obtain a higher weight, while when the user engagement drops, the information of the experience adjustment agent becomes more important. This dynamic adjustment mechanism greatly improves the adaptability and collaboration efficiency of the multi-agent system in the cultural and tourism recommendation scenario.

[0032] The interdependent policy evolution method optimizes the recommendation policy by evaluating the impact of the content updates recommended by multi-agents on each other; Furthermore, traditional multi-agent reinforcement learning methods often update the agent policies in a completely independent or simple summation manner, ignoring the complex interdependent relationships between agent decisions. In the recommendation scenario, the decision of one agent may significantly affect the effects of other agents. For example, content selection affects user engagement, and engagement in turn affects content preference. To address this challenge, this embodiment proposes the interdependent policy evolution method, which is an innovative optimization method considering the mutual influence of agent policies.

[0033] The core idea of the interdependent policy evolution method is to regard the policy update of each agent as a conditional optimization problem, that is, to optimize its own policy under the condition that the policies of other agents are fixed, and at the same time, by introducing a policy stability constraint, to prevent drastic fluctuations in the optimization process.

[0034] Among them, the policy objective function is: , This objective function consists of two key parts: The first part is the expected reward obtained under the joint policy of all agents , where denotes the joint policy of all agents except the -th agent, and is the state-action distribution; The second part is the policy stability constraint, which restricts the difference between the current policy and the reference policy through the KL divergence , and β is a parameter that adjusts the constraint strength. This design ensures that the policy update can not only improve performance but also not deviate too far to cause system instability; denotes the immediate reward obtained when the -th agent takes the action s under the global state , while other agents take the joint action .

[0035] The gradient formula adopted for policy update is: , , , , In the formula, denotes the average reward that the agent expects to obtain under the distribution of the global state s and the joint action a of the agents after all agents run according to the joint policy ; is the communication score of the -th agent; is the policy parameter of the -th agent; is the interdependent advantage function of the -th agent considering other agents except itself, which not only considers the advantage of the agent itself but also incorporates the impact of its decision on other agents; denotes the state-recommended action value function of the -th agent; denotes the state-recommended action value function of the -th agent; j denotes the state-recommended action value function of the -th agent; The state value function of an agent; is the state value function of the j th agent; is the th agent and the j th agent's interdependence weight, indicating the degree of influence of the th agent's decision on the j th agent's goal; MLP is a multi-layer perceptron; Measures the average gradient influence of the th agent's policy change on the j th agent's value function; is the th agent and the j th agent's value function correlation. This interdependence weight calculation method can automatically identify the key dependencies between agents and dynamically adjust them during training.

[0036] Furthermore, in the immersive recommendation scenario, the criteria for evaluating agent performance are diverse, including both the quality of professional task completion and the contribution to the overall experience. The traditional single reward function is difficult to comprehensively measure the performance of agents in a multi-objective environment, especially unable to accurately reflect the collaborative contributions between agents. To address this challenge, the hierarchical differential reward architecture in this embodiment evaluates the performance of multi-agents in a virtual environment through a multi-level reward mechanism.

[0037] Furthermore, the multi-level reward mechanism includes: professional task reward, interaction and collaboration reward, and / or overall experience reward, where For the professional task reward, "professional" means that different agents focus on different professional fields. For example, the content agent focuses on the recommended content, and the experience agent focuses on the user's emotional state and engagement, etc.; the professional task reward is used to evaluate the performance score of the agent in its professional field, and the calculation formula is as follows: , , , In the formula, represents the professional task reward; is the correlation between the recommended content c and the user vector , is the corresponding balance parameter; The diversity of the recommended content c relative to the user's historical vector , is the corresponding balance parameter; Represents the user vector, which contains the user's interests and historical behavior information; Represents the user vector of past history, h Represents Each record in the set; The interactive collaborative reward specifically evaluates how the decision of one agent enhances or weakens the effects of other agents. Different from simply adding individual rewards in traditional methods, the interactive collaborative reward precisely quantifies the mutual influence between agents through counterfactual analysis, and is used to evaluate the performance score of an agent under the influence of the recommended actions of other agents. The calculation formula is as follows: , In the formula, is the interactive collaborative reward; is the collaborative coefficient between the th agent and the j th agent; represents the value function under the global state s , when the j th agent takes the recommended action , and at the same time, other agents except the th agent take the joint action ; represents the value function under the global state , when the s th agent takes the baseline action j , and at the same time, other agents except the th agent take the joint action ; is the baseline action; is the recommended action of agents other than the th agent; In a specific implementation, if the content recommendation agent recommends an exhibit that highly attracts the user's attention, enabling the experience adjustment agent to more effectively adjust the immersion parameters, the content recommendation agent will obtain a positive interactive collaborative reward. This design encourages agents to not only optimize their own goals but also consider how to create favorable conditions for other agents. is the recommended action of agents other than the th agent;

[0038] The overall experience reward is a global reward signal shared by all agents, which evaluates the quality of the overall recommendation experience, including indicators such as user satisfaction, stay time, and willingness to revisit, and is used to evaluate the overall performance score of the multi-agent recommendation actions. The calculation formula is as follows: , In the formula, is the overall experience reward, fglobal a non - linear mapping function that integrates multiple metrics into a single global score; satisfaction is the user satisfaction; duration is the time the user stays in the current recommended content; revisit_intention represents the intention to revisit; The comprehensive reward of the agent is: , In the formula, SUM i is the comprehensive reward function of the th agent; , and are adaptive weights, which are dynamically adjusted according to the system state and learning stage. The adjustment calculation formula is: , In the formula, is the learning rate; t represents time, t - 1 is the previous time state; is the weight optimization objective function, ensuring that the system can automatically find the optimal reward balance point. For example, at the initial stage of learning, more attention may be paid to professional task rewards to build basic capabilities, and as learning progresses, the weight of collaborative rewards will gradually increase to promote agent cooperation.

[0039] Example 2: Figure 2 is a schematic diagram of the process of immersive content recommendation after establishing a preset collaboration mechanism when using the intelligent collaborative recommendation method for multi - modal immersive content in an embodiment of the present invention. In this embodiment, obtaining multi - modal data fusion features and establishing a multi - agent collaboration mechanism includes: Collect multi - modal data associated with multiple modalities of the user in the virtual environment. The multiple modalities include: user behavior, physiological reaction, and display feedback; For each modality, calculate the attention weight of the interaction between this modality and other modalities, and obtain the corresponding fusion feature by weighting the representation after internal attention processing of the self - modality with the calculated attention weight; After obtaining the fusion features, the multi - agents determine the recommendation actions for immersive content according to the fusion features. However, the recommendation actions between different agents affect and restrict each other, often resulting in a reduction in the recommendation effect. To ensure the accuracy and efficiency of the recommendation actions, it is necessary to continuously improve the collaboration ability between agents. Therefore, the multi - agents in this embodiment perform immersive content recommendation through a preset collaboration mechanism. The preset collaboration mechanism includes: a dynamic weight communication network, an interdependent strategy evolution method, and / or a hierarchical differential reward architecture, where The dynamic weight communication network calculates the communication dynamic weights between agents according to the task correlation degree, and determines the communication path with the highest communication score; The interdependent strategy evolution method optimizes the recommendation strategy by evaluating the influence of the content updates recommended by multiple agents on each other; The hierarchical differential reward architecture evaluates the performance of multiple agents in a virtual environment through a multi-level reward mechanism.

[0040] Such as Figure 2 , after the preset cooperation mechanism, the agents of each specialty first calculate the communication dynamic weights between agents according to the task correlation degree through the dynamic weight communication network based on the fusion features and their own states, determine the communication path with the highest communication score, and according to the communication path, each agent exchanges information to establish a comprehensive understanding of the current virtual situation; Based on the understanding of the current virtual situation, multiple agents make collaborative decision-making to recommend immersive content according to the strategy objective function of the interdependent strategy evolution method under the consideration of the expected reward under the joint strategy of multiple agents and the mutual constraints between agents; The system integrates the decisions of each agent into a unified recommendation plan and presents it to the user through an immersive interface; During the interaction between the user and the recommended content, the system continuously collects feedback data, and evaluates the professional task rewards, interaction and collaboration rewards, and / or overall experience rewards of each agent through the hierarchical differential reward architecture, and then obtains a comprehensive reward through the adaptive weights of the three rewards; Based on the collected feedback data and evaluation results, the system then updates the recommendation strategies of each agent through the interdependent strategy evolution method using the interdependent advantage function under the consideration of the influence of other agents.

[0041] Embodiment 3: Figure 3 It is a schematic structural diagram of an electronic device for immersive content recommendation based on multi-modal data interaction in an embodiment of the present invention. Such as Figure 3 shown, the device includes a processor 301, a memory 302, a bus 303, and a computer program stored in the memory 302 and operable on the processor 301. The processor 301 includes one or more processing cores. The memory 302 is connected to the processor 301 through the bus 303. The memory 302 is used to store program instructions. When the processor executes the computer program, it implements the steps in the above method embodiment of Embodiment 1 of the present invention.

[0042] Further, as an executable solution, the electronic device for immersive content recommendation based on multimodal data interaction may be a computer unit, which may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above composition structure of the computer unit is only an example of the computer unit, and does not constitute a limitation on the computer unit. It may include more or fewer components than the above, or combine some components, or different components. For example, the computer unit may further include input / output devices, network access devices, a bus, etc., and the embodiments of the present invention do not make any limitations thereto.

[0043] Further, as an executable solution, the so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the computer unit, and connects various parts of the entire computer unit through various interfaces and lines.

[0044] The memory may be used to store the computer programs and / or modules. The processor realizes various functions of the computer unit by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0045] Although the present invention has been specifically shown and described in connection with preferred embodiments, those skilled in the art should understand that various changes in form and detail can be made to the present invention without departing from the spirit and scope of the present invention as defined by the appended claims, and all such changes are within the scope of protection of the present invention.

Claims

1. An intelligent collaborative recommendation method for multimodal immersive content, characterized in that: The following steps are involved: Collecting multimodal data of a user associated with multiple modalities in a virtual environment, the multiple modalities including: user behavior, physiological response, and display feedback; For each modality, the attention weight of the interaction between the modality itself and other modalities is calculated, and the corresponding fusion feature is obtained by weighting the calculated attention weight with the representation after the attention processing within the modality itself, wherein the fusion feature corresponding to the user behavior is the user interest feature, the fusion feature corresponding to the physiological response is the emotional state feature, and the fusion feature corresponding to the display feedback is the group interaction tendency feature; According to the obtained fusion features, a recommended action is determined using a predefined multi-agent, wherein the multi-agent includes: A content recommendation agent, configured to calculate a first matching degree between the user interest feature and the candidate cultural and travel content, and recommend personalized content based on the user's interest when the first matching degree meets a predetermined first threshold; an experience adjustment agent, configured to calculate a second matching degree between the emotional state characteristic and a preset ideal emotional state, and to adjust a visual effect and / or volume of a display of recommended content according to a relationship between the second matching degree and a predetermined second threshold; an exploration guiding agent, configured to calculate a third matching degree between the user interest feature and the content that has not been recommended when the number of times the current content recommendation exceeds a preset threshold, and recommend the content that has not been recommended when the third matching degree satisfies a third threshold; A social connection agent, for recommending groups with similar interests based on the interaction tendency characteristics of the groups; The attention weight is calculated by the following formula: ， In the formula, Indicates The mode and Attention weights for modal interactions; It is The mode and A specific query projection matrix between modalities; It is The mode and A specific Key projection matrix between modes; Indicates m The mode and The correlation matrix between the modes; and They are The mode and The representation after attention processing within each modality; d K is the dimension of the Key projection matrix; The mode and The first mode is any one of the multiple modes, and the The mode and The modes are different modes.

2. The intelligent collaborative recommendation method for multimodal immersive content according to claim 1, characterized in that: The user behavior data includes: movement trajectory represented by three-dimensional space coordinates, movement speed and dwell time; sight focus represented by gaze point coordinates, gaze duration and scanning path; and operation record represented by click position and operation frequency; The physiological response data includes a heart rate signal represented by a time series heart rate value and a heart rate variability. α Wave, β wave and θ EEG signals represented by wave power spectral density, and EMG signals represented by muscle activation levels; The data displaying the feedback includes voice commands, satisfaction ratings and text reviews.

3. The intelligent collaborative recommendation method for multimodal immersive content according to claim 1, characterized in that: After collecting the multimodal data, the method further includes the steps of extracting features and standardizing the modal data, including: For the non-time series data in the multimodal data, feature extraction is performed using the following formula: ， In the formula, Indicates Modal data The mapping feature of ; tanh is the activation function; is the feature extraction weight matrix, which is used to transform the original modal data Linear transformation to high-dimensional feature space; is the feature bias term; Used to generate the input of the sigmoid activation function; is the gate bias term; For the time series data in the multimodal data, feature extraction is performed using the following formula: ， In the formula ，X time is the original time series data of the modal X raw The time series characteristics of; FFT is fast Fourier transform; Wavelet is wavelet transform; α is the fusion weight coefficient; The method also includes using the following formula to filter the original time series data: ， In the formula, X filtered is the original time series data after noise reduction; Mask is based on threshold Noise identification function; represents the Hadamard product.

4. The intelligent collaborative recommendation method for multimodal immersive content according to claim 1, characterized in that: The fusion feature vector representation of all modalities is calculated by the following formula: ， ， In the formula, H fused is the fusion feature vector representation of all modalities; v interest User interest characteristics; v emotion Characterizes emotional states; v social It is the characteristic of group interaction tendency; Indicates The fusion features of the modalities, when the When a mode is a user behavior, is the user interest feature; When the first mode is a physiological response, is the emotional state characteristic; When the mode is to display feedback, It is the characteristic of group interaction tendency; is the dynamically calculated modal weight, which is activated by the sigmoid activation function The mode and The cosine similarity between the modalities is activated; are the projection parameters.

5. The intelligent collaborative recommendation method for multimodal immersive content according to claim 1, characterized in that: The multiple agents also optimize the recommended content through a collaborative mechanism, wherein the collaborative mechanism includes: a dynamic weight communication network, an interdependent strategy evolution method and / or a hierarchical differentiated reward architecture, wherein: The dynamic weight communication network calculates the dynamic weight of communication between agents according to the task relevance and determines the communication path with the highest communication score; The interdependent strategy evolution method optimizes the recommendation strategy by evaluating the influence of multiple agents on updating recommended content. The hierarchical differentiated reward architecture evaluates the performance of multiple agents in a virtual environment through a multi-level reward mechanism.

6. The intelligent collaborative recommendation method for multimodal immersive content according to claim 5, characterized in that: The communication score is calculated by the following formula: ， ， ， ， ， ， In the formula, M j For the j The communication score of each agent; is the communication weight; For the The agent j The nonlinearly transformed messages sent by the agents; Indicates Recommended actions for each agent; is the activation function; To adjust the original message The weight matrix of is the bias term; Based on the fusion features and context content C i The extracted features represent the state of the agent; Indicates that the The states and recommended actions of each agent are connected so that the transmitted information contains the complete decision context; is the agent-specific nonlinear mapping function, including M Attention head; is the Kronecker product; is the Hadamard product; GELU and Swish are activation functions; is the output projection matrix; For the t The query projection matrix of the attention head; For the t The Key projection matrix of the attention head; For the t The Value projection matrix of the attention head; and are weight and bias respectively; d K is the dimension of the Key projection matrix; Indicates Agent and j The degree of task relevance between agents; ELU is the exponential linear unit; Indicates Agent and j The cosine similarity of the state representations between agents; v and is a learnable parameter, v Used to measure the importance of state features between agents to the communication weight, is the state projection matrix; T is the transpose.

7. The intelligent collaborative recommendation method for multimodal immersive content according to claim 6, characterized in that: The recommendation strategy adopts the following strategy objective function: ， In the formula, Indicates that all agents follow the joint strategy After running, the global state of the agent s and joint actions a In following Under distribution, the agent The average reward expected to be obtained; is the strategy objective function; represents the joint strategy of all agents; Indicates that except The joint strategy of all agents except the agent; s is the global environment state of the agent; Indicates that except Recommended actions for agents other than the agent; is the state-action distribution; β is a parameter that adjusts the constraint strength; To limit the current policy With reference strategy KL divergence of differences; R i Indicates Agents in the global state s Take action , while other agents take joint actions Instant rewards received; The gradient formula used for strategy update is: ， ， ， ， In the formula, is the gradient function; For the The communication score of each agent; For the The strategy parameters of each agent; For the The interdependent advantage function of each agent considering the state representation and recommended action influence of other agents besides itself; Indicates The representation state-recommended action value function of each agent; Indicates j The representation state-recommended action value function of each agent; For the The state value function of an agent; For the j The state value function of an agent; For the Agent and j The interdependence weights between agents; MLP is a multi-layer perceptron; Measure the The effect of the strategy change of the first agent on j The average gradient influence of the value function of each agent; For the Agent and j The correlation between the value functions of the agents.

8. The intelligent collaborative recommendation method for multimodal immersive content according to claim 6, characterized in that: The multi-level reward mechanism includes: professional task rewards, interactive collaboration rewards and / or overall experience rewards, among which: The professional task reward is used to evaluate the performance score of the agent in its professional field. The calculation formula is as follows: ， ， ， In the formula, Indicates professional task rewards; Recommended content c With user vector u The correlation between is the corresponding equilibrium parameter; Recommended content c Relative to the user history vector Diversity, is the corresponding equilibrium parameter; u Represents the user vector, which contains the user's interests and historical behavior information; Represents the user vector set of past history, h express Each record in the collection; The interactive synergy reward is used to evaluate the performance score of an agent under the influence of the recommended actions of other agents. The calculation formula is as follows: ， In the formula, Reward for interactive collaboration; For the Agent and j The coordination coefficient between the agents; Indicates that in the global state s Next, j The agent in Agents take recommended actions a i , except for the Agents other than the agent take joint actions The value function under action; Indicates the global state s Next, j The agent in Agents take baseline actions , except for the Agents other than the agent take joint actions The value function under action; is the benchmark action; For the Agents other than the agent recommend actions; The overall experience reward is used to evaluate the overall performance score of the multi-agent recommended actions. The calculation formula is as follows: ， In the formula, Reward for overall experience, f global is a nonlinear mapping function that integrates multiple indicators into a single global score; satisfaction is user satisfaction; duration is the user's stay time in the current recommended content; revisit_intention represents the intention to revisit; The comprehensive reward of the agent is: ， In the formula, SUM i For the The comprehensive reward function for each agent; , and It is an adaptive weight, which is dynamically adjusted according to the system status and learning stage. The adjustment calculation formula is: ， In the formula, is the learning rate; is the gradient function; Optimize the objective function for the weights; t Indicates time, t -1 is the previous time state.

9. An electronic device, characterized in that: It includes a memory and a processor, the memory stores at least one program, and the at least one program is executed by the processor to implement the intelligent collaborative recommendation method for multimodal immersive content as described in any one of claims 1 to 8.

Images