Aigc cross-medium based metaverse scene dynamic generation method

Abstract

The application relates to the technical field of artificial intelligence generated content and meta-universe scene generation, and discloses a meta-universe scene dynamic generation method based on AIGC cross media, which decomposes source memory scene features into style dimension and content dimension feature vectors through a memory semantic decoupling analyzer, establishes a style mapping relationship of cross-scene contexts by using a memory semantic projection matrix, identifies style-sensitive anchor point nodes and generates an anchor point-medium correlation matrix, ensures cross-media style coordination consistency through a style consistency constraint propagation algorithm, and generates scene content containing 3D models, texture images and audio by using a cross-media AIGC generator. The application solves the technical problems of style feature and scene context coupling, cross-context style inconsistency and cross-media conversion inconsistency, and realizes cross-context and cross-media migration of personalized memory style.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence-generated content and metaverse scene generation technology, and more specifically, to a method for dynamic generation of metaverse scenes based on AIGC across media. Background Technology

[0002] In metaverse virtual scene applications, users often want to transfer the unique style and emotional atmosphere of personal memory scenes they have experienced to new scenes, achieving personalized stylized scene generation. For example, a user may have experienced specific twilight lighting, ambient sound effects, and art style in a virtual Paris street scene and want to apply this emotional atmosphere from their memory to the generation of a virtual Kyoto garden scene.

[0003] Existing style transfer techniques are based on predefined artistic style samples and typically employ neural style transfer algorithms or generative adversarial networks to achieve image-level style transfer. These techniques separate and reassemble the content of the source image from the texture features of the style image to generate an image with the target style.

[0004] However, existing technologies have the following technical shortcomings: First, when the transfer target is a user's personal memory scene, the style characteristics of the user's personal memory scene are highly personalized and tightly coupled with the specific scene context. Memory scene data is usually not fully preserved, only abstract semantic gene encoding exists, making it impossible to accurately extract style parameters. Second, there are contextual differences between different scenes. For example, the urban style of Parisian streets and the natural Zen style of Kyoto gardens are completely different in cultural background and spatial characteristics. Directly applying style transfer will lead to inconsistencies in scene context. Third, metaverse scenes contain multiple media such as 3D models, texture images, and audio. The degree of style conversion varies among different media, leading to cross-media style coordination problems in the generated scene. These factors together result in the generated scene lacking both the emotional resonance expected by the user and the technical shortcomings of cross-media style inconsistency. Summary of the Invention

[0005] This invention provides a method for dynamically generating metaverse scenes based on AIGC across media, which solves the technical problems in related technologies such as the inability to adapt across contexts due to the tight coupling between style features and scene context, style inconsistency due to differences in scene context, and inconsistent degree of style conversion across media.

[0006] This invention provides a method for dynamically generating metaverse scenes based on AIGC across media, comprising the following steps:

[0007] Obtain the source memory anchor data package selected by the user and the target scene description text. The source memory anchor data package includes scene semantic gene encoding, key visual element features, audio atmosphere features, lighting environment parameters, and sentiment tags.

[0008] The source memory anchor data packet is input into the memory semantic decoupling analyzer and decomposed into scene-independent style dimension feature vectors and scene-related content dimension feature vectors. The style dimension feature vectors include color tone, lighting style, emotional tone and sound effect rhythm mode.

[0009] The target scene description text is analyzed using a scene semantic parser, the content constraint set is extracted, a semantic anchor network structure is constructed, style-sensitive anchor nodes that have a significant impact on style expression are identified, and an anchor-media association matrix is ​​generated.

[0010] Construct a memory semantic projection matrix and use a cross-media style encoder to map the style dimension feature vector of the source memory scene to the stylized generation parameter vector of each medium. When mapping, the content constraints of the target scene and the scene context adaptability are considered.

[0011] The sentiment tags in the source memory anchor data package are converted into sentiment modulation vectors. Based on the anchor-media association matrix, the stylization generation parameter vectors and sentiment modulation vectors of each media are assigned to the style-sensitive anchor nodes. The style transfer strength weight between anchor nodes is calculated through the style consistency constraint propagation algorithm.

[0012] For each piece of media content in the target scene, the initial semantic representation and the style feature vector of the corresponding anchor point are fused using an attention weighting method to generate a style-conditional content representation;

[0013] The style-conditional content representations of each medium are input into the cross-media AIGC generator, and the conversion degree of each medium type is adjusted according to the style transfer intensity weight to generate cross-media scene content containing 3D models, texture images and audio.

[0014] Calculate the cross-media style similarity and emotional resonance between the generated target scene and the source memory scene. If the score is lower than the corresponding threshold, identify the media type with the largest style deviation, adjust the mapping weight of the memory semantic projection matrix, regenerate the media content, and output the optimized target scene.

[0015] Among them, the style consistency constraint propagation algorithm calculates the consistency constraint loss of adjacent anchor node pairs and iteratively adjusts the style transfer intensity weight of each anchor point, so that the style conversion degree of anchor nodes with high semantic similarity remains close, ensuring cross-media style coordination and consistency.

[0016] Furthermore, the memory semantic decoupling analyzer is implemented using a feature decoupling encoder, including the following steps:

[0017] Normalization processing is performed on scene semantic gene encoding, key visual element features, audio atmosphere features and lighting environment parameters respectively, and the numerical range of each feature vector is normalized to the interval to eliminate the dimensional differences of different feature types.

[0018] The input layer of the feature decoupling encoder receives the concatenated vector of normalized features. After preliminary feature transformation through the shared feature extraction layer, it is input to the style coding branch and the content coding branch respectively. The style coding branch outputs the style dimension feature vector, and the content coding branch outputs the content dimension feature vector.

[0019] The feature decoupling encoder uses a contrastive learning objective function during training. The first term ensures that the style features are robust to content changes by comparing the distance between the style features of the original sample and the style features of the content perturbation sample. The second term decouples style and content by minimizing the inner product of the style feature vector and the content feature vector. The weight coefficient of the second term is controlled to avoid over-decoupling and decoupling failure.

[0020] Furthermore, the method for identifying style-sensitive anchor nodes includes:

[0021] The style saliency score of each node in the semantic anchor network structure is calculated. The style saliency score is obtained by weighted summation of the node's influence on visual style, influence on audio style, and influence on spatial layout style.

[0022] The influence of a node on each style type is equal to the ratio of the sum of the weights of the corresponding style edges connected to the node to the total out-degree of the node. The edge weights are calculated by normalizing the co-occurrence frequencies of the two semantic concepts in the training dataset.

[0023] The weighting coefficients are determined based on the proportion of semantic tags involving visual elements, audio elements, and spatial layout elements in the target scene description text, ensuring that the style salience score matches the actual media type distribution of the target scene.

[0024] Nodes with style saliency scores higher than a preset threshold are selected as style-sensitive anchor nodes. The correlation strength between each anchor node and each media type is calculated. The correlation strength is equal to the style influence of the anchor node on the corresponding media type, and an anchor-media correlation matrix is ​​generated.

[0025] Furthermore, the method for constructing the memory semantic projection matrix is ​​as follows:

[0026] The memory semantic projection matrix is ​​obtained by multiplying the scene context adaptation transformation matrix, the diagonal weight matrix, and the transpose of the basis vector matrix of the source memory style features. The diagonal elements of the diagonal weight matrix represent the transfer strength of each style dimension.

[0027] The initial values ​​of the diagonal elements are determined by normalizing the absolute values ​​of the corresponding components in the source memory style dimension feature vector. Style dimension components with larger values ​​in the source memory scene obtain higher transfer strength.

[0028] Calculate the contextual similarity between the source memory scene and the target scene, extract the regional cultural features, spatial scale features and functional attribute features of the two scenes, and calculate the semantic distance of each dimension of features. The semantic distance is calculated by the cosine distance of the feature vectors.

[0029] Based on contextual similarity, the transfer intensity weight is dynamically adjusted. For scene pairs with large contextual differences, the transfer intensity of content-related style dimensions is reduced, while the transfer intensity of content-irrelevant style dimensions is increased, ensuring that the transferred scene is consistent with the target context.

[0030] Furthermore, the cross-media style encoder employs a multi-branch coding structure, including a visual media coding branch, an audio media coding branch, and a three-dimensional geometric media coding branch:

[0031] The input layer receives the concatenation of the style dimension feature vector of the source memory and the content constraint vector of the target scene. After extracting the cross-media general style representation through the shared coding layer, it is input into the coding branches of each media type. Each branch outputs the stylization generation parameter vector of the corresponding media type.

[0032] The cross-media style encoder is trained using a multi-task learning mode. The loss function includes style reconstruction loss for each media type and cross-media style consistency loss. The style reconstruction loss calculates the difference between the style features reconstructed by the generated parameters and the original style features. The cross-media style consistency loss maps the stylized generated parameter vectors of different media types to a unified style semantic space through a mapping function, and then constrains the distance between the parameters of different media types in this space to minimize, thus ensuring cross-media style consistency.

[0033] The stylization generation parameter vectors for each media type are obtained by multiplying the memory semantic projection matrix with the source memory style dimension feature vector and adding the media type-specific bias vector, and then calculating through an activation function. The output range of the activation function is a symmetrical interval, which facilitates the bidirectional adjustment of style features.

[0034] Furthermore, the specific steps of the style consistency constraint propagation algorithm include:

[0035] Initialize the style transfer intensity weights for each style-sensitive anchor node, using the normalized value of the magnitude of the assigned stylization generation parameter vector as the initial weights.

[0036] Traverse all adjacent anchor node pairs in the semantic anchor network structure. For two adjacent style-sensitive anchor nodes, calculate their consistency constraint loss. The consistency constraint loss is equal to the absolute value of the difference between the style transfer intensity weights of the two nodes minus the product of the constraint intensity coefficient and the semantic similarity of the two nodes, and take the larger of the maximum value and zero. The semantic similarity of the two nodes is calculated by the cosine similarity of the node feature vectors.

[0037] For anchor pairs with a consistency constraint loss greater than zero, the weight of the node with the larger weight is reduced, and the weight of the node with the smaller weight is increased. The adjustment range is proportional to the product of the consistency constraint loss and the edge weight between the nodes.

[0038] Calculate the total consistency constraint loss of all anchor pairs. If the total loss is less than the preset threshold or the number of iterations reaches the upper limit, output the style transfer intensity weight of each anchor point and terminate the algorithm. Otherwise, return to continue iterating. The preset threshold is set to five percent of the initial total loss.

[0039] Furthermore, the method for generating style-conditional content representations is as follows:

[0040] For each piece of media content in the target scene, obtain the style-sensitive anchor node corresponding to the media content and its stylized generation parameter vector and sentiment adjustment vector according to the anchor point style parameter allocation table;

[0041] The initial semantic representation of the target scene is passed through a learnable linear transformation layer to obtain the query vector, and the feature vectors of each associated anchor node are passed through a learnable linear transformation layer to obtain the key vector.

[0042] The attention weights of each anchor point are obtained by calculating the inner product of the query vector and each key vector and dividing by the square root of the vector dimension, and then by applying an exponential function and normalization.

[0043] The element-wise products of the initial semantic representation and the stylized generation parameter vector and style feature vector of each associated anchor point are weighted by attention weights, and then summed to obtain the stylized content representation.

[0044] The style-conditional content representation retains the semantic content constraints of the target scene but injects the style attributes and emotional atmosphere of the source memory, outputting a set of style-conditional representations of content from various media.

[0045] Furthermore, the cross-media AIGC generator adopts a conditional diffusion model architecture, including a noise encoder, a denoising network, and a media decoder:

[0046] The input layer receives style-conditional content representations and style transfer intensity weights, while the noise encoder maps the input to the latent representation space and adds noise.

[0047] The denoising network injects style-conditional content representations and temporal step embeddings into each layer, and gradually restores the scene content through the iterative denoising process.

[0048] The media decoder decodes the denoised latent representation into specific media data according to the media type. The visual media decoder outputs texture images, the 3D geometry decoder outputs mesh model data and object spatial layout data, and the audio decoder outputs audio waveform data.

[0049] When the memory anchor data packet is missing specific details of key visual element features or audio atmosphere features, a weighted random sampling algorithm is applied based on the user style preference profile to sample the missing style detail parameters from the style feature distribution of the user's historical preferences. The completed style detail parameters are then input as additional conditions into the cross-media AIGC generator.

[0050] Furthermore, the calculation method for cross-media style similarity and emotional resonance is as follows:

[0051] Cross-media style similarity is obtained by calculating the cosine similarity of the style feature vectors of the target scene and the source memory scene in each media type, and then averaging them according to the importance weight of each media type. The importance weight of the media type is determined based on the content proportion of that media type in the target scene.

[0052] The emotional resonance score is obtained by weighted combination of the matching degree of emotional tags and the user's emotional state monitoring score. The matching degree of emotional tags is calculated by the cosine similarity of the emotional tag vectors of the two scenes. The user's emotional state monitoring score is calculated by analyzing the physiological signals and facial expression data of the user when watching the generated scene and then normalized.

[0053] The method for identifying the media type with the greatest style deviation is to calculate the style deviation degree of each media type separately. The style deviation degree is defined as the cosine distance between the style feature vectors of the media type in the target scene and the source memory scene. The media type with the greatest style deviation degree is selected as the media type that needs to be regenerated. At the same time, the anchor node with the highest style transfer intensity weight among the anchor nodes associated with the media type is identified as the anchor node that needs to be adjusted.

[0054] This invention provides a dynamic generation system for metaverse scenes based on AIGC cross-media, comprising the following steps:

[0055] The generated memory-stylized target scene is presented to the user, and the user provides feedback ratings and suggestions for improvement on the degree of memory emotional restoration and cross-media style consistency.

[0056] The user's suggestions are interpreted as adjustments to specific style dimensions or media content, and the corresponding style feature parameters in the memory anchor data package are updated accordingly.

[0057] The memory anchor data package is updated using an incremental learning approach. For style dimensions where the user explicitly points out a deviation, the user's expected style feature values ​​are weighted and fused with the original feature values. The fusion weight is determined by the confidence level of the user's feedback.

[0058] The confidence weight of user feedback is determined by normalizing the average of user feedback ratings on memory emotional fidelity and cross-media style consistency. The higher the user rating, the greater the weight of the user expectation feature in the update.

[0059] The updated parameters are stored in the user's personalized model library to improve the accuracy of future memory style transfer tasks.

[0060] The beneficial effects of this invention are as follows:

[0061] This invention decomposes the features of the source memory scene into style and content dimensions by using a memory semantic decoupling analyzer, making style features independent of specific scene content. This solves the technical problem that the close coupling between style features and scene context leads to the inability to adapt across contexts, and achieves the technical effect of realizing personalized style transfer across scene contexts.

[0062] This invention establishes a mapping relationship between source memory style features and target scene generation parameters through a memory semantic projection matrix. This mapping relationship takes into account scene context adaptability and dynamically adjusts the transfer intensity weight according to the contextual similarity between the source memory scene and the target scene. This solves the technical problem of style inconsistency caused by contextual differences between different scenes and achieves the technical effect of intelligently mapping the warm colors of Parisian streets to the soft sunlight colors of Kyoto gardens and ensuring that the transferred scene is coordinated with the target context.

[0063] This invention identifies key style control points and establishes cross-media constraint relationships by identifying style-sensitive anchor points and using an anchor-media association matrix. It ensures that the style conversion degree of associated anchor points is coordinated and consistent through a style consistency constraint propagation algorithm. This solves the technical problem of inconsistent style conversion degree of 3D models, texture images and audio caused by independent conversion of different media, and achieves the technical effect of cross-media style coordination and consistency.

[0064] This invention solves the technical problem that relying solely on abstract encoding cannot accurately transfer styles by generating missing style detail parameters when the memory anchor data packet only contains abstract semantic gene encoding, through a detail completion method based on user style preference profiles. This achieves the technical effect of completing style details based on user personalized preferences.

[0065] This invention combines emotional resonance assessment with user emotional state monitoring to calculate the degree of matching between the emotional tags of the generated scene and the source memory scene, as well as the user's actual emotional state score. This solves the technical problem of focusing only on visual style while ignoring emotional atmosphere, and achieves the technical effect of ensuring that the migrated scene is not only similar in visual style but also resonates with the source memory scene in emotional atmosphere. Attached Figure Description

[0066] Figure 1 This is a flowchart of a method for dynamically generating metaverse scenes based on AIGC cross-media according to the present invention;

[0067] Figure 2 This is a bar chart comparing the style-content decoupling feature dimensions of this invention;

[0068] Figure 3 This is the anchor-medium correlation strength heat map of the present invention;

[0069] Figure 4 This is a line graph of the style consistency constraint propagation convergence curve of the present invention;

[0070] Figure 5 This is a mixed bar graph of the emotional resonance assessment dimension analysis of the present invention. Detailed Implementation

[0071] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0072] At least one embodiment of the present invention discloses a method for dynamically generating metaverse scenes based on AIGC across media, such as... Figure 1 As shown, it includes the following steps:

[0073] Step 100: Obtain the source memory anchor data packet and the target scene description;

[0074] The system acquires the source memory anchor identifier and target scene description text selected by the user, analyzes the source memory anchor identifier using a natural language scene retrieval tool, and retrieves matching memory anchor data packages from the user memory anchor database. The memory anchor data packages include scene semantic gene encoding, key visual element features, audio atmosphere features, lighting environment parameters, and sentiment tags to generate source memory scene data.

[0075] The aforementioned natural language scene retrieval system employs a dual-encoder structure, comprising a query encoder and an anchor encoder. The query encoder's input layer receives source anchor identifier text, passes it through a pre-trained language model's word embedding layer and multiple Transformer encoding layers, and outputs a query semantic vector. The anchor encoder uses the same structure to process the text descriptions of each anchor in the memory anchor database, outputting a set of anchor semantic vectors. During retrieval, the cosine similarity between the query semantic vector and each anchor semantic vector is calculated, and the anchor data packet with the highest similarity is returned. The natural language scene retrieval system is trained using a contrastive learning mode with cross-entropy loss as the loss function. Positive samples are matching query-anchor pairs, and negative samples are unmatched query-anchor pairs. The training data includes users' historical query and anchor interaction records, and the optimization strategy employs the Adam optimization algorithm.

[0076] It should be noted that scene semantic gene encoding is an abstract representation of the overall semantic features of a scene, using a high-dimensional vector form, including abstract semantic dimensions such as scene type, style tendency, and emotional tone. Key visual element features include color distribution histograms, texture pattern feature vectors, illumination direction vectors, and illumination color temperature values. Audio atmosphere features include audio spectrum distribution, rhythm pattern parameters, and timbre feature vectors. Emotional tags include emotional efficacy value, emotional arousal value, and emotional category identifiers.

[0077] In this embodiment of the application, in order to improve the accuracy of memory anchor retrieval, when using a natural language scene retrieval tool, the user's scene interaction history is also obtained, historical scene identifiers in the history that have a semantic similarity to the target scene description text exceeding a preset threshold are extracted, and the time relevance weight between each historical scene identifier and the source memory anchor identifier is calculated. The time relevance weight is calculated by inversely proportional to the distance between the two on the time axis. The time relevance weight is integrated into the similarity calculation of the retrieval to improve the personalization of the retrieval results.

[0078] Furthermore, the formula for calculating the time correlation weight is as follows: ,in For historical scene indexing, For the first The temporal correlation weight between each historical scene identifier and the source memory anchor identifier, with a value range of [value range missing]. , This represents the time interval between the two on the timeline, in hours. The time decay constant is set to a value of The formula ensures that historical scenes with shorter time intervals receive higher time relevance weights, thus achieving weighted memory retrieval based on time proximity.

[0079] In a specific application example, user Zhang experienced a virtual sunset scene along the Seine River in Paris on the Metaverse platform at 18:30 on October 15, 2024. This scene left a deep and romantic impression on the user. At 14:00 on October 20, 2024, the user wanted to generate a virtual garden scene of Kiyomizu-dera Temple in Kyoto and transfer the emotional atmosphere of the Paris scene to the new scene. The user's input source memory anchor was identified as "Paris Seine River experience on the evening of October 15, 2024", and the target scene description text was "Kiyomizu-dera Temple Japanese garden in Kyoto on an autumn afternoon, including maple trees, stone lanterns, gravel paths, and wooden corridors".

[0080] The natural language scene retrieval engine converts source memory anchor identifiers into 512-dimensional query semantic vectors and retrieves matching anchor data packets with a similarity of 0.89 from the user memory anchor database. The content of the retrieved memory anchor data packets is shown in Table 1.

[0081] Table 1. Contents of the memory anchor data package:

[0082]

[0083] Step 200: Decompose the source memory scene features using a memory semantic decoupling analyzer;

[0084] The source memory scene data is input into the memory semantic decoupling analyzer, which analyzes the scene semantic gene encoding, key visual element features, audio atmosphere features, and lighting environment parameters. These features are decomposed into scene-independent style dimension feature vectors and scene-related content dimension feature vectors. The style dimension feature vectors include color tendency, brushstroke style, emotional tone, and sound effect rhythm pattern, while the content dimension feature vectors include building type, geographical features, and cultural symbols, generating style-content decoupling features.

[0085] It should be noted that the extraction of style dimension feature vectors is based on the principle of feature transferability, meaning that style dimension feature vectors can exist independently of specific scene content. For example, color tone can be transferred from Parisian streets to Kyoto gardens, while architectural type belongs to scene-specific content and cannot be directly transferred. The decomposition process is implemented using a feature decoupling encoder. Before inputting the feature decoupling encoder, the scene semantic gene encoding, key visual element features, audio atmosphere features, and lighting environment parameters are respectively subjected to mean normalization based on the range, unifying and normalizing the numerical range of each feature vector to a uniform value. The interval eliminates the dimensional differences between different feature types. The input layer of the feature decoupling encoder receives the concatenated vector of normalized features. After preliminary feature transformation through the shared feature extraction layer, it is input to the style encoding branch and the content encoding branch respectively. The style encoding branch contains multiple fully connected layers and activation layers, and outputs style dimension feature vectors. The content encoding branch adopts the same layer structure and outputs content dimension feature vectors.

[0086] The aforementioned feature decoupling encoder uses a contrastive learning objective function during training:

[0087]

[0088] in, For sample index, Indicates the first The style dimension feature vector of each sample, Indicates the first Style feature vectors extracted from samples after content perturbation. Represents the content dimension feature vector. Represents the dot product of vectors. The regularization coefficient is . The training sample size is defined by the first term of the contrastive learning objective function, which ensures the robustness of style features to content variations. The second term decouples style and content features by minimizing their correlation. The training process employs supervised learning, using a scene sample dataset containing labeled style and content categories. The optimization strategy utilizes the Adam optimization algorithm, and during training, content perturbation pairs are generated for each sample as data augmentation.

[0089] Furthermore, the specific method for content perturbation is as follows: The building type, geographical features, and cultural symbol components in the content dimension feature vector of the original sample are randomly replaced. During replacement, the corresponding content component of another sample is randomly selected from the training dataset. The replacement ratio is equal to the total number of dimensions of the content dimension feature vector. to This perturbation method ensures that style features remain unchanged while content features change significantly, and is used to train the encoder to extract style features that are robust to content perturbations.

[0090] Furthermore, the regularization coefficient The range of values ​​is This range of values ​​ensures that the weight of the second decoupling constraint term is neither too small, causing it to fail and preventing the decoupling of style and content features, nor too large, weakening the effect of the first robust constraint term and causing style features to be overly sensitive to content disturbances.

[0091] The memory semantic decoupling analyzer performs feature decomposition on the source memory scene data in Table 1. First, the input features are normalized. Scene semantic gene encoding, key visual element features, audio ambient features, and lighting environment parameters are concatenated into a 1280-dimensional input vector, which is then normalized to... The decoupling encoder outputs a 128-dimensional style-dimensional feature vector from its style encoding branch and a 256-dimensional content-dimensional feature vector from its content encoding branch. The decoupling results are shown in Table 2.

[0092] Table 2 Style-Content Decoupling Feature Vectors:

[0093]

[0094] In this embodiment of the application, in order to extract the user's personalized style preference, after step 200 is executed, the user's scene interaction history within the time window before and after the source memory time is also analyzed. The time window is set to 30 minutes before and after the source memory time. The set of scenes visited by the user within the time window is extracted, and the similarity between each scene and the source memory scene in the style dimension feature vector is calculated. For scenes with similarity exceeding the threshold, the K-means clustering algorithm is applied to perform style feature clustering. The input of the K-means clustering algorithm is the set of style dimension feature vectors of scenes with similarity exceeding the threshold, and the output is the style category identifier of each scene and the cluster center vector of each category, generating a user style preference profile. The user style preference profile represents the distribution pattern of style features preferred by the user in a specific emotional state.

[0095] Figure 2 The key component values ​​of style dimension features and content dimension features extracted by the memory semantic decoupling analyzer are shown.

[0096] Step 300: Extract semantic constraints and style-sensitive anchors from the target scene;

[0097] This paper analyzes the target scene description text using a scene semantic parser, extracts the content constraint set of the target scene, including scene type, regional cultural characteristics, and main object categories, and constructs a semantic anchor network structure for the target scene. Nodes in the semantic anchor network structure represent semantic concepts in the scene, and edges represent the relationships between semantic concepts. Style-sensitive anchor nodes that have a significant impact on style expression are identified in the network. The media types covered by each style-sensitive anchor node are calculated, and an anchor-media association matrix is ​​generated. .

[0098] The aforementioned scene semantic parser employs a cascaded structure of sequence labeling and graph construction. The input layer receives the target scene description text and extracts its semantic representation through a pre-trained language model's word embedding layer and multiple Transformer encoding layers. The sequence labeling layer uses a fully connected layer and a softmax classifier to output the probability distribution of each semantic label category corresponding to each text position. By selecting the category index with the highest probability and mapping it to a predefined semantic label vocabulary, semantic labels such as scene type, regional cultural features, and object category are decoded. The graph construction layer generates nodes and edges of the semantic anchor network structure based on the labeling results and predefined semantic association rules. Node feature vectors are initialized by the semantic representation of the corresponding text segment, and edge weights are calculated from the co-occurrence frequency of semantic concepts. The scene semantic parser is trained using supervised learning, with the loss function being the cross-entropy loss of sequence labeling. The training data includes scene description text labeled with scene semantic tags, and the optimization strategy employs the Adam optimization algorithm.

[0099] Furthermore, the predefined semantic association rules include: establishing an association edge when the word distance between two semantic tags in the target scene description text is less than 5 words. The type of the edge is determined according to the combination of semantic tag categories. If both tags are visual types, the edge is a visual type edge; if one is a visual type and the other is an audio type, the edge is a cross-media association edge. The co-occurrence frequency is calculated by counting the number of times two semantic concepts co-occur in the same scene description in the training dataset. The total number of occurrences of each of the two semantic concepts is denoted as follows: and Then the edge weight is calculated as follows: This calculation method ensures that both co-occurrence frequency and occurrence count are normalized.

[0100] It should be noted that the identification of style-sensitive anchor nodes is based on a style saliency score, which is calculated using the following formula:

[0101]

[0102] in, Represents a node Style salience score, This indicates the degree of influence of the node on the visual style. Indicates the degree of influence on audio style. Indicates the degree of influence on the spatial layout style. , , For the weighting coefficients, satisfying .

[0103] The calculation methods for each influence level are as follows: Visual Style Influence Level Equal to node In a semantic anchor network structure, the ratio of the sum of the weights of the visual type edges connected to that node to the total out-degree of that node represents the audio style influence. Equal to node The ratio of the sum of the weights of the connected audio type edges to the total out-degree of the node, representing the influence of spatial layout style. Equal to node The ratio of the sum of the weights of the edges connecting spatial types to the total out-degree of that node. Anchor-medium relationship matrix. elements Indicates the first The anchor node and the first The association strength of each media type, with a value range of [value range missing]. .

[0104] Furthermore, the anchor-medium association matrix elements The calculation method is as follows: where For anchor node indexing, For media type index, if the first If the media type is visual media, then If it is an audio medium, then If it is a three-dimensional geometric medium, then ,in Indicates the first This calculation method establishes a direct mapping relationship between the style influence of anchor nodes on each media type and the strength of their association, ensuring that the strength of the association reflects the degree of contribution of anchor nodes to the style expression of the corresponding media.

[0105] Furthermore, the weighting coefficients , , Determined based on the proportion of media types in the target scenario.

[0106] Specifically, this involves calculating the percentage of semantic tags representing visual elements in the target scene description text, denoted as: The percentage of semantic tags involving audio elements is denoted as . The percentage of semantic tags involving spatial layout elements is denoted as . ,

[0107]

[0108] This determination method ensures that the style saliency score matches the actual media type distribution contained in the target scene.

[0109] In this embodiment, to improve the accuracy of style-sensitive anchor point identification, when constructing the semantic anchor point network structure, a historical generated sample set of similar scenes is also obtained. The actual contribution of each semantic concept node in the sample set in the style transfer task is analyzed. The actual contribution is calculated by comparing the change in style feature vectors before and after the transfer. The historical contribution is incorporated as prior knowledge into the style saliency score calculation, and the revised scoring formula adds a historical contribution term. ,in For historical knowledge weighting coefficients, For nodes The historical contribution statistics.

[0110] Furthermore, the node Historical contribution statistics The calculation formula is: ,in For historical sample index, For historical sample sets containing nodes The number of samples, For the first The corresponding node for each historical sample before style transfer Style feature vectors, The calculated value is the style feature vector after migration. This method measures the actual contribution of a node to style migration by statistically averaging the Euclidean distance between the style feature vectors before and after migration.

[0111] Furthermore, the historical knowledge weighting coefficient The range of values ​​is When the number of samples in the historical sample set is greater than 100, Pick When the sample size is less than 10, Pick When the sample size is between 10 and 100, The value is determined by linear interpolation based on the sample size. This range ensures that the weight of historical knowledge is not too large and thus does not mask the style saliency score calculated based on the current scenario, avoiding excessive interference from historical prior knowledge in the analysis of the current scenario.

[0112] The scene semantic parser analyzes the target scene description text "A Japanese garden at Kiyomizu-dera Temple in Kyoto on an autumn afternoon, including maple trees, stone lanterns, pebble paths, and wooden corridors," extracting the scene type as "Japanese garden," the regional cultural feature as "Kyoto Zen culture," and the main object categories as "maple trees, stone lanterns, pebble paths, and wooden corridors." The constructed semantic anchor network contains 8 nodes connected by 15 edges. The style saliency score for each node is calculated, including the proportion of visual labels. audio tag percentage spatial tag percentage To obtain the weighting coefficients , , Five style-sensitive anchor nodes were identified, and the association strength between each anchor node and three media types was calculated to generate an anchor-media association matrix. The identification results and association matrix are shown in Table 3.

[0113] Table 3. Style-sensitive anchor node identification results and anchor-media association matrix:

[0114]

[0115] Figure 3 It shows the correlation strength matrix between five style-sensitive anchor nodes (N1-maple color, N2-stone lantern shape, N3-light and shadow atmosphere, N4-pebble path texture, N5-ambient sound effect) and three media types (visual media, audio media, and three-dimensional geometric media).

[0116] Step 400: Generate the memory semantic projection matrix and stylization generation parameters;

[0117] Constructing a memory semantic projection matrix The memory semantic projection matrix defines the mapping relationship between the style dimension feature vector of the source memory scene and the generation parameter space of each medium in the target scene. The mapping rule considers the content constraints and contextual adaptability of the target scene. The style mapping function in the representation space of each medium is calculated by using a cross-media style encoder. For visual media, the hue tendency feature of the source memory is mapped to the color adjustment parameters of the target scene. For audio media, the sound effect rhythm pattern of the source memory is mapped to the audio rhythm parameters of the target scene, generating stylized generation parameter vectors for each medium. ,in The first To the Stylization generation parameter vectors for each medium This represents the number of media types.

[0118] The aforementioned cross-media style encoder employs a multi-branch encoding structure. Before inputting into the cross-media style encoder, the style dimension feature vector of the source memory and the content constraint vector of the target scene are Z-score standardized to eliminate the dimensional differences between vectors from different sources. The input layer receives the concatenation of the two standardized vectors, extracts a cross-media general style representation through a shared encoding layer, and then inputs it into the visual media encoding branch, audio media encoding branch, and 3D geometric media encoding branch, respectively. Each branch contains fully connected layers and residual connection structures, outputting the stylized generation parameter vector for the corresponding media. The cross-media style encoder is trained using a multi-task learning mode. The loss function includes the style reconstruction loss for each media and the cross-media style consistency loss. The style reconstruction loss calculates the difference between the style features reconstructed by the generated parameters and the original style features. The cross-media style consistency loss constrains the alignment degree of style parameters for different media in the style semantic space. The optimization strategy employs the Adam optimization algorithm.

[0119] Furthermore, the formula for calculating the style reconstruction loss is as follows:

[0120] ,in For media indexing, For the number of media types, For the first The original style feature vector of each medium, To generate parameter vectors through stylization The reconstructed style feature vector. The formula for calculating the cross-media style consistency loss is: ,in For media indexing, As a mapping function that maps stylized generation parameter vectors to a unified style semantic space, cross-media style consistency loss ensures cross-media style consistency by minimizing the distance between parameters of different media in the semantic space.

[0121] Furthermore, the total loss function of the cross-media style encoder is:

[0122] ,in The weighting coefficients for the style reconstruction loss are set to [value]. , The weighting coefficient for cross-media style consistency loss is set to [value]. This weighting setting ensures that while guaranteeing the accuracy of style reconstruction across media, it also imposes appropriate constraints on cross-media style consistency, avoiding excessive constraints that could limit the expressive power of each media style.

[0123] Furthermore, the mapping function Implemented using a fully connected neural network, the input is a stylization generation parameter vector, and the output is a representation vector in the unified style semantic space. The network contains two fully connected layers. The first layer has twice the number of neurons as the dimension of the stylization generation parameter vector and uses the ReLU activation function. The second layer has the number of neurons equal to the dimension of the unified style semantic space. Without an activation function, this mapping function learns through a neural network to map stylization generation parameter vectors from different media to a unified style semantic space for alignment.

[0124] It should be noted that the memory semantic projection matrix The calculation formula is:

[0125]

[0126] in, Indicates transpose. Let be the basis vector matrix of the source memory style features, with dimension . , As a dimension of style characteristics, This represents the number of style basis vectors. The first To the The transfer intensity of each style dimension This is a diagonal weight matrix, with diagonal elements. Indicates the first The transfer strength of each style dimension, with values ​​ranging from... , The transformation matrix is ​​adapted to the scene context, with dimensions of , Generate parameter dimensions for the target scene. The formula for calculating the stylized generation parameter vector for each medium is as follows:

[0127]

[0128] in, For the first Stylization generation parameter vectors for each medium The style dimension feature vector of the source memory. For the first The bias vector of a medium. This is the activation function.

[0129] Furthermore, the activation function The Tanh function is used and defined as follows: The output range is This activation function ensures that the values ​​of each component of the stylization generation parameter vector are within a symmetrical range, which facilitates the bidirectional adjustment of style features, such as adjusting the hue towards warm or cool colors.

[0130] Furthermore, the scene context adaptation transformation matrix This is achieved through a learnable neural network layer, where the input to the network layer is the content constraint vector of the target scene, and the output is the transformation matrix. During training, a supervised learning mode is adopted, and the loss function is the mean square error between the generated stylized parameter vector and the labeled target style parameter vector. The training data is a set of scene pairs. This method enables the transformation matrix to adaptively adjust the style mapping relationship according to the specific content constraints of the target scene.

[0131] Furthermore, the diagonal element The initial value is based on the first Each style dimension is determined by normalizing the absolute values ​​of its feature vector components in the source memory scene, specifically: ,in For style dimension indexing, The first feature vector representing the style dimension of the source memory One portion, The first feature vector representing the style dimension of the source memory This method ensures that style dimension components with larger values ​​in the source memory scene receive higher transfer strength, and achieves adaptive adjustment of transfer weights based on the saliency of the source memory style features.

[0132] In this embodiment, to achieve intelligent adaptation of scene context, when constructing the memory semantic projection matrix, the contextual similarity between the source memory scene and the target scene is also calculated. Regional cultural features, spatial scale features, and functional attribute features of the two scenes are extracted, and the semantic distance of each dimension of features is calculated. The larger the semantic distance, the greater the contextual difference. The migration strength weight is dynamically adjusted based on the contextual similarity. For scenarios with significant differences in context, reduce the transfer intensity of content-related style dimensions and increase the transfer intensity of content-irrelevant style dimensions to ensure that the transferred scenario is consistent with the target context.

[0133] Furthermore, the semantic distance is calculated as follows: the regional cultural features, spatial scale features, and functional attribute features of the source memory scene and the target scene are encoded into feature vectors respectively. Regional cultural features are converted into vectors using one-hot encoding, while spatial scale features and functional attribute features are encoded numerically. The cosine distance of each dimension of the feature vectors is calculated. The cosine distance is defined as... The range of values ​​is The total semantic distance is obtained by weighted summation of the cosine distances of each dimension. This calculation method quantifies the degree of difference in semantic features between the source memory scene and the target scene.

[0134] Furthermore, the formula for calculating the total semantic distance is as follows:

[0135] ,in , , These are the cosine distances to regional cultural characteristics, spatial scale characteristics, and functional attribute characteristics, respectively. , , These are the corresponding weight coefficients, with values ​​of respectively. , , ,satisfy This weighting ensures that regional cultural characteristics, as the primary dimension of contextual differences, receive the highest weight, while spatial scale and functional attributes, as secondary dimensions, receive lower weights.

[0136] The cross-media style encoder concatenates the 128-dimensional style feature vector from Table 2 with the target scene content constraint vector after Z-score normalization, and inputs them into three encoding branches: visual media, audio media, and 3D geometric media. It calculates the contextual similarity between the source and target scenes and the cosine distance of regional cultural features. Spatial scale characteristic cosine distance Functional attribute feature cosine distance The total semantic distance is obtained. Based on significant contextual differences, the system reduces the transfer intensity of content-related style dimensions and increases the transfer intensity of content-irrelevant dimensions such as color tone and lighting. (Three media types) Corresponding stylization generation parameter vectors are generated for each medium, with the visual media parameter vector having a dimension of 80, the audio media parameter vector having a dimension of 60, and the 3D geometric media parameter vector having a dimension of 40. The detailed composition of each medium's stylization generation parameter vector is shown in Table 4.

[0137] Table 4. Detailed composition of the stylization generation parameter vectors for each media:

[0138]

[0139] Step 500: Generate sentiment modulation vectors and assign them to anchor nodes;

[0140] The emotional tags in the source memory scene data are converted into emotional modulation vectors for the target scene. During the conversion, the emotional efficacy value and emotional arousal value are normalized to the mean based on the range. In the interval, the emotion category identifier is converted into a numerical vector using one-hot encoding. The emotion regulation vector includes the emotion valence regulation component, the emotion arousal regulation component, and the emotion category mapping parameter. Based on the anchor-media association matrix generated in step 300, the stylization generation parameter vector and emotion regulation vector of each media are assigned to the corresponding style-sensitive anchor nodes. Through the style consistency constraint propagation algorithm, the style transfer strength weight between anchor nodes is calculated to ensure that the style conversion degree of the associated anchors remains coordinated, and an anchor style parameter allocation table is generated.

[0141] It should be noted that the input to the style consistency constraint propagation algorithm is the semantic anchor network structure and the initial style transfer strength weights of each anchor point, and the output is the optimal style transfer strength weights of each anchor point that satisfy the consistency constraint. The style consistency constraint propagation algorithm includes the following steps:

[0142] Step S1: Initialize the style transfer intensity weights of each style-sensitive anchor node, and use the normalized value of the modulus of the stylization generation parameter vector assigned in step 500 as the initial weights.

[0143] Step S2: Traverse all adjacent anchor node pairs in the semantic anchor network structure. For two adjacent style-sensitive anchor nodes... and Calculate its consistency constraint loss:

[0144]

[0145] in, and Representing nodes respectively and Style transfer intensity weights, The constraint strength coefficient, The semantic similarity between two nodes is calculated using the node... and eigenvectors and The cosine similarity, i.e. The range of values ​​is ;

[0146] Step S3: For anchor pairs with consistency constraint loss greater than zero, reduce the weight of the node with the larger weight.

[0147] Increase the weight of nodes with smaller weights by a factor proportional to the product of the consistency constraint loss and the edge weights between nodes. The specific adjustment formula is as follows:

[0148]

[0149] in The learning rate is set to a value of , For nodes and Edge weights between them;

[0150] Step S4: Calculate the total consistency constraint loss of all anchor point pairs. If the total loss is less than the preset threshold or the number of iterations reaches the upper limit, output the style migration intensity weight of each anchor point and terminate the style consistency constraint propagation algorithm. Otherwise, return to step S2 to continue iterating.

[0151] Furthermore, the constraint strength coefficient Values This value ensures that when the semantic similarity between two anchor nodes is 1... When they are completely similar, the allowed difference in style transfer intensity weights is: When semantic similarity decreases, the allowed weight difference decreases accordingly, achieving the constraint that anchor nodes with higher similarity have a closer degree of style transfer. The preset threshold is set as a fraction of the initial total loss. The upper limit of the number of iterations is set to In the next iteration, this threshold setting ensures that the style consistency constraint propagation algorithm terminates after the consistency constraint loss has fully converged, avoiding the waste of computational resources caused by excessive iteration.

[0152] The emotional labels of the source memory scenes in Table 1 were converted into emotional regulation vectors. The emotional efficacy value of 0.82 and the emotional arousal value of 0.58 remained unchanged after normalization. The emotional categories "romantic, nostalgic" were converted into 5-dimensional one-hot encoded vectors. Based on the anchor-media association matrix in Table 3, the corresponding sub-parameters of the media stylization generation parameter vectors in Table 4 are assigned to the five style-sensitive anchor nodes according to the association strength. The style consistency constraint propagation algorithm is executed, initializing the style migration strength weight of each anchor point to the normalized value of the parameter vector magnitude. After 23 iterations, the consistency constraint loss decreases from the initial value of 8.64 to 0.43, falling below the threshold of 0.432, at which point the algorithm terminates. The anchor point style parameter allocation results are shown in Table 5.

[0153] Table 5 Anchor Point Style Parameter Allocation Table:

[0154]

[0155] Figure 4 The iterative convergence process of the style consistency constraint propagation algorithm is demonstrated. It is verified that the style consistency constraint propagation algorithm can effectively optimize the style transfer intensity weights between anchor points, ensuring that the degree of style transformation between associated anchor points remains consistent.

[0156] Step 600: Generate a style-conditional content representation;

[0157] For each media content in the target scene, based on the anchor point style parameter allocation table generated in step 500, the style-sensitive anchor point node corresponding to the media content and its stylized generation parameter vector and sentiment adjustment vector are obtained. The initial semantic representation of the target scene is fused with the style feature vector of the corresponding anchor point. During the fusion, an attention weighting method is used to calculate the attention weight of each dimension of the initial semantic representation with the style feature vector, and a style-conditional content representation is generated. The style-conditional content representation retains the semantic content constraints of the target scene but injects the style attributes and emotional atmosphere of the source memory, and outputs the set of style-conditional content representations for each media.

[0158] It should be noted that the fusion formula for style-conditional content representation is:

[0159]

[0160] in, For anchor index, For style-conditional content representation, This serves as the initial semantic representation of the target scene. A set of style-sensitive anchor nodes associated with current media. anchor point Attention weights anchor point Stylization generates parameter vectors, anchor point Style feature vectors, This represents element-wise multiplication. Attention weights. The calculation formula is:

[0161]

[0162] in, Represents an exponential function. Indicates transpose. For anchor index, To start from the initial semantic representation Extracted query vector, anchor point The key vector, anchor point The key vector, For vector dimensions.

[0163] Furthermore, the query vector By initial semantic representation The input is fed into a learnable linear transformation layer, whose transformation matrix has a dimension of . ,in The dimension of the initial semantic representation. The target dimension of the query vector is calculated using the following formula: ,in To query the transformation matrix. The key vector By anchoring Node feature vectors The input is fed into a learnable linear transformation layer, whose transformation matrix has a dimension of . ,in The dimension of the node feature vector is calculated using the following formula: ,in The key transformation matrix is ​​used to ensure that the similarity calculation of the query vector and the key vector is performed in the same semantic space.

[0164] Furthermore, the vector dimension The range of values ​​is The preferred value is This dimensional setting ensures that the attention mechanism has sufficient expressive power to capture complex style semantic relationships, while avoiding excessive computational complexity and overfitting risks caused by excessive dimensionality, thus achieving a balance between expressive power and computational efficiency.

[0165] In this embodiment of the application, in order to improve the accuracy of style injection, when generating style-conditional content representation, the style plasticity of each semantic component in the initial semantic representation is also analyzed. The plasticity represents the range of changes allowed for the component during style transfer. For components with low style plasticity (such as regional cultural objects in the scene), the corresponding style injection intensity is reduced. For components with high style plasticity (such as lighting tone and texture details), the style injection intensity is increased. Through style plasticity weighted modulation, over-stylization is avoided to prevent semantic distortion of the target scene.

[0166] Furthermore, the method for calculating style plasticity is as follows: The variation range of each semantic component in the historical training sample set during the style transfer task is statistically analyzed, and the standard deviation of the style feature vector of each component in the successfully transferred samples is calculated. The larger the standard deviation, the higher the style plasticity of the component. The standard deviation is then normalized to... The interval serves as a style plasticity score. For semantic components in the initial semantic representation, the corresponding style plasticity score is mapped according to their component type. This calculation method determines the degree of variability of each type of semantic component based on historical data statistics.

[0167] Initial semantic representations were generated for each media content in the Kyoto garden scene, including a 512-dimensional visual representation of maple trees, a 384-dimensional geometric representation of stone lanterns, and a 256-dimensional audio representation of the environment. According to the anchor point style parameter allocation table in Table 5, anchor point N1 is associated with the maple tree visual content, anchor point N2 with the stone lantern content, and anchor point N5 with the environment audio content. When calculating attention weights, the query vector... The initial semantic representation is transformed linearly to obtain a 256-dimensional vector, the key vector. A 256-dimensional vector is obtained by transforming the feature vector of the anchor node. Examples of style-conditional fusion of maple tree visual content are shown in Table 6.

[0168] Table 6. Example of generating style-conditional content representation:

[0169]

[0170] Step 700: Invoke the cross-media AIGC generator to generate content for the target scene;

[0171] The style-conditional content representations of each medium generated in step 600 are input into the cross-media AIGC generator. The cross-media AIGC generator adjusts the conversion degree of each medium according to the style transfer intensity weight determined in step 500. For 3D model media, the cross-media AIGC generator outputs the geometric model data and object spatial layout data of the target scene. For texture image media, the cross-media AIGC generator outputs texture image data with applied source memory style. For audio media, the cross-media AIGC generator outputs ambient sound effect data and background music data to generate cross-media scene content.

[0172] The aforementioned cross-media AIGC generator employs a conditional diffusion model architecture, comprising three units: a noise encoder, a denoising network, and a media decoder. The input layer receives style-conditional content representations and style transfer intensity weights. The noise encoder maps the input to the latent representation space and adds noise. The denoising network uses a U-Net structure, including multi-layer convolutional downsampling and upsampling modules, as well as cross-layer skip connections. Style-conditional content representations and temporal embeddings are injected into each layer, gradually recovering scene content through iterative denoising. The media decoder decodes the denoised latent representation into specific media data based on the media type. The visual media decoder outputs texture images, the 3D geometry decoder outputs mesh model data, and the audio decoder outputs audio waveform data. During training, the cross-media AIGC generator uses an inverse denoising objective of the diffusion process. The loss function calculates the mean squared error between the predicted noise and the actual added noise. The training data includes scene-style paired samples, and the optimization strategy employs the Adam optimization algorithm.

[0173] It should be noted that if the memory anchor data packet only contains scene semantic gene encoding but lacks specific details of key visual element features or audio atmosphere features, based on the user style preference profile generated in step 200, a weighted random sampling algorithm is applied to sample the missing style detail parameters from the style feature distribution of the user's historical preferences. The input of the weighted random sampling algorithm is the vector of each cluster center in the user style preference profile and its corresponding occurrence frequency, and the output is the sampled style detail parameter vector. The sampling weight is determined by the occurrence frequency of each cluster center. The completed style detail parameters are input as additional conditions into the cross-media AIGC generator to guide the cross-media AIGC generator to produce scene content that conforms to the user's personalized style preferences.

[0174] In this embodiment of the application, in order to ensure the quality of cross-media content generation, a phased generation strategy is also adopted during the generation process of calling the cross-media AIGC generator. The first phase generates the overall layout of the scene and low-resolution versions of the main objects. The style features of the first phase generation result are extracted and their similarity with the style features of the source memory is calculated. If the similarity is lower than a preset threshold, the style injection weight in the style conditional content representation is adjusted and the first phase generation is re-executed. If the similarity meets the standard, the second phase is entered to generate high-resolution details and secondary scene elements. Through phased verification, the cost of regeneration after the failure of full generation is reduced.

[0175] The cross-media AIGC generator receives the style-conditional content representation from Table 6 as input and uses a conditional diffusion model for denoising and generation. The visual media decoder generates texture image data, the 3D geometry decoder outputs geometric model data and object spatial layout data containing 8356 vertices and 15742 triangles, and the audio decoder generates ambient sound effects data and background music data with a sampling rate of 48kHz. The generation process involves 50 denoising iterations, and the generation results for each medium are shown in Table 7.

[0176] Table 7. Cross-media AIGC generation results:

[0177]

[0178] Step 800: Evaluate style similarity and emotional resonance and perform optimized generation;

[0179] The cross-media style similarity and emotional resonance of the target scene generated in step 700 and the source memory scene are calculated. The style similarity is obtained by extracting the style feature vectors of each medium in the two scenes and calculating the cosine distance. The emotional resonance is calculated by comparing the matching degree of the emotional tags of the two scenes and combining the user's emotional state monitoring data. If the score of style similarity or emotional resonance is lower than the corresponding threshold, the medium with the largest style deviation and the corresponding anchor node are identified. The mapping weight of the medium and the style constraint strength of the corresponding anchor point in the memory semantic projection matrix are adjusted to trigger the regeneration of the content of the medium and output the optimized target scene content.

[0180] Furthermore, the method for identifying the medium with the greatest style deviation is as follows: calculate the style deviation degree of each medium separately, defined as... ,in For the first The style deviation of each medium, with a value range of [value range missing]. The method selects the medium with the largest style deviation as the medium that needs to be regenerated, and identifies the anchor node with the highest style transfer intensity weight among the anchor nodes associated with the medium as the anchor node that needs to be adjusted. This method locates the optimization target by quantifying the style deviation of each medium.

[0181] It should be noted that the formula for calculating cross-media style similarity is:

[0182]

[0183] in, For media indexing, For cross-media style similarity, For the number of media types, For the first The importance weight of each medium and The target scene and the source memory scene are respectively in the first... A vector of stylistic features for each medium. Emotional resonance. The calculation formula is:

[0184]

[0185] in, The degree of matching of sentiment tags is calculated using the cosine similarity of the sentiment tag vectors of the two scenes, with a value range of [value range missing]. , To obtain a user's emotional state monitoring score, the raw monitoring value is calculated by analyzing the user's physiological signals and facial expression data while viewing the generated scene. Then, a mean normalization method based on the range is used to normalize this value to a normalized value. The interval ensures that it is consistent with... The dimensions are consistent. This is a balance coefficient between tag matching and user state, with a value range of [value range missing]. .

[0186] Furthermore, the first The importance weight of each media The importance weight is determined based on the content proportion of the medium in the target scene. For visual media, the importance weight equals the ratio of the number of semantic tags of the visual element to the total number of semantic tags in the target scene. The same calculation method is used for audio and 3D geometric media. The sum of the importance weights of all media satisfies... (in (For media indexing), this determination method ensures that the weight of each media in the style similarity assessment matches its actual importance in the scene. The corresponding threshold for style similarity is set as follows: The threshold corresponding to the emotional resonance is set as follows: When the style similarity or emotional resonance score falls below a corresponding threshold, regeneration is triggered. This threshold is set to ensure that the generated target scene achieves sufficient similarity in style and emotion to the source memory scene. The balance coefficient... Values This value ensures that the weight of the emotional tag matching degree is slightly higher than the weight of the user's emotional state monitoring score, so as to achieve a comprehensive evaluation of objective tag matching and subjective user feelings.

[0187] The Kyoto garden scenes generated in Table 7 were evaluated. Style feature vectors for each medium were extracted, and cosine similarity was calculated between the media and the source memory scenes in Table 1. Visual media similarity was 0.78, audio media similarity was 0.74, and 3D geometric media similarity was 0.82. The similarity was determined based on media importance weights. , , Cross-media style similarity was calculated:

[0188] It is higher than the threshold of 0.7.

[0189] Extract the sentiment tags from the generated scene and calculate the cosine similarity between them and the sentiment tags of the source memory scene. A user wearing an emotion monitoring device watched a scene for 90 seconds, and the raw emotion valence value was 7.8, which was normalized to... Calculate emotional resonance:

[0190] The value is 0.6 above the threshold. The evaluation results are shown in Table 8. Both indicators meet the standards and do not need to be regenerated.

[0191] Table 8. Results of style similarity and emotional resonance assessment:

[0192]

[0193] Figure 5 The study presents a detailed dimensional analysis of the emotional resonance assessment, verifying that the generated scenes not only resemble the source memory scenes in terms of visual style, but also resonate with them in terms of emotional atmosphere.

[0194] This implementation decomposes the features of the source memory scene into style and content dimensions using a memory semantic decoupling analyzer. This makes style features independent of specific scene content, overcoming the problem of incompatibility caused by the tight coupling of style features and scene context. A memory semantic projection matrix establishes a mapping relationship between source memory style features and target scene generation parameters. This mapping relationship considers scene context adaptability, allowing the warm tones of Parisian streets to be intelligently mapped to the soft sunlight tones of a Kyoto garden, and the auditory metaphor of coffee aroma to be mapped to the auditory representation of tea aroma, solving the style inconsistency problem caused by differences in scene context. Key style control points are located and cross-media constraint relationships are established through style-sensitive anchor point identification and anchor-media association matrix. A style consistency constraint propagation algorithm ensures that the style conversion degree of associated anchor points is consistent, avoiding the problem of inconsistent conversion degrees where the 3D model has completed style transfer but the audio still retains the original style due to independent conversion between different media. A detail completion method based on user style preference profiles generates missing style detail parameters when the memory anchor data package only contains abstract semantic gene encoding, solving the problem that relying solely on abstract encoding cannot accurately transfer style. By combining emotional resonance assessment with user emotional state monitoring, this method ensures that the migrated scene not only resembles the original scene in visual style but also resonates with it in emotional atmosphere, thus solving the problem of focusing solely on visual style while neglecting emotional atmosphere. In summary, this implementation method achieves personalized style migration scene generation across contexts, media, and with emotional preservation, based on personal memory and transcending conventional boundaries.

[0195] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for dynamically generating metaverse scenes based on AIGC cross-media, characterized in that, Includes the following steps: Obtain the source memory anchor data package selected by the user and the target scene description text. The source memory anchor data package includes scene semantic gene encoding, key visual element features, audio atmosphere features, lighting environment parameters and sentiment tags. The source memory anchor data packet is input into the memory semantic decoupling analyzer and decomposed into scene-independent style dimension feature vectors and scene-related content dimension feature vectors. The style dimension feature vectors include color tone, lighting style, emotional tone and sound effect rhythm mode. The memory semantic decoupling analyzer is implemented using a feature decoupling encoder, and includes the following steps: Normalization processing is performed on scene semantic gene encoding, key visual element features, audio atmosphere features and lighting environment parameters respectively, and the numerical range of each feature vector is normalized to the interval to eliminate the dimensional differences of different feature types. The input layer of the feature decoupling encoder receives the concatenated vector of normalized features. After preliminary feature transformation through the shared feature extraction layer, it is input into the style coding branch and the content coding branch respectively. The style coding branch outputs the style dimension feature vector, and the content coding branch outputs the content dimension feature vector. The feature decoupling encoder uses a contrastive learning objective function during training. The first term ensures that the style features are robust to content changes by comparing the distance between the style features of the original sample and the style features of the content perturbation sample. The second term decouples style and content by minimizing the inner product of the style feature vector and the content feature vector. The weight coefficient of the second term is controlled to avoid over-decoupling and decoupling failure. The specific method of content perturbation is as follows: randomly replace the building type, geographical features and cultural symbol components in the content dimension feature vector of the original sample, and randomly select the corresponding content component of another sample from the training dataset during the replacement. The target scene description text is analyzed using a scene semantic parser, the content constraint set is extracted, a semantic anchor network structure is constructed, style-sensitive anchor nodes that have a significant impact on style expression are identified, and an anchor-media association matrix is ​​generated. The predefined semantic association rules include: when the word distance between two semantic tags in the target scene description text is less than 5 words, an association edge is established, and the type of the edge is determined according to the combination of the semantic tag categories; The method for identifying the style-sensitive anchor nodes includes: The style saliency score of each node in the semantic anchor network structure is calculated. The style saliency score is obtained by weighted summation of the node's influence on visual style, influence on audio style, and influence on spatial layout style. The influence of a node on each style type is equal to the ratio of the sum of the weights of the corresponding type edges connected to the node to the total out-degree of the node. The edge weights are calculated by normalizing the co-occurrence frequencies of the two semantic concepts in the training dataset. The weighting coefficients are determined based on the proportion of semantic tags involving visual elements, audio elements, and spatial layout elements in the target scene description text, ensuring that the style salience score matches the actual media type distribution included in the target scene. Nodes with style saliency scores higher than a preset threshold are selected as style-sensitive anchor nodes. The correlation strength between each anchor node and each media type is calculated. The correlation strength is equal to the style influence of the anchor node on the corresponding media type. An anchor-media correlation matrix is ​​generated. Construct a memory semantic projection matrix and use a cross-media style encoder to map the style dimension feature vector of the source memory scene to the stylized generation parameter vector of each medium. When mapping, the content constraints of the target scene and the scene context adaptability are considered. The method for constructing the memory semantic projection matrix is ​​as follows: The memory semantic projection matrix is ​​obtained by multiplying the scene context adaptation transformation matrix, the diagonal weight matrix, and the transpose of the basis vector matrix of the source memory style features. The diagonal elements of the diagonal weight matrix represent the transfer strength of each style dimension. The initial values ​​of the diagonal elements are determined by normalizing the absolute values ​​of the corresponding components in the source memory style dimension feature vector. Style dimension components with larger values ​​in the source memory scene obtain higher transfer strength. Calculate the contextual similarity between the source memory scene and the target scene, extract the regional cultural features, spatial scale features and functional attribute features of the two scenes, and calculate the semantic distance of each dimension of features. The semantic distance is calculated by the cosine distance of the feature vectors. The regional cultural features, spatial scale features, and functional attribute features of the source memory scene and the target scene are encoded into feature vectors respectively. The regional cultural features are converted into vectors using one-hot encoding, while the spatial scale features and functional attribute features are encoded using numerical values. Based on contextual similarity, the transfer intensity weight is dynamically adjusted. For scene pairs with large contextual differences, the transfer intensity of content-related style dimension is reduced, while the transfer intensity of content-irrelevant style dimension is increased to ensure that the scene after transfer is consistent with the target context. The sentiment tags in the source memory anchor data package are converted into sentiment modulation vectors. Based on the anchor-media association matrix, the stylization generation parameter vectors and sentiment modulation vectors of each media are assigned to the style-sensitive anchor nodes. The style transfer strength weight between anchor nodes is calculated through the style consistency constraint propagation algorithm. For each piece of media content in the target scene, the initial semantic representation and the style feature vector of the corresponding anchor point are fused using an attention weighting method to generate a style-conditional content representation; The style-conditional content representations of each medium are input into the cross-media AIGC generator, and the conversion degree of each medium type is adjusted according to the style transfer intensity weight to generate cross-media scene content containing 3D models, texture images and audio. Calculate the cross-media style similarity and emotional resonance between the generated target scene and the source memory scene. If the score is lower than the corresponding threshold, identify the media type with the largest style deviation, adjust the mapping weight of the memory semantic projection matrix, regenerate the media content, and output the optimized target scene. Among them, the style consistency constraint propagation algorithm calculates the consistency constraint loss of adjacent anchor node pairs and iteratively adjusts the style transfer intensity weight of each anchor point, so that the style conversion degree of anchor nodes with high semantic similarity remains close, ensuring cross-media style coordination and consistency.

2. The method for dynamically generating metaverse scenes based on AIGC cross-media as described in claim 1, characterized in that, The cross-media style encoder employs a multi-branch coding structure, including a visual media coding branch, an audio media coding branch, and a three-dimensional geometric media coding branch: The input layer receives the concatenation of the style dimension feature vector of the source memory and the content constraint vector of the target scene. After extracting the cross-media general style representation through the shared coding layer, it is input into the coding branches of each media type. Each branch outputs the stylization generation parameter vector of the corresponding media type. The cross-media style encoder is trained using a multi-task learning mode. The loss function includes style reconstruction loss for each media type and cross-media style consistency loss. The style reconstruction loss calculates the difference between the style features reconstructed by the generated parameters and the original style features. The cross-media style consistency loss maps the stylized generated parameter vectors of different media types to a unified style semantic space through a mapping function, and then constrains the distance between the parameters of different media types in this space to minimize, thus ensuring cross-media style consistency. The stylization generation parameter vectors for each media type are obtained by multiplying the memory semantic projection matrix with the source memory style dimension feature vector and adding the media type-specific bias vector, and then calculating through an activation function. The output range of the activation function is a symmetrical interval, which facilitates the bidirectional adjustment of style features.

3. The method for dynamically generating metaverse scenes based on AIGC cross-media as described in claim 1, characterized in that, The specific steps of the style consistency constraint propagation algorithm include: Initialize the style transfer intensity weights for each style-sensitive anchor node, using the normalized value of the magnitude of the assigned stylization generation parameter vector as the initial weights. Traverse all adjacent anchor node pairs in the semantic anchor network structure. For two adjacent style-sensitive anchor nodes, calculate their consistency constraint loss. The consistency constraint loss is equal to the absolute value of the difference between the style transfer intensity weights of the two nodes minus the product of the constraint intensity coefficient and the semantic similarity of the two nodes, and take the larger of the maximum value and zero. The semantic similarity of the two nodes is calculated by the cosine similarity of the node feature vectors. For anchor pairs with a consistency constraint loss greater than zero, the weight of the node with the larger weight is reduced, and the weight of the node with the smaller weight is increased. The adjustment range is proportional to the product of the consistency constraint loss and the edge weight between the nodes. Calculate the total consistency constraint loss of all anchor pairs. If the total loss is less than the preset threshold or the number of iterations reaches the upper limit, output the style transfer intensity weight of each anchor point and terminate the algorithm. Otherwise, return to continue iterating. The preset threshold is set to five percent of the initial total loss.

4. The method for dynamically generating metaverse scenes based on AIGC cross-media as described in claim 1, characterized in that, The method for generating style-conditional content representations is as follows: For each piece of media content in the target scene, obtain the style-sensitive anchor node corresponding to the media content and its stylized generation parameter vector and sentiment adjustment vector according to the anchor point style parameter allocation table; The initial semantic representation of the target scene is passed through a learnable linear transformation layer to obtain the query vector, and the feature vectors of each associated anchor node are passed through a learnable linear transformation layer to obtain the key vector. The attention weights of each anchor point are obtained by calculating the inner product of the query vector and each key vector and dividing by the square root of the vector dimension, and then by applying an exponential function and normalization. The element-wise products of the initial semantic representation and the stylized generation parameter vector and style feature vector of each associated anchor point are weighted by attention weights, and then summed to obtain the stylized content representation. The style-conditional content representation retains the semantic content constraints of the target scene but injects the style attributes and emotional atmosphere of the source memory, outputting a set of style-conditional representations of content from various media.

5. The method for dynamically generating metaverse scenes based on AIGC cross-media as described in claim 1, characterized in that, The cross-media AIGC generator adopts a conditional diffusion model architecture, including a noise encoder, a denoising network, and a media decoder: The input layer receives style-conditional content representations and style transfer intensity weights, while the noise encoder maps the input to the latent representation space and adds noise. The denoising network injects style-conditional content representations and temporal step embeddings into each layer, and gradually restores the scene content through the iterative denoising process. The media decoder decodes the denoised latent representation into specific media data according to the media type. The visual media decoder outputs texture images, the 3D geometry decoder outputs mesh model data and object spatial layout data, and the audio decoder outputs audio waveform data. When the memory anchor data packet is missing specific details of key visual element features or audio atmosphere features, a weighted random sampling algorithm is applied based on the user style preference profile to sample the missing style detail parameters from the style feature distribution of the user's historical preferences. The completed style detail parameters are then input as additional conditions into the cross-media AIGC generator.

6. The method for dynamically generating metaverse scenes based on AIGC cross-media as described in claim 1, characterized in that, The calculation methods for cross-media style similarity and emotional resonance are as follows: Cross-media style similarity is obtained by calculating the cosine similarity of the style feature vectors of the target scene and the source memory scene in each media type, and then averaging them according to the importance weight of each media type. The importance weight of the media type is determined based on the content proportion of that media type in the target scene. The emotional resonance score is obtained by weighted combination of the matching degree of emotional tags and the user's emotional state monitoring score. The matching degree of emotional tags is calculated by the cosine similarity of the emotional tag vectors of the two scenes. The user's emotional state monitoring score is calculated by analyzing the physiological signals and facial expression data of the user when watching the generated scene and then normalized. The method for identifying the media type with the greatest style deviation is to calculate the style deviation degree of each media type separately. The style deviation degree is defined as the cosine distance between the style feature vectors of the media type in the target scene and the source memory scene. The media type with the greatest style deviation degree is selected as the media type that needs to be regenerated. At the same time, the anchor node with the highest style transfer intensity weight among the anchor nodes associated with the media type is identified as the anchor node that needs to be adjusted.

7. A dynamic generation system for metaverse scenes based on AIGC cross-media, used to execute the dynamic generation method for metaverse scenes based on AIGC cross-media as described in any one of claims 1-6, characterized in that, It also includes the following steps: The generated memory-stylized target scene is presented to the user, and the user provides feedback ratings and suggestions for improvement on the degree of memory emotional restoration and cross-media style consistency. The user's suggestions are interpreted as adjustments to specific style dimensions or media content, and the corresponding style feature parameters in the memory anchor data package are updated accordingly. The memory anchor data package is updated using an incremental learning approach. For style dimensions where the user explicitly points out a deviation, the user's expected style feature values ​​are weighted and fused with the original feature values. The fusion weight is determined by the confidence level of the user's feedback. The confidence weight of user feedback is determined by normalizing the average of user feedback ratings on memory emotional fidelity and cross-media style consistency. The higher the user rating, the greater the weight of the user expectation feature in the update. The updated parameters are stored in the user's personalized model library to improve the accuracy of future memory style transfer tasks.

Images