A generative art therapy method based on multi-modal sentiment computing

Abstract

A kind of generative art therapy method based on multi-modal emotion computing, multi-modal emotion data of user are collected, including face video, physiological signal, eye movement track and interactive behavior track, and are preprocessed and feature coding, generate uniform high-dimensional emotion feature vector, construct cross-modal generative adversarial network, generator is input with emotion feature vector as condition, output corresponding artistic image;Discriminator simultaneously distinguishes real image and generated image, and predicts the emotion feature expressed by generated image, the consistency loss between the prediction emotion feature and input emotion feature vector is optimized generator through adversarial loss, realize the nonlinear mapping of " emotion feature-visual element ".The present application encodes multi-modal emotion feature: synchronously collects facial micro-expression, physiological signal, sight track and interactive behavior track, generates high-dimensional emotion feature vector using multi-head attention fusion, realizes the quantitative expression to user subconscious emotion state.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of digital art therapy, specifically a generative art therapy method based on multimodal emotion computing. Background Technology

[0002] Art therapy is a non-pharmacological intervention that promotes psychological recovery through the process of visual art creation. Traditional art therapy relies heavily on the therapist's subjective experience and intuition, lacks quantitative and objective assessment indicators for the patient's emotional state, and different therapists interpret the same patient's work differently, resulting in unstable therapeutic effects.

[0003] In recent years, researchers have begun to explore the use of computer vision technology to analyze patients' facial expressions, body movements, and physiological signals during the artistic creation process, attempting to establish a correlation between emotional states and artistic expression. For example, convolutional neural networks are used to identify basic emotions such as happiness, sadness, and anger, and map them onto a pre-defined color palette. However, existing methods have the following limitations: (1) The emotion-visual mapping is too simplistic: Most methods only map discrete emotion categories to fixed colors or shapes, ignoring the continuity and multidimensionality of emotions (such as intensity, valence, and arousal), and cannot reflect the subtle subconscious emotional state of users.

[0004] (2) Lack of dynamic narrative generation capability: Static image generation cannot meet the needs of PTSD exposure therapy for personalized visual stories with coherent timelines and controllable plots. Existing interactive systems are mostly based on predefined scripts and cannot dynamically adjust the narrative content according to the patient's real-time behavioral trajectory and gaze focus.

[0005] (3) No closed-loop optimization mechanism: There is no continuous feedback correction between the generated content and the user's real emotional state. The system cannot gradually approach the user's personalized emotional expression preferences as the number of interactions increases.

[0006] (4) Lack of privacy protection and cross-institutional collaboration: Existing methods mostly adopt a centralized data processing architecture, which poses a risk of data privacy leakage and cannot achieve cross-institutional collaborative training of treatment data, thus limiting the promotion and application of the model in real clinical scenarios. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art, the present invention provides a generative art therapy method based on multimodal emotion computing to at least partially solve the above-mentioned technical problems.

[0008] The technical solution adopted in this invention is as follows: This invention proposes a generative art therapy method based on multimodal emotion computing, comprising the following steps: Step S1: Collect the user's multimodal emotional data, including facial videos, physiological signals, eye movement trajectories, and interactive behavior trajectories, and perform preprocessing and feature encoding to generate a unified high-dimensional emotional feature vector; Step S2: Construct a cross-modal generative adversarial network. The generator takes the emotion feature vector as a conditional input and outputs the corresponding artistic image. The discriminator distinguishes between real images and generated images and predicts the emotion features expressed by the generated images. The generator is optimized by adversarial loss and consistency loss between the predicted emotion features and the input emotion feature vector, so as to realize the non-linear mapping of "emotion features - visual elements". Step S3: Based on the user's behavior trajectory and eye tracking data, use a sequence generation model to dynamically create an interactive visual narrative, decompose the narrative into a key frame sequence, each frame is generated by the generator according to the emotional characteristics of the current moment, and maintain the emotional coherence across frames through a trajectory attention mechanism; Step S4: Collect users' physiological feedback and subjective ratings on the generated artistic content, construct an emotional consistency reward signal, and update the generator parameters through a meta-learning optimizer so that the generated content gradually approaches the user's subconscious emotional state after multiple interactions. Step S5: Deploy the trained cross-modal generative adversarial network and visual narrative model to a local computing terminal or cloud server to receive user multimodal data in real time and output personalized art therapy content.

[0009] In one embodiment of the present invention, step S1 specifically includes: Facial video is captured by an RGB camera to extract the intensity sequence of micro-expression action units; physiological signals such as heart rate and skin conductance are collected by photoelectric sensors; gaze coordinates and pupil diameter changes are obtained by an eye tracker; and user interaction behavior trajectories (clicks, drags, path movements) are recorded by a touch screen or controller. The following formula is used to standardize and synchronize multimodal data:

[0010] in, For the first The first mode One original feature value, This represents the mean of the modal characteristics. Standard deviation; A unified sentiment feature vector is generated by fusing features from different modalities using a multi-head attention mechanism. The formula is as follows:

[0011]

[0012]

[0013] These are obtained by linear transformation of the feature matrices encoded for each modality. For the number of attention heads, This is the output weight matrix.

[0014] In one embodiment of the present invention, the cross-modal generative adversarial network in step S2 is specifically implemented as follows: generator Using a U-Net structure, it receives random noise vectors. With sentiment feature vector Through the adaptive instance normalization layer Each layer of the decoder is integrated to output a synthetic art image: ; Discriminator The PatchGAN structure is used to distinguish real images. With synthetic images Simultaneously, it outputs the predicted sentiment features of the image through an additional sentiment prediction head; its adversarial loss and sentiment consistency loss are defined as follows: Discriminator loss:

[0016] Generator loss:

[0018] in, Weighting for emotional consistency , represents the sentiment prediction branch of the discriminator; represents the feature matching loss of the intermediate layer of the discriminator; and represents the feature matching weights.

[0019] In one embodiment of the present invention, the dynamic interactive visual narrative generation method in step S3 is as follows: User behavior trajectory ,in Two-dimensional coordinates, For timestamps, along with eye-tracking data Distributed feature extraction is performed using a federated learning framework to generate local narrative state vectors. ; Employing a multimodal emotion recognition model with differential privacy protection, local narrative state vectors are securely aggregated to generate a global narrative feature sequence. :

[0020] in, As of the end of the All local narrative states at the corresponding frame time. For privacy protection parameters, a typical value is... , It is Laplace noise; Emotional features in each frame Input the generator for step S2 Generate corresponding visual narrative frames All frames are played in chronological order to form a dynamic visual narrative; the Transformer decoder uses a trajectory attention mechanism to weight historical behavior trajectories and gaze trajectories to ensure emotional coherence across frames.

[0021] In one embodiment of the present invention, the emotional feedback closed-loop optimization in step S4 specifically includes: Collect user physiological feedback on the currently generated art content: heart rate variability (HRV), skin conductance response (GSR), and subjective ratings. Constructing an emotion-consistent reward signal through a differential privacy mechanism:

[0022] in, For privacy protection parameters, To meet - Laplace noise for differential privacy Sensitivity to the reward function; The generator parameters are updated using the Model-Independent Meta-Learning (MAML) algorithm. Treating each user's different sessions as a task enables cross-device model collaborative optimization while protecting user privacy: Local model update (inner loop):

[0023] Meta-learning optimization (outer loop):

[0024] in, , The first Local training and validation data for each user For reward signals Constructed feedback loss, Learning rate for privacy protection.

[0025] In one embodiment of the present invention, the deployment and real-time inference in step S5 includes: After training, the generator The federated learning model is quantized to 8-bit integer precision using gradient compression technology, and employs... Sparsification, retaining the first gradient with the largest magnitude ; Real-time acquisition of user multimodal data streams, dynamic updating of sentiment feature vectors, and triggering distributed narrative frame generation through an asynchronous aggregation mechanism: After each edge device completes gradient calculation locally, it asynchronously uploads the compressed gradient to the central server according to a fixed time window. The server then performs weighted aggregation and asynchronously distributes the updates to ensure low-latency response. To ensure smooth operation, a multi-threaded parallel architecture is adopted. The data acquisition thread, feature encoding thread, and model inference thread execute in parallel to ensure real-time response performance.

[0026] The beneficial effects of the technical solution of this invention are as follows: This invention utilizes multimodal emotional feature encoding: simultaneously collecting facial micro-expressions, physiological signals, gaze trajectories, and interaction behavior trajectories, and using multi-head attention fusion to generate high-dimensional emotional feature vectors, thereby achieving a quantitative expression of the user's subconscious emotional state.

[0027] This invention uses emotional feature vectors as conditional inputs to establish a nonlinear mapping model of "emotional features - visual elements". The generator adopts a U-Net structure and incorporates emotional features through an adaptive instance normalization (AdaIN) layer, outputting artistic images with corresponding emotional colors, textures and configurations. The discriminator simultaneously evaluates the image's realism and emotional consistency, ensuring the generated content is both aesthetically and psychologically reasonable.

[0028] This invention creates personalized visual stories in real time using a Transformer sequence generation model based on user behavior trajectory and eye tracking data. Each story frame is generated driven by the current emotional features, and the trajectory attention mechanism ensures emotional coherence across frames, providing progressive exposure therapy materials for PTSD patients.

[0029] This invention constructs a reward signal by collecting users' physiological feedback (heart rate variability, skin conductance response) and subjective ratings on the generated content, and uses the Model Independent Meta-Learning (MAML) algorithm to update the generator parameters. It treats each user's different sessions as independent tasks, enabling the system to adaptively approximate the user's subconscious emotional expression needs after multiple interactions.

[0030] This invention achieves efficient collaborative training of treatment data across institutions by employing gradient compression and asynchronous aggregation mechanisms (fixed time window upload), reducing communication bandwidth requirements while ensuring model performance, and ensuring user data security through differential privacy technology.

[0031] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the steps of the generative art therapy method based on multimodal emotion computing proposed in an embodiment of the present invention. Detailed Implementation

[0033] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0034] The following describes a generative art therapy method based on multimodal emotion computing according to an embodiment of the present invention, with reference to the accompanying drawings.

[0035] like Figure 1 As shown, this embodiment of the invention provides a generative art therapy method based on multimodal emotion computing, including the following steps: Step S1: Collect the user's multimodal emotional data, including facial videos, physiological signals, eye movement trajectories, and interactive behavior trajectories, and perform preprocessing and feature encoding to generate a unified high-dimensional emotional feature vector; Step S2: Construct a cross-modal generative adversarial network. The generator takes the emotion feature vector as conditional input and outputs the corresponding artistic image. The discriminator distinguishes between real images and generated images and predicts the emotion features expressed by the generated images. The generator is optimized by adversarial loss and consistency loss between predicted emotion features and input emotion feature vector, so as to achieve non-linear mapping of "emotion features - visual elements". Step S3: Based on the user's behavior trajectory and eye tracking data, use a sequence generation model to dynamically create an interactive visual narrative, decompose the narrative into a key frame sequence, each frame is generated by the generator according to the emotional characteristics of the current moment, and maintain the emotional coherence across frames through a trajectory attention mechanism; Step S4: Collect users' physiological feedback and subjective ratings on the generated artistic content, construct an emotional consistency reward signal, and update the generator parameters through a meta-learning optimizer so that the generated content gradually approaches the user's subconscious emotional state after multiple interactions. Step S5: Deploy the trained cross-modal generative adversarial network and visual narrative model to a local computing terminal or cloud server to receive user multimodal data in real time and output personalized art therapy content.

[0036] In practical applications, the data acquisition end of this invention utilizes an RGB camera, photoelectric sensor, eye tracker, and interactive recording device working in parallel to acquire facial video, physiological signals, eye movement trajectories, and interactive behavior trajectories. Before processing, the raw data undergoes standardization and time synchronization to eliminate temporal deviations caused by differences in sampling rates between different sensors and to map the features of each modality to a unified dimension. Subsequently, a multi-head attention mechanism performs a linear transformation on the feature matrices encoded by each modality, generating a query vector Q, a key vector K, and a value vector V. By calculating attention weights and fusing information from different modalities, a unified high-dimensional emotional feature vector h is output. This vector not only contains static emotional state identifiers but also implicitly contains dynamic emotional change trends, providing multi-dimensional constraints for subsequent generation processes.

[0037] Furthermore, the generated sentiment feature vector h is passed as a conditional input to a cross-modal generative adversarial network. Within the network, the generator employs a U-Net architecture, receiving a random noise vector z and the sentiment feature vector h. An adaptive instance normalization layer integrates the sentiment feature h into each layer of the decoder, adjusting the normalization offset and scaling factor to ensure the structure and texture of the generated image are directly controlled by the sentiment feature. The discriminator uses a PatchGAN structure to distinguish between real and synthetic images at the local patch level, while an additional sentiment prediction head outputs the predicted sentiment feature value ŷ for the corresponding image. During training, adversarial loss encourages the generated image to approximate the real distribution in terms of visual realism, while consistency loss forces the sentiment features expressed by the generated image to remain consistent with the input sentiment feature vector h, ensuring that the generated artistic image is not only aesthetically pleasing but also semantically matches the user's current state.

[0038] In the dynamic interactive visual narrative generation stage, the system uses a sequence generation model to create a keyframe sequence based on the user's behavior trajectory B(t) and gaze tracking data E(t). The federated learning framework extracts local narrative state vectors on each edge device and generates a global narrative feature sequence through a secure aggregation protocol combined with a differential privacy protection mechanism. The Transformer decoder receives the global sequence and the emotional features of the previous moment. It uses a trajectory attention mechanism to weight the historical behavior and gaze data and calculates the emotional features of the current moment, avoiding emotional breaks caused by frame jumps.

[0039] To further optimize the generated content, the system introduces an emotional feedback closed-loop optimization mechanism. This mechanism collects users' physiological feedback on the generated content (such as heart rate variability (HRV) and skin conductance response (GSR)) and subjective ratings. An emotional consistency reward signal is constructed using a differential privacy mechanism. This reward signal comprehensively considers objective physiological indicators and subjective evaluations, and Laplace noise is injected to protect user privacy. A model-independent meta-learning algorithm is used to update the generator parameters, treating each user's different sessions as independent tasks. In the inner loop, the local model updates its parameters based on the training data of the current session; in the outer loop, the meta-learning optimizer summarizes the gradient information from each task and updates the global model parameters.

[0040] Finally, the trained model is deployed to a local computing terminal or cloud server. Gradient compression and sparsity techniques are used to reduce model size and transmission overhead. During the real-time inference phase, the data acquisition thread, feature encoding thread, and model inference thread execute in parallel. An asynchronous aggregation mechanism ensures timely synchronization of parameter updates among distributed nodes. The system continuously receives multimodal data from users, dynamically updates sentiment feature vectors, and triggers narrative frame generation, achieving low-latency real-time response.

[0041] In one specific implementation, step S1 specifically includes: Facial video is captured by an RGB camera to extract the intensity sequence of micro-expression action units; physiological signals such as heart rate and skin conductance are collected by a photoelectric sensor; gaze coordinates and pupil diameter changes are obtained by an eye tracker; and user interaction behavior trajectories (clicks, drags, path movements) are recorded by a touch screen or controller. The following formula is used to standardize and synchronize multimodal data:

[0042] in, For the first The first mode One original feature value, The mean of the modal features. Standard deviation; A unified sentiment feature vector is generated by fusing features from different modalities using a multi-head attention mechanism. The formula is as follows:

[0043]

[0044]

[0045] in, Each modality is obtained by linear transformation of its encoded feature matrix. For the number of attention heads, This is the output weight matrix.

[0046] In a specific application of this invention, an RGB camera is deployed at a fixed position directly in front of the user, continuously capturing facial video streams at a preset frame rate. An internal feature extraction module, based on an action unit coding system, separates the muscle contraction intensity of key areas of the orbicularis oculi and orbicularis oris muscles from the video, forming a micro-expression action unit intensity sequence. A photoelectric sensor is attached to the user's fingertips or earlobes to monitor changes in blood volume pulse waves in real time. After filtering, heart rate and its variability are extracted, and skin conductance levels are collected to reflect the degree of sympathetic nerve excitation. An eye tracker is integrated into the top of the display terminal, emitting an infrared beam and receiving reflected signals to calculate the pupil center coordinates and the dynamic range of diameter changes, constructing a fixation point trajectory and pupil dilation time sequence. A touchscreen or controller records the touch point coordinate sequence, pressure value changes, and movement speed vectors during user operation, generating interactive behavior trajectory data.

[0047] Before entering the feature fusion layer, the aforementioned multi-source heterogeneous data needs to pass through a time synchronization and standardization processing module. The time synchronization module uses an interpolation algorithm based on the hardware clock offset of each sensor to align the raw data with different sampling rates to a unified time axis, eliminating the timing misalignment caused by transmission delay or sampling frequency differences. The standardization module independently calculates the historical statistical mean and standard deviation for each modality feature, mapping the raw feature values ​​to a zero-mean unit variance distribution. This process eliminates the interference of different physiological index dimensional differences on subsequent fusion calculations, ensuring that the intensity of facial micro-expressions, the amplitude of skin conductance response, and the eye movement coordinate values ​​are on the same order of magnitude, providing a balanced input basis for the multi-head attention mechanism.

[0048] Furthermore, the fusion layer employs a multi-head attention mechanism to weighted aggregate the feature matrices encoded from each modality. A linear projection layer then converts the standardized modal features into query vectors, key vectors, and value vectors, respectively. The outputs of all attention heads are concatenated and linearly transformed using the output weight matrix to compress the high-dimensional feature space, generating a unified high-dimensional emotional feature vector. This vector integrates facial muscle movement patterns, autonomic nervous system activity, visual attention allocation tendencies, and body interaction intentions, forming a three-dimensional mathematical description of the user's current psychological state. Through this cross-modal feature fusion strategy, the system can capture complex emotional cues that a single modality cannot reflect. For example, when a facial expression is neutral but the skin conductance response is dramatically elevated, the model can identify potential stress or anxiety, thereby improving the comprehensiveness of emotion recognition and providing data support for the personalized customization of subsequent generative art therapy content.

[0049] In one specific implementation, the cross-modal generative adversarial network in step S2 is implemented as follows: Generator Using a U-Net structure, it receives random noise vectors. With sentiment feature vector Through the adaptive instance normalization layer Each layer of the decoder is integrated to output a synthetic art image: ; Discriminator The PatchGAN structure is used to distinguish real images. With synthetic images Simultaneously, it outputs the predicted value of the sentiment features corresponding to the image through an additional sentiment prediction head. The definitions of adversarial loss and affective consistency loss are as follows: Discriminator loss:

[0050] Generator loss:

[0051] in, Weighting for emotional consistency , This represents the sentiment prediction branch of the discriminator; For the discriminator's intermediate layer feature matching loss, For feature matching weights.

[0052] In specific applications of this invention, during the construction and training of the cross-modal generative adversarial network, the generator G adopts a U-Net encoder-decoder architecture. Its input receives two independent data sources: one is a random noise vector z sampled from a standard normal distribution, used to introduce randomness and diversity into the generated content; the other is a high-dimensional sentiment feature vector h from step S1, which serves as a conditional constraint for content generation.

[0053] In the decoding stage of U-Net, the adaptive instance normalization layer is embedded after each convolutional operation and before the activation function. The emotional feature vector h is mapped to the scaling parameter γ and the offset parameter β through the fully connected layer, which are applied to the feature map statistics of the current layer. This allows the color tone, texture density and composition style of the generated image to respond to changes in emotional features in real time. For example, high arousal emotional features prompt the network to increase the contrast of the image or the sharpness of the lines, while low arousal features guide the generation of soft and smooth visual elements.

[0054] Furthermore, the discriminator D adopts a PatchGAN local discriminant structure, consisting of multiple stacked convolutional layers. The last convolutional layer outputs 1 channel, representing the probability that each pixel block within the receptive field belongs to the real image. To enhance the consistency of sentiment mapping, the discriminator branches off after the penultimate convolutional layer output—the sentiment prediction head. This branch contains several fully connected layers that compress high-dimensional image features and map them back to a space with the same dimension as the input sentiment feature vector, outputting predicted sentiment feature values. During the backpropagation optimization phase, the discriminator loss is defined as the negative of the sum of the expected log-likelihood of the real image and the expected log-likelihood of the generated image, forcing the discriminator to improve its discriminative ability. Simultaneously, the generator's optimization objective not only includes an adversarial loss term to deceive the discriminator but also introduces two regularization constraints. The first is the sentiment consistency loss, which, by minimizing the Euclidean distance between the predicted and input sentiment features, forces the generated image to faithfully reflect the input sentiment state visually, addressing the problem of traditional generative models neglecting conditional semantic consistency.

[0055] In one specific implementation, the dynamic interactive visual narrative generation method in step S3 is as follows: User behavior trajectory ,in Two-dimensional coordinates, For timestamps, along with eye-tracking data Distributed feature extraction is performed using a federated learning framework to generate local narrative state vectors. ; Employing a multimodal emotion recognition model with differential privacy protection, local narrative state vectors are securely aggregated to generate a global narrative feature sequence. :

[0057] in, As of the end of the All local narrative states at the corresponding frame time. For privacy protection parameters, a typical value is... , It is Laplace noise; Emotional features in each frame Input the generator for step S2 Generate corresponding visual narrative frames All frames are played in chronological order to form a dynamic visual narrative; the Transformer decoder uses a trajectory attention mechanism to weight historical behavioral trajectories and eye trajectory to ensure emotional coherence across frames.

[0058] In a specific application of this invention, during the construction of a dynamic interactive visual narrative, the system first processes the user's behavioral trajectory B(t) and gaze tracking data E(t). The behavioral trajectory B(t) includes two-dimensional coordinates (x, y) and a timestamp t, reflecting the user's operation path within the interactive interface; the gaze tracking data E(t) includes the coordinates of the fixation point and pupil changes, reflecting the user's visual attention distribution. Specifically, each edge node uses a local lightweight neural network to encode the original trajectory data, extract local spatiotemporal features, and generate a local narrative state vector.

[0059] Subsequently, the system employs a multimodal sentiment recognition model with differential privacy protection to aggregate the local narrative state vectors of each node. The central server receives encrypted gradients or feature updates from multiple edge nodes. To further prevent inference of individual data from the aggregation results, the system injects Laplacian noise "Lap" (Δf / ε) into the aggregated global narrative feature sequence. Here, ε is a privacy budget parameter, typically set between 0.5 and 2.0 to balance the strength of privacy protection with data availability.

[0060] During the frame-by-frame generation phase, the Transformer decoder receives the global narrative feature sequence and the sentiment features from the previous moment, calculates the sentiment features for the current moment, and integrates a trajectory attention mechanism. This mechanism transforms historical behavior trajectories and gaze trajectories into key vectors and value vectors, setting the current query vector as the preceding sentiment state. By calculating attention weights, the model can dynamically weight important historical interaction segments, such as when a user stares at a certain area for a long time or repeatedly drags a certain path, thereby capturing potential emotional connections.

[0061] Finally, the generated emotional features are passed as conditional input to the generator G trained in step S2, and combined with random noise to synthesize visual narrative frames. All generated frames are arranged and played in chronological order to form a continuous dynamic visual narrative stream.

[0062] In one specific implementation, the emotional feedback closed-loop optimization in step S4 is as follows: Collect user physiological feedback on the currently generated art content: heart rate variability (HRV), skin conductance response (GSR), and subjective ratings. Constructing an emotion-consistent reward signal through a differential privacy mechanism:

[0063] in, For privacy protection parameters, To meet - Laplace noise for differential privacy, Sensitivity to the reward function; The generator parameters are updated using the Model-Independent Meta-Learning (MAML) algorithm. Treating each user's different sessions as a task enables cross-device model collaborative optimization while protecting user privacy: Local model update (inner loop):

[0064] Meta-learning optimization (outer loop):

[0065] in, , The first Local training and validation data for each user For reward signals Constructed feedback loss, Learning rate for privacy protection.

[0066] In specific applications, during the emotional feedback closed-loop optimization stage, the system constructs a quantitative evaluation system through multi-source data fusion to guide model iteration. The acquisition terminal simultaneously obtains the user's physiological indicators and subjective evaluations of the currently generated artistic content. Heart rate variability (HRV) reflects the balance of the autonomic nervous system, skin conductance response (GSR) characterizes the degree of emotional arousal, and subjective score (r_sub) records the user's intuitive feelings. To protect user privacy and prevent reverse engineering attacks, the system employs a differential privacy mechanism to construct the emotional consistency reward signal R. Specifically, the contribution of each component weight to the total signal is first calculated, and after weighted summation, Laplace noise "Lap" (Δr / ε) is injected. Here, ε is set as a privacy budget parameter to control noise intensity; Δr is the reward function sensitivity, ensuring the output satisfies the (ε,0)-differential privacy definition.

[0067] Based on the aforementioned reward signals, each user's independent interaction session is modeled as an independent learning task. Using a federated learning framework, inner-loop optimization is performed locally on each device. On the local training set of the i-th user, a loss function constructed based on the reward signal R is calculated, where α is the local learning rate. Subsequently, the system enters the outer-loop meta-learning optimization stage to achieve cross-device knowledge sharing and improve global generalization capabilities. The central server collects encrypted local update gradients or parameter differences uploaded by each client to prevent the leakage of raw data.

[0068] In one specific implementation, the deployment and real-time inference in step S5 include: After training, the generator The federated learning model is quantized to 8-bit integer precision using gradient compression technology, and employs... Sparsification, retaining the first gradient with the largest magnitude ; Real-time acquisition of user multimodal data streams, dynamic updating of sentiment feature vectors, and triggering distributed narrative frame generation through an asynchronous aggregation mechanism: After each edge device completes gradient calculation locally, it asynchronously uploads the compressed gradient to the central server according to a fixed time window. The server then performs weighted aggregation and asynchronously distributes the updates to ensure low-latency response. To ensure smooth operation, a multi-threaded parallel architecture is adopted. The data acquisition thread, feature encoding thread, and model inference thread execute in parallel to ensure real-time response performance.

[0069] In practical applications, this invention employs gradient compression technology to map model parameters from 32-bit floating-point precision to 8-bit integer precision. By using linear scaling or nonlinear mapping, key numerical distribution characteristics are preserved, thereby reducing memory usage and accelerating matrix operations. Furthermore, redundant gradients close to zero are set to zero, retaining core information influencing the model update direction while reducing data transmission volume, making it suitable for bandwidth-constrained network environments.

[0070] Furthermore, the real-time inference process relies on the continuous acquisition of multimodal data streams and the updating of dynamic emotional feature vectors. The system employs a multi-threaded parallel architecture to ensure low-latency response performance. The data acquisition thread is responsible for synchronously receiving RGB video, physiological signal sensor data, and eye tracker coordinates via USB or wireless interface, performing preliminary timestamp alignment and noise filtering. The feature encoding thread runs in parallel, using a lightweight convolutional neural network or Transformer encoder to transform the raw data into standardized feature vectors, which are then input into the multi-head attention fusion module described in step S1 to generate the current emotional feature vector. The model inference thread receives the emotional vector, combines it with random noise, and calls the quantized generator G for forward propagation to synthesize an artistic healing frame.

[0071] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0072] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A generative art therapy method based on multimodal emotion computing, characterized in that, Includes the following steps: Step S1: Collect the user's multimodal emotional data, including facial videos, physiological signals, eye movement trajectories, and interactive behavior trajectories, and perform preprocessing and feature encoding to generate a unified high-dimensional emotional feature vector; Step S2: Construct a cross-modal generative adversarial network. The generator takes the emotion feature vector as a conditional input and outputs the corresponding artistic image. The discriminator distinguishes between real images and generated images and predicts the emotion features expressed by the generated images. The generator is optimized by adversarial loss and consistency loss between the predicted emotion features and the input emotion feature vector, so as to realize the non-linear mapping of "emotion features - visual elements". Step S3: Based on the user's behavior trajectory and eye tracking data, use a sequence generation model to dynamically create an interactive visual narrative, decompose the narrative into a key frame sequence, each frame is generated by the generator according to the emotional characteristics of the current moment, and maintain the emotional coherence across frames through a trajectory attention mechanism; Step S4: Collect users' physiological feedback and subjective ratings on the generated artistic content, construct an emotional consistency reward signal, and update the generator parameters through a meta-learning optimizer so that the generated content gradually approaches the user's subconscious emotional state after multiple interactions. Step S5: Deploy the trained cross-modal generative adversarial network and visual narrative model to a local computing terminal or cloud server to receive user multimodal data in real time and output personalized art therapy content.

2. The generative art therapy method based on multimodal emotion computing according to claim 1, characterized in that, Step S1 specifically includes: Facial video is captured by an RGB camera to extract the intensity sequence of micro-expression action units; physiological signals such as heart rate and skin conductance are collected by photoelectric sensors; gaze coordinates and pupil diameter changes are obtained by an eye tracker; and user interaction behavior trajectories (clicks, drags, path movements) are recorded by a touch screen or controller. The following formula is used to standardize and synchronize multimodal data: ； in, For the first The first mode One original feature value, This represents the mean of the modal characteristics. Standard deviation; A unified sentiment feature vector is generated by fusing features from different modalities using a multi-head attention mechanism. The formula is as follows: ； in, Each modality is obtained by linear transformation of its encoded feature matrix. For the number of attention heads, This is the output weight matrix.

3. The generative art therapy method based on multimodal emotion computing according to claim 1, characterized in that, The cross-modal generative adversarial network in step S2 is specifically implemented as follows: generator Using a U-Net structure, it receives random noise vectors. With sentiment feature vector Through the adaptive instance normalization layer Each layer of the decoder is integrated to output a synthetic art image: ； Discriminator The PatchGAN structure is used to distinguish real images. With synthetic images Simultaneously, it outputs the predicted value of the sentiment features corresponding to the image through an additional sentiment prediction head. The definitions of adversarial loss and affective consistency loss are as follows: Discriminator loss: ； Generator loss: ； in, Weighting for emotional consistency , This represents the sentiment prediction branch of the discriminator; For the discriminator's intermediate layer feature matching loss, For feature matching weights.

4. The generative art therapy method based on multimodal emotion computing according to claim 1, characterized in that, The dynamic interactive visual narrative generation method in step S3 is as follows: User behavior trajectory ,in Two-dimensional coordinates, For timestamps, along with eye-tracking data Distributed feature extraction is performed using a federated learning framework to generate local narrative state vectors. ; Employing a multimodal emotion recognition model with differential privacy protection, local narrative state vectors are securely aggregated to generate a global narrative feature sequence. : ； in, As of the end of the All local narrative states at the corresponding frame time. For privacy protection parameters, a typical value is... , It is Laplace noise; Emotional features in each frame Input the generator for step S2 Generate corresponding visual narrative frames All frames are played in chronological order to form a dynamic visual narrative; the Transformer decoder uses a trajectory attention mechanism to weight historical behavior trajectories and gaze trajectories to ensure emotional coherence across frames.

5. The generative art therapy method based on multimodal emotion computing according to claim 1, characterized in that, The emotional feedback closed-loop optimization in step S4 specifically involves: Collect user physiological feedback on the currently generated art content: heart rate variability (HRV), skin conductance response (GSR), and subjective ratings. Constructing an emotion-consistent reward signal through a differential privacy mechanism: ； in, For privacy protection parameters, To meet - Laplace noise for differential privacy, Sensitivity to the reward function; The generator parameters are updated using the Model-Independent Meta-Learning (MAML) algorithm. Treating each user's different sessions as a task enables cross-device model collaborative optimization while protecting user privacy: Local model update (inner loop): ； Meta-learning optimization (outer loop): ； in, , The first Local training and validation data for each user For reward signals Constructed feedback loss, Learning rate for privacy protection.

6. The generative art therapy method based on multimodal emotion computing according to claim 1, characterized in that, The deployment and real-time inference in step S5 include: After training, the generator The federated learning model is quantized to 8-bit integer precision using gradient compression technology, and employs... Sparsification, retaining the first gradient with the largest magnitude ; Real-time acquisition of user multimodal data streams, dynamic updating of sentiment feature vectors, and triggering distributed narrative frame generation through an asynchronous aggregation mechanism: After each edge device completes gradient calculation locally, it asynchronously uploads the compressed gradient to the central server according to a fixed time window. The server then performs weighted aggregation and asynchronously distributes the updates to ensure low-latency response. To ensure smooth operation, a multi-threaded parallel architecture is adopted. The data acquisition thread, feature encoding thread, and model inference thread execute in parallel to ensure real-time response performance.

Images