Cultural relic activation self-adaptive control method based on emotion perception and physiological feedback

Abstract

The invention provides a cultural relic activation self-adaptive control method based on emotion perception and physiological feedback, and relates to the technical field of intelligent control and digital display. Multi-modal emotion perception is performed by fusing user languages, group images and physiological signals, cross-modal calibration is performed on the recognition result innovatively by taking physiological feedback as a reference, and the problems that emotion recognition is single and is easily interfered, pain points of real emotions cannot be captured, and the recognition accuracy is high in the prior art are effectively solved. The accuracy and reliability of emotion analysis in a group scene are improved; on the basis, the system can adaptively optimize model parameters according to real-time feedback and display effects of audiences through a closed-loop learning mechanism, so that the crossing from static display to dynamic adaptation is realized, and finally, the accuracy of cultural expression is ensured, and meanwhile, the precision and the infectivity of immersive experience are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent control and digital display technology, specifically to an adaptive control method for the revitalization of cultural relics based on emotion perception and physiological feedback. Background Technology

[0002] Cultural relics carry rich historical and cultural information, and their dynamic display is of great significance for cultural inheritance and education. With the development of digital technology, the use of multimodal means such as sound, light, electricity, and mechanical motion to digitally immerse cultural relics has become a new trend in museums, cultural tourism scenic spots, and other fields.

[0003] Existing digital display systems for cultural relics typically employ the following two technological approaches:

[0004] The first type of system relies on pre-programmed instructions. The system stores fixed presentation scripts in advance, plays corresponding audio and video content, and controls lighting and mechanical components based on trigger commands. This method is simple to implement, but the presented content is monotonous and cannot be adjusted based on audience reactions, resulting in poor interactivity and user experience.

[0005] The second type of system incorporates artificial intelligence technology, attempting to achieve a certain level of intelligent interaction. These systems typically understand user commands through speech recognition or text analysis and generate corresponding display content accordingly. For example, some systems can identify keywords in a user's speech (such as "excited" or "soothing") and invoke preset music and lighting modes accordingly. However, this type of technology has significant limitations: First, its emotion recognition relies entirely on the user's actively input language or text, achieved by analyzing text content or tone of voice. This singular perception method is severely limited in complex group viewing scenarios. For example, the system cannot perceive the state of silent audience members who are not actively interacting, and speech recognition is easily affected by environmental noise, while facial expression recognition is easily affected by lighting, occlusion, and cultural differences, leading to unreliable and inaccurate emotion recognition results. Second, existing technologies lack the ability to capture the audience's deep, genuine emotional state. External expressions and language can be deliberately controlled or concealed, failing to reflect the audience's subconscious emotional reactions, making the generated display scheme potentially disconnected from the audience's inner feelings. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an adaptive control method for the revitalization of cultural relics based on emotion perception and physiological feedback.

[0007] The technical solution adopted in this invention is as follows:

[0008] An adaptive control method for cultural relic revitalization based on emotion perception and physiological feedback includes the following steps:

[0009] S1: Obtain user input data, the input data including natural language text and / or voice data;

[0010] S2: Perform semantic parsing and emotion recognition on the input data to generate semantic parsing results and a first emotion vector;

[0011] S3: Collect image data and physiological signal data of the audience in parallel, process the image data to generate a group emotion distribution vector; at the same time, process the physiological signal data to generate a physiological emotion index that represents emotional arousal and valence.

[0012] S4: Perform cross-modal calibration on the first emotion vector, the group emotion distribution vector, and the physiological emotion index to determine emotion recognition conflict; when a conflict exists, perform weighted correction on the first emotion vector and the group emotion distribution vector based on the physiological emotion index, and output a calibrated group emotion state;

[0013] S5: Access the cultural knowledge and style rule base based on the semantic parsing results to obtain a cultural style template that matches the cultural theme;

[0014] S6: Based on a cross-modal mapping network, the semantic parsing results, the calibrated group emotional state, and the cultural style template are jointly mapped into multimodal control parameters, including music control parameters, lighting control parameters, and mechanical motion control parameters;

[0015] S7: Establish a unified time baseline, perform timing synchronization and phase compensation on the multimodal control parameters, and generate a synchronized control instruction set;

[0016] S8: Send the synchronized control command set to the execution device to drive the audio, lighting and mechanical structure to perform coordinated display;

[0017] S9: Collect real-time feedback data from the execution device and audience status data, and adaptively optimize the model parameters based on the real-time feedback data and audience status data.

[0018] Further, in step S2, semantic parsing and emotion recognition are performed on the input data, including:

[0019] The input data is segmented and subjected to dependency parsing to generate structured text data;

[0020] The structured text data is input into a BERT-based semantic understanding model to generate the semantic parsing results;

[0021] The structured text data is input into a multi-task learning model, which jointly outputs the intent classification result and the emotion intensity regression value to form the first emotion vector.

[0022] Further, in step S3, the image data is processed to generate a group emotion distribution vector, including:

[0023] The facial expression features and body posture features in the image data are extracted using a convolutional neural network.

[0024] The facial expression features and the body posture features are fused to output a probability distribution vector representing the proportion of different emotion categories in the group, which serves as the group emotion distribution vector.

[0025] Processing the physiological signal data to generate a physiological emotion index includes:

[0026] The physiological signal data, including heart rate variability and skin conductance response, is filtered and feature extracted.

[0027] The extracted physiological features are input into a classifier, which outputs a scalar value representing emotional arousal and valence, as the physiological emotion index.

[0028] Furthermore, in step S4, determining the conflict in emotion recognition specifically includes:

[0029] Calculate the first cosine similarity between the first emotion vector and the group emotion distribution vector;

[0030] Calculate the second cosine similarity between the group emotion distribution vector and the physiological emotion index;

[0031] When the first cosine similarity is lower than the first preset threshold and the second cosine similarity is higher than the second preset threshold, it is determined that there is an emotion recognition conflict based on the physiological emotion index.

[0032] Furthermore, in step S4, a weighted correction is performed based on the physiological emotion index, specifically as follows:

[0033] The highest weight is assigned to the physiological emotion index, the lowest weight is assigned to the first emotion vector, and a weighted average is calculated on the first emotion vector, the group emotion distribution vector and the physiological emotion index to output the calibrated group emotion state.

[0034] Further, in step S5, accessing the cultural knowledge and style rule base based on the semantic parsing result includes:

[0035] The cultural knowledge and style rule base is constructed in the form of a knowledge graph, which contains cultural symbol entities and semantic relationships between entities.

[0036] Using the topic concepts in the semantic parsing results as queries, semantic similarity retrieval is performed in the knowledge graph to obtain associated texture styles, color systems, and musical mode data, thereby constructing the cultural style template.

[0037] Further, in step S6, the semantic parsing result, the calibrated group emotional state, and the cultural style template are jointly mapped into multimodal control parameters, including:

[0038] The calibrated group emotional state is concatenated with the cultural style template to generate a contextual feature vector.

[0039] The semantic parsing result is used as the query vector, and the context feature vector is used as the key vector and value vector, which are then input into the Transformer-based cross-modal attention module for computation.

[0040] The output of the cross-modal attention module is decoded by a feedforward neural network into the music control parameters, the lighting control parameters, and the mechanical motion control parameters.

[0041] Further, in step S7, timing synchronization and phase compensation are performed on the multimodal control parameters, including:

[0042] Assign a uniform timestamp to the control parameters of all modes;

[0043] Using the music beat signal as a reference, calculate the phase deviation between the lighting control signal and the mechanical action signal relative to the reference.

[0044] When the phase deviation exceeds the allowable tolerance, the control signal of the non-reference mode is shifted along the time axis to eliminate the phase deviation.

[0045] Further, in step S8, the synchronized control instruction set is sent to the execution device, including:

[0046] Convert the music control parameters into MIDI protocol instructions or audio stream instructions;

[0047] The lighting control parameters are converted into DMX512 protocol data frames;

[0048] The mechanical motion control parameters are converted into PWM pulse signals;

[0049] Control commands from different protocols are stored in a sending queue according to their unified timestamps and then sent to the corresponding execution devices in sequence.

[0050] Further, in step S9, the audience's state data is collected, specifically as follows:

[0051] Return to steps S3 and S4 to collect new image data and physiological signal data, and generate a new calibrated group emotional state;

[0052] The model parameters are adaptively optimized based on the real-time feedback data and the audience's status data, including:

[0053] Calculate the error function between the real-time feedback data and the new calibrated group emotional state and the expected effect data;

[0054] The parameters of at least one model in semantic parsing and emotion recognition, cross-modal calibration, or cross-modal mapping network are optimized using the error function.

[0055] The beneficial effects of this invention are:

[0056] This invention achieves multimodal emotion perception by integrating user language, group images, and physiological signals. It innovatively uses physiological feedback as a benchmark to perform cross-modal calibration of the recognition results, effectively solving the pain points of existing technologies in emotion recognition, such as being singular, susceptible to interference, and unable to capture real emotions. This improves the accuracy and reliability of emotion analysis in group scenarios. On this basis, the system, through a closed-loop learning mechanism, can adaptively optimize model parameters based on real-time audience feedback and display effects, achieving a leap from static display to dynamic adaptation. Ultimately, while ensuring the accuracy of cultural expression, it significantly improves the precision and appeal of the immersive experience. Attached Figure Description

[0057] Figure 1 This is an overall flowchart of the adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to an embodiment of the present invention;

[0058] Figure 2 This is a flowchart of image and physiological signal processing according to an embodiment of the present invention;

[0059] Figure 3 This is a flowchart of cross-modal calibration and conflict determination according to an embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] like Figures 1-3As shown in the figure, the adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to an embodiment of the present invention includes the following steps:

[0062] S1: Obtain user input data, including natural language text. ) and / or voice data ( ).

[0063] The natural language text data is collected from various sources, including touchscreen interactive terminals deployed in museums and cultural tourism scenic spots, as well as input boxes in supporting mobile terminal apps and mini-program text input modules. This data is adapted to diverse interactive scenarios such as users actively searching for cultural relic information, switching display styles, and expressing their viewing experiences. This type of data supports input in multiple languages, including Chinese and English, and the text length can be controlled from 1 to 1000 characters, adapting to everyday conversational interaction needs. Voice data is collected through directional microphone arrays deployed in the display area, built-in microphones in interactive terminals, or authorized microphones on user mobile devices. Voice recording duration can be controlled from 1 to 10 seconds per recording, matching the length of everyday interactive sentences.

[0064] After collecting the raw data, preprocessing is required to optimize data quality. Text data preprocessing includes noise reduction, length verification, and encoding conversion. Specifically, this involves filtering spaces, newlines, and invalid symbols, while retaining Chinese characters, English letters, numbers, and key punctuation marks. All encodings are also uniformly converted to UTF-8 format to avoid character garbled text. Voice data preprocessing employs adaptive noise suppression algorithms (such as spectral subtraction) to filter environmental noise, such as crowd noise and equipment operation sounds, while preserving human voice signals. If the voice duration is less than 1 second, it is padded to 1 second to ensure data consistency. Finally, the audio formats collected from different devices are standardized to 16kHz sampling, 16-bit quantization, mono WAV format and stored as an array. .

[0065] S2: Perform semantic parsing and emotion recognition on the input data to generate semantic parsing results and a first emotion vector. This step transforms the unstructured input data (text / speech) collected in S1 into machine-processable structured information, extracts the user's core needs through semantic parsing, captures the user's subjective state through emotion recognition, and finally outputs standardized semantic parsing results and a first emotion vector.

[0066] Specifically, semantic parsing and emotion recognition are performed on the input data, including:

[0067] S21: Perform word segmentation and dependency parsing on the input data to generate structured text data. For the input data preprocessed in S1, word segmentation is first performed using a dictionary-based and statistically-driven tool, such as jieba or HanLP, adaptable to multiple languages ​​including Chinese and English. For Chinese word segmentation, the dictionary is optimized for the cultural relics field, such as by adding specialized terms like "bronze ware" and "glaze color," avoiding ambiguous segmentation (e.g., "bronze mirror" is not split into "bronze" and "mirror"). The final output is a word sequence. Where k is the word sequence length, defaulting to 1~50. For a single valid word. For example, the unstructured natural language sentence "I want to understand the patterns on Tang Dynasty bronze mirrors and hope to display a more dignified style" is segmented as: [I, want, understand, Tang Dynasty, bronze mirror, patterns, hope, display style, more, dignified].

[0068] Subsequently, using the Stanford Parser dependency parsing tool, the grammatical relations of words in the word sequence, such as subject-verb and verb-object, were analyzed based on probabilistic context-free grammar. This clarified the relationship between core words and modifiers, eliminated semantic ambiguity, and generated a syntactic relation matrix. Matrix elements Indicator and syntactic dependency strength express and There is a direct grammatical relationship, such as "understand" and "patterns" having a verb-object relationship. ; This indicates that there is no direct grammatical relationship, such as "Tang Dynasty" and "hope". Non-zero and non-1 values ​​indicate indirect dependency, such as "presentation style" and "solemn" having a primary-secondary relationship. .

[0069] The final generated structured text data is That is, the combination of word sequences and syntactic relation matrices, where, Provide vocabulary materials to clarify the core concepts mentioned by users, such as Tang Dynasty bronze mirrors, patterns, and solemnity; Provide a grammatical framework to clarify the logical relationships between concepts, such as "understanding" being the core action and "solemnity" modifying "display style".

[0070] S22: Input structured text data into a BERT-based semantic understanding model to generate semantic parsing results.

[0071] In this invention, the BERT model selected is the BERT-base-Chinese pre-trained model, which is suitable for Chinese scenarios. If English input is supported, the BERT-base-uncased model can be used. During input processing, the structured text is... The input is converted to a format acceptable to the BERT model. This involves converting the word sequence W into word IDs (via BERT vocabulary mapping), and incorporating the dependency strength of the syntactic relation matrix R as a positional encoding enhancement term into BERT's positional embeddings to strengthen the influence of syntactic relations on semantics. The hidden layer vector corresponding to the [CLS] token output by the model is the semantic parsing vector. ,in, Each component It represents a semantic feature, such as query-type requirements, style switching requirements, Tang Dynasty cultural themes, etc.

[0072] S23: The structured text data is input into the multi-task learning model. The multi-task learning model jointly outputs the intent classification result and the emotion intensity regression value to form the first emotion vector. The multi-task learning model is used to simultaneously identify the user's interaction intent and emotion intensity, avoiding the one-sidedness of single emotion recognition, and generating a comprehensive first emotion vector A.

[0073] In this invention, the multi-task learning model can adopt a shared encoder and task-specific decoder architecture. The shared encoder is a 3-layer bidirectional LSTM (256 hidden layer dimensions), reusing the feature extraction results of structured text. The two parallel decoders handle the intent classification and emotion intensity regression tasks respectively. The intent categories are core intents preset based on the cultural relic display scenario and can be adjusted as needed. For example, intent categories include cultural relic information query (e.g., "introducing blue and white porcelain"), display style switching (e.g., "changing to an exciting mode"), historical background consultation (e.g., "the uses of Tang Dynasty bronze mirrors"), and focusing on exhibit details (e.g., "magnifying the patterns"), etc. The intent classification vector is output through a Softmax activation function. , ,in This is the logit value output by the decoder. Let represent the probability that the input belongs to the k-th type of intent, and satisfy . In this embodiment of the invention, m=5 basic emotions can be preset, including happiness, calmness, solemnity, curiosity, and indifference; the emotion intensity regression vector is output through the Sigmoid activation function. , This represents the intensity of the q-th emotion category; a larger value indicates a stronger emotion. The final intention classification vector... With the regression vector of emotion intensity Concatenate the rows to obtain the first emotion vector. If c=8 and m=5, then A is a 13-dimensional vector, with the first 8 dimensions representing the probability of intent and the last 5 dimensions representing the intensity of emotion.

[0074] S3: Collect image data and physiological signal data of the audience in parallel. Process the image data to generate a group emotion distribution vector. At the same time, process the physiological signal data to generate a physiological emotion index that represents emotional arousal and valence.

[0075] In this invention, group image data can be collected using a high-definition camera array deployed in the exhibition area. The array must contain at least three cameras to achieve full-view coverage. The equipment can be a 1080P resolution, 10fps wide-angle camera, supporting automatic exposure and white balance adjustment. The collection range covers a typical exhibition scale of 5-50 people, focusing on the facial areas (eyes, mouth, eyebrows, and other key areas for emotional expression) and upper body posture of the audience. After collection, the images are sorted by timestamp in JPG format to form an image sequence. .

[0076] Physiological signal data can be collected through wearable devices (such as smart wristbands) or non-contact sensors, simultaneously acquiring heart rate variability (HRV) signals. and skin conductance response (GSR) signal , HRV reflects emotional fluctuations; for example, HRV decreases when stressed and increases when relaxed. Reflecting emotional arousal, such as increased skin conductivity during excitement, the HRV signal was sampled at a frequency of 250Hz, and the GSR signal at a frequency of 100Hz. The data was then processed using a time-series array. Storage, where t is the system's unified timestamp (unit: ms). and This represents the signal amplitude at the corresponding moment.

[0077] In some embodiments of the present invention, image data is processed to generate a group emotion distribution vector B, specifically including:

[0078] Convolutional neural networks are used to extract facial expression and body posture features from image data. Specifically, the acquired image sequence is first preprocessed. The MTCNN algorithm is used to detect all faces in the image, crop the facial region (size 224×224), and perform posture correction based on key points (eyes, nose, mouth) to eliminate posture interference such as side profile and head tilt. The OpenPose algorithm is used to extract 18 key human body nodes such as head, shoulders, elbows, and waist, filtering out invalid data with severe occlusion, and then Gaussian filtering is used to remove image noise and enhance facial texture and posture contour features. It should be noted that for the common problems of partial occlusion (such as arms crossing and obscuring the shoulders, and the back row audience being obscured by the front row) and complete occlusion (such as the head being obscured and the limbs being completely covered) in group scenes, this invention can adopt a hierarchical processing scheme of key point confidence screening and context-related completion to ensure the effectiveness of posture features. First, each key node output by the OpenPose algorithm carries a confidence score (value ∈ [0,1]), and a confidence threshold is set. (Can be dynamically adjusted according to the lighting in the exhibition hall; lowered to 0.4 in low-light environments), only retaining confidence levels ≥ Key point; for confidence levels < Nodes in the first stage are marked as pending completion and do not directly participate in feature extraction. Secondly, a priori library of human posture topology (including standard limb length ratios for adults / children, joint angle ranges, etc.) is constructed. For nodes to be completed, geometric inference is used to complete them using the spatial coordinates of their adjacent valid nodes. For example, if the shoulder node is valid but the elbow node is missing, the predicted coordinates of the elbow node are calculated based on the prior ratio of upper arm length / shoulder-hip length = 0.35, combined with the coordinates of the shoulder and wrist (if valid). If more than two nodes of the same limb are missing (e.g., the entire arm is occluded), the average of the neighboring postures is used for filling based on the statistical laws of group posture distribution. That is, the average of the posture features of the same part of the same limb from 3-5 unoccluded viewers around the viewer is selected as the compensation feature for the missing part. Finally, an occlusion degree coefficient is defined. ( For unobstructed views, (For complete occlusion), the calculation formula is: ,in For the number of effective key points, This represents the total number of key nodes; in the subsequent feature fusion stage, samples with high occlusion levels ( Reduce the pose feature weights (original) Down to At the same time, the weight of facial expression features was increased (originally...). Rise to This avoids the distortion of features caused by occlusion, which could affect the judgment of emotions.

[0079] To address the issues of overlapping postures (multiple people with intersecting limbs) and identity confusion (difficulty in distinguishing the limb affiliation of adjacent visitors) in densely populated areas (such as the core area of ​​an exhibition hall where the distance between visitors is <0.5m), this invention achieves precise processing through depth information assistance, spatial clustering, and posture instance segmentation. One approach is to add a ToF depth camera (10fps, time-synchronized with an RGB camera, timestamp error ≤30ms) to obtain pixel depth coordinates. Combining RGB image two-dimensional coordinates Constructing three-dimensional spatial coordinates ; By depth threshold Distinguish between front and back rows of audience (front row) Back row First, it achieves layered extraction of poses at different depths, avoiding overlap of limbs between front and back rows. Second, it performs DBSCAN density clustering on key points in 3D space (neighborhood radius 0.3m, minimum number of points 5), and the clustering result is the pose cluster of a single audience member. For key points that still exist across individuals after clustering, they are further distinguished by associating them with limb movement trajectories. By tracking the motion vectors of key points in 5 consecutive frames, key points with the same motion direction are grouped into the same subject, thus achieving pose instance segmentation. Third, to address the problem that individual pose features are easily interfered with in dense scenes, it extracts group statistical features (such as the average tilt angle of group poses, variance of limb movement amplitude, etc.) to supplement the body pose feature vector. This enhances feature robustness and avoids reliance on individual individual pose details. To address extreme occlusion or ultra-dense scenarios, this invention employs a multi-layered fallback strategy to ensure data continuity. Firstly, if a certain area's audience is completely occluded (e.g., by large exhibits) or extremely dense (crowd-blocked), making it impossible to extract effective poses, the average facial emotion features of that area are directly used as a substitute pose feature value, ensuring no data loss occurs in the calculation of the group emotion distribution vector B. By real-time statistical analysis of audience density (unit: people / ㎡) and occlusion ratio (number of occluded audience members / total number of audience members), when density > 3 people / ㎡ or occlusion ratio > 40%, the pose feature fusion weight can be reduced to... Increase the weight of facial features to When the density is less than 1 person / m² and the shading rate is less than 10%, restoration is required. , The original configuration is adapted to changes in data quality.

[0080] Facial expression feature extraction uses ResNet-18 as the base network. The input is a corrected facial image, and the output is... 3D facial expression feature vector , Quantitative information on facial muscle movements, such as upturned corners of the mouth and furrowed brows, is included to represent individual facial emotions. Body posture features are first derived using MobileNet-V2 as the backbone network, coupled with a Part Affinity Fields (PAF) pose detection module. From pre-processed group images (face detection and alignment, Gaussian filtering for noise reduction), the coordinates of 18 key human body nodes for each viewer are accurately output (including head, shoulder, elbow, waist, wrist, knee, and ankle, etc., coordinate format is...). In this step, the core role of MobileNet-V2 is to process image pixel data and extract image-level features to support pose detection, rather than directly processing structured coordinates. Subsequently, for the output discrete coordinate data, a graph convolutional network (GCN) is introduced to extract pose semantic features. First, a human pose map is constructed, and 18 key nodes are used as the node set of the graph. Define the associated edges between nodes based on human anatomy (e.g., left shoulder node). With left elbow node Connected, right shoulder node With right elbow node (connected), forming an adjacency matrix ( Represents a node and There is an anatomical connection. (Indicates no association); then the 18 nodes Coordinates are concatenated to form the initial feature matrix. (Each row corresponds to the two-dimensional coordinate features of a node), input into a 2-layer GCN for topological feature extraction, with the formula for each layer as follows: The first layer GCN uses a normalized adjacency matrix. (in Given an adjacency matrix with self-loops, for (degree matrix) and weight matrix Combined with the ReLU activation function, the intermediate feature matrix is ​​output. The second layer GCN uses As input, through the weight matrix With ReLU activation function output Finally, for Perform global average pooling, which means averaging the 18 node features of each feature channel to obtain a dimension of... Body posture feature vector ,Right now This vector can effectively represent the semantic information of an individual's posture, such as limb orientation, range of motion, and joint angles.

[0081] The facial expression features and body posture features are fused to output a probability distribution vector representing the proportion of different emotion categories in the group, which serves as the group emotion distribution vector.

[0082] Feature fusion can be achieved using a weighted concatenation method, with the following fusion formula:

[0083] ;

[0084] in, It can be set to 0.6 (facial feature weight). It can be set to 0.4 (posture feature weight). This aligns with the cognitive logic that facial expressions are the primary carriers of emotion, while posture is supplementary. Ensure consistent feature scale.

[0085] Finally, the fusion features will be used. The input layer is a fully connected layer and a softmax activation function, and the output is a group sentiment distribution vector with the same dimension as the S2 sentiment dimension. :

[0086] ;

[0087] In the formula, To fuse the weight matrix (obtained through training and optimization). This is the bias vector; where t=5, consistent with the number of emotion dimensions in step S2. and , This represents the proportion of the i-th type of emotion in a group, such as the proportion of "curiosity". "Calm" percentage .

[0088] In some embodiments of the present invention, physiological signal data are processed to generate a physiological mood index. Specifically, it includes:

[0089] The physiological signal data, including heart rate variability and skin conductance response signals, is filtered and feature extracted. Specifically, the raw physiological signals are first preprocessed using a Butterworth low-pass filter with a cutoff frequency of 0.5 Hz to effectively filter power frequency noise and motion artifacts. Then, the signals are segmented according to the image acquisition cycle (100 ms / segment) to ensure temporal alignment with the image data.

[0090] Subsequently, 4D core features (corresponding to) were extracted from the filtered HRV and GSR signals. ), forming physiological feature vectors : ;

[0091] In the formula, The mean value of the HRV signal (unit: ms) reflects heart rate stability (the mean value is higher when the mood is calm). The standard deviation of the HRV signal (unit: ms) reflects the amplitude of heart rate fluctuations (the standard deviation is larger when there are emotional fluctuations). The peak value of the GSR signal (unit: μS) reflects the peak value of emotional arousal (the peak value is higher when excited / nervous). The power spectral density of the GSR signal (unit: The power spectrum reflects the energy distribution of the GSR signal, and when the emotion is stable, the power spectrum is concentrated in the low frequency band.

[0092] The extracted physiological features are input into a classifier, which outputs scalar values ​​representing emotional arousal and valence, as a physiological emotion index.

[0093] physiological feature vectors Input a classifier, which can be a Support Vector Machine (SVM) or a lightweight neural network, and output a two-dimensional physiological mood index. :

[0094] ;

[0095] The classifier weight matrix is... is the bias vector (obtained through training with physiological emotion-annotated data); where, wake-up rate 0 represents low arousal (calm, indifferent), and 1 represents high arousal (excitement, tension); valence -1 represents a negative valence (boredom, discomfort), and 1 represents a positive valence (pleasure, curiosity).

[0096] It should be noted that if physiological signals from multiple audience members are collected, the average of the arousal and valence of all individuals should be taken as the group physiological emotion index to ensure that it represents the overall state of the group.

[0097] S4: Perform cross-modal calibration on the first emotion vector, the group emotion distribution vector, and the physiological emotion index to determine emotion recognition conflict; when a conflict exists, perform weighted correction on the first emotion vector and the group emotion distribution vector based on the physiological emotion index, and output a calibrated group emotion state.

[0098] In this invention, the first emotion vector A is c+m dimensional (number of intention categories + number of emotion dimensions), the group emotion distribution vector B is t dimensional (number of group emotion categories), and the physiological emotion index C is 2 dimensional (arousal - valence). Since these three types of data have different dimensions, they cannot be directly calculated for consistency and need to be transformed into vectors of the same dimension through dimensional mapping. Therefore, A and C can be mapped using a pre-trained mapping matrix. Transform it into a t-dimensional vector (consistent with the B-dimensional vector, t=5 by default) to ensure a one-to-one correspondence between emotion categories.

[0099] Specifically, the first emotion vector A is mapped , ,in, As a mapping matrix, the c+m dimensional "intention + emotion intensity" vector is compressed into a t-dimensional emotion distribution vector, preserving the core features related to group emotion in A through mapping. Physiological emotion index C is mapped. , ,in, As a mapping matrix, the 2-dimensional "arousal-valence" index is expanded into a t-dimensional emotion distribution vector. This mapping is based on the two-dimensional "arousal-valence" model in emotion psychology and establishes a correspondence with specific emotions of type t. For example, "high arousal + positive valence" corresponds to "curiosity" and "happiness". After mapping, the proportion of these two types of emotions increases.

[0100] Ultimately, three types of emotion vectors with the same dimension were obtained: (First emotion vector after mapping) (Group sentiment distribution vector, no mapping required) (Mapped physiological mood index).

[0101] In one embodiment of the present invention, determining an emotion recognition conflict specifically includes: calculating a first cosine similarity between a first emotion vector and a group emotion distribution vector; calculating a second cosine similarity between the group emotion distribution vector and a physiological emotion index; and determining that an emotion recognition conflict based on the physiological emotion index exists when the first cosine similarity is lower than a first preset threshold and the second cosine similarity is higher than a second preset threshold.

[0102] Among them, the first cosine similarity The formula used to measure the degree of matching between A' (mapped individual emotion) and B (overt group emotion) is as follows:

[0103] ,

[0104] In the formula, the molecule The dot product of vectors represents the degree to which two types of vectors are in the same direction; the denominator is the dot product of vectors. is the product of the L2 norms of the two vectors, used for normalization to ensure that the similarity values ​​range from [-1, 1]; where, (default This indicates that individual and group emotions are consistent; for example, if a user says "curious," the group is also likely to express curiosity. This indicates a discrepancy between individual and group emotions; for example, users might describe the emotion as "solemn," while the group might express curiosity.

[0105] The second cosine similarity measure quantifies the degree of matching between the group's explicit emotions and implicit physiological emotions. Physiological emotions serve as an objective benchmark, without subjective concealment. Its calculation formula is:

[0106] ;

[0107] (default This indicates that the group's overt emotions are consistent with their actual physiological emotions. For example, if the group exhibits curiosity, the physiological signals also show high arousal and positive valence. This suggests that the group's overt emotions may be disguised, such as appearing calm on the surface but showing signs of tension through physiological signals.

[0108] In this embodiment of the invention, the core of the conflict is the inconsistency between the individual's input emotion and the objective physiological-group emotion, and the determination formula is as follows:

[0109] ;

[0110] For example, a user enters "I want to appear dignified" ( The proportion of "solemn" was high in the group image, but the group image showed "curiosity" ( The proportion of "curiosity" is high in the body, and physiological signals also support "curiosity" ( and If the similarity is high, it is judged as a conflict, indicating that the user's subjective needs are inconsistent with the group's true emotions.

[0111] In one embodiment of the present invention, a weighted correction is performed based on a physiological emotion index, specifically as follows:

[0112] Assign the highest weight to the physiological emotion index and the lowest weight to the first emotion vector. Calculate the weighted average of the first emotion vector, the group emotion distribution vector, and the physiological emotion index, and output the calibrated group emotion state.

[0113] Weighted correction allocates weights based on objectivity priority to achieve complementary fusion of multi-source data. The weighting follows the principle of: physiological emotions being the most objective (highest weight) → group emotions being the second most objective (medium weight) → individual input emotions being the most subjective (lowest weight). The specific weight parameters are: the weight of the first emotion vector after mapping. (Adjustment range 0.05~0.2), indicating that individual input may be influenced by expression habits and subjective intentions, and has the lowest weight; group emotion distribution vector weight (Adjusted range 0.2~0.4), indicating that the group's overt emotions are statistically significant, but may be disguised; the weights of the mapped physiological emotion index (Adjustment range 0.5~0.7) indicates that physiological signals (HRV, GSR) are not subject to subjective control, best reflect true emotions, and have the highest weight; and .

[0114] Regardless of whether conflicts exist, this invention uses a unified weighted average formula to generate D, only fine-tuning the weights when there are no conflicts, such as increasing... Reduced to 0.5 up to 0.05, Up to 0.45; the weighted fusion formula is: ;

[0115] The calibrated group emotional state is obtained by summing the components. Each component ,and , This represents the proportion of the i-th type of emotion in the calibrated group.

[0116] For example, assuming t=5 (emotion categories: happiness, calmness, solemnity, curiosity, indifference), the mapped first emotion vector (User input tends to be "solemn"); Group sentiment distribution vector (The dominant emotional tendency of the group is "curiosity"); the mapped physiological emotional index (Physiological signals support "curiosity"); weighting , , ; obtained through calculation , , , , Calibration results The core emotion of the group was "curiosity" (accounting for 52%), while also taking into account the user's need for "solemnity" (15%), which is consistent with objective physiology and the group's state.

[0117] S5: Access the cultural knowledge and style rule base based on the semantic parsing results to obtain a cultural style template that matches the cultural theme.

[0118] In one embodiment of the present invention, accessing a cultural knowledge and style rule base based on semantic parsing results specifically includes: the cultural knowledge and style rule base is constructed in the form of a knowledge graph, which contains cultural symbol entities and semantic relationships between entities. Using the topic concepts from the semantic parsing results as queries, semantic similarity retrieval is performed in the knowledge graph to obtain associated texture styles, color systems, and musical mode data, thereby constructing a cultural style template.

[0119] Specifically, a knowledge graph is defined as , where entity set It encompasses core cultural symbol entities in the field of cultural relics, categorized by thematic entities (such as "Tang Dynasty bronze mirrors" and "blue and white porcelain"), stylistic entities (texture patterns such as "scroll patterns," color systems such as "blue and white," and musical modes such as "feather mode"), and related entities (such as "Tang Dynasty" and "ritual vessels"). Semantic relation matrix. Matrix elements Representing entities and The semantic association strength, of which strong association ( For example, "Tang Dynasty bronze mirror" is related to "scrolling grass pattern," "palace style," and "vermilion"; moderately related ( For example, "blue and white porcelain" and "cloud patterns" and "feather patterns"; weak correlation ( For example, "bronze" and "bright yellow" (historically rarely used for bronze).

[0120] To achieve semantic similarity calculation, this invention uses the TransE model to calculate the semantic similarity of each entity. Mapped to a low-dimensional dense vector ( The semantic query vector is generated from the semantic parsing results of step S2. Extract core topic concepts and generate semantic query vectors for retrieval. Thematic concept extraction is based on a dictionary of cultural relics (containing professional terms such as "Tang Dynasty", "bronze mirror", "patterns", and "style"). The top-3 core keywords with the highest weights are selected from the semantic parsing vector S. For example, if S corresponds to "I want to know about the patterns on Tang Dynasty bronze mirrors", the keywords would be "Tang Dynasty", "bronze mirror", and "patterns". Thematic concept vectors are constructed by extracting the semantic components in S that correspond to the core keywords. Dimension (default (Preserving core semantic features). To meet the requirements of similarity calculation, it is necessary to... L2 normalization is performed to make the vector magnitude length 1, ultimately generating a standardized semantic query vector. For example, when the S2 semantic parsing result corresponds to the text "introducing the display style of Song Dynasty blue and white porcelain", the core keywords are "Song Dynasty", "blue and white porcelain" and "display style". After extracting the relevant semantic components and normalizing them, the retrieval vector is obtained.

[0121] Semantic similarity retrieval is the process of searching for entities within a knowledge graph's entity vector set that are similar to or similar to other entities. The top-k style feature entities (texture, color, music) with the highest similarity. In this embodiment of the invention, since both the query vector and the entity vector have been normalized, the similarity calculation formula is as follows:

[0122] ;

[0123] Its retrieval process executes in parallel for three entity subsets: texture style, color system, and musical mode, to avoid cross-category interference, while also setting a similarity threshold. That is, only valid related entities that meet the threshold are retained. The top k entities with the highest similarity in each dimension are selected to form the search results. If there are fewer than k entities that meet the threshold, all of them are retained.

[0124] Cultural style template generation involves extracting and concatenating the attribute data of retrieved textures, colors, and musical entities to generate standardized cultural style templates. Each style feature entity pre-stores a quantized feature vector; specifically, the texture style feature vector... Including metrics such as density and curve complexity, color system feature vectors Includes metrics such as RGB ratio and hue, and musical mode feature vectors. It includes indicators such as scale distribution and rhythmic patterns.

[0125] The three types of feature vectors are concatenated row by row to form a cultural style template. : ;right Perform global L2 normalization to make the template vector magnitude 1, which is used for cross-modal mapping calculation.

[0126] S6: Based on a cross-modal mapping network, semantic parsing results, calibrated group emotional states, and cultural style templates are jointly mapped into multimodal control parameters, including music control parameters, lighting control parameters, and mechanical motion control parameters.

[0127] In one embodiment of the present invention, the semantic parsing results, the calibrated group emotional state, and the cultural style template are jointly mapped to multimodal control parameters, specifically including:

[0128] The calibrated group emotional state is concatenated with cultural style templates to generate a contextual feature vector, providing a key and value for cross-modal attention computation and ensuring the synergistic constraint between emotion and culture.

[0129] Specifically, using the calibrated group emotional state D and cultural style template T as input, the contextual feature vector is formed by directly concatenating them row by row. The mathematical expression is: ;in, , where is the dimension of the key / value vector. After concatenation, it needs to be... Layer normalization is performed to eliminate scale differences in features of different dimensions, thereby improving the stability of model training and inference.

[0130] The final generated context vector combines emotional and cultural constraints, allowing control parameters to align with group emotions, such as high arousal corresponding to faster rhythm and increased brightness, while ensuring that parameters do not deviate from the cultural attributes of cultural relics, such as Tang Dynasty templates corresponding to palace-style music and vermilion lighting.

[0131] The semantic parsing result is used as the query vector, and the context feature vector is used as the key and value vectors, respectively. These are then input into a Transformer-based cross-modal attention module for computation. The output of the cross-modal attention module is decoded into music control parameters, lighting control parameters, and mechanical motion control parameters via a feedforward neural network.

[0132] Specifically, cross-modal attention computation is guided by semantic parsing results. It dynamically allocates weights to context vectors through an attention mechanism, strengthening core relevant features and weakening irrelevant features. The semantic parsing result S is used as the query vector Q, and the normalized context feature vectors are used as the key vector K and value vector V. To improve fusion precision, a multi-head attention mechanism is employed, with up to eight attention heads. Each head maps a high-dimensional vector to a subspace using an independent projection matrix for parallel computation. Single-head attention first obtains attention weights through vector projection, similarity calculation (dividing by the square root of the subspace dimension to avoid gradient vanishing), and Softmax normalization, then weights and sums the value vectors. Subsequently, the outputs of the eight heads are concatenated and mapped back to the original dimension through an output projection matrix, ultimately capturing the complex relationships between semantics, emotion, and culture.

[0133] Control parameter decoding utilizes a feedforward neural network (FFN) to transform the attention-enhanced feature vector into standardized control parameters that can directly drive the device. Specifically, the 36-dimensional vector U output by the FFN is divided into three categories of control parameters based on its dimensions: ;

[0134] in, Music control parameters (16 dimensions) encompass indicators such as mode, rhythm, and volume, and are jointly constrained by cultural mode and emotional arousal. Lighting control parameters (8 dimensions) include primary color tone, brightness, color temperature, etc., which are determined by cultural color and emotional valence. These are mechanical motion control parameters, which are 12-dimensional and involve factors such as motion amplitude, speed, and complexity. They are influenced by cultural texture and emotional arousal.

[0135] All parameters are normalized after decoding to ensure that the value range is [0,1].

[0136] S7: Establish a unified time baseline, perform timing synchronization and phase compensation on multimodal control parameters, and generate a synchronized control instruction set.

[0137] The timing synchronization and phase compensation of multimodal control parameters include: assigning a unified timestamp to the control parameters of all modes; calculating the phase deviation of the lighting control signal and the mechanical action signal relative to the reference reference using the music beat signal as a reference reference; and when the phase deviation exceeds the allowable tolerance, shifting the control signal of the non-reference mode on the time axis to eliminate the phase deviation.

[0138] It's important to note that establishing a unified time baseline is crucial for assigning unique and consistent timestamps to control parameters across all modalities, eliminating synchronization issues caused by local clock deviations. Specifically, a high-precision global clock with an accuracy ≤1ms can be used to synchronize the clocks of all control modules (semantic parsing, emotion processing, parameter generation, and execution-side drivers). Periodic calibration via Network Time Protocol (NTP), such as once per second, eliminates system clock drift, keeping the clock drift error within 0.1ms / minute to ensure clock stability. Timestamps can be in Unix format (unit: ms), using the frame period of the music control parameters (default 10ms / frame) as a benchmark. Each set of data for lighting and mechanical motion parameters, after being divided according to its own frame period, is assigned a timestamp consistent with the most recent music parameter frame. For example, after dividing lighting parameters into 20ms / frames, every two frames of lighting parameters will share the timestamp of one music parameter frame, achieving initial alignment in the time dimension. This globally unified time baseline not only ensures that parameters corresponding to the same timestamp are executed simultaneously but also provides a basis for tracing the correlation between control parameters and audience feedback in closed-loop optimization.

[0139] Musical beats have a strong temporal regularity and can be used as time anchors. By calculating the phase deviation of lighting and mechanical action signals relative to the musical beats, the degree of timing asynchrony can be quantified. Specifically, this is first done from the music control parameters. Extract beat features and calculate the beat cycle based on the tempo component (60~120 BPM) in the music control parameters. For example, 120 BPM corresponds to a beat period of 500 ms; and a sine wave is used to represent the beat intensity (strong beats are peaks, weak beats are troughs), the mathematical formula is: ;

[0140] in, (Signal amplitude, normalized) The first time stamp (in milliseconds); The beat cycle; (Initial phase, strong beat corresponding phase) (k=0,1,2,...).

[0141] Subsequently, the lighting control parameters were analyzed separately. Mechanical motion control parameters Extracting light signals and mechanical action signals Among them, light signals Using the triggering moments of key actions such as brightness switching and hue changes as phase reference points, the model is constructed as a sine wave, with the mathematical expression as follows:

[0142] ;

[0143] In the formula, (amplitude), The light action cycle (unit: ms). This is the start time stamp for the first action of the lighting. This represents the initial phase of the light signal.

[0144] Mechanical action signal Using the triggering moments of key actions such as the start and stop of axis motion and the peak amplitude as phase reference points, the model is a sine wave, and its mathematical expression is:

[0145] ;

[0146] In the formula, (amplitude), The mechanical motion cycle (unit: ms). This is the timestamp of the first action of the machine. This represents the initial phase of the mechanical signal.

[0147] In this invention, the phase deviation is the phase difference between the signal to be calibrated and the reference signal, as shown in the following formula:

[0148] ;

[0149] ;

[0150] In the formula, Due to phase deviation of the light signal, This refers to the phase deviation of the mechanical action signal; the range of the phase deviation value is... (Corresponding time deviation range) (T is the reference period).

[0151] The correspondence between phase deviation and time deviation is as follows: ,

[0152] like hour, Corresponding time deviation .

[0153] In this invention, phase compensation is achieved by shifting the time axis of the light and mechanical action signals to make their initial phase consistent with the reference reference when the phase deviation exceeds the allowable tolerance, thus realizing timing synchronization.

[0154] Among them, the allowable tolerance for phase deviation is ,default value (45°), the corresponding time tolerance can be calculated based on the beat cycle, using the following formula: ;like hour, It takes 62.5ms.

[0155] In this invention, compensation is only performed on non-reference modes (lighting, mechanical actions), while the reference reference (music) remains unchanged; during compensation, the translation amount of the lighting signal is calculated first. (The negative sign indicates reverse translation to eliminate deviation) and translation amount of mechanical action signals. Compensated light signals Original timestamp The corresponding signal is adjusted to Execution; compensated mechanical action signal .

[0156] After compensation, synchronous verification is required to ensure that the accuracy meets the standard. The core verification indicator is that the time deviation of the three types of modal signals is ≤ During verification, the phase deviation between the compensated signal and the musical reference is recalculated. If... If the verification passes, then the translation amount is recalculated and secondary compensation is performed.

[0157] After successful verification, the compensated music, lighting, and mechanical motion control parameters are sorted by timestamp to form a synchronized control instruction set. :

[0158] ;

[0159] Among them, instruction set features are based on timestamps. Arranged in ascending order, the three types of parameters corresponding to the same timestamp are executed synchronously to ensure that "sound, light, and motion" are coordinated and consistent.

[0160] S8: Send the synchronized control command set to the execution device to drive the audio, lighting, and mechanical structures for coordinated display. Sending the synchronized control command set to the execution device includes: converting music control parameters into MIDI protocol commands or audio stream commands; converting lighting control parameters into DMX512 protocol data frames; converting mechanical motion control parameters into PWM pulse signals; storing control commands of different protocols in a transmission queue according to their unified timestamps, and sending them sequentially to the corresponding execution devices.

[0161] In this embodiment of the invention, the control parameters generated in step S6 are standardized numerical vectors (values ​​∈ [0,1]), which need to be converted into industrial / audio communication protocol instructions supported by the audio equipment, lighting fixtures, and mechanical structures respectively, to ensure that the devices can recognize and execute them. Specifically, music control parameters... It needs to be converted to MIDI protocol commands or audio stream commands supported by the audio equipment to adapt to different playback scenarios; among them, MIDI protocol conversion is suitable for synthesized music and instrument sound effects. The MIDI protocol controls the sound source device to generate music through digital signals. The core parameter conversion formula is:

[0162] MIDI note numbers (controlling pitch): ;

[0163] MIDI volume value (controls loudness): ;

[0164] MIDI tempo (controls BPM): ;

[0165] In the formula, This refers to the "mode type" component (∈[0,1]) in the music control parameters. , ; The MIDI protocol standard note number (∈[0,127]) corresponds to the piano key pitch (0=lowest note, 127=highest note); This refers to the "volume level" component (∈[0,1]) in the music control parameters. For the corresponding volume levels (0 = mute, 127 = maximum volume); This refers to the "rhythm speed" component (∈[0,1]) in the music control parameters. Corresponding beat frequency (60~120BPM).

[0166] Will , , It is encapsulated into a command frame according to the MIDI 1.0 protocol format, containing status bytes (such as 0x90 = note on), data byte 1 (note number), and data byte 2 (volume value).

[0167] Audio stream instruction conversion (suitable for pre-recorded cultural sound effects and background music). When it is necessary to play pre-stored cultural audio (such as Tang Dynasty musical instrument performances or explanations related to cultural relics), (Cultural style matching component) is mapped to the index of the pre-stored audio file, such as index 001 for "Tang Dynasty mode"; Mapped to audio playback volume (0~100%). Mapped to playback speed (0.8~1.2x speed); finally encapsulated as audio stream control instructions, such as HTTP LiveStreaming protocol instructions, including file index, volume, speed, and playback start timestamp.

[0168] Lighting control parameters The data is uniformly converted to DMX512 protocol data frames to adapt to professional stage lighting fixtures, such as LED PAR lights and moving head lights. The corresponding conversion formula (taking brightness and color temperature as examples) is as follows:

[0169] DMX brightness data: ;

[0170] DMX color temperature data: ;

[0171] This refers to the "brightness level" component (∈[0,1]) in the lighting control parameters, corresponding to 0~1000 lux; The brightness data is for the DMX512 protocol (∈[0,255]), where 0 = minimum brightness and 255 = maximum brightness. The color temperature component (∈[0,1]) in the lighting control parameters corresponds to 2700K (warm light) ~ 6500K (cool light); The color temperature data (∈[0,255]) of the DMX512 protocol is transformed according to the linear mapping relationship.

[0172] Mechanical motion control parameters Converted into a PWM (Pulse Width Modulation) signal to drive the motor and servo motor of the mechanical structure to perform actions. The conversion formula (taking action speed and amplitude as an example):

[0173] PWM duty cycle (speed control): ;

[0174] PWM duty cycle (amplitude control): ;

[0175] In the formula, The "action speed" component (∈[0,1]) in the mechanical control parameters corresponds to 0.1~1m / s; The duty cycle of the PWM signal (∈[0,100%]) is used. The higher the duty cycle, the faster the motor speed (e.g., 50% duty cycle corresponds to 0.5m / s). The "amplitude of motion" component (∈[0,1]) in the mechanical control parameters corresponds to 0~90°; The duty cycle for servo control is (∈[5%,10%], special adaptation), where 5% corresponds to 0° and 10% corresponds to 90°.

[0176] The motor control can use a 1kHz carrier frequency, and the servo control can use a 50Hz carrier frequency. The motor and servo are driven to perform actions through square wave signals.

[0177] Instruction queue management is crucial for ensuring timing synchronization. All protocol instructions must be managed in an orderly manner according to the unified timestamps allocated by S7. Specifically, instructions of different types, such as MIDI, DMX512, and PWM, are sorted in ascending order using timestamps as the key. Instructions with the same timestamp are grouped together to construct a priority queue. Queue elements include execution timestamp, protocol type, and instruction data. The buffer capacity is no less than 1000 groups, and breakpoint resumption is supported to prevent interruptions during transmission. The triggering mechanism can employ real-time time comparison. When the system clock reaches the timestamp of the instruction at the head of the queue, the transmission of that group of instructions is immediately triggered, with comparison accuracy controlled within 1ms. If an instruction is not transmitted within 50ms of the trigger time, it will automatically retry 3 times. If a retry fails, it will be logged for later investigation.

[0178] During the sequential delivery phase, instructions must be reliably transmitted to the target device via an adapted transmission link. Different devices employ different transmission schemes: MIDI instructions for audio equipment are transmitted via a USB-MIDI interface at a fixed baud rate of 31250bps; audio stream instructions are transmitted via Ethernet TCP / IP, requiring a bandwidth of at least 1Mbps and a latency of no more than 50ms; DMX512 instructions for lighting fixtures are transmitted via XLR-interface DMX cables, supporting daisy-chain connections and a transmission distance of up to 100m; PWM signals for mechanical structures are transmitted via control lines for short distances and extended via an RS485 bus for long distances, with a baud rate of 9600bps. The delivery process employs a combined "broadcast + addressing" mode. Instructions from multiple devices using the same protocol at the same timestamp are broadcast in batches, while personalized mechanical instructions are delivered unicast. Each device is assigned a unique ID to ensure accurate instruction reception.

[0179] After the command is issued, feedback data from the execution end is collected synchronously. This feedback data includes reception status (whether the command was successfully received and the timestamp), execution status (whether the command was executed normally and the timestamp of completion), and device status (operating parameters such as actual volume and brightness, and fault codes such as motor overload warnings). The data collection frequency is consistent with the command issuance frequency, both being 10ms / time. This data is transmitted through the reverse channel of the command issuance link and stored in association with the timestamp and device ID of the corresponding command, forming a command-feedback data pair. This ensures accurate tracking of control effects during subsequent optimization.

[0180] S9: Collect real-time feedback data from the execution device and audience status data, and adaptively optimize the model parameters based on the real-time feedback data and audience status data to achieve closed-loop control. Specifically, collecting audience status data involves returning to execution steps S3 and S4, collecting new image data and physiological signal data, and generating a new calibrated group emotional state.

[0181] In this invention, the acquisition of audience state data fully reuses the image acquisition equipment of S3 and the cross-modal calibration logic of S4. Specifically, image data is still acquired through a 1080P high-definition camera array at a frame rate of 10fps, covering the facial expressions and body postures of the audience in the entire display area; physiological signals are simultaneously acquired through wearable devices or non-contact sensors, including HRV and GSR signals, ensuring alignment with the image data timestamps (error ≤ 50ms). The acquired data is filtered, feature extracted, cosine similarity conflict judged, and weighted to generate a new calibrated group emotional state. This state directly reflects the audience's true feedback on the current display effect, such as whether the group's emotion has shifted from "indifference" to "curiosity" after the display style is adjusted.

[0182] Real-time feedback data from the execution end can be collected through the device's built-in sensors and communication links. The data dimensions cover the device's actual output parameters, operating status indicators (normal / abnormal), energy consumption values, fault codes, etc., ensuring comprehensive monitoring of the device's execution status.

[0183] Adaptive optimization of model parameters based on real-time feedback data and audience state data includes: calculating the error function between real-time feedback data and the new calibrated group emotional state and expected effect data; and using the error function to optimize the parameters of at least one model in semantic parsing and emotion recognition, cross-modal calibration, or cross-modal mapping network.

[0184] Among them, the real-time feedback data from the execution end is The preset expected performance data for the equipment is as follows: (Based on cultural style templates and emotional adaptation requirements, such as expected music volume and expected lighting brightness); the new calibrated group emotional state is... The expected value of sentiment is (e.g., the emotion distribution vector corresponding to "high arousal + positive valence"). The total error function adopts the weighted L2 norm form, as shown in the following formula:

[0185] ;

[0186] in, and These are error weights (default values ​​are 0.3 and 0.7 respectively), used to adjust the contribution ratio of device execution accuracy and emotion matching to the total error. Since emotion matching is the core of the display effect, it is assigned... Higher weight. It is the square of the L2 norm of the vector, calculated as the sum of the squares of the differences between the vector's components. The larger the value, the more significant the deviation between the actual and the expected values.

[0187] Parameter optimization minimizes the total error using gradient descent. This enables dynamic updating of model parameters. The set of parameters to be optimized... The optimization process covers the core models from the preceding steps, including the BERT semantic understanding model and multi-task learning model in S2, the cross-modal calibration weight matrix in S4, and the cross-modal mapping network parameters in S6. The mathematical formula for the optimization process is: ;

[0188] In the formula, These are the model parameters before optimization. For the optimized parameters, The learning rate (default value 0.001, adjustable range 0.0001~0.01) is used to control the step size of parameter updates. Too large a step size will cause parameter oscillations and non-convergence, while too small a step size will result in low optimization efficiency. The total error function with respect to parameters The gradient of the error indicates the direction in which the error increases. Therefore, when updating parameters, adjustments should be made in the opposite direction of the gradient to ensure that the total error continues to decrease.

[0189] The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to embodiments of the present invention integrates user language, group images, and physiological signals for multimodal emotion perception, and innovatively uses physiological feedback as a benchmark to perform cross-modal calibration of the recognition results. This effectively solves the pain points of existing technologies, such as single emotion recognition, susceptibility to interference, and inability to capture real emotions, and improves the accuracy and reliability of emotion analysis in group scenarios. On this basis, the system can adaptively optimize model parameters according to real-time audience feedback and display effects through a closed-loop learning mechanism, realizing the leap from static display to dynamic adaptation. Ultimately, while ensuring the accuracy of cultural expression, it significantly improves the precision and appeal of the immersive experience.

[0190] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An adaptive control method for cultural relic revitalization based on emotion perception and physiological feedback, characterized in that, Includes the following steps: S1: Obtain user input data, the input data including natural language text and / or voice data; S2: Perform semantic parsing and emotion recognition on the input data to generate semantic parsing results and a first emotion vector; S3: Parallel acquisition of image data and physiological signal data of the audience; processing of the image data to generate a group emotion distribution vector; and processing of the physiological signal data to generate a physiological emotion index representing emotional arousal and valence. S4: Perform cross-modal calibration on the first emotion vector, the group emotion distribution vector, and the physiological emotion index to determine emotion recognition conflict; when a conflict exists, perform weighted correction on the first emotion vector and the group emotion distribution vector based on the physiological emotion index, and output a calibrated group emotion state; S5: Access the cultural knowledge and style rule base based on the semantic parsing results to obtain a cultural style template that matches the cultural theme; S6: Based on a cross-modal mapping network, the semantic parsing results, the calibrated group emotional state, and the cultural style template are jointly mapped into multimodal control parameters, including music control parameters, lighting control parameters, and mechanical motion control parameters; S7: Establish a unified time baseline, perform timing synchronization and phase compensation on the multimodal control parameters, and generate a synchronized control instruction set; S8: Send the synchronized control command set to the execution device to drive the audio, lighting and mechanical structure to perform coordinated display; S9: Collect real-time feedback data from the execution device and audience status data, and adaptively optimize the model parameters based on the real-time feedback data and audience status data.

2. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S2, semantic parsing and emotion recognition are performed on the input data, including: The input data is segmented and subjected to dependency parsing to generate structured text data; The structured text data is input into a BERT-based semantic understanding model to generate the semantic parsing results; The structured text data is input into a multi-task learning model, which jointly outputs the intent classification result and the emotion intensity regression value to form the first emotion vector.

3. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S3, the image data is processed to generate a group emotion distribution vector, including: The facial expression features and body posture features in the image data are extracted using a convolutional neural network. The facial expression features and the body posture features are fused to output a probability distribution vector representing the proportion of different emotion categories in the group, which serves as the group emotion distribution vector. Processing the physiological signal data to generate a physiological emotion index includes: The physiological signal data, including heart rate variability and skin conductance response, is filtered and feature extracted. The extracted physiological features are input into a classifier, which outputs a scalar value representing emotional arousal and valence, as the physiological emotion index.

4. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S4, determining the conflict in emotion recognition specifically includes: Calculate the first cosine similarity between the first emotion vector and the group emotion distribution vector; Calculate the second cosine similarity between the group emotion distribution vector and the physiological emotion index; When the first cosine similarity is lower than the first preset threshold and the second cosine similarity is higher than the second preset threshold, it is determined that there is an emotion recognition conflict based on the physiological emotion index.

5. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S4, a weighted correction is performed based on the physiological emotion index, specifically as follows: The highest weight is assigned to the physiological emotion index, the lowest weight is assigned to the first emotion vector, and a weighted average is calculated on the first emotion vector, the group emotion distribution vector and the physiological emotion index to output the calibrated group emotion state.

6. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S5, the cultural knowledge and style rule base is accessed based on the semantic parsing result, including: The cultural knowledge and style rule base is constructed in the form of a knowledge graph, which contains cultural symbol entities and semantic relationships between entities. Using the topic concepts in the semantic parsing results as queries, semantic similarity retrieval is performed in the knowledge graph to obtain associated texture styles, color systems, and musical mode data, thereby constructing the cultural style template.

7. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S6, the semantic parsing result, the calibrated group emotional state, and the cultural style template are jointly mapped into multimodal control parameters, including: The calibrated group emotional state is concatenated with the cultural style template to generate a contextual feature vector. The semantic parsing result is used as the query vector, and the context feature vector is used as the key vector and value vector, which are then input into the Transformer-based cross-modal attention module for computation. The output of the cross-modal attention module is decoded by a feedforward neural network into the music control parameters, the lighting control parameters, and the mechanical motion control parameters.

8. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S7, timing synchronization and phase compensation are performed on the multimodal control parameters, including: Assign a uniform timestamp to the control parameters of all modes; Using the music beat signal as a reference, calculate the phase deviation between the lighting control signal and the mechanical action signal relative to the reference. When the phase deviation exceeds the allowable tolerance, the control signal of the non-reference mode is shifted along the time axis to eliminate the phase deviation.

9. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S8, the synchronized control command set is sent to the execution device, including: Convert the music control parameters into MIDI protocol instructions or audio stream instructions; The lighting control parameters are converted into DMX512 protocol data frames; The mechanical motion control parameters are converted into PWM pulse signals; Control commands from different protocols are stored in a sending queue according to their unified timestamps and then sent to the corresponding execution devices in sequence.

10. The adaptive control method for cultural relic activation based on emotion perception and physiological feedback according to claim 1, characterized in that, In step S9, the audience's status data is collected, specifically as follows: Return to steps S3 and S4 to collect new image data and physiological signal data, and generate a new calibrated group emotional state; The model parameters are adaptively optimized based on the real-time feedback data and the audience's status data, including: Calculate the error function between the real-time feedback data and the new calibrated group emotional state and the expected effect data; The parameters of at least one model in semantic parsing and emotion recognition, cross-modal calibration, or cross-modal mapping network are optimized using the error function.

Images