Campus green space ownership perception evaluation method and system based on multi-modal learning

Abstract

The invention discloses a campus green space ownership perception assessment method and system based on multi-modal learning, and the method specifically comprises the steps: inputting the output results of a visual feature coding network and an auditory feature coding network into an audio-visual modal feature fusion module for fusion processing, and obtaining an environment joint representation; inputting the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation; inputting the environment joint characterization and the physiological and psychological joint characterization into a global fusion module for deep fusion to obtain a final fusion feature; and inputting the final fusion feature into a multi-task prediction channel for processing, and training and optimizing in combination with a loss function to obtain an environment comfort comprehensive scoring result and an explanatory decision basis. According to the invention, comprehensive assessment of campus green space perception is realized, the accuracy and scientificity of assessment are improved, and decision support is provided for optimization of campus space design.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent environmental assessment / environmental behavior assessment, and specifically relates to a campus green space embodied perception assessment method and system based on multimodal learning. Background Art

[0002] Improving the sensory environment of campus space has become an important way to improve the quality of space and an important research topic in the field of environmental perception. However, the current technological development in this field has limitations, which to some extent affects the accuracy and effectiveness of scientific evaluation of campus space environment.

[0003] On the one hand, most existing environmental evaluation systems use a single-modal data analysis model. This model either relies solely on subjective questionnaires to evaluate individual psychological feelings or simply uses physiological indicators such as heart rate variability to reflect environmental pressure. However, this method ignores the complex relationship between audiovisual environment characteristics and multidimensional psychological and physiological parameters. There is a close correlation between audiovisual environment characteristics and people's psychological reactions and physiological states. At present, there is a lack of comprehensive analysis of this correlation, which makes it impossible for existing evaluation systems to comprehensively and accurately evaluate the impact of spatial environment on people's perception.

[0004] On the other hand, traditional multimodal fusion methods have obvious shortcomings when processing multi-source heterogeneous data. These methods usually use simple feature concatenation or weighted averaging to fuse data from different modalities, failing to effectively capture the nonlinear correlation between the temporal characteristics of the audiovisual environment and the dynamic response of physiological signals. This simple fusion method ignores the complex interactive relationship between different modal data, resulting in the fused data being unable to accurately reflect the impact of the spatial environment on human embodied perception. Summary of the invention

[0005] The purpose of the present invention is to provide a campus green space embodied perception evaluation method and system based on multimodal learning, which realizes a comprehensive evaluation of the embodied perception of campus green space, improves the accuracy and scientificity of the evaluation, and provides decision support for optimizing campus space design, so as to solve at least one of the above-mentioned prior art problems.

[0006] In a first aspect, the present invention provides a campus green space embodied perception evaluation method based on multimodal learning, the method specifically comprising: Acquire an environmental perception dataset, a physiological response dataset, and a psychological assessment dataset, wherein the environmental perception dataset includes a visual indicator quantization matrix and an acoustic time series tensor; Inputting the visual index quantization matrix into a visual feature coding network for processing, inputting the acoustic time series tensor into an auditory feature coding network for processing, and inputting the output results of the visual feature coding network and the auditory feature coding network into an audiovisual modality feature fusion module for fusion processing to obtain a joint representation of the environment; Inputting the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation; Inputting the environmental joint representation and the physiological and psychological joint representation into a global fusion module for deep fusion to obtain a final fusion feature; The final fusion features are input into the multi-task prediction channel for processing, and combined with loss function training optimization to obtain the comprehensive scoring result of environmental comfort and explanatory decision-making basis.

[0007] In a second aspect, the present invention provides a campus green space embodied perception evaluation system based on multimodal learning, the system specifically comprising: A data acquisition module, used to acquire an environmental perception data set, a physiological response data set, and a psychological assessment data set, wherein the environmental perception data set includes a visual indicator quantization matrix and an acoustic time series tensor; An audio-visual processing module, used to input the environment perception data set into a visual feature coding network and an auditory feature coding network for processing, and input the output results of the visual feature coding network and the auditory feature coding network into an audio-visual modality feature fusion module for fusion processing to obtain a joint representation of the environment; A physiological and psychological processing module, used for inputting the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation; A global fusion module, used for inputting the environmental joint representation and the physiological and psychological joint representation into the global fusion module for deep fusion to obtain a final fusion feature; The evaluation output module is used to input the final fusion features into the multi-task prediction channel for processing, and combine the loss function training optimization to obtain the comprehensive scoring result of environmental comfort and the explanatory decision basis.

[0008] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor and a computer program stored in the memory. When the computer program is executed on the processor, the campus green space embodied perception assessment method based on multimodal learning as described in the above method is implemented.

[0009] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the campus green space embodied perception assessment method based on multimodal learning as described in the above method.

[0010] Compared with the prior art, the present invention has at least one of the following technical effects: 1. The present invention realizes a comprehensive evaluation of the embodied perception of campus green space by integrating multimodal data such as environmental perception, physiological response and psychological evaluation, improves the accuracy and scientificity of the evaluation, and provides decision support for optimizing campus space design.

[0011] 2. The present invention innovatively constructs a multimodal dynamic perception and interpretable evaluation system, showing significant technical advantages in the field of intelligent evaluation of environmental comfort. The dynamic coupling of environmental parameters and physiological feedback is achieved through an adaptive weight distribution mechanism, effectively capturing the nonlinear correlation characteristics of multi-source information in complex scenes; the multi-task joint reasoning architecture breaks through the semantic decoupling bottleneck of traditional models, and simultaneously outputs comprehensive evaluation indexes and multi-dimensional interpretable components, significantly improving the traceability of decision-making basis; the hierarchical normalization fusion strategy overcomes the difficulty of cross-modal feature alignment, ensuring high-precision mapping of physical environment quantitative indicators and subjective perception evaluation in a unified semantic space. This technical system provides intelligent decision-making support for optimizing campus space design through closed-loop feedback optimization.

[0012] 3. The present invention is based on a real-time weight distribution mechanism of a dual-channel neural network, and constructs a dynamic feature fusion framework through collaborative analysis of environmental physical parameters and human physiological indicators. This technology breaks through the traditional static weighting mode and adopts a joint measurement mechanism of environmental perception confidence and physiological feedback sensitivity to achieve autonomous adjustment of audiovisual features and biological signal contributions. It strengthens the dominance of physical features under extreme environmental parameter conditions, and gives priority to human feedback information in a state of significant physiological stress, effectively balancing the interactive effects of objective environmental attributes and subjective perception, and significantly improving the robustness of evaluation in complex scenarios.

[0013] 4. The present invention constructs a three-level attention-driven feature interaction system to achieve progressive fusion from micro-feature matching to macro-semantic association. The local correlation between visual parameters and pupil dynamics is captured through fine-grained cross-modal dot product attention, and the nonlinear mapping relationship between acoustic features and psychological scores is modeled using multi-head cross-attention. Finally, decision-level feature optimization is achieved through a gated residual network. This architecture breaks through the limitations of traditional single-scale fusion, eliminates feature conflicts between modalities while retaining the spatiotemporal characteristics of the original data, and provides a unified expression space for deep collaboration of multi-source heterogeneous data.

[0014] 5. The present invention designs a decoupled joint training framework, which decomposes the environmental comfort assessment into a topological structure with a main decision network and a multi-dimensional interpretation network in parallel. The feature space dissociation technology is used to achieve independent modeling of visual contribution, acoustic comfort and physiological fitness, and the mutual information constraint mechanism is combined to ensure the orthogonality of each explanatory factor. While maintaining the prediction accuracy, the system provides a physically interpretable decision-making basis for intelligent environmental control.

[0015] 6. The present invention utilizes the local feature extraction layer and the LSTM timing layer to extract multi-scale acoustic features from the acoustic timing tensor and capture the temporal dependencies to form an accurate acoustic embedding, providing important information for the construction of the joint representation of the environment.

[0016] 7. The present invention realizes the effective fusion of visual embedding and acoustic embedding through a cross-modal Transformer encoder and a dynamic weight gating unit, forms a joint representation of the environment containing audio-visual environment information, and improves the comprehensive understanding of environmental characteristics.

[0017] 8. The present invention co-encodes the psychological assessment data set and the physiological response data set to form a physiological and psychological joint representation that includes the individual's psychological and physiological states, providing a key basis for evaluating the impact of the environment on human perception.

[0018] 9. The present invention extracts multi-scale physiological time series features from the physiological response dataset through the dilated time series convolution layer and the gated attention pooling layer, and calculates the importance score of each time step to form an accurate physiological state embedding.

[0019] 10. The present invention achieves effective integration of physiological state embedding and psychological state embedding by dynamically weighting and correcting the two, forming a joint representation of the individual's comprehensive physiological and psychological state.

[0020] 11. The present invention realizes the deep fusion of environmental joint representation and physiological and psychological joint representation through fine-grained feature matching, semantic-level relationship modeling, decision-level dynamic aggregation and other steps, forming a final fusion feature containing rich information. At the same time, through environmental impact assessment and human feedback compensation, the accuracy and practicality of the fusion feature are further improved.

[0021] 12. The present invention maps the final fusion features to the evaluation space through the three-layer fully connected network of the main prediction channel, outputs the comprehensive score result of comfort, and provides a direct basis for the quantitative evaluation of environmental comfort; the final fusion features are analyzed for interpretable decision basis through the auxiliary channel, and multi-dimensional explanatory factors are output, which provides decision support for the interpretation and understanding of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1It is a flow chart of a campus green space embodied perception evaluation method based on multimodal learning provided by an embodiment of the present invention; Figure 2 It is a structural schematic diagram of a multimodal learning model of a campus green space embodied perception evaluation method based on multimodal learning provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a campus green space embodied perception evaluation system based on multimodal learning provided by an embodiment of the present invention; Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0024] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0025] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0027] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0028] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0029] In the embodiment of the present application, the execution subject of the process includes a terminal device, which includes but is not limited to: a server, a computer, a smart phone, a tablet computer, and other devices capable of executing the method disclosed in the present application. Figure 1 A schematic diagram of a flow chart of a campus green space embodied perception evaluation method based on multimodal learning disclosed in an embodiment of the present invention is shown, and is described in detail as follows: S101, acquiring an environmental perception dataset, a physiological response dataset, and a psychological assessment dataset, wherein the environmental perception dataset includes a visual indicator quantization matrix and an acoustic time series tensor.

[0030] In this embodiment, the three types of core indicator data required include: (1) Environmental physical indicators: multimodal environmental characteristics covering vision (seven-dimensional quantitative parameters such as green view rate, openness, green coverage, color richness, water body ratio, tree species richness, and facility richness) and acoustics (complex bird sounds, human voices, sound pressure levels of mechanical sounds, and MFCC spectra); (2) Physiological response indicators: including time series signals of eye movement (number of fixations, fixation time, average number of blinks, pupil diameter), EEG (α / β waves), ECG (heart rate HR, heart rate variability HRV), and skin conductance (skin conductivity SCL); (3) Psychological evaluation indicators: The comprehensive scoring data of visual satisfaction (seven dimensions, including naturalness, beauty, extensibility, peace, diversity, compatibility, and fluctuation), auditory satisfaction (seven dimensions, including pleasure, pleasantness, clarity, disorder, richness, fluency, and naturalness) and audio-visual coordination based on factor analysis constitute the psychological representation of human perception of the environment.

[0031] Based on the above core indicators, environmental perception datasets, physiological response datasets and psychological assessment datasets were collected. Among them, the environmental perception dataset includes visual indicator quantization matrix and acoustic time series tensor. The visual indicator quantization matrix is ​​a 7-dimensional environmental feature vector composed of green view rate (0-1 proportional value), openness (three-dimensional space index), green coverage rate (0-1 proportional value), HSV color richness (hue saturation brightness statistics), water body proportion (0-1 proportional value), tree species richness (discrete count), and facility richness (discrete count). The acoustic time series tensor includes the time domain waveforms of composite bird sounds, human voices, and mechanical sounds (sampling rate ≥ 44.1kHz) and their corresponding equivalent sound levels LAeq (dB(A)) and MFCC spectrum features (24 dimensions / frame).

[0032] The physiological response dataset includes eye movement trajectory matrix, EEG feature tensor and electrocardiogram-skin joint tensor. The eye movement trajectory matrix includes the multi-channel time series of gaze coordinate point sequence (x, y), gaze duration (ms), pupil diameter change curve (mm / s), and blink interval (ms). The EEG feature tensor includes the power spectral density matrix (64 leads × time window) of α wave (8-13Hz) and β wave (14-30Hz). The electrocardiogram-skin joint tensor includes the minute-level physiological parameter sequence of heart rate variability (LF / HF ratio), RR interval standard deviation (ms), and skin conductance level (μS).

[0033] S102, input the visual indicator quantization matrix into the visual feature coding network for processing, input the acoustic time series tensor into the auditory feature coding network for processing, and input the output results of the visual feature coding network and the auditory feature coding network respectively into the audio-visual modality feature fusion module for fusion processing to obtain a joint representation of the environment.

[0034] In this embodiment, the visual index quantization matrix is ​​input into the visual feature encoding network. The network includes input layer, convolution layer, pooling layer, fully connected layer and other structures, which can automatically learn and extract key features in the image. Through the layer-by-layer processing of the network, the visual embedding is finally output, that is, a low-dimensional vector representation containing image information.

[0035] The acoustic time series tensor is input into the auditory feature encoding network. The network includes input layer, LSTM (Long Short-Term Memory) layer, fully connected layer and other structures, which can process time series data and extract the temporal dependencies therein. Through the layer-by-layer processing of the network, the acoustic embedding is finally output, which is a low-dimensional vector representation containing sound information.

[0036] The visual embedding and the acoustic embedding are input into the fusion module. The module can fuse the two embeddings using methods such as attention mechanism, concatenation, weighted summation, etc. Through the fusion process, the final output is a joint representation of the environment, that is, a comprehensive representation containing visual and auditory information.

[0037] In this embodiment, by processing the visual and auditory data separately and fusing them into a joint representation of the environment, the environmental characteristics of the green space can be more comprehensively reflected, which helps to improve the accuracy of the evaluation and make the evaluation results more in line with the actual situation. Since the visual and auditory data are processed separately and fused into a joint representation of the environment, it is easier to analyze the impact of different senses on environmental comfort, which helps to enhance the interpretability of the evaluation results and make the evaluation process more transparent and understandable.

[0038] S103, inputting the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation.

[0039] In this embodiment, the physiological and psychological joint representation integrates physiological and psychological information, providing researchers with a more comprehensive perspective to understand the response of individuals to specific environmental stimuli, and helping to reveal the potential associations and interaction mechanisms between physiology and psychology. By training and optimizing the physiological and psychological co-encoder, the model is able to learn the complex relationship between physiological and psychological responses, which makes the model more accurate in predicting the individual's future physiological or psychological responses.

[0040] S104, inputting the environmental joint representation and the physiological and psychological joint representation into a global fusion module for deep fusion to obtain a final fusion feature.

[0041] In this embodiment, the final fusion feature integrates environmental, physiological and psychological information, providing researchers with a more comprehensive perspective to understand the overall response of individuals under specific environmental stimuli, and helping to reveal the potential associations and interaction mechanisms between the environment, physiology and psychology. The global fusion module can more accurately predict the overall response of individuals under future environmental stimuli by learning the deep associations between different representations, which is of great significance for formulating personalized intervention measures and improving individuals' adaptability and quality of life.

[0042] S105, inputting the final fusion feature into the multi-task prediction channel for processing, and combining with the loss function training optimization to obtain the comprehensive scoring result of the environmental comfort and the explanatory decision basis.

[0043] In this embodiment, the fused features are input into a multi-task prediction channel. This channel includes two subtasks: one is to predict the comprehensive score of environmental comfort, and the other is to generate explanatory decision basis. In order to handle these two tasks simultaneously, a multi-task learning model can be designed, which includes a shared hidden layer and two output layers corresponding to different tasks. In the task of predicting the comprehensive score of environmental comfort, the output layer can be a regression model for outputting a continuous score value. In the task of generating explanatory decision basis, the output layer can be a classification model or a sequence generation model for outputting discrete decision labels or descriptive texts.

[0044] In order to train a multi-task learning model, a comprehensive loss function needs to be designed.

[0045] For main task loss , the comfort score regression task uses the mean square error (MSE), the formula is as follows: ; Among them, the symbol and Respectively represent The actual comfort score of each sample and the model-predicted comfort score are compared. The predicted score is calculated through the main prediction channel.

[0046] For auxiliary task loss , the explanatory factor regression task uses smooth L1 loss (SmoothL1): ; Among them, the symbol and Respectively represent The actual sub-dimension scores of the explanatory factors and the sub-dimension scores predicted by the model.

[0047] Based on the above, the regularization term is calculated by using the logical consistency regularization , the regularization term Constrain the mutual information between explanatory factors to prevent dimensional redundancy.

[0048] ; Finally, a multi-objective optimization framework is used to balance the main and auxiliary tasks, and the total loss Using weighted summation: ; Among them, when the auxiliary score deviates significantly from the main evaluation result (difference > 2 standard deviations), the model self-check mechanism is triggered to recalibrate the feature weights.

[0049] In terms of optimizer, AdamW is used, with initial learning rate and weight decay. For learning rate scheduling, a cosine annealing strategy is adopted with a period of 10 epochs and a minimum learning rate. During the training process, we use the back propagation algorithm to update the weight parameters of the model to minimize the comprehensive loss function. Through continuous iterative training, the model will gradually learn how to predict the comprehensive score of environmental comfort and generate explanatory decision-making basis based on the input features. Finally, the trained model is applied to new data input to predict the comprehensive score of environmental comfort and generate explanatory decision-making basis. These results can help users better understand the comfort status of the indoor environment and make corresponding adjustments or improvements.

[0050] In this embodiment, through the multi-task learning model, the performance of two tasks can be optimized at the same time, thereby improving the prediction accuracy of the comprehensive score of environmental comfort. In addition, feature fusion technology also helps to comprehensively utilize information at different levels to further improve the prediction ability of the model. The task of generating explanatory decision-making basis enables the model to not only predict results, but also provide decision support, which helps users better understand the prediction results of the model and make corresponding decisions. The multi-task prediction channel combined with loss function training optimization can provide users with accurate and explainable environmental comfort evaluation results, thereby improving user experience and satisfaction.

[0051] In some embodiments, reference Figure 2 , the audio-visual modality feature fusion module includes a cross-modal Transformer encoder and a dynamic weight gating unit; in the above step S102, the output results of the visual feature encoding network and the auditory feature encoding network are input into the audio-visual modality feature fusion module for fusion processing to obtain a joint representation of the environment, specifically including: Inputting the visual index quantization matrix into the first input layer, performing maximum and minimum normalization processing on the visual index quantization matrix, and obtaining a normalized visual index quantization matrix; The standardized visual index quantization matrix is ​​mapped to a high-dimensional space using a first fully connected layer to obtain the original projection features. The first fully connected layer satisfies: ; in, Represents the first visual feature mapping result, Represents the second visual feature mapping result, and represents the weight matrix of the visual feature map, and Respectively represent the bias vector of each fully connected layer of the first fully connected layer, represents the GELU activation function; A self-attention layer is used to capture the interaction between each indicator of the standardized visual indicator quantization matrix to form an attention feature. The self-attention layer satisfies: ; ; ; in, , and Represents query, key and value respectively, , and Respectively , and The projection matrix, Represents the calculation result of single-head attention, represents the scaling factor, Represents the attention feature, Represents a splicing operation, Indicates that the first attention is used to focus on low-order projection features, Indicates that the second attention is used to focus on high-order semantic features, Indicates that the third attention head is used to focus on residual features and cross-head interactions, represents the fusion matrix of multi-head attention output, Represents the Softmax function; The original projection feature and the attention feature are input into the first output layer for residual connection, and the visual embedding is output. The visual embedding is ,in, represents visual embedding, Representation layer normalization operation.

[0052] In this embodiment, the processing steps of the visual feature encoding network include an input layer, a fully connected projection, and a self-attention mechanism.

[0053] Input layer: 7-dimensional visual indicator vector , corresponding to the green viewing rate , openness Equal quantitative parameters, after maximum and minimum standardization: ; in, For the The raw measurements of the visual indicators, and The corresponding theoretical or statistical maximum / minimum value of each indicator is used for standardization. The The output results after the indicators are standardized.

[0054] Fully connected projection: Sparse features are mapped to high-dimensional space through two layers of MLP, and the activation function is GELU: ; in, , both are weight matrices of visual feature mapping, which respectively map the input dimension 7 to a 128-dimensional latent space and then expand it to 256 dimensions. , The bias vectors corresponding to the two fully connected layers are added together by multiplying the weight matrix through the broadcast mechanism. It is the result of 7→128-dimensional projection + GELU operation, which is used to decouple low-order features and separate basic physical properties. It is the result of 128→256-dimensional projection + GELU operation, which is used for high-order semantic abstraction and extraction of complex environment features for nonlinear enhancement; hierarchical mapping gradually separates indicator coupling through the intermediate 128-dimensional latent space.

[0055] Aiming at the discrete visual parameters unique to this field (such as green view rate as discrete sampling value), a phased progressive mapping strategy is proposed to solve the over-smoothing problem of traditional single-layer MLP on low-dimensional sparse data. Through two layers of MLP, the sparse 7-dimensional visual indicators (green view rate, openness, etc.) are mapped to a 256-dimensional latent space to achieve feature decoupling and information concentration. The first layer (7→128 dimensions) breaks through the representation bottleneck of traditional single-layer projection and separates low-order visual attributes; the second layer (128→256 dimensions) uses GELU activation function to enhance nonlinear representation capabilities and extract high-order semantic features.

[0056] Self-attention mechanism: Capture the interaction between indicators through three-head attention and generate attention features , the 256-dimensional features are evenly divided into 3 groups (85 dimensions each), and attention is calculated in independent subspaces. Then, the output of each group of attention is restored to 256 dimensions through concatenation and linear projection: ; ; ; in, , are the projection matrices of query, key, and value, respectively, mapping hidden features to the attention calculation space, and is a scaling factor, which is determined by the number of heads (3) and the feature dimension (256) of the multi-head attention mechanism and is used to stabilize the distribution of attention weights; is the result of single-head attention calculation, and is the fusion matrix of the multi-head attention output. The 3-head attention captures the interaction relationship of different types of indicators through parallel subspace modeling. Focusing on low-order projection features, it is used to quantify the linear relationship of indicators, such as calculating the complementarity between green view rate and openness; Focus on high-order semantic features to achieve perceptual semantic coupling, such as the synergistic effect of openness and color richness; The focus is on residual features and cross-head interactions to achieve dynamic environmental adaptation, that is, adjusting weights according to different situations, such as strengthening the weight of openness in densely built-up areas and strengthening the weight of green view rate in natural landscape areas.

[0057] Construct a 3-head self-attention layer in the 256-dimensional latent space through a dynamic weight matrix , quantify the potential correlation between visual indicators, and explore the implicit coupling rules between multiple indicators. Through the multi-head mechanism, local details (single-head focusing on color-space correlation) and global semantics (multi-head joint modeling landscape coordination) are separated.

[0058] Based on the above, the self-attention features and the original projection features are combined to output the residual connection. : ; Fully connected projection and self-attention form a complementary mechanism - the former solves feature sparsity, and the latter eliminates the empirical bias of manually preset weights. The two jointly realize end-to-end mapping from physical parameters to psychological semantics. The fully connected projection module and self-attention mechanism of this design form a specific encoding paradigm for environmental perception data through progressive feature decoupling and dynamic indicator relationship modeling.

[0059] In some embodiments, reference Figure 2 , the auditory feature encoding network includes a second input layer, a local feature extraction layer, an LSTM timing layer and a second output layer; in the above step S102, the acoustic timing tensor is input into the auditory feature encoding network for processing, specifically including: Inputting the sound pressure level time series and MFCC spectrum of the acoustic time series tensor into the second input layer, segmenting and splicing into joint acoustic features through a sliding window; Multi-scale acoustic features are extracted from the joint acoustic features using a multi-layer dilated convolution of a local feature extraction layer, wherein the local feature extraction layer satisfies ,in, Indicates Layer dilated convolution output, Indicates Layer dilated convolution output, represents the convolution kernel, Represents the number of channels of the convolutional layer, One-dimensional convolution operation, ReLU activation function. The temporal dependency of the multi-scale acoustic features is captured by the LSTM timing layer to form an acoustic embedding. The LSTM timing layer satisfies: ; ; in, Indicates The forward state concatenation result of time steps is Indicates The backward state concatenation result of time steps is: Indicates The forward state concatenation result of time steps is Indicates The backward state concatenation result of time steps is Represents the output of the 3rd layer dilated convolution; The acoustic embedding is ,in, represents the acoustic embedding, Represents the total number of windows generated after the sliding window is split. Represents the final output sequence of the LSTM timing layer; The acoustic embedding is output through the second output layer.

[0060] In this embodiment, the processing steps of the auditory feature encoding network include input preprocessing, 1D-CNN local feature extraction and LSTM time series modeling.

[0061] Input preprocessing: sound pressure level time series and MFCC spectrum , segmented by sliding window (window length 2s, step length 1s) and then spliced ​​into joint features .in is the total duration of the acoustic signal (number of time steps), is the total number of windows generated after the sliding window is split. The window length is 2 seconds, and the number of sampling points in the window is ( is the sampling rate), the original signal is divided into windows, is the step length points; For the feature concatenation operation, the sound pressure level (1 dimension) and MFCC (20 dimensions) are merged into a 21-dimensional joint feature along the feature dimension.

[0062] 1D-CNN local feature extraction: Three layers of dilated convolution (dilation rate = 1, 2, 4) are used to extract multi-scale acoustic features. The dilation rate increases (1→2→4) to gradually expand the receptive field, capturing instantaneous events, short-term patterns, and long-term regularities respectively. The output of each layer is as follows: ; in , output For the Layer dilation convolution output, the receptive field expands exponentially with the dilation rate (layer 1: 5 points, layer 2: 9 points, layer 3: 17 points), ; is the convolution kernel, is the number of channels of the convolutional layer, and each layer outputs 64 feature maps.

[0063] LSTM time series modeling: Bidirectional LSTM captures long-term dependencies, and the unidirectional hidden state dimension is 128. The modeling formula is as follows: ; ; Based on the above, bidirectional splicing is performed to obtain the bidirectional LSTM at the time step The hidden state , the dimension of the forward and backward states after concatenation is 256, and the final output sequence is ; Gradually compress the time dimension (4 times) through the convolution step, balance detail retention and computational efficiency, and improve training speed.

[0064] Final Acoustic Embed The average pooling operation on the time axis is used to compress the time series features into global statistics, retaining the steady-state characteristics and dynamic patterns of the acoustic environment: ; The encoding network achieves fine-grained analysis of acoustic comfort through multi-scale time-frequency joint modeling. The dilated convolutional layer of 1D-CNN gradually expands the receptive field (5→9→17 time points) to capture multi-granular features from transient noise to periodic acoustic events; the bidirectional LSTM modeling modeling long-term evolution of acoustic parameters captures time dependency, and the time axis average pooling condenses the time series features into semantically clear acoustic embeddings to improve the robustness to burst noise. At the same time, the 256-dimensional embedding With visual embedding Dimension alignment provides compatibility guarantee for subsequent cross-modal fusion.

[0065] In some embodiments, reference Figure 2, the audio-visual modality feature fusion module includes a cross-modal Transformer encoder and a dynamic weight gating unit; in the above step S102, the output results of the visual feature encoding network and the auditory feature encoding network are input into the audio-visual modality feature fusion module for fusion processing to obtain a joint representation of the environment, specifically including: A cross-modal Transformer encoder is used to perform cross-modal attention calculation on the visual embedding and the acoustic embedding, and the cross-modal Transformer encoder satisfies: ; ; ; in, , and Represents query, key and value respectively, , and Respectively , and The projection matrix, represents the cross-modal attention weight matrix, represents the fusion feature result, represents the scaling factor; A dynamic weight gating unit is used to adjust the modal contribution of the visual embedding and the acoustic embedding to obtain a joint representation of the environment, where the joint representation of the environment satisfies: ; ; in, represents the gating coefficient, represents the parameter matrix used to vectorize the visual embedding and acoustic embedding, represents the Sigmoid activation function, Represents a joint representation of the environment.

[0066] In this embodiment, the processing steps of the audio-visual modality feature fusion module include a spatial alignment strategy and dynamic weight gating.

[0067] Spatial alignment strategy: Design a cross-modal Transformer encoder with visual embedding as query and acoustic embedding as key-value: ; ; ; in , are the projection matrices of the query, key, and value, respectively, and the scaling factor To stabilize cross-modal attention computation; is the cross-modal attention weight matrix, which quantifies the retrieval strength of visual features over acoustic features; The value projection of acoustic features encodes the semantic expression of auditory information. This strategy breaks through the traditional symmetrical cross-attention and designs a visual query-dominated retrieval mode to improve the evaluation accuracy of landscape-dominated environments (such as campus environments). In order to fuse the feature results, the structure adopted can retain the residual features, inject cross-modal corrections while retaining the basic visual features to avoid information loss.

[0068] The difference from the visual feature encoder is that the query source here is the visual embedding , the Key-Value source is acoustic embedding ; Unlike the projection matrix of the visual feature encoder, which is used for feature reorganization within the same modality, the projection matrix corresponding to the audio-visual modality feature fusion module is used to achieve cross-modal feature alignment and establish audio-visual semantic associations. The input sources are inconsistent (intra-modal feature interaction and inter-modal feature retrieval), and the weight learning targets are also inconsistent (visual self-attention and cross-modal attention).

[0069] Compared with the traditional symmetric cross-attention method, the present invention adopts vision-dominated cross-attention, with only vision as the query and hearing as the key-value. It reduces the amount of calculation through one-way retrieval and is more suitable for landscape-dominated campus scenes. At the same time, it adopts visual semantic constraint weight distribution to capture key audio-visual associations.

[0070] Dynamic weight gating: The modal contribution is adjusted by adaptive gating coefficients based on environmental characteristics, and the Sigmoid activation function is used to , output gating coefficient : ; ; Parameter Matrix The concatenated 512-dimensional vector is compressed into a 1-dimensional scalar to achieve global weight distribution at the modality level; the gate coefficient It is dynamically generated by the joint features of the environment, rather than fixed weights, and globally adjusts the contribution of the audiovisual modalities to reduce the evaluation error of the model when there is audiovisual conflict, and finally outputs a joint representation of the environment. .

[0071] This part realizes the adaptive fusion of cross-modal features in the environmental context through spatial alignment strategy and dynamic weight gating. The cross-modal Transformer uses visual embedding as the query vector to retrieve semantically relevant components from the acoustic embedding (such as strengthening the negative impact of low-frequency noise when visual openness is high); dynamic gating generates modal weights through environmental features and automatically adjusts the dominant mode in audio-visual conflict scenes (such as landscape accompanied by traffic noise). The visual encoder (decoupled self-attention) and the auditory encoder (hierarchical temporal modeling) complement each other, and semantic alignment is achieved through the cross-modal fusion module, constructing a joint representation system for complex perceptual environments.

[0072] In some embodiments, reference Figure 2 The physiological and psychological collaborative encoder includes a psychological feature projection layer, a physiological temporal coding network and a collaborative attention fusion layer; in the above step S103, the physiological response data set and the psychological assessment data set are input into the physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation, which specifically includes: The psychological assessment data set is input into the psychological feature projection layer for processing to obtain psychological state embedding, and the psychological feature projection layer satisfies: ; ; ; in, The first MLP layer represents the psychological feature projection layer and is used to achieve primary feature decoupling. The second MLP layer represents the psychological feature projection layer and is used to achieve high-level semantic abstraction. represents the psychological state embedding, , and Represents the weight matrix of each layer of MLP in the psychological feature projection layer, , and Represents the bias vector of each layer of MLP in the psychological feature projection layer, represents the psychological indicator vector, represents the tanh activation function, express The real matrix space of , express The real matrix space of , express The real matrix space of ; Inputting the physiological response data set into a physiological temporal coding network for processing to obtain a physiological state embedding; The psychological state embedding and the physiological state embedding are fused by using a collaborative attention fusion layer to obtain a joint physiological and psychological representation.

[0073] In this embodiment, the physiological and psychological collaborative encoder realizes the joint modeling of human physiological response and subjective psychological state through temporal feature extraction and dynamic weight allocation mechanism.

[0074] In the psychological feature projection layer, the psychological indicator vector Includes visual satisfaction , Auditory Satisfaction , audio-visual coordination , after Z-score normalization, it is input into the three-layer MLP: ; ; ; First floor To achieve primary feature decoupling, the 3D psychological indicators are mapped to 16-dimensional space, and the coupling of subjective perception is initially separated; the second layer Achieve high-level semantic abstraction, further extract 32-dimensional features, and capture the potential interactions of psychological indicators. The third layer outputs the psychological state embedding , is the hyperbolic tangent activation function, constraining the output to interval, enhancing feature stability. The psychological feature projection layer maps discrete psychological indicators (3D) into a continuous embedding space. Through the hierarchical mapping of 3→16→32→64 dimensions, it gradually separates the coupling effect of the perceptual dimension, and nonlinear transformation decouples the implicit dimension of subjective perception to improve semantic discrimination. The ReLU activation function enhances sparsity, and the tanh output layer suppresses the feature amplitude to avoid modal imbalance in the subsequent fusion stage.

[0075] In some embodiments, reference Figure 2 , the physiological time series coding network includes a third input layer, an expanded time series convolution layer and a gated attention pooling layer; further, the physiological response data set is input into the physiological time series coding network for processing to obtain a physiological state embedding, specifically including: Inputting the physiological response data set into the third input layer and aligning it into a physiological time series tensor according to the sampling rate; A dilated temporal convolutional layer is used to extract multi-scale physiological temporal features from the physiological temporal tensor, and the dilated temporal convolutional layer satisfies: ; in, represents the first dilated temporal convolutional layer The output feature vector of the dilated convolution layer, represents the first dilated temporal convolutional layer The output feature vector of the dilated convolution layer, represents the first dilated temporal convolutional layer Layer dilated convolution at time step The output feature vector of Indicates The convolution kernel of the layer A weight matrix, represents the expansion coefficient, represents an integer index, Indicates The bias vector of the dilated convolution layer, represents the convolution kernel; The importance score of the multi-scale physiological time series features of each time step is calculated through the gated attention pooling layer to obtain the physiological state embedding, which satisfies: ; ; in, represents physiological state embedding, Represents the time step Embedding physiological state The contribution weight of represents a fixed time step, Indicates a value from 1 to The variable time step, The 4th dilated convolution layer of the dilated temporal convolution layer is represented by the time step The output feature vector of The 4th dilated convolution layer of the dilated temporal convolution layer is represented by the time step The output feature vector of represents the time step, represents the learnable weights, express The transpose of Represents a 64-dimensional real matrix space.

[0076] In this embodiment, the processing steps of the physiological temporal encoding network include input structuring, dilated temporal convolution (TCN), and gated attention pooling.

[0077] Input structuring: Multi-source physiological signals are aligned into time series tensors according to sampling rates ,in (eye movement 4D + EEG 2D + ECG 2D + skin conduction 2D), time step (corresponding to 2 minutes of data, 1Hz sampling).

[0078] Dilated Temporal Convolution (TCN): A 4-layer causal dilated convolution is used to extract multi-scale features, with a dilation coefficient of In order , convolution kernel size , number of channels 64: ; in, For the Layer dilated convolution at time step The output feature vector of For the The convolution kernel of the layer A weight matrix, is an integer index used for causal expansion constraints, which only depend on the current and historical time steps. The The bias vector of the layer dilation convolution provides a baseline offset for each output channel, enhancing the model's robustness to input distribution shifts. The output feature is finally achieved , retaining the original time series length.

[0079] Gated Attention Pooling: via Learnable Weights Calculate the importance score for each time step: ; ; For the 4th layer TCN at time step The output feature vector of is the time step Final physiological embedding The contribution weight of the physiological timing encoding network is . The physiological timing encoding network extracts temporal dynamic features from multimodal physiological signals (eye movement, EEG, ECG, and skin electricity). The causal dilation convolution captures multi-granular responses from transient events (such as pupil contraction, time scale 0.1-0.5 seconds) to long-period patterns (such as skin conductance baseline drift, time scale > 30 seconds); the gated attention pooling adaptively focuses on key physiological events (such as eye movement saccades corresponding to attention shifts). This part is based on hybrid timing modeling, integrating the local feature extraction capabilities of TCN and the global weight allocation of the attention mechanism to reduce the detection delay of sudden physiological events (such as a sudden increase in heart rate); the physiological signal alignment strategy designs a unified sampling rate (1Hz) and timing length (T=120), and reduces the sampling rate differences of multiple devices through an interpolation algorithm.

[0080] Furthermore, the fusing of the psychological state embedding and the physiological state embedding to obtain a physiological and psychological joint representation specifically includes: Dynamically weight the physiological state embedding to obtain physiological feature enhancement, where the physiological feature enhancement satisfies: ; ; in, represents the gating coefficient, represents the trainable parameter matrix, Indicates enhanced physiological characteristics, represents the element-wise multiplication operator, Represents the Sigmoid activation function; The psychological state embedding is modified to obtain a psychological characteristic modification, where the psychological characteristic modification satisfies: ; ; in, Represents the cross-modal attention weight matrix from physiological signals to psychological features, represents the cross-modal association matrix, represents the transpose of the mental state embedding, Indicates psychological characteristics modification; The physiological feature enhancement and the psychological feature correction are fused to obtain a physiological and psychological joint representation, which is: ,in, Represents a joint physiological and psychological representation, Representation layer normalization operation.

[0081] In this embodiment, the dynamic alignment of psychological cognitive state and physiological response signal is achieved through a bidirectional interactive attention mechanism, and the specific steps include: Psychological guidance enhances physiological characteristics (positive pathway): Construct a bidirectional gating unit to embed the psychology vector With physiological embedding vector After concatenation, the gating coefficient matrix is ​​generated through the fully connected layer, and the Sigmoid function is used to constrain the gating value in the interval [0,1] to form a feature selection mask; (element-by-element multiplication) realizes the dynamic weighting of physiological characteristics. The following formulas are the gating coefficients Generative and physiological enhancement : ; ; in, It is a trainable parameter matrix that realizes the projection of 128-dimensional concatenated vectors to 64-dimensional gated space and generates feature-level local masks, which can perform fine-grained enhancement on specific frequency bands or time points of physiological signals; is the element-wise multiplication operator; is the Sigmoid activation function. The gate coefficient of this part Its function is to locally enhance the sensitive areas of physiological characteristics and enhance the characteristics related to psychological state in physiological signals.

[0082] Physiological feedback psychological characteristics modification (reverse pathway): Construct a cross-modal attention matrix, calculate the soft alignment weights of physiological signals to psychological indicators, and recalibrate the original psychological embedding vector through attention weights. The attention weights are calculated as follows: Modification for psychological characteristics: ; ; in, is a cross-modal association matrix that learns the potential mapping relationship between physiological and psychological features; Softmax is a normalized exponential function that ensures that the sum of the weights of each row is 1; Represents the cross-modal attention weight matrix from physiological signals to psychological features, which is used to model the dynamic correction relationship between physiological features and psychological states, and finally realize the correction of output psychological features .

[0083] Joint Representation Generation: This part breaks through the traditional one-way fusion paradigm (physiological → psychological or psychological → physiological), fuses the output of the two-way pathway, retains the original features through residual connection, suppresses the dimensional difference between modalities through layer normalization, and generates a joint feature representation with cross-modal consistency. First, the output vectors of the two channels are residually connected, and the training process is stabilized by layer normalization (LayerNorm).

[0084] ; The physiological and psychological collaborative encoder achieves fine modeling of human multimodal responses through a two-way dynamic interaction mechanism and hierarchical feature decoupling. Through the three-level fusion architecture formed by the psychological projection layer (static semantics), physiological TCN (dynamic timing), and collaborative attention (cross-modal interaction), multi-granularity spatiotemporal alignment is achieved. For the non-stationary characteristics of physiological signals (such as baseline drift of ECG signals), layer normalization replaces BatchNorm to improve training stability in few-sample scenarios. Its innovation is reflected in the cross-modal closed-loop correction strategy and the timing-semantic joint representation framework.

[0085] In some embodiments, reference Figure 2The global fusion module includes a fine-grained feature matching unit, a semantic-level relationship modeling unit, a decision-level dynamic aggregation unit, an environmental impact assessment unit, and a human feedback compensation unit; in the above step S104, the environmental joint representation and the physiological and psychological joint representation are input into the global fusion module for deep fusion to obtain the final fusion feature, which specifically includes: The physiological and psychological joint representation is upgraded to obtain the upgraded physiological and psychological joint representation ,in, Represents a joint physiological and psychological representation, It represents the physiological and psychological joint representation after dimensionality increase. represents the dimension-raising projection matrix, represents the GELU activation function; The fine-grained feature matching unit is used to decompose the joint representation of the environment into visual subspace and auditory subspace, and then combined with the physiological and psychological joint representation after dimensionality increase. Perform fine-grained feature matching; A bidirectional cross-attention mechanism using semantic-level relation modeling units is used to establish semantic connections between environmental joint representation and physiological-psychological joint representation. The gated residual network of the decision-level dynamic aggregation unit is used to adjust the contribution ratio of the environmental joint representation and the physiological and psychological joint representation, and the environmental joint representation and the physiological and psychological joint representation are fused according to the contribution ratio. The gated residual network satisfies: ; ; in, Indicates the contribution ratio of real-time adjustment of environment and human characteristics, represents the dynamic gating weight matrix, represents the joint representation of the environment, Indicates The fusion feature of the iteration represents the The fusion features of the iteration, represents a multi-layer perceptron, Represents the Sigmoid activation function; The dual-channel neural network of the environmental impact assessment unit is used to analyze the decision-making contribution of the environmental joint representation and the physiological and psychological joint representation to obtain the environmental dominance coefficient. The dual-channel neural network satisfies: ; in, represents the environmental impact assessment network in the dual-channel neural network, represents the human response evaluation network in the dual-channel neural network, represents the environmental dominance coefficient; Adopt human body feedback compensation unit to environmental dominant coefficient Perform correction processing to obtain a human body correction coefficient, which satisfies ,in, represents the human body correction factor, represents the adaptive gain parameter, represents the transposition of the physiological and psychological joint representation after dimensionality increase, ReLU activation function. Dynamic weighted fusion is performed according to the environment dominant coefficient and the human body correction coefficient to obtain a final fusion feature, and the final fusion feature satisfies: ; in, represents the final fusion feature, represents the fusion features finally output by the decision-level dynamic aggregation unit, Representation layer normalization operation.

[0086] In this embodiment, this module realizes the deep integration of audio-visual environment characteristics and human physiological and psychological responses through multi-level feature interaction and dynamic weight control mechanism, and finally outputs the comprehensive evaluation results of campus environment comfort. The specific technical implementation includes feature space alignment and calibration, hierarchical feature interaction mechanism and adaptive weight allocation.

[0087] In order to solve the problem of cross-modal feature scale differences, a bidirectional projection and distribution constraint strategy is adopted. First, the physiological and psychological characteristics are upgraded to expand the 64-dimensional collaborative representation to 256 dimensions through nonlinear mapping to match the audio-visual environment feature dimension. The GELU activation function is used to enhance the expression ability: ; in For high-dimensional collaborative representation, the 64-dimensional joint features Mapping to 256-dimensional space to enhance cross-modal interaction capabilities; To increase the dimension of the projection matrix, we learn the nonlinear mapping rules from physiological and psychological characteristics to high-dimensional space.

[0088] Next, we perform dimensionality reduction screening of environmental features: we retain the key components of audiovisual coding through an interpretable gating mechanism and generate a 128-dimensional streamlined representation. The gating mask is dynamically generated by sorting the feature importance, retaining the top 50% of the most contributing features.

[0089] The hierarchical feature interaction mechanism realizes multi-granularity information fusion by constructing a three-level attention network, which includes fine-grained feature matching, semantic-level relationship modeling, and decision-level dynamic aggregation.

[0090] Fine-grained feature matching: Decompose the audiovisual environment features into visual and auditory subspaces, and perform dot product attention calculations with physiological and psychological features. This layer focuses on capturing cross-modal local correlations, such as the correlation pattern between green vision rate and heart rate changes.

[0091] Semantic-level relationship modeling: Through the bidirectional cross-attention mechanism, the semantic connection between the physical parameters of the environment and the subjective perception of the human body is established. A multi-head design (4 heads) is used to parallelly calculate multiple groups of interactive relationships to improve the ability to analyze complex scenes.

[0092] Decision-level dynamic aggregation: Feature iterative optimization is achieved by designing a gated residual network. The formula is as follows: ; ; in, Represents a dynamic gating weight matrix, which is used to learn the joint mapping relationship between environmental features and physiological and psychological collaborative representations and generate gating coefficients , Adjust the contribution ratio of environment and human characteristics in real time, and finally achieve the first The fusion features of the iteration , gradually optimize cross-modal representation through residual connections; is the fusion feature of the previous iteration: retain the historical fusion state to achieve the continuity of feature evolution, and the number of iterations .

[0093] The adaptive weight allocation mechanism realizes intelligent adjustment of fusion weights by dynamically evaluating the interaction between environmental characteristics and human responses. Specifically, it includes the following contents: A dual-channel neural network is designed to analyze the decision-making contribution of audio-visual environment characteristics and physiological and psychological responses respectively. Among them, the environmental assessment network focuses on the physical significance of acoustic indicators (sound pressure level, spectral complexity) and visual parameters (green view rate, spatial openness). The decision-making influence of environmental characteristics is analyzed through a dual-channel neural network. When the sound pressure level is >65dB or the green view rate is <25%, the system automatically increases the weight coefficient of the environmental characteristics to reflect the physical dominant effect under extreme environmental conditions. is the environmental dominant coefficient, which is dynamically calculated by a dual-channel neural network: ; in, represents the environmental impact assessment network, quantifying the decision-making contribution of the audio-visual environment to comfort; Represents the human response evaluation network, quantifying the ability of physiological and psychological characteristics to modify comfort.

[0094] The feature correlation compensation term is introduced to make corrections based on the fluctuation amplitude of physiological indicators to enhance the sensitivity of the model to human stress state. When the coefficient of variation of physiological indicators is >0.3 (reflecting a significant stress state), the feedback compensation pathway is activated. Correction factor for the human body: ; in The initial value is obtained by learning historical data for the adaptive gain parameter. This design ensures that when the user is in a state of high physiological stress, the human body response characteristics are compensated with additional weights, avoiding the evaluation bias caused by relying solely on environmental parameters.

[0095] The final fusion feature is calculated by the following formula to achieve dynamic weighted fusion of environmental features and human responses: ; Based on the weighted sum of the environmental interaction features and the human response features, the fused 256-dimensional features are normalized. This operation stabilizes the training process and makes the model insensitive to changes in feature scale.

[0096] In some embodiments, reference Figure 2 , the multi-task prediction channel includes a main prediction channel and an auxiliary channel; in the above step S105, the final fusion feature is input to the multi-task prediction channel for processing, specifically including: The final fusion feature is mapped to the evaluation space using a three-layer fully connected network of the main prediction channel, and a comprehensive comfort rating result is output, which satisfies: ; ; ; in, represents the output of the first layer of the fully connected network of the main prediction channel, represents the output of the second layer of the fully connected network of the main prediction channel, It represents the comprehensive score of comfort. , and Represents the weight matrix of each layer of the fully connected network, , and Represent the bias vector of each layer of the fully connected network, represents the GELU activation function, Represents the Dropout regularization function, which represents Represents the Sigmoid function; The auxiliary channel is used to analyze the interpretability decision basis of the final fusion feature and output a multidimensional explanation factor, which satisfies: ; ; ; in, Represents visual contribution analysis, represents the acoustic comfort analysis, represents physiological fitness analysis, , and Respectively represent the weight matrices of visual contribution analysis, acoustic comfort analysis, and physiological fitness analysis, represents the ReLU activation function, represents the tanh activation function, express The real matrix space of .

[0097] In this embodiment, the main prediction channel maps the fusion features to the evaluation space through a three-layer fully connected network and outputs the comprehensive comfort score result. , the three-layer structure is as follows: The first layer (256→128 dimensions): high-order feature extraction layer, responsible for extracting high-order interactive features, using GELU activation function to enhance nonlinear expression capabilities, and 128-dimensional hidden layer features to retain key decision information: ; The second layer (128→64 dimensions): robustness enhancement layer, which performs feature dimensionality reduction and introduces Dropout (p=0.3) to suppress overfitting and improve the generalization ability of the model: ; Output layer (64 → 1 dimension): Bisect the mapping layer and use the Sigmoid function ( ) constrains the result to the interval [0,1] and linearly expands it to a 0-10 scale.

[0098] ; In the above formula, , , Represents the weight matrix of each layer, , and They correspond to the bias vectors of the three fully connected layers respectively. This part first integrates the global features It is mapped to a 128-dimensional latent space, further reduced to a 64-dimensional robust feature space, and finally the third layer outputs the predicted value of the comfort score.

[0099] Then, a parallel branch network is constructed to output multidimensional explanatory factors to quantify the contribution of each dimension, and the auxiliary channel realizes indicator decoupling through feature decomposition technology.

[0100] Visual contribution analysis (0-5 points): Focus on analyzing the weighted impact of indicators such as green view rate and spatial layout. , the calculation method is as follows, the special network focuses on spatial frequency characteristics (0.5-4Hz) and color histogram distribution, the weight matrix After L1 regularization: ; Acoustic comfort analysis (0-3 points): quantify the negative effects of parameters such as background noise and sound field uniformity, , the calculation method is as follows, integrating global features Acoustic characteristics : ; Physiological fitness analysis (0-2 points): reflects the degree of matching between human biological rhythms and environmental parameters. , the calculation method is as follows, based on the physiological and psychological characteristics after dimensionality upgrade , constrain the output range by the hyperbolic tangent function: ; Reference Figure 3 An embodiment of the present invention provides a campus green space embodied perception evaluation system 3 based on multimodal learning, which specifically includes: A data acquisition module 301 is used to obtain an environmental perception data set, a physiological response data set and a psychological assessment data set, wherein the environmental perception data set includes a visual indicator quantization matrix and an acoustic time series tensor; The audio-visual processing module 302 is used to input the environment perception data set into the visual feature coding network and the auditory feature coding network for processing, and input the output results of the visual feature coding network and the auditory feature coding network into the audio-visual modality feature fusion module for fusion processing to obtain a joint representation of the environment; A physiological and psychological processing module 303 is used to input the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation; A global fusion module 304 is used to input the environmental joint representation and the physiological and psychological joint representation into the global fusion module for deep fusion to obtain a final fusion feature; The evaluation output module 305 is used to input the final fusion features into the multi-task prediction channel for processing, and combine the loss function training optimization to obtain the comprehensive scoring result of environmental comfort and the explanatory decision basis.

[0101] It is understandable that if Figure 1 The contents of the embodiment of the campus green space embodied perception evaluation method based on multimodal learning shown in the embodiment are all applicable to the embodiment of the campus green space embodied perception evaluation system based on multimodal learning. The functions specifically implemented by the embodiment of the campus green space embodied perception evaluation system based on multimodal learning are similar to those in the embodiment of the campus green space embodied perception evaluation system based on multimodal learning. Figure 1 The embodiment of the campus green space embodied perception evaluation method based on multimodal learning shown in FIG. 1 is the same as that of FIG. 1 , and the beneficial effects achieved are the same as those of FIG. Figure 1 The beneficial effects achieved by the embodiment of the campus green space embodied perception assessment method based on multimodal learning are also the same.

[0102] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0103] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0104] Reference Figure 4 The embodiment of the present invention further provides a computer device 4, comprising: a memory 402 and a processor 401 and a computer program 403 stored in the memory 402. When the computer program 403 is executed on the processor 401, the campus green space embodied perception evaluation method based on multimodal learning as described in any one of the above methods is implemented.

[0105] The computer device 4 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 4 may include, but is not limited to, a processor 401 and a memory 402. Those skilled in the art will appreciate that Figure 4It is only an example of the computer device 4 and does not constitute a limitation on the computer device 4. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0106] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0107] In some embodiments, the memory 402 may be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 402 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 4. Further, the memory 402 may also include both an internal storage unit and an external storage device of the computer device 4. The memory 402 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 402 may also be used to temporarily store data that has been output or is to be output.

[0108] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for evaluating embodied perception of campus green space based on multimodal learning as described in any one of the above methods is implemented.

[0109] In this embodiment, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0110] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0111] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0112] In the embodiments disclosed in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A campus green space embodied perception evaluation method based on multimodal learning, characterized by: The method specifically comprises: Acquire an environmental perception dataset, a physiological response dataset, and a psychological assessment dataset, wherein the environmental perception dataset includes a visual indicator quantization matrix and an acoustic time series tensor; Inputting the visual index quantization matrix into a visual feature coding network for processing, inputting the acoustic time series tensor into an auditory feature coding network for processing, and inputting the output results of the visual feature coding network and the auditory feature coding network into an audiovisual modality feature fusion module for fusion processing to obtain a joint representation of the environment; Inputting the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation; Inputting the environmental joint representation and the physiological and psychological joint representation into a global fusion module for deep fusion to obtain a final fusion feature; The final fusion features are input into the multi-task prediction channel for processing, and combined with loss function training optimization to obtain the comprehensive scoring result of environmental comfort and explanatory decision-making basis.

2. The method according to claim 1, characterized in that: The visual feature encoding network includes a first input layer, a first fully connected layer, a self-attention layer and a first output layer; the step of inputting the visual indicator quantization matrix into the visual feature encoding network for processing specifically includes: Inputting the visual index quantization matrix into the first input layer, performing maximum and minimum normalization processing on the visual index quantization matrix, and obtaining a normalized visual index quantization matrix; Using a first fully connected layer to map the standardized visual index quantization matrix to a high-dimensional space to obtain original projection features; A self-attention layer is used to capture the interaction between each indicator of the standardized visual indicator quantization matrix to form an attention feature; The original projection features and the attention features are input into the first output layer for residual connection, and the visual embedding is output.

3. The method according to claim 2, characterized in that The auditory feature encoding network includes a second input layer, a local feature extraction layer, an LSTM timing layer, and a second output layer; the step of inputting the acoustic timing tensor into the auditory feature encoding network for processing specifically includes: Inputting the sound pressure level time series and MFCC spectrum of the acoustic time series tensor into the second input layer, segmenting and splicing into joint acoustic features through a sliding window; Extracting multi-scale acoustic features from the joint acoustic features using multi-layer dilated convolution of a local feature extraction layer; The temporal dependency of the multi-scale acoustic features is captured by the LSTM timing layer to form an acoustic embedding; The acoustic embedding is output through the second output layer.

4. The method according to claim 3, characterized in that The audiovisual modality feature fusion module includes a cross-modal Transformer encoder and a dynamic weight gating unit; the output results of the visual feature encoding network and the auditory feature encoding network are input into the audiovisual modality feature fusion module for fusion processing to obtain a joint representation of the environment, specifically including: Using a cross-modal Transformer encoder to perform cross-modal attention calculation on the visual embedding and the acoustic embedding; A dynamic weight gating unit is used to adjust the modal contribution of the visual embedding and the acoustic embedding to obtain a joint representation of the environment.

5. The method according to claim 1, characterized in that The physiological and psychological collaborative encoder includes a psychological feature projection layer, a physiological temporal coding network and a collaborative attention fusion layer; The step of inputting the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation specifically includes: Inputting the psychological assessment data set into the psychological feature projection layer for processing to obtain a psychological state embedding; Inputting the physiological response data set into a physiological temporal coding network for processing to obtain a physiological state embedding; The psychological state embedding and the physiological state embedding are fused by using a collaborative attention fusion layer to obtain a joint physiological and psychological representation.

6. The method according to claim 5, characterized in that The physiological temporal coding network includes a third input layer, an expanded temporal convolution layer, and a gated attention pooling layer; The step of inputting the physiological response data set into a physiological temporal coding network for processing to obtain a physiological state embedding specifically includes: Inputting the physiological response data set into the third input layer and aligning it into a physiological time series tensor according to the sampling rate; Extracting multi-scale physiological timing features from the physiological timing tensor using an expanded timing convolutional layer; The importance scores of multi-scale physiological timing features at each time step are calculated through a gated attention pooling layer to obtain the physiological state embedding.

7. The method according to claim 6, characterized in that The fusing of the psychological state embedding and the physiological state embedding to obtain a physiological and psychological joint representation specifically includes: Dynamically weighting the physiological state embedding to obtain physiological feature enhancement; Modifying the psychological state embedding to obtain a psychological feature modification; The physiological feature enhancement and the psychological feature correction are fused to obtain a physiological and psychological joint representation.

8. The method according to claim 1, characterized in that The global fusion module includes a decision-level dynamic aggregation unit, an environmental impact assessment unit, and a human feedback compensation unit; the inputting of the environmental joint representation and the physiological and psychological joint representation into the global fusion module for deep fusion to obtain the final fusion feature specifically includes: The physiological and psychological joint representation is dimensionally upgraded, and fine-grained feature matching and semantic connection establishment are performed with the environmental joint representation; The gated residual network of the decision-level dynamic aggregation unit is used to adjust the contribution ratio of the environmental joint representation and the physiological and psychological joint representation, and the environmental joint representation and the physiological and psychological joint representation are fused according to the contribution ratio. The gated residual network satisfies: ； ； in, Indicates the contribution ratio of real-time adjustment of environment and human characteristics, represents the dynamic gating weight matrix, represents the joint representation of the environment, It represents the joint physiological and psychological representation after dimensionality increase. represents the fusion feature of the tth iteration, Indicates The fusion features of the iteration, represents a multi-layer perceptron, Represents the Sigmoid activation function; The dual-channel neural network of the environmental impact assessment unit is used to analyze the decision-making contribution of the environmental joint representation and the physiological and psychological joint representation to obtain the environmental dominance coefficient; The human body feedback compensation unit is used to correct the environment dominant coefficient to obtain the human body correction coefficient; Dynamic weighted fusion is performed according to the environment dominant coefficient and the human body correction coefficient to obtain a final fusion feature.

9. The method according to claim 8, characterized in that The multi-task prediction channel includes a main prediction channel and an auxiliary channel; the inputting the final fusion feature into the multi-task prediction channel for processing specifically includes: A three-layer fully connected network of the main prediction channel is used to map the final fusion features to the evaluation space, and output a comprehensive comfort score result; An auxiliary channel is used to perform interpretable decision basis analysis on the final fusion features and output a multi-dimensional explanatory factor.

10. Campus green space embodied perception evaluation system based on multimodal learning, characterized by: The system specifically comprises: A data acquisition module, used to acquire an environmental perception data set, a physiological response data set, and a psychological assessment data set, wherein the environmental perception data set includes a visual indicator quantization matrix and an acoustic time series tensor; An audio-visual processing module, used to input the environment perception data set into a visual feature coding network and an auditory feature coding network for processing, and input the output results of the visual feature coding network and the auditory feature coding network into an audio-visual modality feature fusion module for fusion processing to obtain a joint representation of the environment; A physiological and psychological processing module, used for inputting the physiological response data set and the psychological assessment data set into a physiological and psychological collaborative encoder for processing to obtain a physiological and psychological joint representation; A global fusion module, used for inputting the environmental joint representation and the physiological and psychological joint representation into the global fusion module for deep fusion to obtain a final fusion feature; The evaluation output module is used to input the final fusion features into the multi-task prediction channel for processing, and combine the loss function training optimization to obtain the comprehensive scoring result of environmental comfort and the explanatory decision basis.

Images