A stage environment self-adapting adjustment method and device

Abstract

The application relates to the technical field of automatic control of stage environments, and in particular to a stage environment self-adaptive adjustment method and device, which comprises the following steps: collecting multi-modal environment data in real time; converging the multi-modal environment data to an edge computing node adjacent to a stage; extracting independent space-time features of each mode of data in parallel through a lightweight deep learning model in the edge computing node, and performing correlation processing to generate multi-modal environment integrated features; inputting the multi-modal environment integrated features into a multi-modal deep learning decision model with an embedded cross-attention module; outputting a set of collaborative control instructions for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning by the multi-modal deep learning decision model; and performing linkage adjustment on the multiple heterogeneous stage execution devices under the control of the edge computing node according to the set of collaborative control instructions. The application helps to realize efficient fusion and real-time processing of multi-modal environment data and collaborative self-adaptive adjustment of multiple heterogeneous execution devices.

Description

Technical Field

[0001] This application relates to the field of automatic stage environment control technology, and in particular to a stage environment adaptive adjustment method and device. Background Technology

[0002] With the continuous development of modern stage performances, stage environment control has evolved from independent adjustment of lighting and sound to comprehensive control involving multiple devices and elements, placing higher demands on the real-time performance, accuracy, and adaptability of stage environment control.

[0003] Currently, stage environment adjustment mainly relies on control methods based on preset scripts or closed-loop control methods based on single sensor feedback. Preset script-based methods cannot dynamically adjust to real-time changes in the stage environment and lack adaptive adjustment capabilities; single sensor feedback methods can only collect single environmental parameters, making it difficult to comprehensively reflect multi-dimensional stage environment information, and the deep correlations between different modal data are not fully explored and utilized. Furthermore, existing solutions mostly adopt a centralized cloud processing architecture, requiring all sensor data to be uploaded to cloud servers for processing and decision-making. This results in problems such as large data transmission latency, high network bandwidth consumption, and system failure under network outage conditions, making it difficult to meet the stringent requirements of millisecond-level real-time response in stage performances.

[0004] Therefore, there is an urgent need for a technical solution that can achieve efficient fusion and real-time processing of multimodal environmental data, and accordingly perform collaborative adaptive adjustment of multiple heterogeneous execution devices, in order to solve the technical problems existing in the above-mentioned technologies. Summary of the Invention

[0005] Therefore, it is necessary to provide a stage environment adaptive adjustment method and device that can achieve efficient fusion and real-time processing of multimodal environmental data and collaborative adaptive adjustment of multiple heterogeneous execution devices, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a stage environment adaptive adjustment method, the method comprising: Multimodal environmental data is collected in real time by a high-density sensor array deployed on the stage. The multimodal environmental data includes at least temperature field, humidity field, illuminance, smoke concentration, airflow speed, sound field, and actor physiological state data. The multimodal environment data is aggregated to an edge computing node near the stage. Within the edge computing node, a lightweight deep learning model is used to extract the independent spatiotemporal features of each modality in parallel. The independent spatiotemporal features are then correlated to generate integrated multimodal environment features. The feature components corresponding to each modality in the integrated multimodal environment features are kept independent and distinguishable according to the feature dimension index. The multimodal environment integration features are input into a multimodal deep learning decision model with an embedded cross-attention module. The multimodal deep learning decision model outputs a set of collaborative control instructions for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning. According to the collaborative control instruction set, the multiple heterogeneous stage execution devices are coordinated and adjusted under the control of the edge computing node.

[0007] In one embodiment, the multimodal deep learning decision model, which integrates the multimodal environment features into a cross-attention module, outputs a set of collaborative control instructions for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning. The integrated features of the multimodal environment are input into a multimodal deep learning decision model. The multimodal deep learning decision model adopts a feature-level fusion architecture, which includes multiple independent modal encoders, a cross-modal interaction layer, and a multi-task decoder. The multiple independent modal encoders respectively receive the corresponding modal components in the integrated features of the multimodal environment, which are divided according to the preset feature dimension index, and extract the high-level semantic features of each modality. The cross-modal interaction layer embeds a lightweight multi-head cross-attention module, which takes the high-level semantic features of each modality as input, calculates the correlation weight between any two modal features through the cross-attention mechanism, realizes cross-modal feature interaction and information fusion in a unified semantic space, captures the implicit coupling relationship between different modalities, and completes cross-modal causal reasoning. The multi-task decoder receives the fused representation output by the cross-modal interaction layer and outputs control parameters corresponding to multiple heterogeneous execution devices in parallel to form a collaborative control instruction set.

[0008] In one embodiment, the step of using the high-level semantic features of each modality as input, calculating the correlation weight between any two modal features through a cross-attention mechanism, realizing cross-modal feature interaction and information fusion in a unified semantic space, capturing the implicit coupling relationship between different modalities, and completing cross-modal causal reasoning includes: The high-level semantic features of each modality are mapped to a unified embedding space through linear projection to obtain the query vector, key vector, and value vector corresponding to each modality. For any two modalities, a cross-attention mechanism is used for processing. The dot product of the query vector of the first modality and the key vector of the second modality is used as the original relevance measure. After scaling and normalization, the attention weight matrix between the two modalities is obtained. The value vector of the second modality is weighted and aggregated using the attention weight matrix to generate cross-modal interaction information obtained by the first modality from the second modality; By fusing cross-modal interaction information from various modalities, a fused representation containing the coupling relationship between any two modalities is formed, thus completing cross-modal causal reasoning.

[0009] In one embodiment, the step of weighting and aggregating the value vector of the second modality using the attention weight matrix to generate cross-modal interaction information obtained by the first modality from the second modality includes: The attention weight matrix is ​​multiplied with the value vector of the second modality to obtain the weighted aggregated second modality information; The second modal information is subjected to a nonlinear transformation to obtain the transformed cross-modal information; The transformed cross-modal information is residually concatenated with the high-level semantic features of the first modality to generate cross-modal interaction information obtained by the first modality from the second modality. The cross-modal interaction information encodes the contextual influence of the second modality on the first modality, so that complementary information from the second modality is incorporated into the feature representation of the first modality.

[0010] In one embodiment, the process of aggregating the multimodal environment data to an edge computing node adjacent to the stage, extracting independent spatiotemporal features of each modality data in parallel using a lightweight deep learning model within the edge computing node, and performing correlation processing on the independent spatiotemporal features to generate integrated multimodal environment features includes: The multimodal environment data is aggregated to an edge computing node near the stage. Within the edge computing node, a lightweight deep learning model that has undergone model pruning and quantization is configured for each modality of the multimodal environment data. The lightweight deep learning models run in parallel within the edge computing node and independently extract the temporal dependency features and / or spatial distribution features of the corresponding modality data to form independent spatiotemporal features of each modality. Based on the attention mechanism, the attention weight matrix between independent spatiotemporal features of different modalities is calculated to quantify the strength of the inter-modal correlation in the temporal and spatial dimensions. Based on the attention weight matrix, the independent spatiotemporal features of each modality are weighted and integrated to obtain multimodal environment integrated features. The multimodal environment integrated features retain the feature components corresponding to each modality and encode the spatiotemporal correlation between different modalities.

[0011] In one embodiment, the step of weighting and integrating the independent spatiotemporal features of each modality according to the attention weight matrix to obtain multimodal environment integrated features includes: The attention weight matrix is ​​decomposed into a time-dimensional attention weight submatrix and a spatial-dimensional attention weight submatrix; The independent spatiotemporal features of each modality are weighted and integrated along the time dimension using the time-dimensional attention weight submatrix to obtain the time-dimensional integrated features. The independent spatiotemporal features of each modality are weighted and integrated along the spatial dimension using the spatial-dimensional attention weight submatrix to obtain the spatial-dimensional integrated features. Through a gating integration mechanism, the integration weights of the time-dimensional integration features and the spatial-dimensional integration features are adaptively calculated based on the dynamic confidence levels of the time-dimensional integration features and the spatial-dimensional integration features under the current feature distribution. Based on the integration weights, multimodal environment integration features are generated.

[0012] In one embodiment, the step of adaptively calculating the integration weights of the temporal and spatial integrated features based on the dynamic confidence levels of the temporal and spatial integrated features under the current feature distribution through a gated integration mechanism, and generating multimodal environment integrated features based on the integration weights, includes: By using a gating integration mechanism, the confidence levels of the time-dimensional integration features and the spatial-dimensional integration features are evaluated based on their respective statistical characteristics. Based on the confidence level, the integration weights of the time-dimensional integrated features and the spatial-dimensional integrated features are adaptively determined, such that the allocation of the integration weights is adjusted as the feature distributions of the time-dimensional integrated features and the spatial-dimensional integrated features change. Based on the integration weights, the time-dimensional integration features and the spatial-dimensional integration features are weighted and integrated to generate multimodal environment integration features.

[0013] In one embodiment, the step of coordinating and adjusting the plurality of heterogeneous stage execution devices under the control of the edge computing node according to the collaborative control instruction set includes: The edge computing node distributes the control parameter vectors in the collaborative control instruction set to the corresponding heterogeneous stage execution equipment in parallel through a standardized industrial control protocol. The heterogeneous stage execution equipment includes at least lighting equipment, ventilation and smoke extraction equipment, and stage machinery equipment. While distributing the collaborative control instruction set, the edge computing node continuously receives real-time multimodal environmental data from the sensor array as a feedback signal for linkage adjustment. Based on the feedback signal, the edge computing node dynamically corrects the collaborative control instruction set.

[0014] Secondly, this application also provides a stage environment adaptive adjustment device, the device comprising: The environmental data acquisition module is used to collect multimodal environmental data in real time through a high-density sensor array deployed on the stage. The multimodal environmental data includes at least temperature field, humidity field, illuminance, smoke concentration, airflow speed, sound field, and actor physiological state data. The modal data integration module is used to aggregate the multimodal environment data to an edge computing node adjacent to the stage. Within the edge computing node, a lightweight deep learning model is used to extract the independent spatiotemporal features of each modal data in parallel, and the independent spatiotemporal features are correlated to generate multimodal environment integrated features. The control command output module is used to input the integrated features of the multimodal environment into a multimodal deep learning decision model with an embedded cross-attention module. The multimodal deep learning decision model outputs a set of collaborative control commands for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning. The equipment linkage adjustment module is used to perform linkage adjustment of the multiple heterogeneous stage execution devices under the control of the edge computing node according to the collaborative control instruction set.

[0015] In summary, this application includes the following beneficial technical effects: By deploying a high-density sensor array on the stage to collect multimodal environmental data in real time, a comprehensive perception of multidimensional information about the stage environment is achieved, overcoming the shortcomings of traditional solutions that rely on a single perception dimension and fail to reflect the full picture of the complex stage environment. By aggregating multimodal environmental data to edge computing nodes near the stage, lightweight deep learning models are used within these nodes to extract independent spatiotemporal features of each modality in parallel. These independent spatiotemporal features are then correlated to generate integrated multimodal environmental features. This approach significantly reduces data transmission latency by leveraging the proximity of edge computing to the data source. Furthermore, the parallel extraction and correlation processing fully uncovers the deep relationships between different modalities, avoiding information loss caused by traditional single-modal processing or coarse fusion, and significantly improving the accuracy and richness of environmental representation. The multimodal deep learning decision model outputs a collaborative control instruction set for multiple heterogeneous execution devices through cross-modal feature interaction and inference. This enables multiple heterogeneous devices, such as lighting, ventilation and smoke extraction, and stage machinery, to obtain consistent control parameters based on unified environmental perception results. This solves the problem in existing technologies where each device operates independently and lacks unified scheduling, making true linkage difficult. By adjusting multiple heterogeneous stage execution devices in a coordinated manner under the control of edge computing nodes according to the collaborative control instruction set, the entire adjustment process can be completed at the edge, from perception and decision-making to execution. This eliminates the need to rely on cloud servers, avoids the risk of failure under network outage conditions, and ensures the synchronization of multi-device responses. Thus, it fully realizes the efficient fusion and real-time processing of multimodal environmental data and the collaborative adaptive adjustment of multiple heterogeneous execution devices. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an adaptive adjustment method for the stage environment in one embodiment; Figure 2 This is a flowchart illustrating the adaptive adjustment method for the stage environment in another embodiment; Figure 3 This is a structural block diagram of a stage environment adaptive adjustment device in one embodiment. Detailed Implementation

[0017] This invention provides a method and apparatus for adaptive adjustment of the stage environment.

[0018] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0019] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the stage environment adaptive adjustment method in this invention includes: The S100 collects multimodal environmental data in real time through a high-density sensor array deployed on the stage.

[0021] Specifically, a high-density sensor array is deployed at multiple key locations on the stage. This array includes temperature sensors, humidity sensors, light sensors, smoke concentration sensors, airflow velocity sensors, sound acquisition sensors, and actors' physiological state monitoring sensors. Each sensor continuously collects corresponding environmental parameters at a preset sampling frequency, forming multimodal environmental data. This multimodal environmental data includes at least temperature field, humidity field, illuminance, smoke concentration, airflow velocity, sound field, and actors' physiological state data. Specifically, temperature field data reflects the temperature distribution in different areas of the stage; humidity field data reflects the humidity distribution in the stage space; illuminance data reflects the lighting intensity in different areas of the stage; smoke concentration data reflects the distribution density of smoke particles in the stage space; airflow velocity data reflects the stage ventilation and airflow; sound field data reflects the sound pressure level and audio distribution at different locations on the stage; and actors' physiological state data reflects the physical condition information of the performers. This multimodal environmental data is transmitted in real time to the data processing node via wired or wireless means, achieving comprehensive perception of multi-dimensional information about the stage environment.

[0022] S200 aggregates multimodal environment data to edge computing nodes near the stage. Within the edge computing nodes, a lightweight deep learning model is used to extract independent spatiotemporal features of each modality in parallel, and the independent spatiotemporal features are correlated to generate integrated multimodal environment features.

[0023] Specifically, the collected multimodal environmental data is aggregated to an edge computing node deployed near the stage. This edge computing node possesses certain computing and storage capabilities and can independently complete data processing tasks. Within the edge computing node, a corresponding lightweight deep learning model is configured for each modality of data, including temperature, humidity, illuminance, smoke concentration, airflow velocity, sound field, and actor physiological state data. These lightweight deep learning models undergo model pruning and quantization to run efficiently under the limited computing power of the edge computing node, and the models execute in parallel without interference. Each lightweight deep learning model extracts features from the corresponding input modal data, specifically extracting temporal dependence features (such as temperature change trends, smoke concentration accumulation or dissipation patterns over time) and spatial distribution features (such as spatial differences in temperature fields across different stage areas, uneven distribution of illuminance in different spatial locations), thus forming independent spatiotemporal characteristics for each modality. Based on this, cross-modal correlation processing is performed on the independent spatiotemporal features of each modality. Specifically, an attention weight matrix is ​​calculated between features of different modalities through an attention mechanism to quantify the mutual correlation strength of each modality in the temporal and spatial dimensions. Then, the independent spatiotemporal features of each modality are weighted and integrated according to the attention weight matrix to generate multimodal environment integrated features. This integrated feature not only retains the feature components corresponding to each modality to ensure the independent distinguishability of each modality information, but also encodes the spatiotemporal correlation between different modalities, so that the integrated features can be fully utilized in the subsequent decision-making process.

[0024] The S300 integrates multimodal environment features into a multimodal deep learning decision model with an embedded cross-attention module. Through cross-modal feature interaction and reasoning, the multimodal deep learning decision model outputs a set of collaborative control instructions for multiple heterogeneous execution devices.

[0025] Specifically, the integrated features of the multimodal environment are input into a pre-trained multimodal deep learning decision model. This model incorporates a cross-attention module, enabling deep interaction processing of the multimodal features. The integrated features are first decoupled into feature components corresponding to each modality. The decision model then performs high-level semantic extraction on each modal component using a multimodal encoder. The cross-attention module then enables cross-modal interaction between any two modal features, calculating the correlation weights between modalities to capture the implicit coupling relationships between different modalities and complete cross-modal causal inference. Based on the inference results, the multi-task decoder outputs control parameters corresponding to multiple heterogeneous stage execution devices in parallel, forming a collaborative control instruction set. This set contains the specific control parameters required for each device, such as color temperature, brightness, and illumination angle adjustment parameters for lighting equipment; wind speed, air volume, and wind direction adjustment parameters for ventilation and smoke extraction equipment; and lifting, moving, and rotating motion parameters for stage machinery.

[0026] The S400, based on the collaborative control instruction set, performs coordinated adjustment of multiple heterogeneous stage execution devices under the control of edge computing nodes.

[0027] Specifically, edge nodes distribute the control parameter vectors from the collaborative control command set to corresponding heterogeneous stage execution devices in parallel via standardized industrial control protocols. These heterogeneous stage execution devices include at least lighting equipment, ventilation and smoke extraction equipment, and stage machinery. Upon receiving the corresponding control parameters, each device synchronously executes its adjustment action, achieving coordinated adjustment. For example, when an actor's position changes, the edge computing node can, based on real-time perceived actor position information and stage environment data, synchronously adjust the illumination direction and brightness of the lighting equipment, the airflow direction and speed of the ventilation and smoke extraction equipment, and the platform position of the stage machinery, enabling all devices to work together to adapt to the current performance requirements.

[0028] In one embodiment, such as Figure 2 As shown, S300 includes: S310 integrates multimodal environment features into a multimodal deep learning decision model; S320: Multiple independent modal encoders receive the components of the corresponding modalities from the integrated features of the multimodal environment according to the preset feature dimension index, and extract the high-level semantic features of each modality. S330, a lightweight multi-head cross-attention module is embedded in the cross-modal interaction layer. It takes the high-level semantic features of each modality as input, calculates the correlation weight between any two modal features through the cross-attention mechanism, realizes cross-modal feature interaction and information fusion in a unified semantic space, captures the implicit coupling relationship between different modalities, and completes cross-modal causal reasoning. S340, the multi-task decoder receives the fused representation output from the cross-modal interaction layer, and outputs in parallel the control parameters corresponding to multiple heterogeneous execution devices to form a cooperative control instruction set.

[0029] Specifically, multimodal environment integration features are input into a multimodal deep learning decision model. During the generation of these integrated features, the feature components corresponding to each modality remain independently distinguishable in the feature vector according to a preset feature dimension index. That is, the temperature field feature components, humidity field feature components, illuminance feature components, smoke concentration feature components, airflow velocity feature components, sound field feature components, and actor physiological state feature components each occupy a specific dimension interval in the feature vector, without overlapping. This ensures that the feature information corresponding to each modality can be accurately identified and extracted in subsequent processing. This multimodal deep learning decision model adopts a feature-level fusion architecture, and its overall structure includes three main components: multiple independent modal encoders, a cross-modal interaction layer, and a multi-task decoder. Among them, the multiple independent modal encoders correspond to each modal feature component and are used to perform high-level semantic extraction on the input modal features. Specifically, firstly, after inputting the integrated features of the multimodal environment into the multimodal deep learning decision model, the model divides the integrated features according to a preset feature dimension index, separating the feature components corresponding to temperature field, humidity field, illuminance, smoke concentration, airflow velocity, sound field, and actor physiological state data. These feature components are then fed into corresponding independent modal encoders. Each independent modal encoder employs a lightweight neural network structure, specifically designed for the data characteristics of its corresponding modality, to extract high-level semantic features for that modality. The independent modal encoders run in parallel within the model, without interfering with each other, each outputting a high-level semantic feature representation of its corresponding modality. These high-level semantic features possess stronger abstract expressive power and semantic information compared to the original feature components. Secondly, the high-level semantic features of each modality output by the independent modal encoders are jointly input into the cross-modal interaction layer. This layer embeds a lightweight multi-head cross-attention module, which employs a grouped sparse computation structure. Using the high-level semantic features of each modality as the processing object, it calculates the correlation weight between any two modal features through a cross-attention mechanism. Specifically, for each modal combination, the cross-attention module uses a feature from one modality as the query and a feature from the other modality as the key and value, calculating the attention weight between them to quantify the association strength between the two modalities. In this way, the cross-modal interaction layer can achieve full interaction and information fusion between modal features in a unified semantic space, effectively capturing the implicit coupling relationships between different modalities, such as the physical correlation between temperature field changes and smoke concentration diffusion, and the synergistic relationship between illuminance and the actor's physiological state. Based on this, the cross-modal interaction layer completes cross-modal causal reasoning and outputs a fused representation after full interaction and fusion. Finally, the fused representation output from the cross-modal interaction layer is fed into a multi-task decoder, which employs a multi-output head structure, with different output heads corresponding to different heterogeneous stage execution devices.After receiving the fused representation, the multi-task decoder decodes and generates corresponding control parameters for each heterogeneous execution device in parallel. These parameters include, but are not limited to, color temperature, brightness, and illumination angle control parameters for lighting equipment; wind speed, air volume, and wind direction control parameters for ventilation and smoke extraction equipment; and lifting height, movement position, and rotation angle control parameters for stage machinery equipment. These control parameters are output in parallel, collectively forming a collaborative control instruction set for multiple heterogeneous execution devices.

[0030] In one embodiment, using high-level semantic features of each modality as input, a cross-attention mechanism is used to calculate the correlation weight between any two modal features. This enables cross-modal feature interaction and information fusion in a unified semantic space, capturing the implicit coupling relationships between different modalities and completing cross-modal causal inference, including: The high-level semantic features of each modality are mapped to a unified embedding space through linear projection to obtain the query vector, key vector, and value vector corresponding to each modality. For any two modalities, a cross-attention mechanism is used for processing. The dot product of the query vector of the first modality and the key vector of the second modality is used as the original relevance measure. After scaling and normalization, the attention weight matrix between the two modalities is obtained. The value vector of the second modality is weighted and aggregated through the attention weight matrix to generate the cross-modal interaction information obtained by the first modality from the second modality. The cross-modal interaction information of each modality is fused to form a fused representation containing the coupling relationship between any two modalities, thus completing cross-modal causal inference.

[0031] Specifically, in the cross-modal interaction layer, the high-level semantic features of each modality output by the independent modal encoders are first linearly projected to a unified embedding space. Specifically, for each modality's high-level semantic feature, it is transformed into a query vector, key vector, and value vector using three independent linear projection matrices. Since the query vectors, key vectors, and value vectors of all modalities are mapped to the same embedding space through linear projection, the features of different modalities are comparable and interactive in this space, laying the foundation for subsequent cross-modal attention computation. Based on this, for any two modalities, a cross-attention mechanism is used for cross-modal interaction processing. Specifically, the query vector of the first modality and the key vector of the second modality are multiplied by a dot product. The dot product result is used as a measure of the original correlation between the first and second modalities. The magnitude of this dot product value reflects the degree of correlation between a certain position or feature in the first modality and the corresponding position or feature in the second modality; the larger the dot product value, the stronger the correlation between the two. Subsequently, all calculated dot product values ​​are scaled by dividing them by a scaling factor (usually the square root of the key vector dimension) to control the magnitude of the dot product values ​​and prevent the gradient from entering the saturation region during subsequent normalization operations due to excessively large values. After scaling, the dot product values ​​are normalized using the softmax function, converting them into a probability distribution form so that the sum of the attention weights at each query position across all key positions is 1, thus obtaining the attention weight matrix between the two modalities. After obtaining the attention weight matrix between the first and second modalities, this matrix is ​​used to perform weighted aggregation on the value vector of the second modality. Specifically, matrix multiplication is performed between the attention weight matrix and the value vector of the second modality. Each weight value in the weight matrix corresponds to the weighting coefficient at the corresponding position in the value vector. By weighted summation, cross-modal interaction information obtained by the first modality from the second modality is generated. This cross-modal interaction information essentially represents the responsive adjustments that the features of the first modality should make under the influence of the context of the second modality. The above process is repeated for all possible modal combinations; that is, for each modal combination, the interaction computation between query, key, and value is completed according to the cross-attention mechanism. During this process, each modality can obtain corresponding cross-modal interaction information from all other modalities. The cross-modal interaction information obtained by each modality is fused with the original features of the modality itself to form a fused representation containing the coupling relationship between any two modalities. This fused representation not only preserves the high-level semantic information of each modality itself but also fully encodes the bidirectional influence relationships between modalities, such as the physical correlation between temperature field and smoke concentration, the heat transfer correlation between airflow velocity and temperature field, and the stress correlation between sound field and actor's physiological state.

[0032] In one embodiment, weighted aggregation of the value vector of the second modality using an attention weight matrix to generate cross-modal interaction information obtained by the first modality from the second modality includes: The attention weight matrix is ​​multiplied with the value vector of the second modality to obtain the weighted aggregated second modality information. The second modality information is then subjected to a nonlinear transformation to obtain the transformed cross-modal information. The transformed cross-modal information is then residually connected with the high-level semantic features of the first modality to generate the cross-modal interaction information obtained by the first modality from the second modality. The cross-modal interaction information encodes the contextual influence of the second modality on the first modality, so that the feature representation of the first modality incorporates complementary information from the second modality.

[0033] Specifically, firstly, a matrix multiplication operation is performed between the attention weight matrix and the value vector of the second modality. Each row of the attention weight matrix corresponds to a query position in the query vector of the first modality, and each column corresponds to a key position in the key vector of the second modality. The weight values ​​in the matrix represent the attention intensity between each query position in the first modality and each key position in the second modality. By multiplying the attention weight matrix with the value vector of the second modality, the features at each position in the value vector are weighted and summed according to their corresponding attention weights, thus obtaining the weighted aggregated second modality information. This information is essentially a feature representation aligned with the query positions of the first modality, formed by recombining the feature information of the second modality according to the attention weights. Secondly, the weighted aggregated second modality information is input into a nonlinear transformation layer for nonlinear transformation processing. Through nonlinear mapping, the weighted aggregated second modality information is transformed from the original representation space to a higher-order semantic space, enhancing the expressive power and nonlinear fitting ability of the features, making it easier to fuse with the features of the first modality at the semantic level. After the nonlinear transformation, the transformed cross-modal information is obtained. Finally, the transformed cross-modal information is residually joined with the high-level semantic features of the first modality. Specifically, the transformed cross-modal information is added element-wise to the high-level semantic features of the first modality, and the result is the cross-modal interaction information obtained by the first modality from the second modality. During the residual joining process, the transformed cross-modal information is superimposed as a residual term onto the original high-level semantic features of the first modality, so that the feature representation of the first modality incorporates complementary information from the second modality. Through the introduction of residual joining, this cross-modal interaction information encodes the contextual influence of the second modality on the first modality. That is, after considering the overall state information of the second modality, the first modality features should make corresponding adjustments. For example, when the temperature field data changes, the features of the smoke concentration modality will be adjusted accordingly to reflect the impact of temperature changes on smoke diffusion; when the actor's position moves, the features of the illuminance modality will be adjusted accordingly to adapt to the change in lighting requirements. The residual structure ensures that the original high-level semantic features of the first modality are preserved, while additional complementary information from the second modality is superimposed. This maintains the integrity of the features of the first modality itself and achieves effective integration of cross-modal information, avoiding the loss of original semantic information during cross-modal interaction.

[0034] In one embodiment, multimodal environment data is aggregated to an edge computing node near the stage. Within the edge computing node, a lightweight deep learning model is used to extract independent spatiotemporal features of each modality in parallel. These independent spatiotemporal features are then correlated to generate integrated multimodal environment features, including: Multimodal environment data is aggregated to edge computing nodes near the stage. Within each edge computing node, a lightweight deep learning model, pruned and quantized, is configured for each modality of the multimodal environment data. These lightweight deep learning models run in parallel within the edge computing nodes, independently extracting temporal dependency features and / or spatial distribution features of their respective modal data, forming independent spatiotemporal features for each modality. Based on an attention mechanism, an attention weight matrix is ​​calculated between the independent spatiotemporal features of different modalities to quantify the inter-modal correlation strength in both temporal and spatial dimensions. According to the attention weight matrix, the independent spatiotemporal features of each modality are weighted and integrated to obtain integrated multimodal environment features. These integrated features retain the feature components corresponding to each modality and encode the spatiotemporal correlation between different modalities.

[0035] Specifically, multimodal environmental data is first aggregated via wired or wireless networks to edge computing nodes deployed near the stage. These edge computing nodes, located close to the data source, possess independent computing, storage, and network communication capabilities, enabling them to independently process, analyze, and make decisions without relying on cloud servers. By aggregating data at the edge rather than in the cloud, the physical distance for data transmission is significantly shortened, effectively reducing latency. Inside each edge computing node, a corresponding lightweight deep learning model is configured for each modality of the multimodal environmental data. Specifically, for temperature field data, humidity field data, illuminance data, smoke concentration data, airflow velocity data, sound field data, and actor physiological state data, a separate and dedicated lightweight deep learning feature extraction model is deployed. These lightweight deep learning models undergo model pruning and quantization before deployment to the edge computing nodes. Model pruning reduces the number of parameters and computational load by removing redundant connections or neurons; model quantization further compresses the model size and accelerates the inference process by converting floating-point precision weight parameters into low-bit-width integer parameters. After the above processing, each lightweight deep learning model can run efficiently under the limited computing power and storage resources of edge computing nodes, and has low inference latency, meeting the stringent real-time requirements of stage scenes. Each lightweight deep learning model runs in parallel within the edge computing node, that is, the feature extraction process of all modalities is carried out simultaneously without blocking each other. Each lightweight deep learning model independently receives the raw data of the corresponding modality and extracts the temporal dependency features and / or spatial distribution features of the modality data. Specifically, for modal data with obvious temporal evolution patterns (such as the changing trend of temperature field over time, the accumulation or dissipation of smoke concentration as the performance progresses, the fluctuation of actors' physiological state with the performance intensity, etc.), the model focuses on extracting its temporal dependency features to capture the dynamic change pattern and evolution law of the modality in the time dimension; for modal data with significant spatial distribution characteristics (such as the spatial non-uniformity of illumination in different stage areas, the distribution difference of temperature field in different spatial locations, etc.), the model focuses on extracting its spatial distribution features to characterize the distribution of the modality in various locations on the stage space. After the above extraction process, each modality data forms a corresponding independent spatiotemporal feature, which comprehensively reflects the state information of each modality in both time and space dimensions.Based on this, cross-modal correlation processing is performed on the extracted independent spatiotemporal features of each modality. Specifically, based on an attention mechanism, an attention weight matrix is ​​calculated between the independent spatiotemporal features of different modalities. This attention weight matrix is ​​used to quantify the inter-modal correlation strength between each modality in the temporal and spatial dimensions, that is, how changes in one modality in the temporal dimension affect the spatiotemporal feature performance of another modality, and how the spatial distribution of one modality correlates with the spatial distribution of other modalities. Through this attention weight matrix, the complex cross-modal dependencies between modalities can be comprehensively captured. Finally, based on the calculated attention weight matrix, the independent spatiotemporal features of each modality are weighted and integrated. In the weighted integration process, modal features with larger weight values ​​in the attention weight matrix have a more significant impact on the integration result, thereby achieving effective fusion of cross-modal information. After weighted integration, a multimodal environment integrated feature is generated. In terms of data structure, the feature retains the feature components corresponding to each modality, namely, the temperature field component, humidity field component, illuminance component, smoke concentration component, airflow velocity component, sound field component, and actor physiological state component are still independent and distinguishable, which facilitates subsequent decision-making models to identify and process them separately. At the same time, the relative relationships and correlation information between each feature component have been encoded into the overall structure of the integrated feature through the attention weight matrix, so that the integrated feature not only contains the state information of each modality itself, but also contains the spatiotemporal correlation between different modalities.

[0036] In one embodiment, the independent spatiotemporal features of each modality are weighted and integrated based on the attention weight matrix to obtain multimodal environment integrated features, including: The attention weight matrix is ​​decomposed into a temporal attention weight submatrix and a spatial attention weight submatrix. The independent spatiotemporal features of each modality are weighted and integrated along the temporal dimension using the temporal attention weight submatrix to obtain the temporal integrated features. Similarly, the independent spatiotemporal features of each modality are weighted and integrated along the spatial dimension using the spatial attention weight submatrix to obtain the spatial integrated features. Through a gating integration mechanism, the integration weights of the temporal and spatial integrated features are adaptively calculated based on their dynamic confidence levels under the current feature distribution. Based on these integration weights, multimodal environment integrated features are generated.

[0037] Specifically, after calculating the attention weight matrix between independent spatiotemporal features of different modalities through the attention mechanism, this attention weight matrix is ​​first decomposed to separate a temporal dimension attention weight submatrix and a spatial dimension attention weight submatrix. The temporal dimension attention weight submatrix focuses on characterizing the interdependence of features across modalities in the time series, i.e., the degree of influence of a modality's state change at a historical moment on the current features of another modality. The spatial dimension attention weight submatrix focuses on characterizing the interdependence of features across modalities in the stage space layout, i.e., the degree of influence of a modality's feature distribution at a specific spatial location on the features of another modality at the same or adjacent spatial locations. Using the decomposed temporal dimension attention weight submatrix, the independent spatiotemporal features of each modality are weighted and integrated along the time dimension. The features of each modality at different times are weighted and combined according to the temporal dimension attention weights to obtain the temporal dimension integrated features. These features centrally reflect the co-evolutionary laws of each modality in the time dimension. Simultaneously, the spatial dimension attention weights are used... The submatrix weights and integrates the independent spatiotemporal features of each modality along the spatial dimension. It combines the features of each modality at different spatial locations according to spatial dimension attention weights to obtain spatial dimension integrated features, which centrally reflect the collaborative distribution patterns of each modality in the spatial dimension. Based on this, a gating integration mechanism further fuses the temporal and spatial dimension integrated features. This mechanism first evaluates the dynamic confidence of the temporal and spatial dimension integrated features based on their statistical characteristics under the current feature distribution, i.e., determining the reliability and information richness of each feature under the current environmental state. Then, it adaptively calculates the integration weights of the temporal and spatial dimension integrated features based on their confidence levels, giving higher-confidence integrated features a larger weight proportion and lower-confidence integrated features a smaller weight proportion. Finally, it weights and combines the temporal and spatial dimension integrated features according to the calculated integration weights to generate multimodal environment integrated features.

[0038] In one embodiment, through a gated integration mechanism, the integration weights of the temporal and spatial integrated features are adaptively calculated based on the dynamic confidence levels of the temporal and spatial integrated features under the current feature distribution. Based on these integration weights, multimodal environment integrated features are generated, including: Through a gating integration mechanism, the confidence levels of the time-dimensional and spatial-dimensional integrated features are evaluated based on their respective statistical characteristics. Based on the confidence levels, the integration weights of the time-dimensional and spatial-dimensional integrated features are adaptively determined, so that the allocation of integration weights adjusts with the changes in the feature distributions of the time-dimensional and spatial-dimensional integrated features. Based on the integration weights, the time-dimensional and spatial-dimensional integrated features are weighted and integrated to generate multimodal environment integrated features.

[0039] Specifically, after weighting and integrating the independent spatiotemporal features of each modality along the time and space dimensions using the temporal and spatial dimension attention weight sub-matrices respectively, to obtain temporal and spatial integrated features, a gating integration mechanism is further used to fuse the temporal and spatial integrated features. This gating integration mechanism first performs statistical characteristic analysis on the temporal and spatial integrated features respectively, calculating the feature mean, variance, energy distribution, and dynamic range of feature amplitude in their corresponding dimensions. These statistical measures are used to comprehensively evaluate the stability and reliability of each feature under the current feature distribution, and then the confidence levels of the temporal and spatial integrated features are obtained respectively. The higher the confidence level, the more stable the feature distribution of the corresponding integrated feature under the current environmental state, the richer the information content, and the less noise interference. The lower the confidence level, the worse the information reliability of the corresponding integrated feature. After obtaining the confidence level, the gating integration mechanism adaptively determines the integration weights of the temporal and spatial integrated features based on this confidence level. Specifically, integrated features with higher confidence levels are assigned larger integration weights, while integrated features with lower confidence levels are assigned smaller integration weights. The allocation of these integration weights is dynamically adjusted as the feature distribution of each integrated feature changes. When the feature distribution changes over time or the environment changes, the confidence level is reassessed, and the integration weights are redistributed accordingly, thus achieving dynamic adaptability in weight allocation. Finally, based on the determined integration weights, the temporal and spatial integrated features are weighted and integrated, that is, element-wise weighted summation is performed according to their respective integration weights to generate multimodal environmental integrated features that integrate temporal and spatial information. This allows the final integrated features to dynamically adjust the fusion ratio of spatiotemporal information according to the quality of the current feature distribution, achieving a more accurate and robust environmental representation.

[0040] In one embodiment, the coordinated adjustment of multiple heterogeneous stage performance devices under the control of an edge computing node, according to a collaborative control instruction set, includes: Edge computing nodes distribute the control parameter vectors in the collaborative control instruction set to the corresponding heterogeneous stage execution devices in parallel through standardized industrial control protocols. The heterogeneous stage execution devices include at least lighting equipment, ventilation and smoke extraction equipment, and stage machinery. While distributing the collaborative control instruction set, the edge computing nodes continuously receive real-time multimodal environmental data from the sensor array as feedback signals for linkage adjustment. Based on the feedback signals, the edge computing nodes dynamically correct the collaborative control instruction set.

[0041] Specifically, after the multimodal deep learning decision model outputs a collaborative control instruction set, the edge computing node parses the collaborative control instruction set and extracts the control parameter vectors corresponding to different heterogeneous stage execution devices. These include control parameter vectors for lighting equipment (such as color temperature, brightness, horizontal rotation angle, and vertical rotation angle), control parameter vectors for ventilation and smoke extraction equipment (such as wind speed, air volume, and wind direction), and control parameter vectors for stage machinery equipment (such as lifting platform height, moving device displacement, and rotating device rotation angle). The edge computing node encapsulates these control parameter vectors according to the standardized industrial control protocols supported by the corresponding devices (such as the DMX512 lighting control protocol and the Modbus industrial communication protocol) and distributes them simultaneously to each heterogeneous stage execution device in parallel. The system prepares to issue corresponding control commands, and each device synchronously executes adjustment actions upon receiving the control commands, thereby achieving parallel linkage of multiple devices. While the edge computing node distributes the collaborative control command set in parallel, it continuously receives multimodal environmental data collected in real time from the sensor array. This real-time collected data serves as feedback signals for the linkage adjustment, reflecting the actual environmental state changes after the current adjustment action is executed. The edge computing node compares the feedback signals with the current desired target environmental state. If there is a deviation between the actual environmental state and the target state, the issued collaborative control command set is dynamically corrected based on the feedback signal. That is, in the next control cycle, the control parameters of each device are adjusted according to the feedback deviation, thereby achieving closed-loop adjustment based on real-time feedback and ensuring that the stage environment state is always maintained within the desired target range.

[0042] In one embodiment, such as Figure 3 As shown, a stage environment adaptive adjustment device is provided, including: an environmental data acquisition module 10, a modal data integration module 20, a control command output module 30, and an equipment linkage adjustment module 40, wherein: The environmental data acquisition module 10 is used to acquire multimodal environmental data in real time through a high-density sensor array deployed on the stage. The multimodal environmental data includes at least temperature field, humidity field, illuminance, smoke concentration, airflow speed, sound field and actor physiological state data. The modal data integration module 20 is used to aggregate multimodal environment data to the edge computing node near the stage. Within the edge computing node, a lightweight deep learning model is used to extract the independent spatiotemporal features of each modality data in parallel, and the independent spatiotemporal features are correlated to generate multimodal environment integration features. In the multimodal environment integration features, the feature components corresponding to each modality are kept independent and distinguishable according to the feature dimension index. The control command output module 30 is used to input multimodal environment integration features into a multimodal deep learning decision model with an embedded cross-attention module. The multimodal deep learning decision model outputs a set of collaborative control commands for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning. The equipment linkage adjustment module 40 is used to perform linkage adjustment on multiple heterogeneous stage execution devices under the control of the edge computing node according to the collaborative control instruction set.

[0043] In one embodiment, the control command output module 30 is further configured to input multimodal environment integration features into a multimodal deep learning decision model. The multimodal deep learning decision model adopts a feature-level fusion architecture, including multiple independent modal encoders, a cross-modal interaction layer, and a multi-task decoder. The multiple independent modal encoders respectively receive the components of the corresponding modal from the multimodal environment integration features divided according to a preset feature dimension index, and extract the high-level semantic features of each modality. The cross-modal interaction layer embeds a lightweight multi-head cross-attention module, which takes the high-level semantic features of each modality as input, calculates the correlation weight between any two modal features through a cross-attention mechanism, realizes cross-modal feature interaction and information fusion in a unified semantic space, captures the implicit coupling relationship between different modalities, and completes cross-modal causal reasoning. The multi-task decoder receives the fused representation output by the cross-modal interaction layer and outputs control parameters corresponding to multiple heterogeneous execution devices in parallel to form a collaborative control command set.

[0044] In one embodiment, the control instruction output module 30 is further configured to map the high-level semantic features of each modality to a unified embedding space through linear projection to obtain the query vector, key vector, and value vector corresponding to each modality; for any two modalities, a cross-attention mechanism is used for processing, and the dot product of the query vector of the first modality and the key vector of the second modality is used as the original relevance measure. After scaling and normalization, the attention weight matrix between the two modalities is obtained; the value vector of the second modality is weighted and aggregated through the attention weight matrix to generate the cross-modal interaction information obtained by the first modality from the second modality; the cross-modal interaction information of each modality is fused to form a fusion representation containing the coupling relationship between any two modalities, thereby completing cross-modal causal reasoning.

[0045] In one embodiment, the control instruction output module 30 is further configured to perform matrix multiplication between the attention weight matrix and the value vector of the second modality to obtain weighted aggregated second modality information; perform nonlinear transformation on the second modality information to obtain transformed cross-modality information; and perform residual connection between the transformed cross-modality information and the high-level semantic features of the first modality to generate cross-modality interaction information obtained by the first modality from the second modality, wherein the cross-modality interaction information encodes the contextual influence of the second modality on the first modality, so that complementary information from the second modality is incorporated into the feature representation of the first modality.

[0046] In one embodiment, the modal data integration module 20 is further configured to aggregate multimodal environment data to an edge computing node adjacent to the stage. Within the edge computing node, a lightweight deep learning model, pruned and quantized, is configured for each modality in the multimodal environment data. The lightweight deep learning models run in parallel within the edge computing node, independently extracting the temporal dependency features and / or spatial distribution features of the corresponding modal data to form independent spatiotemporal features of each modality. Based on the attention mechanism, an attention weight matrix is ​​calculated between the independent spatiotemporal features of different modalities to quantify the inter-modal correlation strength in the temporal and spatial dimensions. According to the attention weight matrix, the independent spatiotemporal features of each modality are weighted and integrated to obtain multimodal environment integrated features. The multimodal environment integrated features retain the feature components corresponding to each modality and encode the spatiotemporal correlation between different modalities.

[0047] In one embodiment, the modal data integration module 20 is further configured to decompose the attention weight matrix into a temporal dimension attention weight submatrix and a spatial dimension attention weight submatrix; use the temporal dimension attention weight submatrix to perform weighted integration of the independent spatiotemporal features of each modality along the time dimension to obtain temporal dimension integrated features, and use the spatial dimension attention weight submatrix to perform weighted integration of the independent spatiotemporal features of each modality along the spatial dimension to obtain spatial dimension integrated features; through a gating integration mechanism, based on the dynamic confidence of the temporal dimension integrated features and the spatial dimension integrated features under the current feature distribution, adaptively calculate the integration weights of the temporal dimension integrated features and the spatial dimension integrated features, and generate multimodal environment integrated features based on the integration weights.

[0048] In one embodiment, the modal data integration module 20 is further configured to, through a gating integration mechanism, evaluate the confidence levels of the temporal and spatial integrated features based on their respective statistical characteristics; adaptively determine the integration weights of the temporal and spatial integrated features according to the confidence levels, such that the allocation of integration weights is adjusted as the feature distributions of the temporal and spatial integrated features change; and perform weighted integration of the temporal and spatial integrated features based on the integration weights to generate multimodal environment integrated features.

[0049] In one embodiment, the device linkage adjustment module 40 is further configured to distribute the control parameter vectors in the collaborative control instruction set to the corresponding heterogeneous stage execution devices in parallel through a standardized industrial control protocol. The heterogeneous stage execution devices include at least lighting equipment, ventilation and smoke extraction equipment, and stage machinery equipment. While distributing the collaborative control instruction set, the edge computing node continuously receives real-time multimodal environmental data from the sensor array as feedback signals for linkage adjustment. Based on the feedback signals, the edge computing node dynamically corrects the collaborative control instruction set.

[0050] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A stage environment adaptive adjustment method, characterized in that, include: Multimodal environmental data is collected in real time by a high-density sensor array deployed on the stage. The multimodal environmental data includes at least temperature field, humidity field, illuminance, smoke concentration, airflow speed, sound field, and actor physiological state data. The multimodal environment data is aggregated to an edge computing node near the stage. Within the edge computing node, a lightweight deep learning model is used to extract the independent spatiotemporal features of each modality in parallel. The independent spatiotemporal features are then correlated to generate integrated multimodal environment features. The feature components corresponding to each modality in the integrated multimodal environment features are kept independent and distinguishable according to the feature dimension index. The multimodal environment integration features are input into a multimodal deep learning decision model with an embedded cross-attention module. The multimodal deep learning decision model outputs a set of collaborative control instructions for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning. According to the collaborative control instruction set, the multiple heterogeneous stage execution devices are coordinated and adjusted under the control of the edge computing node.

2. The stage environment adaptive adjustment method according to claim 1, characterized in that, The multimodal deep learning decision model, which integrates the features of the multimodal environment into an embedded cross-attention module, outputs a set of collaborative control instructions for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning. The integrated features of the multimodal environment are input into a multimodal deep learning decision model. The multimodal deep learning decision model adopts a feature-level fusion architecture, which includes multiple independent modal encoders, a cross-modal interaction layer, and a multi-task decoder. The multiple independent modal encoders respectively receive the components of the corresponding modal from the integrated features of the multimodal environment according to the preset feature dimension index, and extract the high-level semantic features of each modality; The cross-modal interaction layer embeds a lightweight multi-head cross-attention module, which takes the high-level semantic features of each modality as input, calculates the correlation weight between any two modal features through the cross-attention mechanism, realizes cross-modal feature interaction and information fusion in a unified semantic space, captures the implicit coupling relationship between different modalities, and completes cross-modal causal reasoning. The multi-task decoder receives the fused representation output by the cross-modal interaction layer and outputs control parameters corresponding to multiple heterogeneous execution devices in parallel to form a collaborative control instruction set.

3. The stage environment adaptive adjustment method according to claim 2, characterized in that, The process of using high-level semantic features of each modality as input, calculating the correlation weight between any two modal features through a cross-attention mechanism, realizing cross-modal feature interaction and information fusion in a unified semantic space, capturing the implicit coupling relationship between different modalities, and completing cross-modal causal reasoning includes: The high-level semantic features of each modality are mapped to a unified embedding space through linear projection to obtain the query vector, key vector, and value vector corresponding to each modality. For any two modalities, a cross-attention mechanism is used for processing. The dot product of the query vector of the first modality and the key vector of the second modality is used as the original relevance measure. After scaling and normalization, the attention weight matrix between the two modalities is obtained. The value vector of the second modality is weighted and aggregated using the attention weight matrix to generate cross-modal interaction information obtained by the first modality from the second modality; By fusing cross-modal interaction information from various modalities, a fused representation containing the coupling relationship between any two modalities is formed, thus completing cross-modal causal reasoning.

4. The stage environment adaptive adjustment method according to claim 3, characterized in that, The step of weighting and aggregating the value vector of the second modality using the attention weight matrix to generate cross-modal interaction information obtained by the first modality from the second modality includes: The attention weight matrix is ​​multiplied with the value vector of the second modality to obtain the weighted aggregated second modality information; The second modal information is subjected to a nonlinear transformation to obtain the transformed cross-modal information; The transformed cross-modal information is residually concatenated with the high-level semantic features of the first modality to generate cross-modal interaction information obtained by the first modality from the second modality. The cross-modal interaction information encodes the contextual influence of the second modality on the first modality, so that complementary information from the second modality is incorporated into the feature representation of the first modality.

5. The stage environment adaptive adjustment method according to claim 1, characterized in that, The process of aggregating the multimodal environment data to an edge computing node adjacent to the stage, extracting independent spatiotemporal features of each modality data in parallel using a lightweight deep learning model within the edge computing node, and performing correlation processing on the independent spatiotemporal features to generate integrated multimodal environment features includes: The multimodal environment data is aggregated to an edge computing node near the stage. Within the edge computing node, a lightweight deep learning model that has undergone model pruning and quantization is configured for each modality of the multimodal environment data. The lightweight deep learning models run in parallel within the edge computing node and independently extract the temporal dependency features and / or spatial distribution features of the corresponding modality data to form independent spatiotemporal features of each modality. Based on the attention mechanism, the attention weight matrix between independent spatiotemporal features of different modalities is calculated to quantify the strength of the inter-modal correlation in the temporal and spatial dimensions. Based on the attention weight matrix, the independent spatiotemporal features of each modality are weighted and integrated to obtain multimodal environment integrated features. The multimodal environment integrated features retain the feature components corresponding to each modality and encode the spatiotemporal correlation between different modalities.

6. The stage environment adaptive adjustment method according to claim 5, characterized in that, The step of weighting and integrating the independent spatiotemporal features of each modality based on the attention weight matrix to obtain the multimodal environment integrated features includes: The attention weight matrix is ​​decomposed into a time-dimensional attention weight submatrix and a spatial-dimensional attention weight submatrix; The independent spatiotemporal features of each modality are weighted and integrated along the time dimension using the time-dimensional attention weight submatrix to obtain the time-dimensional integrated features. The independent spatiotemporal features of each modality are weighted and integrated along the spatial dimension using the spatial-dimensional attention weight submatrix to obtain the spatial-dimensional integrated features. Through a gating integration mechanism, the integration weights of the time-dimensional integration features and the spatial-dimensional integration features are adaptively calculated based on the dynamic confidence levels of the time-dimensional integration features and the spatial-dimensional integration features under the current feature distribution. Based on the integration weights, multimodal environment integration features are generated.

7. The stage environment adaptive adjustment method according to claim 6, characterized in that, The step of using a gated integration mechanism to adaptively calculate the integration weights of the temporal and spatial integrated features based on their dynamic confidence levels under the current feature distribution, and then generating multimodal environment integrated features based on these integration weights includes: By using a gating integration mechanism, the confidence levels of the time-dimensional integration features and the spatial-dimensional integration features are evaluated based on their respective statistical characteristics. Based on the confidence level, the integration weights of the time-dimensional integrated features and the spatial-dimensional integrated features are adaptively determined, such that the allocation of the integration weights is adjusted as the feature distributions of the time-dimensional integrated features and the spatial-dimensional integrated features change. Based on the integration weights, the time-dimensional integration features and the spatial-dimensional integration features are weighted and integrated to generate multimodal environment integration features.

8. The stage environment adaptive adjustment method according to claim 1, characterized in that, The step of coordinating and adjusting the multiple heterogeneous stage execution devices under the control of the edge computing node according to the collaborative control instruction set includes: The edge computing node distributes the control parameter vectors in the collaborative control instruction set to the corresponding heterogeneous stage execution equipment in parallel through a standardized industrial control protocol. The heterogeneous stage execution equipment includes at least lighting equipment, ventilation and smoke extraction equipment, and stage machinery equipment. While distributing the collaborative control instruction set, the edge computing node continuously receives real-time multimodal environmental data from the sensor array as a feedback signal for linkage adjustment. Based on the feedback signal, the edge computing node dynamically corrects the collaborative control instruction set.

9. A stage environment adaptive adjustment device, characterized in that, include: The environmental data acquisition module is used to collect multimodal environmental data in real time through a high-density sensor array deployed on the stage. The multimodal environmental data includes at least temperature field, humidity field, illuminance, smoke concentration, airflow speed, sound field, and actor physiological state data. The modal data integration module is used to aggregate the multimodal environment data to an edge computing node adjacent to the stage. Within the edge computing node, a lightweight deep learning model is used to extract the independent spatiotemporal features of each modality data in parallel, and the independent spatiotemporal features are correlated to generate multimodal environment integrated features. The feature components corresponding to each modality in the multimodal environment integrated features are kept independent and distinguishable according to the feature dimension index. The control command output module is used to input the integrated features of the multimodal environment into a multimodal deep learning decision model with an embedded cross-attention module. The multimodal deep learning decision model outputs a set of collaborative control commands for multiple heterogeneous execution devices through cross-modal feature interaction and reasoning. The equipment linkage adjustment module is used to perform linkage adjustment of the multiple heterogeneous stage execution devices under the control of the edge computing node according to the collaborative control instruction set.

Images