Multi-modal data gan generative physics-enhanced method and system fusing physical laws

Abstract

This invention relates to the field of multimodal data processing technology, and more particularly to a generative physical augmentation method for multimodal data using GANs that integrates physical laws. The method involves acquiring multimodal sample data of a construction site's safety production environment, constructing a multimodal coding network to map the features of each modality to a unified latent feature space, extracting conditional control information and encoding it as conditional constraint vectors, and using a generative adversarial network to generate synthetic multimodal data. The generator and discriminator networks are optimized through adversarial training. Augmented multimodal data is generated using the trained generator, and then filtered and merged into the original sample data to construct an augmented dataset. This method improves the diversity and realism of multimodal data and enhances its adaptability to complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of multimodal data processing technology, and in particular to a multimodal data GAN generative physics enhancement method and system that integrates physical laws. Background Technology

[0002] In the field of multimodal data augmentation for construction site safety environments, current conventional practices mainly rely on traditional data augmentation techniques or single-modal data generation methods based on generative adversarial networks (GANs). Traditional data augmentation typically employs geometric transformations, color jittering, and noise addition for image modalities, and temporal perturbations or random sampling for environmental sensing modalities. These methods process different modalities independently, lacking modeling and utilization of the inherent correlations between multimodal data. More advanced approaches attempt to input multimodal data into GANs and generate synthetic samples through end-to-end training. This includes building generators and discriminators separately for image and sensor data, or simply concatenating multimodal feature vectors and inputting them into the generator. However, such methods usually treat multimodal data as independent feature sources, pursuing only visual or temporal realism while ignoring the coupling relationships and mutual constraints between different modalities in the physical scene.

[0003] The generated multimodal data lacks consistency at the level of physical laws. In construction site safety scenarios, the actions of personnel and the status of equipment in images are dependent on data such as temperature, humidity, and vibration collected by environmental sensors. For example, when welding sparks occur, the infrared sensor should simultaneously record a high-temperature signal. However, existing methods fail to explicitly or implicitly embed these physical rules into the generation process, leading to contradictions between visual and sensor data in the synthesized samples. For example, an image may show rain, but the humidity sensor indicates dryness. Such false samples can mislead the judgment ability of subsequent models.

[0004] Existing methods struggle to accurately control the semantic scene attributes of generated samples. Construction site safety scenarios present diverse hazardous conditions, such as falls from heights and mechanical collisions. Current generation methods cannot effectively guide the generation process by using specific safety production conditions (such as "working at heights") as constraints, resulting in a disconnect between the distribution of synthetic data and the target scene, leading to insufficient coverage and relevance. These shortcomings severely limit the practicality of augmented data in complex safety detection and early warning tasks. Summary of the Invention

[0005] This invention provides a method and system for generative physics enhancement of multimodal data using GANs that integrates physical laws, which can solve the problems in the prior art.

[0006] A first aspect of this invention provides a multimodal data GAN generative physics augmentation method that integrates physical laws, comprising: Multimodal sample data under the construction site safety production environment is acquired, including image modal data and environmental sensing modal data. A multimodal coding network is constructed, and features are extracted from each modal sample data. The features of each modality are mapped to a unified latent feature space, and the spatial representations of different modal features are aligned to obtain multimodal joint latent features. Conditional control information for safe production scenarios is extracted from multimodal sample data. This information is then encoded into conditional constraint vectors. A multimodal generative adversarial network (GAN) containing a generator network and a discriminator network is constructed. The multimodal joint latent features and conditional constraint vectors are input into the generator network to generate synthetic multimodal data. Synthetic multimodal data and real multimodal sample data are input into the discriminator network. The adversarial loss is calculated based on the discrimination results. Adversarial training is carried out by alternately optimizing the parameters of the generator network and the discriminator network until the training converges. The generator network, after training and convergence, generates enhanced multimodal data for a specified scenario. The quality of the enhanced multimodal data is evaluated, and the enhanced multimodal data that meets the quality threshold is selected. The selected enhanced multimodal data is then merged with the original multimodal sample data to construct an enhanced multimodal training dataset.

[0007] A multimodal coding network is constructed to extract features from sample data of each modality. These features are then mapped to a unified latent feature space, aligning the spatial representations of different modalities to obtain joint latent features of the multimodality, including: An image coding branch and a sensing coding branch are constructed. The image coding branch consists of multiple convolutional layers and pooling layers stacked alternately. Multi-layer convolutional feature extraction is performed on the image modal data to capture the hierarchical visual representation from local texture to global semantics of the image layer by layer, and the image feature map is output. The sensing coding branch is composed of multiple gated loop unit layers stacked together. It extracts temporal features from environmental sensing modal data at each time. The gated loop unit layer uses update gate and reset gate to calculate the information retention weight of the sensing data at the corresponding time. Based on the information retention weight, it filters key information that is effective in identifying safe production scenarios and outputs sensing temporal features. The image feature map and the sensing time-series feature are respectively mapped to a preset unified dimension space through a fully connected projection layer to obtain the image projection feature and the sensing projection feature. A cross-modal similarity matrix is ​​calculated based on image projection features and sensor projection features. A contrast alignment loss is constructed through the cross-modal similarity matrix. With the goal of minimizing the contrast alignment loss, the projection parameters and coding parameters of each coding branch are updated using gradient backpropagation to make the semantic distribution of image projection features and sensor projection features consistent in a preset unified dimensional space. The aligned image projection features and sensor projection features are then fused element-wise to obtain multimodal joint latent features.

[0008] Conditional control information for safe production scenarios is extracted from multimodal sample data, and this information is encoded into conditional constraint vectors, including: Based on the types of construction behaviors and object interaction states in the construction site safety production environment, multiple safety production scenario categories are predefined, and each safety production scenario category is mapped to a unique category label vector; Spatial visual features are extracted from image modal data to identify the target object categories, spatial positional relationships and motion state information between target objects, and generate visual scene semantic features. Sensing semantic features are extracted from environmental sensing modal data to identify the temporal variation trend, peak abnormal deviation degree, and frequency abnormal fluctuation characteristics of values ​​in each sensing channel, generating sensing scene semantic features. The visual scene semantic features and the sensing scene semantic features are fused across modal features. The intermodal interaction weight between the visual scene semantic features and the sensing scene semantic features is calculated by an attention mechanism. Based on the intermodal interaction weight, the visual scene semantic features and the sensing scene semantic features are weighted and fused into a fused scene semantic feature. The matching calculation is performed based on the semantic features of the fused scene and the predefined safety production scene category label vector to determine the probability of belonging to each safety production scene category. The safety production scene category with the highest belonging probability is selected as the condition control information, and the category label vector corresponding to the condition control information is used as the condition constraint vector.

[0009] Encoding conditional control information into a conditional constraint vector also includes: During the training of the multimodal generative adversarial network, the discriminator network outputs the data authenticity judgment result, and additionally outputs the scenario discrimination classification result of the safety production scenario to which the synthetic multimodal data belongs; Based on the degree of difference between the scene discrimination and classification results of each synthetic multimodal data and the target safety production scene category corresponding to the input condition constraint vector, the scene condition fit index is calculated; for synthetic multimodal data belonging to each safety production scene category, the average scene condition fit index of all synthetic multimodal data under that category is calculated, and the safety production scene category with the average scene condition fit index lower than the preset fit benchmark is marked as the difficult scene category to generate. For safety production scenario categories that are marked as difficult to generate, the corresponding scenario weight coefficients are dynamically increased in the next round of adversarial training, and the updated scenario weight coefficients are re-encoded into the condition constraint vector of the next training round. A closed-loop adaptive refinement mechanism is formed, in which the conditional constraint vector is driven by discriminator feedback between adversarial training rounds, enabling the generator network to automatically enhance the fidelity of generating difficult scene categories in subsequent training rounds.

[0010] A multimodal generative adversarial network (GAN) is constructed, comprising a generator network and a discriminator network. The joint latent features of the multimodal data and the conditional constraint vectors are input into the generator network to generate synthetic multimodal data, including: The joint latent features of the multimodal modes are concatenated with the conditional constraint vector to obtain the joint input vector. A generator network with an encoder-decoder structure is constructed. The encoder part consists of multiple stacked fully connected layers, which perform layer-by-layer nonlinear transformation on the joint input vector and extract intermediate features layer by layer. The decoder part consists of multiple stacked upsampling layers, which perform layer-by-layer upsampling and reconstruction on the generated intermediate features. During the upsampling process of the decoder part, synthetic image modal data and synthetic environment sensing modal data are generated respectively. In each upsampling layer of the decoding part, a cross-modal consistency constraint unit is set. The cross-modal consistency constraint unit extracts the intermediate feature map of the synthetic image modal data generated by the current upsampling layer and the intermediate feature sequence of the synthetic environment sensing modal data, respectively, calculates the modal mutual information value between the intermediate feature map and the intermediate feature sequence, constructs the inter-layer cross-modal consistency loss term of each upsampling layer based on the modal mutual information value, and superimposes the inter-layer cross-modal consistency loss term of each upsampling layer with the generative adversarial loss to obtain the comprehensive objective function of the generator network. The generator network parameters are optimized by minimizing the comprehensive objective function, and the synthetic image modal data and the synthetic environment sensing modal data are constrained to maintain cross-modal consistency at the semantic level.

[0011] The multimodal joint latent features and conditional constraint vectors are input into the generator network to generate synthetic multimodal data, which also includes: Before upsampling reconstruction in each upsampling layer, the conditional constraint vector is input into the conditional modulation network corresponding to the upsampling layer. The conditional modulation network is composed of multiple fully connected layers stacked together, which performs nonlinear feature transformation on the input conditional constraint vector. The conditional modulation network outputs the channel-wise scaling factor vector and the channel-wise bias vector corresponding to the upsampling layer. The channel-wise scaling factor vector is aligned with the channel dimension of the intermediate feature generated by the current layer of the upsampling layer, and the channel-wise bias vector is aligned with the channel dimension of the intermediate feature generated by the current layer of the upsampling layer. The feature values ​​of each channel in the current layer generated intermediate features of the upsampling layer are multiplied by the scaling factor corresponding to the channel according to the corresponding channel position and then added to the bias value corresponding to the channel. This completes the channel-wise adaptive conditional modulation operation of the conditional constraint vector on the layer generated intermediate features of the upsampling layer, and the conditionally modulated layer generated intermediate features are obtained. Each upsampling layer uses its conditionally modulated layer to generate intermediate features to continue subsequent upsampling reconstruction calculations, so that the conditional constraint vectors participate in the conditional regulation of the generation and reconstruction process at each upsampling level of the generator network, thereby achieving layer-by-layer fine-grained conditional control over the safety production scenario to which the synthesized multimodal data belongs.

[0012] A second aspect of the present invention provides a multimodal data GAN generative physics augmentation system that integrates physical laws, comprising: The feature alignment unit is used to acquire multimodal sample data under the construction site safety production environment, including image modal data and environmental sensing modal data. It constructs a multimodal coding network, extracts features from each modal sample data, maps each modal feature to a unified latent feature space, aligns the spatial representations of different modal features, and obtains multimodal joint latent features. The condition generation unit is used to extract condition control information of safety production scenarios from multimodal sample data, encode the condition control information into condition constraint vectors, construct a multimodal generative adversarial network containing a generator network and a discriminator network, and input the multimodal joint latent features and condition constraint vectors into the generator network to generate synthetic multimodal data. The adversarial training unit is used to input synthetic multimodal data and real multimodal sample data into the discriminator network, calculate the adversarial loss based on the discrimination results, and perform adversarial training by alternately optimizing the parameters of the generator network and the discriminator network until the training converges. The data augmentation unit is used to generate augmented multimodal data for a specified scenario using the generator network after training convergence, to evaluate the quality of the augmented multimodal data, to select augmented multimodal data that meets the quality threshold, and to merge the selected augmented multimodal data with the original multimodal sample data to construct an augmented multimodal training dataset.

[0013] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0015] This approach effectively addresses the scarcity and incompleteness of multimodal data in construction site safety environments, significantly enhancing the realism and diversity of augmented data. By constructing a multimodal coding network, images and sensor data are mapped to a unified latent feature space, achieving precise alignment of cross-modal features and avoiding the loss or misalignment of modal information inherent in traditional methods. The synergistic effect of conditional constraint vectors and generative adversarial networks ensures that synthetic data not only maintains physical consistency across modalities but also generates highly realistic augmented samples for specific safety scenarios, drastically reducing the manpower and equipment costs of data collection.

[0016] The adversarial training strategy based on the alternating optimization mechanism ensures a dynamic balance between the generator and discriminator, effectively suppressing training instability issues such as mode collapse and gradient vanishing. The generator, after training convergence, can stably output high-quality multimodal synthetic data. Low-confidence samples are removed through quality assessment and screening mechanisms, further ensuring the reliability of the augmented data. The training dataset, merged with the original data, achieves a dual improvement in modality coverage and sample quantity, providing a richer training foundation for subsequent deep learning models. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the generative physics enhancement method using GANs to integrate physical laws from multimodal data. Figure 2 A schematic diagram of the process for generating synthetic multimodal data. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0020] refer to Figure 1 and Figure 2 The present invention provides a multimodal data GAN generative physics augmentation method that integrates physical laws, comprising: Multimodal sample data under the construction site safety production environment is acquired, including image modal data and environmental sensing modal data. A multimodal coding network is constructed, and features are extracted from each modal sample data. The features of each modality are mapped to a unified latent feature space, and the spatial representations of different modal features are aligned to obtain multimodal joint latent features. Conditional control information for safe production scenarios is extracted from multimodal sample data. This information is then encoded into conditional constraint vectors. A multimodal generative adversarial network (GAN) containing a generator network and a discriminator network is constructed. The multimodal joint latent features and conditional constraint vectors are input into the generator network to generate synthetic multimodal data. Synthetic multimodal data and real multimodal sample data are input into the discriminator network. The adversarial loss is calculated based on the discrimination results. Adversarial training is carried out by alternately optimizing the parameters of the generator network and the discriminator network until the training converges. The generator network, after training and convergence, generates enhanced multimodal data for a specified scenario. The quality of the enhanced multimodal data is evaluated, and the enhanced multimodal data that meets the quality threshold is selected. The selected enhanced multimodal data is then merged with the original multimodal sample data to construct an enhanced multimodal training dataset.

[0021] A multimodal coding network is constructed to extract features from sample data of each modality. These features are then mapped to a unified latent feature space, aligning the spatial representations of different modalities to obtain joint latent features of the multimodality, including: An image coding branch and a sensing coding branch are constructed. The image coding branch consists of multiple convolutional layers and pooling layers stacked alternately. Multi-layer convolutional feature extraction is performed on the image modal data to capture the hierarchical visual representation from local texture to global semantics of the image layer by layer, and the image feature map is output. The sensing coding branch is composed of multiple gated loop unit layers stacked together. It extracts temporal features from environmental sensing modal data at each time. The gated loop unit layer uses update gate and reset gate to calculate the information retention weight of the sensing data at the corresponding time. Based on the information retention weight, it filters key information that is effective in identifying safe production scenarios and outputs sensing temporal features. The image feature map and the sensing time-series feature are respectively mapped to a preset unified dimension space through a fully connected projection layer to obtain the image projection feature and the sensing projection feature. The cross-modal similarity matrix is ​​calculated based on image projection features and sensor projection features. The contrast alignment loss is constructed through the cross-modal similarity matrix. With the goal of minimizing the contrast alignment loss, the projection parameters and coding parameters of each coding branch are updated by gradient backpropagation, so that the semantic distribution of image projection features and sensor projection features is consistent in a preset unified dimensional space. The aligned image projection features and the sensing projection features are fused using element-wise weighted fusion to obtain multimodal joint latent features.

[0022] When constructing a multimodal coding network, independent coding branches are designed to address the heterogeneity between image modal data and environmental sensing modal data, fully utilizing the inherent structural characteristics of each modality. The image coding branch employs a convolutional neural network structure, consisting of multiple convolutional and pooling layers stacked alternately. Each convolutional layer uses several 3×3 convolutional kernels to perform sliding convolution operations on the image, extracting local spatial features. The pooling layer uses max pooling to downsample the feature map, gradually compressing the spatial resolution and expanding the receptive field. Shallow convolutional layers primarily capture low-level local visual features such as edges, textures, and colors in the image. As the network depth increases, deeper convolutional layers gradually integrate local features to form more abstract semantic representations, such as perceiving high-level semantic information like the wearing status of safety helmets, personnel posture, and boundaries of dangerous areas in a construction site scene. Through the alternating stacking of multiple convolutional and pooling layers, the image coding branch ultimately outputs an image feature map containing rich hierarchical visual information. This feature map preserves the local structural information of the image in the spatial dimension while converging multi-scale semantic representations in the channel dimension.

[0023] The sensor coding branch is specifically designed to address the temporal characteristics of environmental sensor modal data, consisting of multiple stacked Gated Recurrent Unit (GRU) layers. Environmental sensor modal data contains sequential information on the temporal changes of multidimensional physical quantities such as temperature, humidity, gas concentration, and noise intensity, exhibiting significant temporal dependence. The GRU dynamically filters temporal information through two gating mechanisms: an update gate and a reset gate. The update gate controls the proportion of historical information retained in the current hidden state; when the update gate output value is close to 1, a large amount of historical state is retained, suitable for sensor signals with stable long-term trends. The reset gate determines the degree of forgetting of historical information when calculating candidate hidden states; when the reset gate output value is close to 0, a large amount of historical state is discarded, enabling the network to respond to abrupt changes in sensor data. The two gating mechanisms work together to dynamically calculate the information retention weights of sensor data at each time point, effectively filtering out sensor signal segments that are crucial for identifying safe production scenarios, such as sudden increases in gas concentration and abnormal temperature fluctuations, while suppressing redundant information such as stable background noise. Multi-layer GRU stacking further enhances the model's ability to model complex temporal patterns, ultimately outputting a sensing temporal feature vector that fully encodes the dynamic safety state information in the sensing sequence.

[0024] Since image feature maps and sensing temporal features originate from different modalities, their feature dimensions and distribution spaces differ significantly, making direct cross-modal fusion impossible. Therefore, fully connected projection layers are applied to the features of both modalities respectively, mapping them to a pre-defined unified dimensional space. Let the vector dimension obtained after global average pooling compression of the image feature map be... The output dimension of the sensing time-series features is The unified target dimension is The image projection layer is achieved through a parameter matrix. Linearly transform the image features to In 3D space, the sensing projection layer uses a parameter matrix. Linearly transform the sensing features to the same In 3D space, the image projection features are obtained respectively. With sensing projection features After linear transformation, the projection layer undergoes batch normalization to eliminate the numerical scale bias of different modal features, bringing the two projection features into similar numerical ranges and providing a stable numerical basis for subsequent cross-modal alignment.

[0025] In a unified dimensional space, a cross-modal similarity matrix is ​​calculated based on image projection features and sensor projection features to quantify the semantic correspondence between features of different modalities. For a given set of features... For a batch of samples, the image projection feature matrix is: The sensing projection feature matrix is Cross-modal similarity matrix The Middle Line number The elements of the column are obtained by calculating the normalized inner product: in Indicates the first Image projection features of each sample Indicates the first Sensing projection features of each sample, A larger value indicates that the two are semantically more similar. (Based on the similarity matrix) Construct contrast alignment loss Using a symmetric cross-entropy approach, the goal is to maximize the similarity between the image projection features and the sensor projection features of the same sample, while minimizing the similarity with the cross-modal features of other samples within the batch. Specifically, this is expressed as: in Temperature is a hyperparameter used to adjust the smoothness of the similarity distribution; a smaller value indicates a lower degree of smoothness. The value makes the model more sensitive to distinguishing between positive and negative sample pairs. This is to minimize... To optimize the objective, a gradient backpropagation mechanism is used to propagate the loss gradient back along the network computation graph layer by layer, synchronously updating the parameters of each projection layer in the image coding branch and the sensor coding branch. , The parameters of each coding layer drive the output features of the two branches to form a semantically consistent distribution structure in a unified dimensional space, so that image features and sensor features from the same construction site safety production scenario are close to each other in the feature space, while cross-modal features from different scenarios are far apart.

[0026] After completing cross-modal alignment training, the aligned image projection features With sensing projection features Element-level weighted fusion is performed to obtain multimodal joint latent features. Weighted fusion utilizes a learnable fusion weight vector. The integration process is as follows: in This represents element-wise multiplication. The value range of each element in the model is constrained to between 0 and 1 by the Sigmoid activation function, enabling the model to adaptively adjust the contribution ratio of the image modality and the sensing modality in each dimension of the feature space. Under good lighting conditions and high image quality in a construction site scene, The weights corresponding to the visual semantic dimension tend to be larger, giving higher fusion weights to image features; while in scenarios where image quality is compromised, such as dust obscuring the image or low visibility, the weights of the sensing modalities are correspondingly increased, ensuring that the multimodal joint latent features can effectively represent the comprehensive state of the safety production scenario under various complex working conditions. Fusion weight vector As network parameters, they participate in the overall training process, automatically learning the optimal modality fusion strategy through end-to-end optimization, and finally outputting multimodal joint latent features. This will serve as input to the subsequent generator network, driving the generation of synthetic multimodal data.

[0027] Conditional control information for safe production scenarios is extracted from multimodal sample data, and this information is encoded into conditional constraint vectors, including: Based on the types of construction behaviors and object interaction states in the construction site safety production environment, multiple safety production scenario categories are predefined, and each safety production scenario category is mapped to a unique category label vector; Spatial visual features are extracted from image modal data to identify the target object categories, spatial positional relationships and motion state information between target objects, and generate visual scene semantic features. Extract sensor semantic features from environmental sensor modal data, identify the time change trend of values ​​of each sensor channel, the degree of abnormal deviation of peak values ​​and the abnormal fluctuation of frequency, and generate sensor scene semantic features. Cross-modal feature fusion is performed between visual scene semantic features and sensor scene semantic features. The intermodal interaction weights between visual scene semantic features and sensor scene semantic features are calculated by an attention mechanism. Based on the intermodal interaction weights, the visual scene semantic features and sensor scene semantic features are weighted and fused into fused scene semantic features. The matching calculation is performed based on the semantic features of the fused scene and the predefined safety production scene category label vector to determine the probability of belonging to each safety production scene category. The safety production scene category with the highest belonging probability is selected as the condition control information, and the category label vector corresponding to the condition control information is used as the condition constraint vector.

[0028] In the context of construction site safety, the complexity of construction scenarios lies in the high diversity of various construction behaviors and object interactions. To enable generative adversarial networks (GANs) to generate corresponding synthetic multimodal data according to a specified scenario type, it is necessary to extract conditional control information that can characterize the semantics of the scenario from the original multimodal sample data and encode it into structured conditional constraint vectors for use by the generator network.

[0029] Predefined safety production scenario categories form the foundation for constructing conditional control information. Based on common construction activities in the construction site safety environment (such as high-altitude operations, hoisting, hot work, welding, earthwork excavation, and edge protection) and object interaction states (such as the proximity distance between personnel and machinery, whether personnel are wearing safety protective equipment, and the intersection status of vehicles and pedestrians), several safety production scenario categories are predefined. Each scenario category is assigned a unique category label vector, represented using either one-hot encoding or learnable embedding encoding. When the number of scenario categories is small, one-hot encoding maintains clear distinction between categories; when the number of scenario categories is large and there are semantic hierarchical relationships between categories, learnable embedding encoding can be used to map each category to a low-dimensional dense vector space, making semantically similar scenario categories closer in the vector space. The dimension of the category label vector is consistent with the dimension of the subsequent conditional constraint vectors so that it can be directly input into the conditional modulation layer of the generator network.

[0030] When extracting spatial visual features from image modal data, a convolutional backbone network with multi-scale receptive fields is used to extract features layer by layer from the input image. This captures low-level visual information such as edges and textures at shallow layers and semantically rich high-level features at deeper layers. For object category recognition, a target detection branch locates and classifies typical objects in the image, such as people, safety helmets, reflective vests, tower cranes, excavators, and scaffolding, outputting a category confidence vector for each object. For spatial relationship analysis, based on the bounding box coordinates of each object, the relative distance, overlap, and azimuth information between objects are calculated, quantifying these spatial relationships into relationship feature vectors. For motion state information extraction, when the input is a continuous frame sequence, the motion speed and direction of each object are calculated using optical flow estimation or frame differencing methods; when the input is a single-frame image, a pose estimation network infers the limb posture state of a person (e.g., standing, bending, climbing), thereby indirectly representing the motion state. After concatenating the target category features, spatial relationship features, and motion state features, and mapping them through a fully connected layer, a visual scene semantic feature vector is obtained, denoted as . Its dimensions are .

[0031] When extracting semantic features from environmental sensing modal data, feature mining is performed from three dimensions for the time-series data of each sensing channel. Regarding temporal trends, the moving average and linear trend slope are calculated for the time-series signal of each sensing channel to determine whether the signal exhibits a continuously rising, continuously falling, or stable fluctuation trend, encoding the trend category as a discrete feature. Regarding peak anomaly deviation, the deviation factor of the maximum and minimum values ​​of each sensing channel within the observation window relative to the historical baseline mean is calculated. When the deviation factor exceeds a preset threshold, it is marked as an abnormal peak event, and the normalized value of the deviation factor is used as the anomaly degree feature. Regarding frequency anomaly fluctuation features, a Fast Fourier Transform is performed on the time-series signal of each sensing channel to extract the main frequency components and their energy proportions, identifying abnormal high-frequency vibration components or low-frequency periodic anomalies in the signal, and using the frequency domain feature vector as the frequency fluctuation feature. After concatenating the three types of sensing features, time-series modeling is performed through a bidirectional long short-term memory network or a one-dimensional convolutional network to output a semantic feature vector of the sensing scene, denoted as . Their dimensions are the same It is consistent with the dimension of the semantic feature vector of the visual scene to facilitate subsequent cross-modal fusion calculations.

[0032] Cross-modal fusion of visual scene semantic features and sensor scene semantic features is achieved through an attention mechanism. Specifically, it will... and Using these as the query vector and key-value vector respectively, calculate the intermodal interaction weights between them. The calculation process is as follows: The query matrix is ​​obtained after linear transformation. ,Will The bond matrices are obtained after linear transformation. AND-value matrix The interaction weights of the visual modality to the sensing modality are obtained by scaling dot product attention. Similarly, As a query vector As a key-value vector, the interaction weights of the sensing modality to the visual modality are calculated. .based on Weighted aggregation of semantic features of the sensing scene yields a visually enhanced semantic representation of the sensing scene. ;based on Weighted aggregation of semantic features of the visual scene yields a sensor-enhanced visual semantic representation. .Will and After concatenation, the data is compressed to the target dimension via a linear projection layer to obtain the fused scene semantic feature vector. This bidirectional interactive attention design enables the semantic information of the visual modality and the sensing modality to complement each other, avoiding the bias problem of a single modality dominating the fusion result. It is particularly effective in scenarios where there are abnormal linkages between human behavior and sensing values.

[0033] Based on the semantic features of the fusion scene The algorithm matches and calculates the probability of belonging to each safety production scenario category by matching the vector with predefined safety production scenario category labels. The input scenario classification head consists of several fully connected layers and batch normalization layers. Finally, the softmax activation function is used to output the probability distribution vector of each safety production scenario category. ,in The The element represents the current multimodal sample belonging to the th element. The probability of each safety production scenario category. (Selection) The category with the highest probability value is used as the conditional control information for the current sample, and the category label vector corresponding to that category is directly output as the conditional constraint vector. When one-hot encoding is used, the conditional constraint vector is a sparse vector with a value of 1 only at the corresponding category position and a value of 0 at all other positions; when learnable embedding encoding is used, the conditional constraint vector is the corresponding row vector retrieved from the embedding matrix. The conditional constraint vector and the multimodal joint latent features are concatenated and input into the generator network, enabling the generator network to generate synthetic multimodal data with scene semantic consistency under the constraints of a specified safety production scenario category, thereby achieving targeted data augmentation for specific scarce hazardous scenarios.

[0034] Encoding conditional control information into a conditional constraint vector also includes: During the training of the multimodal generative adversarial network, the discriminator network outputs the data authenticity judgment result, and additionally outputs the scenario discrimination classification result of the safety production scenario to which the synthetic multimodal data belongs; The scenario condition fit index is calculated based on the degree of difference between the scenario discrimination and classification results of each synthetic multimodal data and the target safety production scenario category corresponding to the input condition constraint vector. For the synthetic multimodal data belonging to each safety production scenario category, the average scenario condition fit index of all synthetic multimodal data under that category is calculated. Safety production scenario categories with an average scenario condition fit index lower than the preset fit benchmark are marked as difficult-to-generate scenario categories. For safety production scenario categories that are marked as difficult to generate, the corresponding scenario weight coefficients are dynamically increased in the next round of adversarial training, and the updated scenario weight coefficients are re-encoded into the condition constraint vector of the next training round. A closed-loop adaptive refinement mechanism is formed, in which the conditional constraint vector is driven by discriminator feedback between adversarial training rounds, enabling the generator network to automatically enhance the fidelity of generating difficult scene categories in subsequent training rounds.

[0035] During the training of a multimodal generative adversarial network (GAN), the discriminator network undertakes a dual output task: on the one hand, it outputs a binary discrimination result indicating whether the input data is a real sample; on the other hand, it additionally outputs a classification prediction result for the safety production scenario category to which the synthesized multimodal data belongs. This dual-output structure is achieved by branching two parallel fully connected output paths at the top level of the discriminator network. The authenticity discrimination branch outputs a single scalar probability value, while the scenario classification branch outputs a probability distribution vector with a dimension equal to the total number of safety production scenario categories. The scenario classification branch uses the softmax activation function, ensuring that the sum of all components of the output vector is 1, and each component corresponds to a probability estimate of the synthetic sample belonging to a specific safety production scenario category. The two branches share the low-level feature extraction part of the discriminator network, ensuring that the scene semantic understanding ability and authenticity discrimination ability develop synergistically within the same feature space, avoiding feature redundancy caused by independent branches.

[0036] The scene condition fit index is calculated based on the consistency measure between the output of the discriminator's scene classification branch and the condition constraint vector corresponding to the current synthetic sample. Let the predicted probability distribution vector output by the synthetic multimodal data on the scene classification branch be... The dimension is the total number of categories of safe production scenarios. The one-hot label vector of the target safety production scenario category encoded by the corresponding condition constraint vector is: The scene condition fit index of a single synthetic sample Defined as the negative of the cross-entropy loss between the predicted probability distribution and the target category label, i.e. ,in This is the label value corresponding to the target category (1 for the target category and 0 for the rest). The smaller the value, the more the scene a synthetic sample belongs to matches its generation conditions, meaning the generation quality is higher at the scene condition level. The larger the value, the weaker the generator network's ability to control conditions in that scene category, and the more obviously the generated synthetic data deviates from the target scene.

[0037] After each training round, for all synthetic multimodal data belonging to each safety production scenario category, the mean value of the scenario condition fit index of all synthetic samples under that category is calculated. ,in This is the scene category index, with a value range of [value range missing]. to .Will Matching the preset benchmark threshold Comparison: If a certain scene category of Exceed If the category is not specified, it will be marked as a difficult scenario category and added to the difficult category set. . The initial value can be set based on the statistical distribution of the fit index of each category in the early stage of training. It is usually taken as a certain multiple of the initial mean of all categories. As training progresses, an exponential moving average method can also be used to adjust the initial value. It is dynamically updated to adapt to the improvement of overall generation capabilities during training.

[0038] For safety production scenario categories marked as difficult to generate, the corresponding scenario weight coefficients are dynamically increased in the next training round. Let the... Scene categories in rounds The corresponding scene weight coefficient is Then in the first In each round, for those belonging to the set of difficult categories The weights of each category are updated according to the following rules: ,in This is the weight gain adjustment coefficient, which controls the magnitude of the weight increase. This is the difference between the mean fit of this category in the current round and the fit benchmark, i.e. For scene categories that do not belong to the set of difficult categories, their weight coefficients remain unchanged or are slightly reduced at a certain decay rate to maintain the relative balance between the weights of each category and avoid the degradation of generation ability due to excessively low weights in non-difficult categories. After normalization, the updated weight coefficients of all scene categories are re-encoded into the conditional constraint vectors for the next training round. This ensures that when the generator network samples training batches, the conditional constraint vectors from difficult scene categories are sampled with a higher probability, thereby automatically strengthening the generation training intensity for those scenes.

[0039] The update method for the condition constraint vector is specifically manifested as follows: the scene weight coefficient vector is updated. As the sampling probability distribution of the conditional constraint vectors for each category, during the construction of each training batch, weighted sampling is performed from the conditional constraint vector library for each category according to this distribution, so that the conditional constraint vectors of the difficult scene category occupy a higher proportion in the batch. At the same time, for the difficult scene category, in addition to the scene category encoding component, its conditional constraint vector can also include a component derived from the current round. The difficulty of the normalized calculation is emphasized by the scalar. This is used to further enhance the generator's sensitivity to the scene conditions at the vector level. The calculation method is as follows ,when This indicates that the category is in a difficult state. It is concatenated to the end dimension of the corresponding conditional constraint vector and used as an additional conditional emphasis signal input to the generator network.

[0040] The aforementioned mechanism forms a closed-loop adaptive refinement process for the conditional constraint vectors, driven by discriminator feedback across adversarial training rounds. After each training round, the discriminator outputs the scene classification results for each synthesized sample. These results, after calculating the fit index and identifying the hard category, directly drive changes in the sampling weights and encoding content of the conditional constraint vectors in the next round, thus affecting the distribution of training signals received by the generator network in the next round. This cross-round feedback mechanism automatically identifies bottleneck scenarios without manual intervention and allocates training resources to them, ensuring that the generator network's ability to generate various safe production scenarios tends to be balanced throughout the training process.

[0041] In real-world construction site safety scenarios, the data distribution across different scenario categories is inherently unbalanced. For example, the number of original samples for hazardous scenarios such as high-altitude operations and confined space operations is often far fewer than for conventional ground operations. This results in the generative adversarial network's ability to control the conditions of rare hazardous scenarios being significantly weaker than that of common scenarios in the early stages of training. Through the aforementioned adaptive refinement mechanism, rare hazardous scenario categories are identified as difficult-to-generate scenario categories early in the training process, and their corresponding scenario weight coefficients are subsequently enhanced. This allows the generator network to achieve more targeted optimization in subsequent training, thereby generating more high-quality synthetic samples of rare hazardous scenarios in the augmented dataset. This compensates for the deficiencies of the original dataset in these categories, improving the class coverage balance and overall data quality of the final multimodal training dataset.

[0042] A multimodal generative adversarial network (GAN) is constructed, comprising a generator network and a discriminator network. The joint latent features of the multimodal data and the conditional constraint vectors are input into the generator network to generate synthetic multimodal data, including: The joint latent features of the multimodal modes are concatenated with the conditional constraint vector to obtain the joint input vector. A generator network with an encoder-decoder structure is constructed. The encoder part consists of multiple stacked fully connected layers. The joint input vector is subjected to layer-by-layer nonlinear transformation, and intermediate features are extracted and generated layer by layer. The decoding part consists of multiple stacked upsampling layers, which upsample and reconstruct the intermediate features layer by layer. During the upsampling process of the decoding part, synthetic image modal data and synthetic environment sensing modal data are generated respectively. In each upsampling layer of the decoding part, a cross-modal consistency constraint unit is set. The cross-modal consistency constraint unit extracts the intermediate feature map of the synthetic image modal data generated by the current upsampling layer and the intermediate feature sequence of the synthetic environment sensing modal data, respectively, calculates the modal mutual information value between the intermediate feature map and the intermediate feature sequence, constructs the inter-layer cross-modal consistency loss term of each upsampling layer based on the modal mutual information value, and superimposes the inter-layer cross-modal consistency loss term of each upsampling layer with the generative adversarial loss to obtain the comprehensive objective function of the generator network. The generator network parameters are optimized by minimizing the comprehensive objective function, and the synthetic image modal data and the synthetic environment sensing modal data are constrained to maintain cross-modal consistency at the semantic level.

[0043] Before entering the generator network, the multimodal joint latent features and conditional constraint vectors need to undergo a vector concatenation operation to merge them along the feature dimension, forming a joint input vector. Specifically, the multimodal joint latent features... Cross-modal alignment and fusion have already been completed in the preceding module, carrying joint semantic information of vision and sensing in the construction site safety production scenario; the condition constraint vector compactly encodes condition control information such as the category attributes, hazard level, and environmental state of the target generation scene. The two are directly concatenated along the feature dimension to obtain the joint input vector. Its dimension is the sum of the dimension of the joint latent features of the multimodal data and the dimension of the conditional constraint vector. This concatenation method enables the generator network to be aware of the conditional constraint information throughout the generation process, avoiding the dilution of conditional information in the deep network, thereby ensuring that the final generated synthetic multimodal data is consistent with the target conditions in terms of scene semantics.

[0044] The generator network employs a symmetric encoder-decoder structure. The encoder part consists of multiple fully connected layers stacked sequentially, each followed by a batch normalization layer and a non-linear activation function (such as LeakyReLU) on the joint input vector. A layer-by-layer nonlinear transformation is performed. As the number of layers increases, the feature dimension gradually expands. The network abstracts the intermediate semantic representations required for generation layer by layer in the high-dimensional latent space. Let the intermediate features generated by the final encoding layer be denoted as... The number of layers and neurons in each layer of the encoding part is configured according to the data scale and the complexity of the generation task, typically using 4 to 6 fully connected layers, with the number of neurons gradually increasing from the input dimension to the target hidden layer dimension. The decoding part consists of multiple upsampling layers stacked together. As input, through layer-by-layer upsampling reconstruction, the compact intermediate features are progressively unfolded into a multimodal output with full spatial resolution and temporal length. The decoding section structurally employs two parallel decoding branches: the image decoding branch is responsible for layer-by-layer upsampling reconstruction of the synthesized image modal data, and the sensor decoding branch is responsible for layer-by-layer upsampling reconstruction of the synthesized environmental sensor modal data. The two branches interact and constrain each other across upsampling layers through cross-modal consistency constraint units, ultimately outputting the synthesized image modal data respectively. Synthetic environmental sensing modal data .

[0045] The cross-modal consistency constraint unit is the core innovative mechanism of this method, and it is set in each upsampling layer of the decoding part. In the... Each upsampling layer, the image decoding branch outputs the intermediate feature map of the current layer. Its spatial size gradually increases with the number of upsampling layers; the sensing decoding branch outputs the intermediate feature sequence of the current layer. Its time series length gradually extends with the increase of the number of upsampling layers. The cross-modal consistency constraint unit respectively... Perform global average pooling to obtain a compact representation vector of the current layer's image modality. ;right Perform a time-series global average pooling operation to obtain a compact representation vector of the current layer's sensing mode. .based on and Calculate the modal mutual information value between the intermediate representations of two modes. .

[0046] Modal mutual information values ​​are calculated using a neural network-based mutual information estimation method (such as the MINE estimator). Specifically, a lightweight mutual information estimation network is set up to... and The concatenated vector is used as the input of the positive sample pair, with After random shuffling The concatenated vector is used as input for the negative sample pair, and the lower bound of mutual information is estimated by maximizing the expected score difference between the positive and negative sample pairs. Mutual information value The larger the value, the richer the semantic information shared between the intermediate features of the image modality and the intermediate features of the sensing modality in the current upsampling layer, and the stronger the cross-modal consistency.

[0047] Based on the modal mutual information values ​​calculated from each upsampling layer, an inter-layer cross-modal consistency loss term is constructed. It is defined as taking the negative of the mutual information value, that is By minimizing this loss term, the network is driven to maximize cross-modal mutual information among intermediate features of each layer, thereby continuously imposing cross-modal consistency constraints at each upsampling stage of the generation process. The total cross-modal consistency loss is obtained by weighted summation of the inter-layer cross-modal consistency loss terms of all upsampling layers. ,Right now ,in This represents the total number of upsampling layers in the decoding section. For the first The interlayer loss weight coefficients can be configured differently according to the feature resolution or semantic abstraction level of each layer. Generally, shallow layers (low resolution, high semantic abstraction layers) are given larger weights, while deep layers (high resolution, low semantic abstraction layers) are given smaller weights, so as to dominate the consistency constraint at the semantic level rather than the pixel level.

[0048] The overall objective function of the generator network consists of the superposition of generative adversarial loss and total cross-modal consistency loss. The generative adversarial loss drives the generator network to deceive the discriminator network, making the synthesized multimodal data statistically approximate the real multimodal sample data. The total cross-modal consistency loss imposes additional constraints on multiple intermediate layers in the generation process, ensuring that the synthesized image modal data and the synthesized environment sensing modal data maintain cross-modal consistency at the semantic level. Overall Objective Function Represented as ,in To generate adversarial losses, This is the weight adjustment coefficient for the cross-modal consistency loss, used to balance the relative strength between the adversarial training objective and the cross-modal consistency constraint objective. In actual training, The value of is tuned using the generation quality evaluation metric on the validation set, and is usually set between 0.1 and 1.0.

[0049] To minimize the comprehensive objective function To achieve this, the generator network parameters are updated using the backpropagation algorithm. In construction site safety scenarios, image modal data typically contains visual information such as the location, posture, and protective equipment status of construction workers, while sensor modal data contains time-series readings of physical quantities such as ambient temperature, noise intensity, and toxic gas concentration. If the two types of modal data lack consistency constraints during generation, semantic contradictions can easily arise, such as synthesized images showing high-temperature work scenarios while synthesized sensor data shows normal temperature readings. This leads to the introduction of incorrect training signals into the generated augmented data. The cross-modal consistency constraint unit fundamentally suppresses such cross-modal semantic inconsistencies by continuously applying mutual information maximization constraints across multiple upsampling layers. This ensures that the final generated synthetic multimodal data maintains an inherent consistency between image semantics and sensor physical quantities that conforms to the physical laws of real construction site scenarios, thereby improving the overall quality of the augmented multimodal dataset and the training effect of downstream safety perception models.

[0050] The multimodal joint latent features and conditional constraint vectors are input into the generator network to generate synthetic multimodal data, which also includes: Before upsampling reconstruction in each upsampling layer, the conditional constraint vector is input into the conditional modulation network corresponding to the upsampling layer. The conditional modulation network is composed of multiple fully connected layers stacked together, which performs nonlinear feature transformation on the input conditional constraint vector. The conditional modulation network outputs the channel-wise scaling factor vector and the channel-wise bias vector corresponding to the upsampling layer. The channel-wise scaling factor vector is aligned with the channel dimension of the intermediate feature generated by the current layer of the upsampling layer, and the channel-wise bias vector is aligned with the channel dimension of the intermediate feature generated by the current layer of the upsampling layer. The feature values ​​of each channel in the current layer generated intermediate features of the upsampling layer are multiplied by the scaling factor corresponding to the channel according to the corresponding channel position and then added to the bias value corresponding to the channel. This completes the channel-wise adaptive conditional modulation operation of the conditional constraint vector on the layer generated intermediate features of the upsampling layer, and the conditionally modulated layer generated intermediate features are obtained. Each upsampling layer uses its conditionally modulated layer to generate intermediate features to continue subsequent upsampling reconstruction calculations, so that the conditional constraint vectors participate in the conditional regulation of the generation and reconstruction process at each upsampling level of the generator network, thereby achieving layer-by-layer fine-grained conditional control over the safety production scenario to which the synthesized multimodal data belongs.

[0051] In the decoding and reconstruction stage of the generator network, the conditional constraint vector not only participates in feature fusion at the initial input but also needs to be continuously involved at each upsampling layer to apply fine-grained conditional control to the intermediate generated features of that layer. To this end, an independent conditional modulation network is configured for each upsampling layer. This network consists of multiple fully connected layers stacked sequentially, with a nonlinear activation function (such as LeakyReLU) inserted between every two adjacent fully connected layers. This allows the network to perform sufficient nonlinear feature transformation on the input conditional constraint vector, thereby mapping high-level semantic conditional information to modulation parameters that match the feature space of the current upsampling layer. The input to the conditional modulation network is always the same conditional constraint vector, but the conditional modulation network corresponding to each upsampling layer has its own independent parameters. Therefore, the modulation networks at different levels extract their respective required modulation information from the conditional constraint vector, achieving hierarchical adaptive conditional expression.

[0052] The output of the conditional modulation network consists of two vectors: a channel-wise scaling factor vector and a channel-wise bias vector. Let the number of channels in the current upsampling layer that generate intermediate features be... Then the scaling factor vector output by the conditional modulation network corresponding to this layer With bias vector The dimensions are all with Alignment. Specifically, the last layer of the fully connected layer stack sets the output dimension to... Then the output is split along the channel dimension into the first... After peacekeeping Dimensions, respectively, serve as scaling factor vectors. With bias vector This dimension alignment design ensures that subsequent channel-by-channel modulation operations are mathematically rigorous and that there is no issue of dimension mismatch.

[0053] For the intermediate features generated by the current upsampling layer, let their shape be... (Image decoding branch) or (Sensor Decoding Branch), in which , These represent the spatial height and width of the feature map, respectively. The time length of the sensing feature sequence is given. The channel-by-channel adaptive conditional modulation operation is performed as follows: For the first... One channel ( Extract all feature values ​​corresponding to that channel, and multiply each feature value by the scaling factor vector. The Middle element In addition to the bias vector The Middle element This completes the affine transformation of the channel's features. For all After performing the above operations on each channel, the conditionally modulated intermediate features of the layer generation are obtained. Its shape remains consistent with that before modulation. The above modulation process can be represented as follows: ,in For the first modulation The eigenvalues ​​of the channel, For the modulated first The characteristic values ​​of the channel.

[0054] This channel-wise affine modulation design originates from the core ideas of methods such as Spatial Adaptive Normalization (SPADE) and Conditional Batch Normalization (CBN). However, it is further extended here to each upsampling level in a multimodal generation scenario, and the modulation parameters are dynamically generated entirely by the conditional constraint vectors, rather than using fixed learnable parameters. This means that for the conditional constraint vectors of different safety production scenarios, the conditional modulation network will output different scaling factors and biases, thereby allowing the generator network to apply different channel-level modulations to the same intermediate feature under different scenario conditions, guiding the generation process to shift towards the feature distribution of the target scenario. For example, when the conditional constraint vector corresponds to the "high-altitude operation" scenario, the output of the conditional modulation network is... and The visual texture channel and sensor signal channel related to high-altitude operation scenarios will be enhanced; when the condition constraint vector corresponds to the "limited space operation" scenario, the modulation parameters will instead highlight the feature channel related to the confined environment, so that the generated synthetic multimodal data closely resembles the real statistical characteristics of the target scene in terms of visual appearance and sensor numerical distribution.

[0055] At the engineering implementation level, the number of fully connected layers in a conditional modulation network is typically set to 2 to 4, and the hidden layer dimension can be set to 2 to 4 times the dimension of the conditional constraint vector to ensure sufficient nonlinear mapping capability. For the image decoding branch and the sensing decoding branch, each upsampling layer can share the same conditional modulation network (parameter sharing) or be configured with independent conditional modulation networks (parameter independence). The former has fewer parameters, while the latter is more targeted for conditional modulation of the two modalities. In multimodal data generation tasks in actual construction site safety production scenarios, since the image modality and the sensing modality differ significantly in feature space structure, it is recommended to configure independent conditional modulation networks for the two decoding branches to allow the intermediate features generated for each modality to undergo more targeted conditional modulation.

[0056] After performing conditional modulation, each upsampling layer generates intermediate features using the conditionally modulated layer. The original intermediate features are replaced, and upsampling reconstruction calculations are performed on this layer (such as transposed convolution, bilinear interpolation upsampling, or deconvolution). Subsequent layers follow the same principle. Since each upsampling layer independently performs conditional modulation, the semantic information of the conditional constraint vectors continuously participates in the reconstruction process at each stage of the generator network—shallow, middle, and deep—rather than being injected only once at the input. The conditional modulation of shallow upsampling layers mainly affects the low-frequency global structure (such as the overall layout of the scene and the macroscopic trend of the sensor signal), the conditional modulation of middle upsampling layers affects mid-frequency texture details (such as the appearance features of personnel and equipment and the local fluctuation patterns of the sensor signal), and the conditional modulation of deep upsampling layers affects high-frequency details (such as edge sharpness and the instantaneous noise characteristics of the sensor signal). This achieves layer-by-layer fine-grained conditional control from coarse to fine, ensuring that the final generated synthetic multimodal data conforms to the conditional constraints of the target safe production scenario across multiple frequency bands.

[0057] During adversarial training, the parameters of the conditional modulation network participate in gradient backpropagation and optimization updates along with the remaining parameters of the generator network, eliminating the need for a separate training strategy. The gradients generated by the discriminator network based on the authenticity judgment results of the synthetic multimodal data and the scene condition fit evaluation results are backpropagated to the conditional modulation network through the upsampling reconstruction layer. This prompts the conditional modulation network to continuously optimize its mapping relationship from the conditional constraint vector to the modulation parameters. Ultimately, this enables the generator network to accurately respond to the scene semantic information carried by the conditional constraint vector at each level, generating high-quality, high-scene-fidelity synthetic multimodal data. This provides a stable and reliable data source for enhancing multimodal training data in a construction site safety production environment.

[0058] A second aspect of the present invention provides a multimodal data GAN generative physics augmentation system that integrates physical laws, comprising: The feature alignment unit is used to acquire multimodal sample data under the construction site safety production environment, including image modal data and environmental sensing modal data. It constructs a multimodal coding network, extracts features from each modal sample data, maps each modal feature to a unified latent feature space, aligns the spatial representations of different modal features, and obtains multimodal joint latent features. The condition generation unit is used to extract condition control information of safety production scenarios from multimodal sample data, encode the condition control information into condition constraint vectors, construct a multimodal generative adversarial network containing a generator network and a discriminator network, and input the multimodal joint latent features and condition constraint vectors into the generator network to generate synthetic multimodal data. The adversarial training unit is used to input synthetic multimodal data and real multimodal sample data into the discriminator network, calculate the adversarial loss based on the discrimination results, and perform adversarial training by alternately optimizing the parameters of the generator network and the discriminator network until the training converges. The data augmentation unit is used to generate augmented multimodal data for a specified scenario using the generator network after training convergence, to evaluate the quality of the augmented multimodal data, to select augmented multimodal data that meets the quality threshold, and to merge the selected augmented multimodal data with the original multimodal sample data to construct an augmented multimodal training dataset.

[0059] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0060] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0061] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating physics-enhanced data using a multi-modal data GAN that incorporates physical laws, the method comprising: include: Multimodal sample data under the construction site safety production environment is acquired, including image modal data and environmental sensing modal data. A multimodal coding network is constructed, and features are extracted from each modal sample data. The features of each modality are mapped to a unified latent feature space, and the spatial representations of different modal features are aligned to obtain multimodal joint latent features. Conditional control information for safe production scenarios is extracted from multimodal sample data. This information is then encoded into conditional constraint vectors. A multimodal generative adversarial network (GAN) containing a generator network and a discriminator network is constructed. The multimodal joint latent features and conditional constraint vectors are input into the generator network to generate synthetic multimodal data. Synthetic multimodal data and real multimodal sample data are input into the discriminator network. The adversarial loss is calculated based on the discrimination results. Adversarial training is carried out by alternately optimizing the parameters of the generator network and the discriminator network until the training converges. The generator network, after training and convergence, generates enhanced multimodal data for a specified scenario. The quality of the enhanced multimodal data is evaluated, and the enhanced multimodal data that meets the quality threshold is selected. The selected enhanced multimodal data is then merged with the original multimodal sample data to construct an enhanced multimodal training dataset.

2. The method of claim 1, wherein, A multimodal coding network is constructed to extract features from sample data of each modality. These features are then mapped to a unified latent feature space, aligning the spatial representations of different modalities to obtain joint latent features of the multimodality, including: An image coding branch and a sensing coding branch are constructed. The image coding branch consists of multiple convolutional layers and pooling layers stacked alternately. Multi-layer convolutional feature extraction is performed on the image modal data to capture the hierarchical visual representation from local texture to global semantics of the image layer by layer, and the image feature map is output. The sensing coding branch is composed of multiple gated loop unit layers stacked together. It extracts temporal features from environmental sensing modal data at each time. The gated loop unit layer uses update gate and reset gate to calculate the information retention weight of the sensing data at the corresponding time. Based on the information retention weight, it filters key information that is effective in identifying safe production scenarios and outputs sensing temporal features. The image feature map and the sensing time-series feature are respectively mapped to a preset unified dimension space through a fully connected projection layer to obtain the image projection feature and the sensing projection feature. A cross-modal similarity matrix is ​​calculated based on image projection features and sensor projection features. A contrast alignment loss is constructed through the cross-modal similarity matrix. With the goal of minimizing the contrast alignment loss, the projection parameters and coding parameters of each coding branch are updated using gradient backpropagation to make the semantic distribution of image projection features and sensor projection features consistent in a preset unified dimensional space. The aligned image projection features and sensor projection features are then fused element-wise to obtain multimodal joint latent features.

3. The method of claim 1, wherein, Conditional control information for safe production scenarios is extracted from multimodal sample data, and this information is encoded into conditional constraint vectors, including: Based on the types of construction behaviors and object interaction states in the construction site safety production environment, multiple safety production scenario categories are predefined, and each safety production scenario category is mapped to a unique category label vector; Spatial visual features are extracted from image modal data to identify the target object categories, spatial positional relationships and motion state information between target objects, and generate visual scene semantic features. Sensing semantic features are extracted from environmental sensing modal data to identify the temporal variation trend, peak abnormal deviation degree, and frequency abnormal fluctuation characteristics of values ​​in each sensing channel, generating sensing scene semantic features. The visual scene semantic features and the sensing scene semantic features are fused across modal features. The intermodal interaction weight between the visual scene semantic features and the sensing scene semantic features is calculated by an attention mechanism. Based on the intermodal interaction weight, the visual scene semantic features and the sensing scene semantic features are weighted and fused into a fused scene semantic feature. The matching calculation is performed based on the semantic features of the fused scene and the predefined safety production scene category label vector to determine the probability of belonging to each safety production scene category. The safety production scene category with the highest belonging probability is selected as the condition control information, and the category label vector corresponding to the condition control information is used as the condition constraint vector.

4. The method of claim 3, wherein, Encoding conditional control information into a conditional constraint vector also includes: During the training of the multimodal generative adversarial network, the discriminator network outputs the data authenticity judgment result, and additionally outputs the scenario discrimination classification result of the safety production scenario to which the synthetic multimodal data belongs; Based on the degree of difference between the scene discrimination and classification results of each synthetic multimodal data and the target safety production scene category corresponding to the input condition constraint vector, the scene condition fit index is calculated; for synthetic multimodal data belonging to each safety production scene category, the average scene condition fit index of all synthetic multimodal data under that category is calculated, and the safety production scene category with the average scene condition fit index lower than the preset fit benchmark is marked as the difficult scene category to generate. For safety production scenario categories that are marked as difficult to generate, the corresponding scenario weight coefficients are dynamically increased in the next round of adversarial training, and the updated scenario weight coefficients are re-encoded into the condition constraint vector of the next training round. A closed-loop adaptive refinement mechanism is formed, in which the conditional constraint vector is driven by discriminator feedback between adversarial training rounds, enabling the generator network to automatically enhance the fidelity of generating difficult scene categories in subsequent training rounds.

5. The method of claim 1, wherein, A multimodal generative adversarial network (GAN) is constructed, comprising a generator network and a discriminator network. The joint latent features of the multimodal data and the conditional constraint vectors are input into the generator network to generate synthetic multimodal data, including: The joint latent features of the multimodal modes are concatenated with the conditional constraint vector to obtain the joint input vector. A generator network with an encoder-decoder structure is constructed. The encoder part consists of multiple stacked fully connected layers, which perform layer-by-layer nonlinear transformation on the joint input vector and extract intermediate features layer by layer. The decoder part consists of multiple stacked upsampling layers, which perform layer-by-layer upsampling and reconstruction on the generated intermediate features. During the upsampling process of the decoder part, synthetic image modal data and synthetic environment sensing modal data are generated respectively. In each upsampling layer of the decoding part, a cross-modal consistency constraint unit is set. The cross-modal consistency constraint unit extracts the intermediate feature map of the synthetic image modal data generated by the current upsampling layer and the intermediate feature sequence of the synthetic environment sensing modal data, respectively, calculates the modal mutual information value between the intermediate feature map and the intermediate feature sequence, constructs the inter-layer cross-modal consistency loss term of each upsampling layer based on the modal mutual information value, and superimposes the inter-layer cross-modal consistency loss term of each upsampling layer with the generative adversarial loss to obtain the comprehensive objective function of the generator network. The generator network parameters are optimized by minimizing the comprehensive objective function, and the synthetic image modal data and the synthetic environment sensing modal data are constrained to maintain cross-modal consistency at the semantic level.

6. The method of claim 5, wherein, The multimodal joint latent features and conditional constraint vectors are input into the generator network to generate synthetic multimodal data, which also includes: Before upsampling reconstruction in each upsampling layer, the conditional constraint vector is input into the conditional modulation network corresponding to the upsampling layer. The conditional modulation network is composed of multiple fully connected layers stacked together, which performs nonlinear feature transformation on the input conditional constraint vector. The conditional modulation network outputs the channel-wise scaling factor vector and the channel-wise bias vector corresponding to the upsampling layer. The channel-wise scaling factor vector is aligned with the channel dimension of the intermediate feature generated by the current layer of the upsampling layer, and the channel-wise bias vector is aligned with the channel dimension of the intermediate feature generated by the current layer of the upsampling layer. The feature values ​​of each channel in the current layer generated intermediate features of the upsampling layer are multiplied by the scaling factor corresponding to the channel according to the corresponding channel position and then added to the bias value corresponding to the channel. This completes the channel-wise adaptive conditional modulation operation of the conditional constraint vector on the layer generated intermediate features of the upsampling layer, and the conditionally modulated layer generated intermediate features are obtained. Each upsampling layer uses its conditionally modulated layer to generate intermediate features to continue subsequent upsampling reconstruction calculations, so that the conditional constraint vectors participate in the conditional regulation of the generation and reconstruction process at each upsampling level of the generator network, thereby achieving layer-by-layer fine-grained conditional control over the safety production scenario to which the synthesized multimodal data belongs.

7. A multimodal data GAN generative physics augmentation system that integrates physical laws, used to implement the method as described in any one of claims 1-6, characterized in that, include: The feature alignment unit is used to acquire multimodal sample data under the construction site safety production environment, including image modal data and environmental sensing modal data. It constructs a multimodal coding network, extracts features from each modal sample data, maps each modal feature to a unified latent feature space, aligns the spatial representations of different modal features, and obtains multimodal joint latent features. The condition generation unit is used to extract condition control information of safety production scenarios from multimodal sample data, encode the condition control information into condition constraint vectors, construct a multimodal generative adversarial network containing a generator network and a discriminator network, and input the multimodal joint latent features and condition constraint vectors into the generator network to generate synthetic multimodal data. The adversarial training unit is used to input synthetic multimodal data and real multimodal sample data into the discriminator network, calculate the adversarial loss based on the discrimination results, and perform adversarial training by alternately optimizing the parameters of the generator network and the discriminator network until the training converges. The data augmentation unit is used to generate augmented multimodal data for a specified scenario using the generator network after training convergence, to evaluate the quality of the augmented multimodal data, to select augmented multimodal data that meets the quality threshold, and to merge the selected augmented multimodal data with the original multimodal sample data to construct an augmented multimodal training dataset.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images