Text vision multi-modal data synthesis method and device for coal industry risk scene

Abstract

The invention relates to the technical field of safety monitoring and intellectualization in the coal industry, and discloses a text vision multi-modal data synthesis method and device for a risk scene in the coal industry, and the method comprises the steps: constructing a semantic mapping rule, and outputting a mapping function from a text physical quantity parameter to a vision generation control variable; performing enhancement preprocessing based on a multi-scale Retinex theory on the acquired original image; constructing an industry adaptive cross-modal generation model; performing iterative training on the model based on a mixed constraint loss function including industry rule constraints and physical rationality constraints; migration enhancement is executed for a rare scene, and basic model parameters are loaded and introduced into an external knowledge base for fine tuning; and finally, semantic and physical dual quality verification is performed on the synthesized data and formatted output is performed. According to the method, through double constraints of physics and industry rules, the problems of high risk in the coal industry, scarcity of rare scene samples and logic missing of generated data are solved, and the sense of reality of data and the adaptability of application are improved.

Description

Technical Field

[0001] This invention relates to the field of safety monitoring and intelligent technology in the coal industry, specifically to a method and apparatus for synthesizing textual and visual multimodal data in risk scenarios in the coal industry. Background Technology

[0002] As a crucial guarantee of my country's energy security, the safety monitoring and emergency response during coal production are of paramount importance. With the advancement of intelligent construction in the coal industry, technologies such as AI-based visual analysis and intelligent scheduling decision-making are gradually becoming core means of identifying high-risk underground scenarios such as gas exceedances, roof collapses, equipment failures, and personnel violations. However, the performance improvement of these intelligent technologies heavily relies on large-scale, high-quality, and accurately labeled text-visual multimodal data, but in practical industry applications, obtaining high-quality training data faces significant challenges.

[0003] First, data acquisition in high-risk underground scenarios faces extremely high natural barriers. Because disasters such as gas explosions, water inrushes, and roof collapses are low-probability, unpredictable events, real samples are already extremely scarce. Furthermore, the harsh underground environment, complex lighting conditions, and stringent requirements for the explosion-proof performance of acquisition equipment make it difficult to obtain real image data covering various extreme conditions through conventional methods, resulting in a severe shortage of data samples.

[0004] Secondly, existing general-purpose text-visual multimodal synthesis technologies have not yielded ideal results in the coal industry. Because general models lack the professional knowledge graph and physical rules inherent in the coal industry, they cannot accurately understand industry terminology and the unique physical characteristics of the underground environment. This results in generated scene data that is often disconnected from the real underground environment. For example, light attenuation and fluid movement trajectories in the generated images often violate common sense physics, and there are semantic discrepancies between the risk parameters described in the text and the generated visual representations. This lack of semantic consistency makes it difficult for the synthesized data to meet the accuracy training requirements of downstream AI models.

[0005] Furthermore, for rare and highly destructive scenarios such as water seepage and large-scale roof collapse, existing few-sample synthesis techniques often struggle to capture the scene's unique textures and structural features with extremely limited sample support, resulting in missing or distorted details in the generated data. Simultaneously, the results generated by existing synthesis techniques typically lack unified industry standard interfaces, and the data formats and specifications are difficult to directly adapt to existing coal mine safety monitoring platforms or mainstream AI training frameworks, leading to low data reuse rates and severely hindering the improvement of efficiency in coal industry safety monitoring model training, intelligent scheduling simulation, and emergency drills. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a method and apparatus for synthesizing textual visual multimodal data in risk scenarios in the coal industry. This solves the problems of a lack of real data in high-risk and rare disaster scenarios in the coal industry, as well as the lack of physical logic and semantic inconsistencies in data generated by existing general synthesis technologies, which makes it impossible to meet the training requirements of intelligent safety monitoring models.

[0007] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a method for synthesizing textual visual multimodal data in risk scenarios within the coal industry, comprising the following steps: Construct semantic mapping rules: parse safety regulations in the coal industry, establish conversion logic from unstructured text descriptions to structured text sequences, and generate mapping functions from text physical quantity parameters to visual generation control variables; Acquire enhanced training data: Collect multimodal data of real-world underground scenes, perform enhanced preprocessing based on multi-scale Retinex theory on the original images in the multimodal data, separate the illumination component and the reflection component, and produce high signal-to-noise ratio visual ground truth data; Constructing a generative network architecture: Building an industry-adaptive cross-modal generative model, which includes a semantic parsing module for extracting text features, a visual generation module for synthesizing images, and a discriminative module for adversarial training, and establishing an initial model architecture; Perform hybrid constraint training: Use the mapping function as the basis for logical constraints and the visual ground truth data as training samples to iteratively train the initial model architecture; through the iterative training, produce a hybrid constraint loss function that includes industry rule constraint loss and physical rationality constraint loss, update the model parameters, and produce a basic generative model; Perform rare scene transfer enhancement: For rare high-risk scenes with insufficient sample size, load the model parameters of the basic generative model as initial weights, introduce an external knowledge base, and use the few sample data of rare scenes to fine-tune the basic generative model to produce a dedicated generative model. Validation and output data: Synthetic data is generated using the dedicated generative model, and the synthetic data undergoes dual quality validation in both semantic and physical dimensions. The validated data is then output in a formatted manner.

[0008] Furthermore, during the hybrid constraint training process, this invention introduces industry rule constraint loss to ensure that the generated content conforms to the logic defined in the coal mine safety regulations. The specific principle is as follows: based on the mapping function, key risk parameters in the input text are mapped to theoretical visual attribute values; simultaneously, attribute statistical extraction of preset regions is performed on the images generated by the model to obtain the actual generated visual attribute values. The system calculates the absolute value of the difference between the actual generated visual attribute values ​​and the theoretical visual attribute values ​​by taking the absolute value of the difference, and uses a hinge loss mechanism to penalize the portion of the absolute value of the difference that exceeds a preset physical tolerance threshold. This mechanism forces the generative network to strictly adhere to the physical quantity settings in the text description, avoiding discrepancies between the generated images and the description parameters.

[0009] Furthermore, to address the issue of generated images violating physical principles, this invention introduces a physical rationality constraint loss. This loss calculates the gradient distribution of the brightness field of the generated image along the radial direction of the point light source using a gradient calculation method. This gradient distribution is then fitted and compared with a theoretical illumination attenuation curve derived based on the inverse square law to obtain an illumination constraint loss term, thereby ensuring the realism of the point light source illumination in the well. In addition, an optical flow vector field in the fluid region of the generated video sequence is calculated using an optical flow calculation method. Anomalies in the optical flow vector field whose vertical component direction is opposite to the direction of gravity are identified, and these anomalies are penalized to obtain a gravity constraint loss term. This ensures that the motion trajectories of fluids such as water and smoke conform to the laws of gravity.

[0010] Furthermore, addressing the data quality issues caused by uneven lighting and low contrast in the downhole environment, this invention employs an enhancement preprocessing method based on multi-scale Retinex theory during the data acquisition and enhancement training phase. Specifically, the acquired raw images are converted to the logarithmic domain, and convolved with the logarithmic domain images using a Gaussian kernel function. Weighted summation is then used to estimate the ambient lighting components. Subsequently, the ambient lighting components are extracted from the raw images using logarithmic subtraction, and a color restoration factor is introduced to correct the reflection components. This processing effectively removes the influence of uneven lighting, restores the reflective properties of the objects themselves, and provides high-quality ground truth visual data for the model.

[0011] Furthermore, the visual generation module employs a deconvolutional neural network as its backbone architecture, with skip connections between the encoder and decoder. This allows high-frequency texture features extracted by the shallow network to be directly cascaded to the corresponding deep generative network, preserving the complex texture details of the underground tunnels. Simultaneously, long short-term memory (LSTM) network units are embedded in the generative network. The hidden states and cell states of the previous frame constrain pixel generation in the current frame, and an optical flow prediction branch maintains the temporal consistency of the video sequence, ensuring the continuity of actions between consecutive frames.

[0012] Furthermore, to address the problem of model convergence difficulties caused by the scarcity of samples in rare high-risk scenarios (such as water inrush and large-scale roof collapse), this invention implements rare scenario transfer enhancement. Specifically, the weights of the shallow and mid-level convolutional filters responsible for extracting general texture features in the basic generative model are locked, while only the high-level semantic parsing and decoding layers are unfrozen. An elastic regularization constraint term is added to the loss function of the fine-tuned training. This constraint term, calculated using the Fisher information matrix, limits the deviation of the model parameters from the initial weights. This strategy preserves the model's ability to generate general downhole environments while achieving rapid adaptation to rare features, effectively suppressing catastrophic forgetting.

[0013] Furthermore, an external knowledge injection mechanism is introduced in the rare scene transfer enhancement. The system constructs an external knowledge base containing descriptions of the physical features of rare scenes and encodes these descriptions into high-dimensional knowledge vectors using an encoding method. During fine-tuning training, the high-dimensional knowledge vectors are injected into the unfrozen network layer of the generative model through an adaptive instance normalization module. This enables the model to guide the generation of visual features based on the injected physical rule features even in the absence of visual samples, achieving a supplementary shift from rule-driven to data-driven approaches.

[0014] Furthermore, to ensure the usability of the final output data, this invention performs dual quality verification. For semantic verification, a pre-trained CLIP model is used to extract the visual feature vector of the generated image and the text feature vector of the input text. The similarity between the visual feature vector and the text feature vector is calculated using cosine similarity, and samples with similarity below a preset semantic threshold are discarded. For physical verification, an object detection algorithm is used to identify the entity categories and bounding box coordinates in the generated image, verifying whether the coexistence relationships and spatial locations of entities conform to preset scene logic rules, thereby automatically discarding semantically inconsistent or physically non-compliant invalid data.

[0015] Furthermore, the formatted output can be automatically adapted to downstream applications. When the application scenario is artificial intelligence model training, the verified images and their corresponding bounding box coordinates or instance segmentation masks are encapsulated into a standard dataset format, and the synthesis parameters are marked in the metadata. When the application scenario is intelligent scheduling device simulation, the verified continuous image sequence is encoded into a video stream, and a JSON format event description file containing precise timestamps, risk level identifiers, and trigger source readings is generated simultaneously, directly supporting emergency drills of the scheduling system.

[0016] A second aspect of the present invention provides a text-visual multimodal data synthesis device for risk scenarios in the coal industry, comprising: The semantic modeling and feature mapping module is configured to parse industry regulations and construct mapping functions, converting unstructured text into structured sequences containing visual control variables, providing logical constraints for model training. The multimodal data preprocessing module is configured to acquire real data and perform image enhancement based on multi-scale Retinex theory, producing high signal-to-noise ratio visual ground truth data. The industry-adaptive generative model building module is configured to build a deep neural network architecture that includes text semantic parsing units, visual generation units, and discrimination units, providing an initial model architecture. The hybrid constraint training optimization module is configured to receive the mapping function and the visual ground truth data, train the initial model architecture based on the backpropagation algorithm, update the model parameters by calculating the industry rule constraint loss and the physical rationality constraint loss, and produce a basic generative model. The rare scene transfer enhancement module is configured to load the parameters of the base generative model, combine rare scene few-sample data and external knowledge base to perform fine-tuning training, and produce a dedicated generative model. The data validation and formatted output module is configured to receive the raw data generated by the dedicated generation model, perform semantic and physical dual logical validation, and output standardized data packets according to the requirements of downstream applications.

[0017] This invention provides a method and apparatus for synthesizing textual and visual multimodal data in risk scenarios within the coal industry. It offers the following advantages: 1. This invention constructs semantic mapping rules based on coal mine safety regulations and introduces a hybrid loss function that includes industry rule constraints and physical rationality constraints, forcing the generative network to follow underground physical laws and operating specifications. This mechanism effectively solves the problem of physical logic fallacies in general generative models in professional fields, ensures high semantic consistency and high visual fidelity between the generated visual images and text descriptions, and improves the usability and accuracy of synthetic data in the training of artificial intelligence visual models.

[0018] 2. This invention adopts a rare scenario migration enhancement strategy, and utilizes elastic regularization constraints and external knowledge base injection mechanism to achieve rapid and low-cost migration from general scenario models to specific disaster scenario models. This method can generate diverse disaster evolution sequences under very few sample conditions, and achieves comprehensive coverage of high-risk scenarios such as gas over-limit and geological disasters. It significantly reduces the cost and risk of obtaining extreme working condition data and fills the data gap in the industry.

[0019] 3. This invention incorporates multi-scale Retinex theory for enhanced preprocessing at the data acquisition end, effectively removing noise interference from the low-light environment underground and improving the signal-to-noise ratio of the data. At the output end, through dual quality verification and formatted output modules, it can automatically generate standard datasets or video streams and JSON event files with precise time-series markers as needed. This standardized output format is seamlessly compatible with existing coal mine safety monitoring platforms and mainstream AI training frameworks, significantly improving data reuse and directly supporting intelligent scheduling simulation and emergency drills. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the device module of the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 This invention provides a method for synthesizing textual and visual multimodal data in risk scenarios within the coal industry, comprising the following steps: Step S100 involves constructing semantic modeling and feature mapping rules for high-risk coal mining scenarios. This step, based on coal mine safety regulations and industry risk classification standards, establishes a standardized scenario feature dictionary. This dictionary covers five major risk scenarios: gas exceeding limits, geological disasters, equipment failures, personnel violations, and fires.

[0023] For each risk scenario, a correspondence is defined between a text feature set and a visual feature set. The text feature set includes the risk event name, physical parameters, target object status, spatial location, and time trend. The visual feature set includes ambient illumination distribution, dust or smoke concentration texture, equipment morphological distortion, and the target's relative spatial location.

[0024] In this step, a mapping function is established to map text parameters to visual attributes, transforming abstract parameters in the text description into control variables for visual generation. For video data containing temporal information, a correspondence between text time descriptions and video frame sequences is established, and a time synchronization error threshold is set.

[0025] Step S200: Perform multimodal real data acquisition and differential preprocessing. Obtain real downhole monitoring video streams, sensor logs, and simulation experimental data as the raw dataset.

[0026] For text data, data cleaning operations are performed to remove duplicate and semantically ambiguous text, correct non-standard terminology, and convert unstructured text into a standard format. Sequence labeling methods are used to label the core entities in the text.

[0027] For visual data, images whose sharpness meets a preset variance threshold are selected. An image enhancement algorithm based on Retinex theory is used to decompose and enhance the original downhole images, preserving the object reflection components as inherent texture features, and correcting the illumination components to construct the ground truth visual data required for training.

[0028] Step S300: Construct an industry-adaptive cross-modal generation model. This model includes a text semantic parsing module, a visual generation module, and a discrimination module.

[0029] The text semantic parsing module uses a pre-trained language model to encode the input structured text, and uses a multi-head self-attention mechanism to extract semantic features of key parameters to generate text embedding vectors.

[0030] The visual generation module receives text embedding vectors and noise vectors as input. This module comprises a deconvolutional network backbone, a low-light enhancement branch, and a temporal prediction branch. The low-light enhancement branch employs a skip connection structure to transfer shallow texture features to the deeper network. The temporal prediction branch introduces long short-term memory network units to process temporal states and combines optical flow estimation to predict the next frame of the image.

[0031] The discrimination module includes an authenticity discrimination unit and a semantic consistency discrimination unit, which are used to determine whether the generated image conforms to the real underground visual style and whether the generated image is semantically consistent with the input text, respectively.

[0032] Step S400: Train a cross-modal generative model based on a hybrid constraint loss function. During model training, the total loss function is calculated to optimize the model parameters. The total loss function consists of adversarial loss, semantic consistency loss, industry rule constraint loss, and physical rationality constraint loss.

[0033] Adversarial loss is used to encourage the generator to produce realistic images. Semantic consistency loss uses contrastive learning to calculate the distance between the generated visual feature vector and the text vector. Industry rule constraint loss, based on the mapping function established in step S100, penalizes visual attributes in the generated image that deviate from the preset mapping range. Physical plausibility constraint loss, tailored to the specific physical environment of the well, constrains the attenuation gradient of image brightness with distance from the center of the light source, and constrains the vertical component direction of the suspended object motion vector.

[0034] Step S500: Perform few-shot transfer augmentation for rare scenarios. For rare high-risk scenarios with a sample size below a preset threshold, use the parameters of the feature extractor trained on common scenarios as initialization parameters.

[0035] An external knowledge base vector containing textual descriptions of rare scenes is introduced and fused into the generator's feature map. The backbone network weights are fixed, and the generation layer is fine-tuned using rare scene samples to achieve feature transfer.

[0036] Step S600: Perform data quality verification and formatted output. The quality of the synthesized data is evaluated using a combination of automatic and manual verification. Automatic verification calculates the semantic matching score between the generated data and the text description. Physical compliance verification uses object detection algorithms to identify the reasonableness of the physical location of objects in the generated image.

[0037] For validated data, the output is adapted to the needs of downstream applications. If used for AI model training, it outputs a labeled dataset with bounding boxes and segmentation masks; if used for intelligent scheduling simulation, it outputs video streams and timestamp-synchronized alarm event metadata. Sensitive information in the images is removed before output to meet data security requirements.

[0038] See attached document Figure 2 This invention provides a text-visual multimodal data synthesis device for risk scenarios in the coal industry. The device is implemented based on a computer processing architecture and includes: The semantic modeling and feature mapping module is configured to perform the digitization and structuring of industry knowledge. This module includes a parameter parsing unit, which decomposes the input unstructured text description into risk event identifiers, sets of physical quantity parameters, and sets of states. The module also includes a mapping calculation unit, which runs a pre-defined mapping function. This module converts continuous physical values ​​in the text into control variables required for visual generation (such as pixel grayscale values ​​and texture density values), and converts discrete state descriptions into geometric deformation template indices. Furthermore, it includes a timeline alignment unit to calculate key event time points in the text description and set time synchronization thresholds for the visual frame sequence.

[0039] The multimodal data preprocessing module is configured to perform cleaning and enhancement of the raw data. This module includes a text cleaning unit for performing entity sequence annotation based on the BIO annotation method. At its core is an image enhancement unit based on Retinex theory, configured to perform multi-scale Gaussian filtering convolution operations on the acquired downhole raw images, estimate the ambient illumination component, and separate the object reflection component through logarithmic subtraction. The module also includes a quality screening unit for calculating the Laplacian variance of the image and automatically removing low-quality samples with sharpness below a preset threshold.

[0040] The industry-adaptive generative model building module is configured to load and run deep neural network architectures. This module specifically includes the following sub-units: Text Semantic Encoding Unit: Based on the improved BERT architecture, it integrates a multi-head self-attention mechanism and is configured to map structured text sequences into text embedding vectors containing industry semantic dependencies.

[0041] Visual Generation Unit: Built on a deconvolutional network and U-Net architecture. This unit includes a low-light enhancement branch that concatenates shallow feature maps to deep decoding layers via skip connections. This unit also includes a temporal prediction branch that integrates a Long Short-Term Memory (LSTM) network and an optical flow prediction layer to predict the pixel distribution of the next frame based on the hidden state of the previous frame.

[0042] The dual discriminant unit comprises a ground truth discriminator and a semantic consistency discriminator. The ground truth discriminator is configured to calculate the probability that the input image belongs to a true ground truth distribution; the semantic consistency discriminator is configured to calculate the matching degree between the image feature vector and the text embedding vector.

[0043] The hybrid constraint training optimization module is configured to update model parameters based on the backpropagation algorithm. This module integrates a loss function calculation engine for real-time calculation of the weighted total loss, which consists of adversarial loss, semantic consistency loss, industry rule constraint loss, and physical rationality constraint loss. This module is specifically configured to perform physical logic verification, calculate the gradient distribution of the generated image brightness along the radial direction of the light source, and calculate the deviation of the fluid region's motion vector from the direction of gravity. The calculation results are then converted into gradient penalty signals and fed back to the visual generation unit.

[0044] The Rare Scene Transfer Enhancement Module is configured to handle few-shot generation tasks. This module includes a parameter freeze controller to lock the weights of shallow and mid-layer convolutional kernels in the generative model during fine-tuning. It also includes an external knowledge injection interface to read structured physical description vectors from an external knowledge base and inject them into the generative network through an adaptive instance normalization layer. Furthermore, this module is configured to compute elastic regularization loss, limiting the deviation of model parameters from source domain parameters during fine-tuning to suppress catastrophic forgetting.

[0045] The data validation and formatted output module is configured to perform post-processing and encapsulation of the generated data. This module includes an automatic validation unit that integrates the CLIP model and object detection algorithms (such as YOLO) to calculate semantic cosine similarity and verify the physical location rationality of detected key entities. The module also includes an interface adaptation unit configured to convert validated images and metadata into PascalVOC / COCO dataset format, or encode consecutive frame sequences into H.264 / H.265 format video streams with timestamp indexes, and simultaneously output a JSON event description file containing risk level and trigger source information.

[0046] The modules mentioned above interact with each other via a device bus or internal memory. The logical functions of the modules can be executed by a general-purpose central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA) and combinations thereof.

[0047] In one embodiment of the present invention, step S100, which involves constructing semantic modeling and feature mapping rules for high-risk coal mining scenarios, specifically includes the following execution process: First, based on the "Coal Mine Safety Regulations" and industry risk classification standards, the device establishes a standardized scenario feature dictionary (denoted as...). This process involves parsing unstructured procedure text to extract the definition domains of five major high-risk scenarios: gas exceeding limits, geological disasters, equipment failures, personnel violations, and fires. For each specific scenario category... The device defines its set of features. From a subset of text features and visual feature subset Composition. Among them, the text feature subset. The structured definition is a vector space containing a unique identifier for the risk event, key physical parameters, a description of the target object's state, spatial coordinates, and temporal evolution trends. For example, in a gas exceedance scenario, textual features not only include the label "gas," but also specific concentration values ​​(e.g., 1.5%), duration, and the rate of change in sensor readings. Visual feature subsets... It is defined as including the ambient illuminance distribution matrix, the concentration texture features of the medium (such as dust and smoke), the geometric distortion parameters of the equipment or tunnel, and the relative position coordinates of the target object in three-dimensional space.

[0048] Based on this, a deterministic mapping function from the text parameter space to the visual attribute space is constructed. The mapping function is designed to transform abstract textual descriptions into visual control variables that can be executed by the generative model, thereby eliminating semantic ambiguity in cross-modal synthesis. The device employs differentiated mapping logic for different types of parameters. For continuous numerical parameters, linear or nonlinear numerical transformation relationships are established. Let the key physical parameters in the textual description be... Its corresponding visual presentation attributes are The mapping relationship is defined as follows: ; in, This represents the normalized or nonlinear transformation function of the parameters. and These are transformation coefficients determined based on an industry expert knowledge base. Taking gas concentration as an example, This is a concentration percentage value. This corresponds to the pixel opacity (alpha channel value) or grayscale value of the color channel of the generated smoke layer. The visual gain slope is used to control the increase of smoke concentration with gas levels. For discrete state parameters, a state encoding table is constructed to directly map state descriptions in the text, such as equipment deviation or personnel not wearing safety helmets, to preset geometric deformation templates or skeletal key point configurations.

[0049] Furthermore, for video data synthesis involving dynamic changes, step S100 also includes establishing temporal semantic synchronization rules. The device parses the time-series information in the text description to determine the start time, peak time, and end time of the risk event, which are denoted as the set of key time points of the event. Simultaneously, on the timeline of visual generation, a set of timestamps is defined for corresponding visual feature mutation frames (such as frames where smoke spreads to its maximum extent or frames where equipment malfunctions). To ensure that the generated video content strictly follows the evolution logic described in the text, the device sets the following timing synchronization constraints: ; in, , , This is the maximum permissible time synchronization error threshold. This threshold is set based on the frame rate standard of the coal mine monitoring device, for example, 10 milliseconds, to ensure that the alarm time displayed on the visual screen and the alarm time recorded in the device log are precisely consistent in downstream intelligent scheduling simulation applications.

[0050] Through the aforementioned mapping rules, the device transforms unstructured industry scenario descriptions into computer-readable structured configuration parameters, storing these parameters in JSON or XML format as input constraints for subsequent generative models. This structured mapping mechanism ensures that the visual data generated in subsequent steps is strictly anchored semantically to the safety standards of the coal industry, avoiding the illusionary phenomena that violate physical common sense caused by the lack of prior industry knowledge in general generative models.

[0051] In one embodiment of the present invention, step S200, which involves performing multimodal real data acquisition and differential preprocessing, specifically includes the following execution process: First, during the real-world data acquisition phase, the device acquires raw data from multiple heterogeneous sources to build a basic training library. For text data, sources include coal mine accident investigation reports published on government websites, authoritative industry research materials, internal sensor alarm logs, and manually recorded daily safety inspection logs. For visual data, sources primarily include real-time video streams from underground explosion-proof monitoring cameras, playback of historical risk event monitoring videos, and images captured in simulated test scenarios designed for specific rare scenarios (such as sudden water inrush or large-scale roof collapse). During the acquisition process, considering the unique characteristics of underground operations, all acquisition equipment complies with intrinsically safe explosion-proof standards for mining. Furthermore, for a single scenario category, data samples are collected covering different parameter combinations (such as different lighting angles and different equipment models) to ensure data diversity.

[0052] Secondly, deep preprocessing is performed on the collected text data. The device first cleans the text using a pre-built standard coal industry terminology dictionary, automatically identifying and correcting typos and non-standard terms, and removing duplicate records and semantically ambiguous invalid text. Subsequently, the unstructured descriptive text is converted into a unified JSON or XML standard format, and missing spatiotemporal fields (such as roadway numbers and timestamps) are completed. To support fine-grained semantic understanding in subsequent models, the device uses the BIO (Begin, Inside, Outside) sequence labeling method to annotate core entities in the text. Specifically, the device labels words in the text as B-Equip (first letter of equipment name), I-Equip (middle letter of equipment name), B-Param (first letter of parameter value), I-Param (middle letter of parameter value), etc., thereby transforming a continuous text stream into a sequence of entities with semantic attributes.

[0053] Next, a quality-based screening process is performed on the acquired visual data. The device uses the Laplacian operator to perform convolution operations on the images to calculate the image variance, which is then used as a sharpness evaluation metric. When the image variance is below a preset threshold... If the image is blurry, it is determined to be blurry and removed. Simultaneously, the proportion of occlusion in the image is detected, and samples where the occlusion area of ​​the target object exceeds a preset proportion (e.g., 30%) are removed.

[0054] Finally, to address the common problems of low illumination, uneven point source lighting, and high noise in the screened downhole images, the device performs image enhancement processing based on multi-scale Retinex theory. This processing aims to decompose the image into illumination and reflection components, remove the effects of uneven illumination, and restore the true reflection properties (i.e., texture features) of objects.

[0055] Let the original downhole image be in spatial coordinates The pixel intensity at that location is According to Retinex theory, the image is modeled as an ambient light component. With the reflection component of the object surface The product of: ; To separate these two components, the device converts the image to the logarithmic domain for computation: ; Among them, the illumination component This typically manifests as low-frequency information in an image. The device estimates the illumination components by convolving a multi-scale Gaussian filter function with the original image. Define the... Gaussian wrap function of scale for: ; in, The standard deviation of the Gaussian kernel is used to control the smoothness of changes in illumination. The square of the distance. Natural exponential function, This is a normalization constant. The device employs... Gaussian kernels of different scales (e.g.) The image is convolved at large, medium, and small scales respectively, and then weighted and summed to obtain an accurate estimate of the illumination components. ; in, For the first Weighting coefficients for each scale.

[0056] After obtaining an estimate of the illumination component, the device extracts the reflection component using logarithmic subtraction. To avoid excessive noise amplification in dark areas or overexposure in bright areas in the enhanced image, the device further introduces a color restoration factor. The reflection component is corrected, and gamma correction is applied to the extracted illumination component to adjust the dynamic range. The final enhanced image is generated. It not only preserves the inherent texture details of the underground rock walls, coal seams and equipment, but also corrects local brightness anomalies caused by direct light or shadow from the miner's lamp, serving as high-quality visual ground truth data for subsequent cross-modal generative model training.

[0057] In one embodiment of the present invention, step S300, which involves constructing an industry-adaptive cross-modal generation model, specifically includes the construction and execution logic of the following sub-modules: The text semantic parsing module's core component is a domain-specific pre-trained language encoder based on the Transformer architecture (referred to as Coal-BERT in this paper). Unlike general-purpose language models, this encoder is pre-trained using an industry corpus containing coal mine safety regulations, accident cases, and equipment operation manuals, enabling it to have a deep semantic understanding of industry terms such as gas, roof delamination, and idler rollers.

[0058] Specifically, the module receives the structured text sequence after preprocessing in step S200. As input, in order to accurately capture the long-distance dependencies between risk levels (such as high gas levels) and spatiotemporal information (such as return airways) in the text, the module employs a multi-head self-attention mechanism for feature extraction.

[0059] For each attention head, calculate the query matrix. Key matrix Sum matrix The calculation formula is as follows: ; in, Normalization function, The normalized attention weights ( Results) and Matrix multiplication performs a weighted summation of value vectors. All are generated by linear transformation of the embedding vectors of the input text sequence. This is a scaling factor used to prevent the gradient from vanishing due to an excessively large dot product result. calculate Each query vector in The dot product of all key vectors is calculated. After aggregation through multiple attention layers and processing by a feedforward neural network, the module finally outputs a fixed-dimensional text semantic embedding vector. This vector It not only encodes the literal information of the text, but also the logical relationships between entities (for example, establishing the causal semantics between gas concentration and alarm status), which serve as the semantic condition anchors for subsequent visual generation.

[0060] Visual generation module: Visual generation module (denoted as) It is constructed as a conditional generative network, whose input includes text semantic embedding vectors. and random noise vectors that follow a normal distribution (Used to characterize nondeterministic background textures in a scene, such as randomly distributed coal dust). The architecture of this module integrates a deconvolutional network for feature upsampling, a U-Net skip connection structure for low-light detail preservation, and a recurrent neural network structure for temporal consistency control.

[0061] Texture feature enhancement mechanism in low-light environments: To address the problem of image detail loss due to weak lighting and low contrast in underground scenes, the generator introduces skip connections unique to the U-Net architecture between the feature extraction path (encoder end) and the feature reconstruction path (decoder end). This connection directly concatenates high-frequency texture features extracted by the shallow network (such as rock wall roughness and coal seam bedding) into the deep generated feature map.

[0062] Temporal coherence generation mechanism: For video data generation, the generator introduces Long Short-Term Memory (LSTM) network units to maintain the temporal continuity between frames, preventing flickering or motion jumps in the generated video. Let... The generation of a time step is influenced by the hidden state of the previous time step. and cell state Control. Input features at the current moment. Depend on With current time noise It was obtained through fusion.

[0063] The state update formula for an LSTM cell is: ; in, for The hidden state at all times In order to be in The cell state at any given moment. Based on the updated hidden state. The generator progressively enlarges the feature map size using transposed convolutional layers (Deconvolution Layers). Finally, the first generation Visual images of frames

[0064] ; in, Decoder function.

[0065] In addition, to further enhance the physical realism of dynamic scenes (such as flowing water and smoke diffusion), the generator end is connected to an optical flow prediction branch in parallel, which outputs the predicted optical flow field to constrain the displacement vectors of adjacent frame pixels to conform to the laws of physical motion.

[0066] Discrimination module: Discriminant module (denoted as) It is designed as a dual-discrimination architecture, containing a truth-based discrimination unit. semantic consistency discrimination unit The two work together to provide gradient feedback for adversarial training. (1) Authenticity discrimination unit This unit is a deep convolutional neural network used to perform binary classification tasks. Its input is a real downhole image. Or generate an image The output is the probability that the image belongs to the true distribution. This unit focuses on low-level statistical features of the image, such as the plausibility of the illumination distribution, noise distribution patterns, and the sharpness of object edges, forcing the generator to produce images that conform to the downhole visual style.

[0067] Semantic consistency discrimination unit ( This unit is used to evaluate the matching degree between an image and a text description. It first maps the input image into a visual feature vector using an image encoder, and then matches this vector with the text embedding vector. The images are concatenated, and finally, a fully connected layer outputs the matching confidence score. If the generated image is realistic (e.g., a clear image of a motor), but lacks the key features described in the text (e.g., the text requests smoke from the motor, which is not shown in the image), then the problem persists. The output will have a low confidence level, thus penalizing the generator for ignoring textual semantics.

[0068] In one embodiment of the present invention, step S400, which involves training a cross-modal generative model based on a hybrid constraint loss function, specifically includes the following execution process: During the model training phase, the device alternately updates the generator using the backpropagation algorithm. and discriminator The parameters are specified. To ensure that the generated visual data is not only visually realistic but also accurately corresponds to the text description in terms of semantic content, and strictly complies with the safety regulations and physical laws of the coal mining industry, this invention constructs a multi-dimensional hybrid loss function. The total loss function is defined as the weighted sum of the four sub-loss functions: ; in, , , and These are the weight hyperparameters for each loss term, used to balance the optimization weight among visual realism, semantic consistency, rule compliance, and physical plausibility. Adversarial losses Semantic loss, Rule loss, Physical loss.

[0069] Adversarial loss function ( Calculation of ) Adversarial loss is used to drive the generator Generate images that conform to the real data distribution, and train the discriminator simultaneously. This method distinguishes between real and generated images. This embodiment employs an adversarial loss based on Wasserstein distance or standard cross-entropy. The calculation formula is as follows: ; in, The mathematical expectation symbol represents the mean of a specific data distribution. From a dataset of real downhole images (distributed as follows) Sample a real image from ) , Authenticity discriminator For real images The output represents the probability that the image is judged as real. The training objective is to make this probability close to 1. From a standard noise distribution (such as a Gaussian distribution) Sample a random noise vector in ) , Distribution of text embedding vectors Sampling a text condition , generator With noise and text conditions As input, generate a composite image. Authenticity discriminator The output of the synthesized image represents the probability that the synthesized image is judged as real. The goal of training the generator is to make this probability close to 1, while the goal of training the discriminator is to make this probability close to 0. Logarithmic function. The entire formula is a binary cross-entropy loss used in adversarial games between the generator and the discriminator.

[0070] In this process, the generator The attempt is to minimize this loss (i.e., to deceive the discriminator), while the discriminator... Trying to maximize the loss.

[0071] This loss term ensures that the generated images closely approximate real downhole monitoring footage in terms of texture, lighting, and noise patterns.

[0072] Semantic consistency loss function Calculation of ) This loss term constrains the semantic accuracy of the generated content, preventing the model from generating images unrelated to the text description (e.g., generating a coal mining machine when the text describes a conveyor belt). The device utilizes a pre-trained image encoder (with a reusable discriminator). Feature extraction layer extracts and generates images Mapped to internal visual feature vectors for adversarial training Then, the vector is compared with the input text embedding vector. Cosine distance in high-dimensional space: ; in, Text embedding vector The L2 norm (i.e., the length of the vector), Visual feature vectors The L2 norm.

[0073] By minimizing The device forces the feature vectors of the generated image to converge with the feature vectors of the text in the semantic space, thereby ensuring that the generated visual object is consistent with the subject described in the text.

[0074] Industry rule constraint loss function ( Calculation of ) This is the core constraint mechanism of the present invention, designed to transform the mapping logic of the "Coal Mine Safety Regulations" established in step S100 into a differentiable mathematical penalty term. The device operates based on the mapping function... Numerical verification is performed on the generated results of key parameters. Let the first parameter in the text description be... The key parameters are (e.g., gas concentration), according to the mapping function Its theoretical corresponding visual attribute value is (For example, the theoretical value of the corresponding smoke layer opacity). The device generates the image. Perform attribute statistical extraction operations on a specific region (such as calculating the pixel mean or histogram statistics of the region of interest, ROI) to obtain the actual generated visual attribute values. .

[0075] Industry rule-constrained loss is defined as the portion of the difference between the actual and theoretical values ​​that exceeds the tolerance range. ; in, For a generation task involved The loss of each key parameter is summed. The first one extracted from the generated image A real visual attribute value (such as the average smoke concentration calculated through image processing), According to the text description, the first Parameters Through a preset mapping function The calculated theoretical visual attribute values, For the first The physical tolerance threshold set by each parameter. Take the larger value between the expression inside the parentheses and 0.

[0076] This formula represents a hinge loss mechanism: as long as the generated visual attributes are within the allowable error range... Within this range, the loss is 0; once it exceeds this range (i.e., violates the industry-standard mapping logic), the model will be subject to a linear penalty. This forces the model to learn a strict correspondence between gas concentration values ​​and smoke concentration in the image.

[0077] Physical rationality constraint loss function ( Calculation of ) This loss term is specifically designed to address the characteristics of point source lighting and the physical environment in underground mines, aiming to eliminate image artifacts that violate physical laws. Given that underground roadways are typically illuminated only by point sources such as miners' lamps or explosion-proof lights, the light intensity should exhibit a non-linear decay with distance. The device calculates the brightness field of the generated image. radial direction of the light source gradient And compare it with the theoretical decay curve derived from the inverse square law of physics: ; in, Brightness field in generated image radial distance of the light source The gradient, i.e., the rate of change of brightness with distance. A scaling factor related to the light source intensity is an adjustable hyperparameter. The mathematical expression of the inverse square law of light intensity in physics: intensity is inversely proportional to the square of the distance. A very small positive constant (e.g., 1e-6) is used to prevent when When the denominator is zero, It is the square of the L2 norm, which is the mean square error (MSE) of the difference between the actual luminance gradient and the theoretical luminance gradient.

[0078] If the brightness of the generated image does not exhibit a reasonable attenuation when moving away from the light source (e.g., it remains unusually bright at a distance), this loss term will increase.

[0079] Furthermore, for video generation involving dynamic fluids (such as water and dust), the device calculates the optical flow vector field of the fluid region. According to gravity constraints, the vertical motion component of the fluid It should primarily be downwards. The physical constraint loss also includes a penalty for motion vectors that violate the direction of gravity: ; in, Gravitational physical loss, All pixels in the image identified as fluid regions The set, At pixel At that point, the component of the motion vector in the vertical direction (y-axis) calculated by the optical flow method. The total gravity violation penalty is obtained by summing up the gravity losses of all pixels within the fluid region.

[0080] If upward fluid movement is detected (violating gravity), a loss occurs. Ultimately... This is a weighted sum of illumination and gravity constraints. Through joint optimization of the aforementioned hybrid loss function, the model parameters... Through continuous adjustments during training, the generator was eventually able to output high-quality multimodal synthetic data while meeting the strict rules and physical laws of the coal industry.

[0081] In one embodiment of the present invention, step S500, which involves performing few-sample transfer enhancement for rare scenarios, specifically includes the following execution process: targeting high-risk scenarios in the coal industry with extremely small sample sizes (denoted as the target domain). For example, in cases of sudden water seepage or large-area roof collapse, the device does not train the model from scratch, but instead employs a few-shot learning strategy based on parameter transfer and knowledge enhancement. This strategy utilizes common scenarios (denoted as the source domain). For example, the model parameters that have been fully trained on routine inspections and minor deviations. This serves as a foundation to address the model non-convergence problem caused by a lack of data in the target domain.

[0082] Parameter initialization and network freeze strategy: The device will first be in the source domain Generative model parameters that converge during training The weights are loaded into the target model as initial weights. Based on common features of the underground visual environment (such as the semi-enclosed structure of tunnels, point light source illumination features, and common textures of various cables and pipes), the device implements a layered freezing strategy. Specifically, the device freezes the weights of shallow and mid-level convolutional filters in the generator network, keeping them unchanged during fine-tuning. These layers are responsible for extracting and generating common geometric edges, texture primitives, and lighting rendering patterns. Only the high-level semantic parsing layers and task-specific decoding layers of the generator are unfrozen, keeping their parameters updatable. This strategy ensures that the model's ability to generate common environmental backgrounds is not compromised when encountering rare scene-specific features (such as water jet patterns).

[0083] External knowledge base-guided feature injection: To accurately generate physically consistent scenes even with extremely limited visual samples (e.g., only <10 reference images), the device introduces an external knowledge base embedding mechanism. The device constructs an external knowledge base (ExternalKnowledgeBase) containing detailed physical descriptions of rare scenes. This knowledge base includes structured text extracted from accident cases (e.g., water jets spraying in a parabolic pattern, accompanied by a bulging of the base plate).

[0084] The device uses a text encoder to map this prior knowledge into high-dimensional knowledge vectors. In the generator's computation process, an Adaptive Instance Normalization (AdalN) module or a feature concatenation module is introduced to normalize the knowledge vector. This is injected into the unfrozen network layers. This allows the model to deduce reasonable visual features based on the injected textual logic even in the absence of visual references, achieving a smooth transition from data-driven to rule-driven approaches.

[0085] Fine-tuning and catastrophic forgetting inhibition: Data on rare scenarios During fine-tuning training, to prevent the model from overfitting to a very small number of samples while forgetting the general knowledge learned from the source domain (i.e., catastrophic forgetting), the device constructs a dedicated few-sample fine-tuning loss function. .

[0086] The loss function consists of two parts: the first part is the hybrid constraint loss for rare samples in the target domain (i.e., as described in step S400). However, only in The first part (top) is used to capture rare scene features; the second part is the parameter regularization constraint, used to limit the deviation of model parameters from the initial parameters. Its calculation formula is as follows: ; in, Represents the target domain. , These are the model parameters in the current fine-tuning process. In the pre-trained model on the source domain, the first... Fixed values ​​for each parameter For the fixed parameters of the pre-trained model in the source domain, This is the elastic regularization coefficient, used to control the strength of retaining old knowledge. Let be the importance weight calculated based on the Fisher Information Matrix, representing the weight of the th element. The importance of each parameter to the source domain task. If a parameter is critical to the generation of a general environment (e.g., a parameter controlling illumination attenuation), its... A large value forces the parameter to remain almost unchanged during fine-tuning; conversely, for non-critical parameters, they are allowed to be updated over a wider range to adapt to rare scene characteristics.

[0087] Through the above mechanism, the device can quickly adapt a general model into a special model with specific rare disaster simulation capabilities, provided only a small number of rare scene samples, while maintaining a high-quality downhole realism in the generated image background.

[0088] In one embodiment of the present invention, step S600, which involves performing data quality verification and formatted output, specifically includes the following execution process: The device is configured to use an automated dual verification mechanism to screen the quality of the generated multimodal data and encapsulate the qualified data into a standardized interface format according to different downstream application scenarios.

[0089] Semantic consistency verification based on cross-modal retrieval model: The device incorporates a pre-trained CLIP (Contrastive Language-Image Pre-training) model as an automatic evaluator. This process aims to quantify the degree of matching between the generated visual image and the input text description in the semantic space, thereby eliminating semantically biased data.

[0090] Specifically, the device will generate a synthetic image The image encoder of the CLIP model is input to extract external validation visual feature vectors. At the same time, the corresponding original text description Input text encoder, extract text feature vectors The device calculates the cosine similarity between two vectors as a semantic matching score. : ; in, vector sum vector The norm of .

[0091] The device sets a semantic qualification threshold. (For example, 0.75). When If the generated data is determined to be a semantically distorted sample (e.g., text describing a torn belt but the image only showing a blurry shadow), the device directly discards the sample and triggers the generator to resample the noise vector. Regenerate.

[0092] Physical compliance verification based on object detection: For images that pass semantic verification, the device further performs physical object existence and location validity verification. The device invokes general object detection algorithms based on YOLO or Faster R-CNN architectures to identify and locate key entities (such as people, mining trucks, and support equipment) in the generated images, obtaining the coordinates of the detection boxes. and category confidence.

[0093] The device compares the detection results with the scene feature set defined in step S100. The verification logic includes: Entity existence verification: If the text description includes "not wearing a self-rescue device", the detection algorithm must identify the miner entity in the image and cannot identify the self-rescue device entity in the corresponding area.

[0094] Spatial logic verification: For a scenario where the ceiling collapses, the device calculates the center coordinates of the falling object detection frame. .like If a rock is located in the lower half of the image (ground area) and there are no corresponding signs of detachment above it, it is considered to be a physical anomaly (such as a rock appearing out of nowhere) and is removed.

[0095] Downstream application adaptation and formatted output: After the above dual verification of qualified data, the device performs formatted encapsulation through the data adaptation interface according to the user's preset application requirements: Scenario 1: For training data output from an AI recognition model, if the application mode is configured for model training, the device automatically generates a label file that strictly corresponds to the image pixel coordinates. The device converts the object category, bounding box coordinates, and instance segmentation mask determined during the generation process into a standard dataset format. Output can be in PASCALVOC format (XML file) or MSCOCO format (JSON file). For synthetic data, the device automatically tags the Synthetic attribute in the output metadata and includes synthetic parameters (such as light intensity values ​​and smoke concentration levels) to facilitate downstream models in Curriculum Learning or Domain Adaptation Training.

[0096] Scenario 2: For simulation exercise output of the intelligent scheduling device, if the application mode is configured as simulation exercise, the device will encode the continuously generated image sequence into a standard video stream (such as H.264 or H.265 encoding format) and generate a time-synchronized event metadata stream. The device constructs a JSON event description file containing timestamp indexes, the structure of which is defined as follows: Timecode: A millisecond-level timestamp that is precisely synchronized with video frames.

[0097] Event_ID: A unique identifier for risk events as defined in the "Coal Mine Safety Regulations".

[0098] Risk_Level: Risk level generated based on text description (e.g., general hazard, major hazard).

[0099] Trigger_Source: Simulates the sensor type and reading that triggered the alarm (e.g., gas sensor T1: 2.1%).

[0100] In this way, the present invention outputs not only pixel images, but also rich media data containing structured semantic information, which can be directly read by existing safety monitoring screen devices or AI analysis hosts in coal mines, and used to test the emergency response logic of the scheduling device under zero-risk conditions.

Claims

1. A method for synthesizing textual and visual multimodal data in risk scenarios in the coal industry, characterized in that, Includes the following steps: Step S100: Construct semantic mapping rules: parse the safety regulations of the coal industry, establish the conversion logic from unstructured text description to structured text sequence, and generate a mapping function from text physical quantity parameters to visual generation control variables; Step S200: Obtain enhanced training data: Collect multimodal data of real-world underground scenes, perform enhanced preprocessing based on multi-scale Retinex theory on the original images in the multimodal data, separate the illumination component and the reflection component, and produce high signal-to-noise ratio visual ground truth data. Step S300: Constructing the generative network architecture: Building an industry-adaptive cross-modal generative model, which includes a semantic parsing module for extracting text features, a visual generation module for synthesizing images, and a discriminative module for adversarial training, and establishing an initial model architecture; Step S400: Perform hybrid constraint training: Use the mapping function as the basis for logical constraints and the visual ground truth data as training samples to iteratively train the initial model architecture; through the iterative training, produce a hybrid constraint loss function that includes industry rule constraint loss and physical rationality constraint loss, update the model parameters, and produce a basic generative model; Step S500: Perform rare scene transfer enhancement: For rare high-risk scenes with insufficient sample size, load the model parameters of the basic generative model as initial weights, introduce an external knowledge base, and use the rare scene small sample data to fine-tune the basic generative model to produce a dedicated generative model. Step S600: Verify and output data: Generate synthetic data using the dedicated generation model, perform dual quality verification of the synthetic data in terms of semantic and physical dimensions, and output the verified data in a formatted manner.

2. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The calculation process for the industry rule constraint loss includes: Based on the mapping function, key risk parameters in the input text are mapped to theoretical visual attribute values; Perform attribute statistical extraction on the image generated by the model in a preset region to obtain the actual generated visual attribute values; The absolute value of the difference between the actual generated visual attribute value and the theoretical visual attribute value is calculated by taking the absolute value of the difference. By utilizing the hinge loss mechanism, a penalty is applied to the portion of the difference whose absolute value exceeds a preset physical tolerance threshold, resulting in the industry rule constraint loss.

3. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The calculation process for the physical rationality constraint loss includes: The gradient distribution of the brightness field of the generated image along the radial direction of the point light source is calculated by the gradient calculation method. The gradient distribution is then fitted and compared with the theoretical illumination attenuation curve derived based on the inverse square law to obtain the illumination constraint loss term. The optical flow vector field of the fluid region in the video sequence is calculated using an optical flow calculation method. Anomaly vectors whose vertical component direction is opposite to the gravity direction are identified in the optical flow vector field. The anomalous vectors are penalized to obtain a gravity constraint loss term.

4. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The enhanced preprocessing based on multi-scale Retinex theory in step S200 specifically includes: The acquired raw images are converted to the logarithmic domain; The ambient illumination component is estimated by convolving the Gaussian kernel function with the logarithmic domain image and then using weighted summation. The ambient light component is extracted from the original image by logarithmic subtraction, and the reflection component is corrected by introducing a color restoration factor to obtain the enhanced visual ground truth data.

5. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The construction features of the visual generation module in step S300 are as follows: A deconvolutional neural network is used as the backbone architecture, and a skip connection is set between the encoder and decoder to directly cascade the high-frequency texture features extracted by the shallow network to the corresponding deep generative network. Long Short-Term Memory (LSTM) network units are embedded in the generative network. The hidden state and cell state of the previous frame are used to constrain the pixel generation of the current frame. An optical flow prediction branch is combined to maintain the temporal consistency of the video sequence.

6. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The specific process of fine-tuning the training in step S500 includes: The weights of the shallow and mid-level convolutional filters responsible for extracting general texture features in the basic generative model are locked, and only the high-level semantic parsing layer and decoding layer are unfrozen; An elastic regularization constraint term is added to the loss function of the fine-tuning training. The constraint term limits the deviation of the model parameters from the initial weights by means of the Fisher information matrix.

7. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The process of introducing an external knowledge base in step S500 includes: Construct the external knowledge base containing descriptions of physical features of rare scenes, and encode the descriptions into high-dimensional knowledge vectors using an encoding method; During the fine-tuning training process, the high-dimensional knowledge vector is injected into the unfrozen network layer of the generative model through the adaptive instance normalization module to guide the generation of visual features.

8. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The dual quality verification in step S600 specifically includes: Semantic verification: The visual feature vector of the generated image and the text feature vector of the input text are extracted using a pre-trained CLIP model. The similarity between the visual feature vector and the text feature vector is calculated using cosine similarity. Samples with similarity lower than a preset semantic threshold are removed. Physical verification: Using object detection algorithms to identify entity categories and bounding box coordinates in the generated image, verifying whether the coexistence relationship and spatial location of entities conform to the preset scene logic rules.

9. The text-visual multimodal data synthesis method for risk scenarios in the coal industry according to claim 1, characterized in that, The formatted output in step S600 is specifically manifested as follows: When the application scenario is training an artificial intelligence model, the verified images and their corresponding bounding box coordinates or instance segmentation masks are encapsulated into a standard dataset format, and the synthesis parameters are marked in the metadata. When the application scenario is a simulation of an intelligent scheduling device, the verified continuous image sequence will be encoded into a video stream, and a JSON format event description file containing a precise timestamp, risk level identifier, and trigger source reading will be generated simultaneously.

10. A text-visual multimodal data synthesis device for risk scenarios in the coal industry, characterized in that, A text-visual multimodal data synthesis and assembly method applied to the risk scenarios in the coal industry as described in any one of claims 1-9, comprising: The semantic modeling and feature mapping module is configured to parse industry regulations and construct mapping functions, converting unstructured text into structured sequences containing visual control variables, providing logical constraints for model training. The multimodal data preprocessing module is configured to acquire real data and perform image enhancement based on multi-scale Retinex theory, producing high signal-to-noise ratio visual ground truth data. The industry-adaptive generative model building module is configured to build a deep neural network architecture that includes text semantic parsing units, visual generation units, and discrimination units, providing an initial model architecture. The hybrid constraint training optimization module is configured to receive the mapping function and the visual ground truth data, train the initial model architecture based on the backpropagation algorithm, update the model parameters by calculating the industry rule constraint loss and the physical rationality constraint loss, and produce a basic generative model. The rare scene transfer enhancement module is configured to load the parameters of the base generative model, combine rare scene few-sample data and external knowledge base to perform fine-tuning training, and produce a dedicated generative model. The data validation and formatted output module is configured to receive the raw data generated by the dedicated generation model, perform semantic and physical dual logical validation, and output standardized data packets according to the requirements of downstream applications.

Images