A multi-modal long tail scene automatic generation method, system, device and medium

Abstract

The application discloses a multi-modal long-tail scene automatic generation method, system, device and medium, relates to the field of artificial intelligence, and comprises the following steps: constructing a long-tail scene semantic knowledge base of a target field; based on the long-tail scene semantic knowledge base, generating a natural language description of each scene by using a large language model, and adjusting the generation probability of each scene by using a long-tail distribution reweighting mechanism in the generation process; converting the natural language description into a parameterized representation, generating multi-modal content of each scene based on a physical engine and a diffusion model, and introducing a physical constraint loss and a multi-modal semantic alignment loss in the generation process, so that the motion, force and interaction of objects in the generated multi-modal content conform to the laws of Newtonian mechanics and thermodynamics, and the cosine similarity between different modal content and the parameterized representation is maximized, so as to obtain a multi-modal long-tail scene. The application can automatically generate high-quality, physically credible and multi-modal aligned long-tail scene data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, system, device and medium for automatically generating multimodal long-tail scenes. Background Technology

[0002] With the widespread application of artificial intelligence (AI) technology in autonomous driving, high-precision manufacturing, and process industries, the robustness and safety of AI models have become key challenges. In these fields, long-tail extreme scenarios (such as pedestrians crossing the road in a rainstorm during autonomous driving, rare equipment failures in precision manufacturing, and abnormal operating conditions in process industries) have low probability of occurrence, but if they do happen, they can lead to serious safety accidents and economic losses. Existing long-tail extreme scenario generation technologies mainly suffer from the following shortcomings: (1) The cost of data collection is extremely high: Long-tail scenarios are difficult to obtain through real-world collection, and manual annotation is costly and inefficient.

[0003] (2) Poor physical realism: The scenes generated by traditional Generative Adversarial Networks (GAN) and diffusion models often have physical contradictions (such as floating objects and unreasonable motion trajectories).

[0004] (3) Multimodal inconsistency: There is a lack of semantic consistency between different modalities such as generated images, point clouds, and sensor data.

[0005] (4) Insufficient long-tail coverage: Existing generation methods tend to generate common scenes, making it difficult to effectively cover extreme cases at the tail of the distribution.

[0006] The emergence of large language models has provided a new approach to solving the above problems, but the content they generate lacks precise physical constraints and multimodal expressive capabilities. Therefore, there is an urgent need for an automatic multimodal long-tail scene generation technology that combines semantic guidance from large language models with physical constraints. Summary of the Invention

[0007] The purpose of this application is to provide a method, system, device and medium for automatically generating multimodal long-tail scene data, which can automatically generate high-quality, physically reliable and multimodally aligned long-tail scene data.

[0008] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for automatically generating multimodal long-tail scenes, including: Construct a long-tail scenario semantic knowledge base for the target domain; Based on the long-tail scene semantic knowledge base, a large language model is used to generate natural language descriptions for each scene, and a long-tail distribution reweighting mechanism is used to adjust the generation probability of each scene during the generation process. The natural language description of each scenario is converted into a parameterized representation of the corresponding scenario; Based on the parameterized representation of each scene, multimodal content for each scene is generated using a physics engine and a diffusion model. Physical constraint loss and multimodal semantic alignment loss are introduced during the generation process to ensure that the motion, forces, and interactions of objects in the generated multimodal content conform to the laws of Newtonian mechanics and thermodynamics. At the same time, the cosine similarity between different modal contents and parameterized representations is maximized to obtain multimodal long-tailed scenes.

[0009] Secondly, this application provides a multimodal long-tail scene automatic generation system, including: The knowledge base construction module is used to build a long-tail scenario semantic knowledge base for the target domain; The natural language generation module is used to generate natural language descriptions for each scenario based on the long-tail scenario semantic knowledge base using a large language model, and to adjust the generation probability of each scenario using a long-tail distribution reweighting mechanism during the generation process. The natural language conversion module is used to convert the natural language description of each scenario into a parameterized representation of the corresponding scenario; The scene generation module is used to generate multimodal content for each scene based on the parameterized representation of each scene, the physics engine and the diffusion model. During the generation process, physical constraint loss and multimodal semantic alignment loss are introduced to ensure that the motion, force and interaction of objects in the generated multimodal content conform to the laws of Newtonian mechanics and thermodynamics. At the same time, the cosine similarity between different modal contents and parameterized representations is maximized to obtain multimodal long-tail scenes.

[0010] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for automatically generating multimodal long-tailed scenes.

[0011] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for automatically generating multimodal long-tailed scenes.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects: By constructing a long-tail scene semantic knowledge base and introducing a long-tail distribution reweighting mechanism, the generation probability of rare or extreme scenes is significantly improved, effectively solving the problem of insufficient scene coverage caused by the uneven distribution of long-tail data in the real world. By combining a large language model with parameterized representation, a precise mapping from natural language to controllable parameters is achieved, providing a semantically faithful foundation for subsequent content generation. Furthermore, by integrating a physics engine and a diffusion model, and introducing physical constraint loss and multimodal semantic alignment loss, the motion, forces, and interactions of objects in the generated multimodal content strictly follow the laws of Newtonian mechanics and thermodynamics, avoiding common physical inconsistencies in traditional methods, while ensuring high semantic consistency between different modalities, maximizing the cosine similarity between modal content and parameterized representation. Thus, this application can automatically generate high-quality, physically reliable, and multimodally aligned long-tail scene data, providing scarce and realistic edge cases for model training in fields such as autonomous driving and robot simulation, significantly improving the model's generalization ability and robustness under long-tail distribution. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is an application environment diagram of a multimodal long-tail scene automatic generation method according to an embodiment of this application.

[0015] Figure 2 This is a flowchart illustrating a method for automatically generating multimodal long-tail scenes according to an embodiment of this application.

[0016] Figure 3 This is an overall architecture diagram of a method for automatically generating multimodal long-tail scenes provided in an embodiment of this application.

[0017] Figure 4 This is a schematic diagram illustrating the calculation process of physical constraint loss in one embodiment of this application.

[0018] Figure 5 This is an example diagram of generating long-tail scenarios in the field of autonomous driving in one embodiment of this application.

[0019] Figure 6 This is an example diagram of a high-precision manufacturing defect scenario generated in one embodiment of this application.

[0020] Figure 7This is an example diagram illustrating the generation of abnormal operating conditions in the process industry, as shown in one embodiment of this application.

[0021] Figure 8 This is a schematic diagram of the functional modules of a multimodal long-tail scene automatic generation system provided in an embodiment of this application.

[0022] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The purpose of this application is to automatically generate high-quality data that conforms to physical laws, is multimodal consistent, and covers a wide range of long-tail scenarios.

[0025] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] The multimodal long-tail scene automatic generation method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 101 communicates with server 102 via a network. A data storage system can store the data that server 102 needs to process. The data storage system can be set up independently, integrated into server 102, or placed in the cloud or on another server. Terminal 101 can send a long-tail scene semantic knowledge base of the target domain to server 102, and server 102 generates multimodal long-tail scenes based on the target domain's long-tail scene semantic knowledge base. Server 102 can then feed back the obtained multimodal long-tail scenes to terminal 101.

[0027] The terminal 101 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 102 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0028] In one exemplary embodiment, such as Figure 2 and Figure 3As shown, a method for automatically generating multimodal long-tail scenes is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 102 as an example, the explanation includes the following steps 201 to 205.

[0029] Step 201: Construct a long-tail scene semantic knowledge base for the target domain.

[0030] In a specific application example, accident reports, expert knowledge, and historical anomaly data from the target domain (autonomous driving, precision manufacturing, process industry) are collected. Then, a large language model is used for knowledge extraction and scene classification to construct a structured long-tail scene semantic knowledge base. Key attributes, constraints, and risk levels of each scene are defined and integrated into the long-tail scene semantic knowledge base. The long-tail scene semantic knowledge base includes data such as key attributes, constraints, risk levels, accident reports, and images for each scene.

[0031] Step 202: Based on the long-tail scene semantic knowledge base, a large language model is used to generate a natural language description for each scene, and a long-tail distribution reweighting mechanism is used to adjust the generation probability of each scene during the generation process. Further semantic verification and removal are performed on the natural language descriptions to ensure the uniqueness and rationality of the scenes.

[0032] This application proposes a long-tail distribution reweighting mechanism based on inverse frequency weighting to increase the generation probability of rare scenarios and ensure that the distribution of generated data can effectively cover the long tail part.

[0033] In a specific application example, adjusting the generation probability of each scenario using a long-tail distribution reweighting mechanism includes the following steps 21 to 23.

[0034] Step 21: Filter the knowledge in the long-tail scene semantic knowledge base to construct a training dataset. The training dataset includes knowledge samples from various scenarios.

[0035] Step 22: Calculate the long-tail distribution reweighting coefficient for each scenario based on the training dataset.

[0036] Specifically, the following formula is used to calculate the first... Long-tail distribution reweighting coefficients for similar scenarios ; in, For the first The reweighting coefficients for the long-tail distribution of similar scenarios. The total number of knowledge samples in the training dataset, For the training dataset, the first Number of knowledge samples for each scenario This is a smoothing coefficient used to control the strength of the reweighting. ,when When all categories have the same weight; when At that time, the weight is inversely proportional to the class frequency. By adjusting... The value of can balance the generation probability of common and rare scenarios.

[0037] Step 23: Based on the long-tail distribution reweighting coefficient of each scenario, reweight the probability of generating each scenario using the large language model to adjust the generation probability of each scenario.

[0038] Specifically, the following formula is used to generate the first... The probabilities of similar scenarios are reweighted: ; in, For the reweighted first The generation probability of the scene type Generate the first for large language models The probability of similar scenarios The total number of scene categories. For the first The reweighting coefficients for the long-tail distribution of similar scenarios. Generate the first for large language models The probability of a given scene type. This formula obtains a new generation probability distribution by multiplying the original generation probability by a reweighted coefficient and normalizing it, which significantly increases the generation probability of rare scenes, thus effectively covering the long tail portion.

[0039] This application constrains the natural language descriptions generated by the large language model by integrating the generation probability of each scene after reweighting into the prompt word engineering of the large language model, thereby adjusting the generation probability of each scene and improving the generation probability of rare scenes.

[0040] Step 203 involves converting the natural language description for each scenario into a parameterized representation for that scenario. Specifically, this conversion is achieved through the prompt word engineering of the large language model.

[0041] Step 204: Based on the parameterized representation of each scene, multimodal content for each scene is generated using a physics engine and a diffusion model. Physical constraint loss and multimodal semantic alignment loss are introduced during the generation process to ensure that the motion, forces, and interactions of objects in the generated multimodal content conform to the laws of Newtonian mechanics and thermodynamics. At the same time, the cosine similarity between different modal contents and parameterized representations is maximized to obtain multimodal long-tailed scenes.

[0042] In a specific application example, the multimodal content includes images, point clouds, videos, and sensor data.

[0043] This application introduces physical constraint loss and multimodal semantic alignment loss during the generation process to ensure the physical realism and modal consistency of the generated content. The physical constraint loss ensures that the motion, forces, and interactions of objects in the generated scene conform to Newtonian mechanics and thermodynamics. The multimodal semantic alignment loss ensures consistent semantic information across different modalities, such as generated images, point clouds, and sensor data.

[0044] like Figure 4 As shown, the physical constraint losses include kinematic constraint losses, dynamic constraint losses, thermodynamic constraint losses, and collision constraint losses. Kinematic constraint losses ensure that the position, velocity, and acceleration of an object conform to the laws of kinematics. Dynamic constraint losses ensure that the forces and motion of an object conform to Newton's second law. Thermodynamic constraint losses ensure that thermodynamic parameters such as temperature and pressure conform to the laws of thermodynamics. Collision constraint losses ensure that collisions and contacts between objects conform to the laws of physics.

[0045] The formula for the physical constraint loss is: ; in, For physical constraint loss, For kinematic constraint loss, For dynamic constraint loss, For thermodynamic constraint loss, For collision constraint loss, These are the weighting coefficients for the kinematic constraint loss. These are the weighting coefficients for the dynamic constraint loss. This is the weighting coefficient for thermodynamic constraint loss. These are the weighting coefficients for the collision constraint loss, which can be adjusted according to the application scenario.

[0046] The formula for the kinematic constraint loss is: ; in, T The total number of frames in the video. For the object in the first t The position vector of the frame. , Used to specify the dimension and range of values ​​of a vector, representing the three-dimensional set of real numbers. For the object in the first t -1 frame's position vector For the object in the first t -1 frame velocity vector , The time interval between two adjacent frames. For the object in the first t Acceleration vector of frame -1 , This is the square of the L2 norm, used here to measure the difference between the predicted and actual values. Kinematic constraint loss ensures that the object's trajectory conforms to the laws of uniformly accelerated linear motion, avoiding unrealistic motion phenomena such as instantaneous teleportation or sudden acceleration.

[0047] The formula for the dynamic constraint loss is: ; in, The mass (kg) of the current object is assigned by the scene parameterization module based on the object type. For the object in the first t The acceleration vector of the frame. For the object in the first t The three-dimensional external force vector (gravity, friction, air resistance, braking force, etc.) acting on the frame. For the object in the first t The three-dimensional internal forces or structural constraints acting on the frame (such as equipment structural stress and vehicle body rigidity constraints). This is the square of the L2 norm, used here to quantify the mismatch between the generated force and acceleration. By using dynamic constraint loss, the acceleration and force of objects in the generated scene can be forced to strictly match Newton's second law, eliminating dynamic errors such as "acceleration without external force, no deceleration under force, and weightlessness during suspension."

[0048] The formula for the thermodynamic constraint loss is: ; in, This represents the total number of timing steps for the process parameters. For the first The change in heat of the system at any given time (J). The specific heat capacity of the medium (J / (kg·℃)) is automatically matched according to the medium type. For medium quality , The target temperature change For the first Pressure change over time (MPa) Volume of enclosed equipment (m³) 3 ), The amount of gaseous substance (mol). The gas constant is an ideal gas constant, with a fixed value of 8.314 J / (mol·K). For the first The change in temperature at any given time (°C). The first half of the thermodynamic constraint loss is based on the heat formula to constrain the rationality of temperature changes, eliminating errors such as temperature jumps and temperature rise without a heat source; the second half is based on the ideal gas law to constrain the temperature-pressure linkage relationship, adapting to long-tail anomaly scenarios in industry such as chemical reactors, high-temperature processing equipment, and pressure pipelines, ensuring that the generated time-series process data fully conforms to industrial thermodynamic mechanisms.

[0049] The formula for the collision constraint loss is: ; in, This represents the total number of objects in the scene. For objects i The radius (or the radius of the enclosing sphere). For objects The radius (or the radius of the enclosing sphere). For objects i The position vector, For objects j The position vector, The function aims to maximize the value of the input, meaning that loss only occurs when two objects penetrate each other. It is an L2 norm. The collision constraint loss ensures that objects do not penetrate each other. When the distance between two objects is less than the sum of their radii, the loss value increases sharply, thus penalizing unrealistic collision behavior.

[0050] The multimodal semantic alignment loss is: ; in, For multimodal semantic alignment loss, The total number of modes, The number of semantic objects in the scene. For the first The first mode Feature vectors of semantic objects In the parameterized representation, the first Feature vectors of semantic objects for and The cosine similarity between features is used. Multimodal semantic alignment loss ensures that all modalities accurately represent the same semantic content by maximizing the cosine similarity between features of different modalities and text features. For example, in autonomous driving scenarios, "red car" in an image, "red car" in a point cloud, and "red car" in radar data should have similar feature representations.

[0051] Step 205 evaluates multimodal long-tail scenarios from four dimensions: physical realism, semantic consistency, task relevance, and diversity. The evaluation results are then fed back to the large language model and the diffusion model for iterative optimization. Specifically, the training process of the diffusion model is adjusted based on the evaluation results, and the prompt words of the large language model are modified. Furthermore, high-quality multimodal long-tail scenarios are added to the training dataset.

[0052] The physical authenticity is evaluated using physical constraint loss, the semantic consistency is evaluated using multimodal semantic alignment loss, the task relevance is evaluated using performance metrics of the diffusion model (such as accuracy, recall, and F1 score), and the diversity is determined using scene coverage.

[0053] This application also provides the application process of the multimodal long-tail scene automatic generation method in the fields of autonomous driving, high-end precision manufacturing and process control.

[0054] (a) Generation of long-tail scenarios for autonomous driving.

[0055] In the field of autonomous driving, this application can be used to generate various extreme weather and complex traffic scenarios, such as pedestrians crossing the road in heavy rain, vehicles skidding in snow, and obstacles on unlit roads at night. The specific implementation steps are as follows: (1) Collect autonomous driving accident reports, traffic regulations and expert knowledge to build a semantic knowledge base containing 100+ long-tail scenarios.

[0056] (2) Use the GPT-4o large language model to generate scene descriptions and set... The long-tailed distribution reweighted coefficients increase the probability of generating rare scenes.

[0057] (3) Convert the scene description into a parameterized representation, including vehicle type, pedestrian location, weather conditions, light intensity, etc.

[0058] (4) Generate multimodal data based on the Unity physics engine and the Stable Diffusion XL model, including RGB images, lidar point clouds, millimeter-wave radar data and inertial measurement unit (IMU) data.

[0059] (5) During the generation process, set The physical constraint loss weights are calculated, with a focus on collision and dynamic constraints.

[0060] (6) Use the Contrastive Language-Image Pre-training (CLIP) model to calculate the multimodal semantic alignment loss to ensure semantic consistency between different modalities.

[0061] (7) Automatically evaluate the generated scenes and select high-quality scenes that are physically realistic and semantically consistent.

[0062] (8) Use the generated scene data to train the autonomous driving perception and decision-making model to improve the robustness of the model in extreme scenarios.

[0063] like Figure 5 It showcases multimodal data generated in the field of autonomous driving. Figure 5 Part (a) in the image is an RGB image of pedestrians crossing the road in the rain. Specifically, it is a street view of a city road in the rain, showing a scene of pedestrians crossing a zebra crossing in the rain. This is a typical long-tail, high-risk scenario in the field of autonomous driving. The generated visual modal data shows that, guided by a large language model and constrained by physical parameters, the rainy environment, pedestrian posture, and vehicle position in the generated scene all conform to the laws of real traffic scenarios, solving the problem of difficult data collection in extreme weather scenarios.

[0064] Figure 5 Part (b) of the diagram contains the LiDAR point cloud data for the corresponding scene. Presented as a point cloud, it clearly outlines the contours of pedestrians, roads, vehicles, and trees, demonstrating the LiDAR's 3D perception of the scene. The generated LiDAR modal data highlights the core role of multimodal semantic alignment loss: the pedestrian positions and road structures in the point cloud completely correspond to the RGB images, ensuring consistent descriptions of the same scene from different sensors. This data can be directly used to train a multimodal fusion perception model.

[0065] Figure 5 Part (c) is the millimeter-wave radar heatmap for the corresponding scene. Presented as a grayscale heatmap, it demonstrates the millimeter-wave radar's perception of the scene, clearly identifying two pedestrian targets and illustrating the interference effect of rain on the radar signal. The generated millimeter-wave radar modal data shows that the signal strength and target distribution are consistent with the RGB image and point cloud data, fully simulating the real-world perception effect of radar in rainy conditions. This provides high-fidelity training data for the multi-sensor fusion model of autonomous driving.

[0066] Figure 5 Part (d) in the figure is the vehicle IMU acceleration time-series curve, with the horizontal axis representing the time step and the vertical axis representing the IMU acceleration value. The curve shows the acceleration fluctuations of a vehicle driving on a wet and slippery road surface in rainy weather, simulating the bumps and braking process of driving in the rain. The generated inertial sensor time-series data shows that the curve's fluctuation pattern and peak value conform to the vehicle dynamics characteristics on a rainy road surface, demonstrating the application of physical constraint loss and ensuring that the generated data conforms to the real vehicle motion patterns. This data can be directly used to train autonomous driving decision-making and control models.

[0067] (ii) Generation of defect scenarios in high-precision manufacturing.

[0068] In the field of high-precision manufacturing, this application can be used to generate various rare product defects and equipment failure scenarios, such as micro-cracks in semiconductor chips, dimensional deviations in precision mechanical parts, and abnormal movements of industrial robots. The specific implementation steps are as follows: (1) Collect defect reports, equipment maintenance records and process parameters in the precision manufacturing process, and build a semantic knowledge base containing 50+ types of defect scenarios.

[0069] (2) Use a large language model to generate a description of the defect scenario and set... The long-tailed distribution of reweighted coefficients focuses on extremely rare defect types.

[0070] (3) Convert the scene description into a parameterized representation, including defect type, defect location, defect size, process parameters, etc.

[0071] (4) Generate multimodal data based on finite element analysis (FEA) and generative adversarial networks, including industrial camera images, X-ray inspection images, vibration sensor data and temperature sensor data.

[0072] (5) During the generation process, thermodynamic constraint loss and material mechanical constraint loss are introduced to ensure that the shape and distribution of defects conform to the laws of materials science.

[0073] (6) Use multimodal semantic alignment loss to ensure that the data obtained by different detection methods have consistent defect semantics.

[0074] (7) Automatically evaluate the generated defect scenarios and select high-quality defect data that meet industrial standards.

[0075] (8) Use the generated defect data to train the industrial defect detection model to improve the model's detection accuracy for rare defects.

[0076] like Figure 6 It showcases multimodal data generated in the field of high-precision manufacturing. Figure 6 Part (a) shows an example of a scanning electron microscope (SEM) image of surface cracks on a semiconductor wafer / chip, illustrating two typical semiconductor structures: square chips and circular wafers. The surface exhibits typical crack / microcrack morphology, a rare defect scenario that is extremely difficult to capture in precision manufacturing. This defect generation is based on material mechanical constraints. Through physical constraint losses (such as material stress distribution and crack propagation laws), it ensures that the generated crack morphology, orientation, and distribution conform to the characteristics of the silicon wafer material, rather than being randomly generated "false defects."

[0077] Figure 6 Part (b) of the image shows multimodal defect detection data (X-ray + SEM). The left side shows the X-ray image, revealing the radial distribution characteristics of the defects; the right side shows the SEM image of the same wafer, displaying densely distributed microcracks in a radial pattern. This embodies the multimodal semantic alignment loss of this application: for the same defect scene, the features (crack location, distribution range) remain consistent under both X-ray and SEM modes, ensuring the semantic uniformity of the generated data under different detection devices, and allowing it to be directly used to train a multimodal defect detection model.

[0078] Figure 6 Part (c) is the block diagram of the physical / process parameter control logic for defect generation. The dashed box shows how process parameters (such as temperature / stress curves) drive defect generation. The upper curve represents traditional process parameters (A), and the lower curve represents NFET (new process parameters / constraints). The process is: parameter input → physical constraint unit → generation of SEM defect image, reflecting the linkage between "process-physics-defect". Corresponding to the "parametric representation conversion + physical engine unit", the defect scene description generated by the large language model is converted into controllable process parameters, and defect generation is driven by physical simulation to ensure that the defect generation process conforms to the manufacturing process rules.

[0079] Figure 6 Part (d) shows the process parameter time-series curves (defect process data), with time on the horizontal axis and parameter values ​​on the vertical axis. Two sets of time-series data are displayed: ① A large sine wave: periodic changes in process parameters (such as temperature / stress); ② The noise curve below: small fluctuations in the defect generation process. The location marked "NFET" corresponds to... Figure 6 The novel process constraints in section (c) demonstrate the effectiveness of this application in generating not only static defect images but also dynamic time-series sensor data (such as vibration and temperature curves), fully reproducing the process from defect generation to development, and meeting the training requirements of time-series data-based fault early warning models in industry.

[0080] (iii) Generation of abnormal operating conditions in process industries.

[0081] In the process industry, this application can be used to generate various abnormal operating conditions, such as over-temperature and over-pressure in chemical reactors, pipeline leaks, and valve malfunctions. The specific implementation steps are as follows: (1) Collect accident reports, historical data of distributed control systems (DCS), and safe operating procedures from process industries to build a semantic knowledge base containing 80+ types of abnormal operating conditions.

[0082] (2) Use a large language model to generate abnormal operating condition descriptions and set... The long-tailed distribution reweighting coefficients balance the generation probabilities of common and rare anomalies.

[0083] (3) Convert the scenario description into a parameterized representation, including the anomaly type, location of occurrence, severity, and changes in process parameters.

[0084] (4) Generate multimodal data based on process simulation software (such as Aspen Plus) and time series generation model, including time series curves of process parameters such as temperature, pressure, and flow rate, industrial videos and alarm information.

[0085] (5) During the generation process, thermodynamic constraint loss and hydrodynamic constraint loss are introduced to ensure that the changes in process parameters conform to chemical engineering principles.

[0086] (6) Use multimodal semantic alignment loss to ensure semantic consistency between process parameter curves and video content.

[0087] (7) Automatically evaluate the generated abnormal operating conditions and select high-quality abnormal data that meet industrial safety standards.

[0088] (8) Use the generated abnormal data to train the process monitoring and fault diagnosis model to improve the model’s early warning capability for abnormal operating conditions.

[0089] like Figure 7 It showcases multimodal data generated in the process industry. Figure 7 Part (a) shows the monitoring screen of the abnormal operating condition of the chemical reactor, presenting a screenshot of the monitoring video of the chemical reactor production site. The image is marked "abnormal operating condition of chemical reactor with over-temperature and over-pressure", which is a typical high-risk long-tail scenario in the process industry. The generated visual modal data shows that, guided by a large language model and physical constraints, abnormal operating condition images that conform to the industrial site layout and equipment appearance were generated, solving the problem of difficulty in capturing and collecting real abnormal scenes.

[0090] Figure 7 Part (b) shows the temperature time-series curve of the reactor, with time on the horizontal axis and temperature on the vertical axis. The curve exhibits abnormal fluctuations: small fluctuations in the early stage followed by a sharp peak, simulating the abnormal temperature rise process under over-temperature conditions. The generated sensor data modes demonstrate the application of thermodynamic constraint loss: the rate of change and peak size of the curves conform to the thermodynamic laws of the chemical reactor, rather than being randomly generated invalid data, and can be directly used to train the temperature early warning model.

[0091] Figure 7Part (c) shows the reactor pressure-time curve, with time on the horizontal axis and pressure on the vertical axis. The curve exhibits periodic abnormal fluctuations, simulating pressure oscillations under overpressure conditions. This is further analyzed using another source of process parameter time-time data, compared with... Figure 7 The temperature curves shown in part (b) constitute the complete process data of the "over-temperature and over-pressure" condition, demonstrating the ability of this application to generate multimodal time series data and meeting the training requirements of industrial fault diagnosis models.

[0092] Figure 7 Part (d) in the table presents the equipment alarm information sequence in tabular form, with 0 / 1 markings representing the triggering status of different alarm points and logical symbols at the bottom indicating the correlation between alarm events. The application of multimodal semantic alignment loss is demonstrated through the corresponding generated event sequence modalities: the alarm sequence corresponds one-to-one with the abnormal nodes of the temperature and pressure curves, ensuring that different modal data provide completely consistent descriptions of the same abnormal operating condition.

[0093] This application leverages the commonsense reasoning and semantic generation capabilities of Large Language Models (LLMs), combined with strict physical constraints and multimodal consistency verification, to achieve automated and high-quality generation of long-tail extreme scenarios in autonomous driving, high-precision manufacturing, and process industries. This application proposes three core mathematical models: a physical constraint loss function, a multimodal semantic alignment loss, and a long-tail distribution reweighting mechanism. These models address the issues of physical inaccuracies, modal inconsistencies, and insufficient coverage of long-tail scenarios inherent in traditional generation methods. Practical results demonstrate that the long-tail scenarios generated by this application outperform existing methods in terms of physical realism, semantic consistency, and task performance improvement. This significantly reduces data acquisition costs for industrial scenarios and enhances the robustness and security of AI models.

[0094] Based on the same inventive concept, this application also provides a multimodal long-tail scene automatic generation system for implementing the above-mentioned multimodal long-tail scene automatic generation method. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal long-tail scene automatic generation system provided below can be found in the limitations of the multimodal long-tail scene automatic generation method above, and will not be repeated here.

[0095] In one exemplary embodiment, such as Figure 8 As shown, a multimodal long-tail scene automatic generation system is provided, including the following functional modules: The knowledge base construction module 801 is used to build a long-tail scene semantic knowledge base for the target domain. Specifically, the long-tail scene semantic knowledge base module can store and manage long-tail scene knowledge in the target domain, and supports knowledge retrieval and updates.

[0096] The natural language generation module 802 is used to generate natural language descriptions for each scenario based on the long-tail scenario semantic knowledge base using a large language model, and to adjust the generation probability of each scenario using a long-tail distribution reweighting mechanism during the generation process. Specifically, the natural language generation module generates diverse scenario natural language descriptions based on the long-tail scenario semantic knowledge base, while introducing a long-tail distribution reweighting mechanism.

[0097] The natural language conversion module 803 is used to convert the natural language description of each scenario into a parameterized representation of the corresponding scenario.

[0098] The scene generation module 804 is used to generate multimodal content for each scene based on the parameterized representation of each scene, the physics engine and the diffusion model. In the generation process, physical constraint loss and multimodal semantic alignment loss are introduced to make the motion, force and interaction of objects in the generated multimodal content conform to the laws of Newtonian mechanics and thermodynamics. At the same time, the cosine similarity between different modal contents and parameterized representations is maximized to obtain multimodal long-tailed scenes.

[0099] The Scene Evaluation and Optimization Module 805 is used to automatically evaluate multimodal long-tail scenes from four dimensions: physical authenticity, semantic consistency, task relevance, and diversity. The evaluation results are then fed back to the large language model and diffusion model for parameter updates and model optimization.

[0100] This application deploys a multimodal long-tail scene automatic generation system as a cloud service or edge device, supports user-defined scene parameters and constraints, and provides API interfaces for easy integration with existing AI training platforms.

[0101] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores a semantic knowledge base of long-tail scenes in the target domain. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal long-tail scene automatic generation method.

[0102] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment to which the present application is applied. Specific computer equipment may include, for example, [the following is a list of possible additional structures]. Figure 9 The diagram shows more or fewer components, or combinations of certain components, or different component arrangements.

[0103] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0104] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0106] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0107] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0108] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0109] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for automatically generating multimodal long-tail scenes, characterized in that, include: Construct a long-tail scenario semantic knowledge base for the target domain; Based on the aforementioned long-tail scene semantic knowledge base, a large language model is used to generate natural language descriptions for each scene, and a long-tail distribution reweighting mechanism is employed to adjust the generation probability of each scene during the generation process. This adjustment includes: filtering knowledge from the long-tail scene semantic knowledge base to construct a training dataset; the training dataset includes knowledge samples from various scenes; and calculating the long-tail distribution reweighting coefficient for each scene based on the training dataset. Specifically, the formula is used... Calculate the first The reweighting coefficients for the long-tail distribution of the class scenario; where, For the first The reweighting coefficients for the long-tail distribution of similar scenarios. The total number of knowledge samples in the training dataset, For the training dataset, the first Number of knowledge samples for each scenario This is a smoothing coefficient used to control the strength of the reweighting. Based on the long-tail distribution reweighting coefficient of each scenario, the probability of generating each scenario by the large language model is reweighted to adjust the generation probability of each scenario. The natural language description of each scenario is converted into a parameterized representation of the corresponding scenario; Based on the parameterized representation of each scene, multimodal content for each scene is generated using a physics engine and a diffusion model. Physical constraint loss and multimodal semantic alignment loss are introduced during the generation process to ensure that the motion, forces and interactions of objects in the generated multimodal content conform to the laws of Newtonian mechanics and thermodynamics. At the same time, the cosine similarity between different modal contents and parameterized representations is maximized to obtain multimodal long-tail scenes. The multimodal content includes images, point clouds, videos, and sensor data; the physical constraint loss includes kinematic constraint loss, dynamic constraint loss, thermodynamic constraint loss, and collision constraint loss. The formula for the physical constraint loss is: ； in, For physical constraint loss, For kinematic constraint loss, For dynamic constraint loss, For thermodynamic constraint loss, For collision constraint loss, These are the weighting coefficients for the kinematic constraint loss. These are the weighting coefficients for the dynamic constraint loss. This is the weighting coefficient for thermodynamic constraint loss. These are the weighting coefficients for the collision constraint loss; The formula for the kinematic constraint loss is: ； in, T The total number of frames in the video. For the object in the first t The position vector of the frame. For the object in the first t -1 frame's position vector For the object in the first t -1 frame velocity vector The time interval between two adjacent frames. For the object in the first t Acceleration vector of frame -1 The square of the L2 norm; The formula for the dynamic constraint loss is: ； in, The mass of the current object, For the object in the first t The acceleration vector of the frame. For the object in the first t The three-dimensional external force vector acting on the frame. For the object in the first t The three-dimensional internal forces acting on the frame; The formula for the thermodynamic constraint loss is: ； in, This represents the total number of timing steps for the process parameters. For the first The change in the system's heat at any given time. For the specific heat capacity of the medium, For the quality of the medium, The target temperature change For the first The change in pressure over time. For the volume of the sealed equipment, For the amount of gaseous substance, Let be the ideal gas constant. For the first The change in temperature over time; The formula for the collision constraint loss is: ； in, This represents the total number of objects in the scene. For objects i radius, For objects radius, For objects i The position vector, For objects j The position vector, To find the maximum value function, It is an L2 norm.

2. The method for automatically generating multimodal long-tail scenes according to claim 1, characterized in that, The following formula is used to generate the first... The probabilities of similar scenarios are reweighted: ； in, For the reweighted first The generation probability of the scene type For the first The reweighting coefficients for the long-tail distribution of similar scenarios. Generate the first for large language models The probability of similar scenarios The total number of scene categories. For the first The reweighting coefficients for the long-tail distribution of similar scenarios. Generate the first for large language models The probability of a scenario.

3. The method for automatically generating multimodal long-tail scenes according to claim 1, characterized in that, The multimodal semantic alignment loss is: ； in, For multimodal semantic alignment loss, The total number of modes, The number of semantic objects in the scene. For the first The first mode Feature vectors of semantic objects In the parameterized representation, the first Feature vectors of semantic objects It is an L2 norm.

4. The method for automatically generating multimodal long-tail scenes according to claim 1, characterized in that, The method further includes: The multimodal long-tail scenario is evaluated from four dimensions: physical realism, semantic consistency, task relevance and diversity. The evaluation results are fed back to the large language model and the diffusion model for iterative optimization. The physical authenticity is evaluated using physical constraint loss, the semantic consistency is evaluated using multimodal semantic alignment loss, the task relevance is evaluated using the performance metrics of the diffusion model, and the diversity is determined using scene coverage.

5. A multimodal long-tail scene automatic generation system, applied to the multimodal long-tail scene automatic generation method according to any one of claims 1-4, characterized in that, The system includes: The knowledge base construction module is used to build a long-tail scenario semantic knowledge base for the target domain; The natural language generation module is used to generate natural language descriptions for each scenario based on the long-tail scenario semantic knowledge base using a large language model, and to adjust the generation probability of each scenario using a long-tail distribution reweighting mechanism during the generation process. The natural language conversion module is used to convert the natural language description of each scenario into a parameterized representation of the corresponding scenario; The scene generation module is used to generate multimodal content for each scene based on the parameterized representation of each scene, the physics engine and the diffusion model. During the generation process, physical constraint loss and multimodal semantic alignment loss are introduced to ensure that the motion, force and interaction of objects in the generated multimodal content conform to the laws of Newtonian mechanics and thermodynamics. At the same time, the cosine similarity between different modal contents and parameterized representations is maximized to obtain multimodal long-tail scenes.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the multimodal long-tail scene automatic generation method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for automatically generating multimodal long-tailed scenes as described in any one of claims 1-4.

Images