A multi-modal three-dimensional modeling and virtual environment simulation system based on an industrial scene

Abstract

The application discloses a kind of multi-modal three-dimensional modeling and virtual environment simulation system based on industrial scene, it is related to virtual environment construction technical field, the present application realizes industrial scene digitization simulation and optimization, collects laser point cloud and other various data and completes space-time calibration and heterogeneous fusion;Knowledge graph driven industrial semantic modeling module constructs the industrial ontology knowledge graph containing multiple constraints, generates three-dimensional model with semantic constraint and completes interference check;Cross-domain unified physics engine simulation module synchronously executes multi-domain simulation, realizes data intercommunication and strong linkage, judges index conflict in real time;Virtual-real synchronous digital twinning mapping module establishes two-way real-time mapping, supports PLC debugging and parameter correction;Autonomous decision optimization output module identifies problem and traces source, outputs optimal scheme.Solve multi-source data fusion, multi-domain simulation linkage and other pain points, realize virtual-real depth fusion, improve industrial production precision and efficiency, help digital transformation.

Description

Technical Field

[0001] This invention relates to the field of virtual environment construction technology, specifically to a multimodal 3D modeling and virtual environment simulation system based on industrial scenarios. Background Technology

[0002] Various industrial sectors are accelerating their transformation towards digitalization and intelligence. 3D modeling and virtual environment simulation have become the core infrastructure for the full lifecycle management of industrial scenarios. They are widely used in many equipment-intensive and structurally complex industrial fields such as automobile manufacturing, energy and chemical industry, aerospace, and intelligent warehousing. They provide accurate data support and technical assurance for factory design, construction, production operation and maintenance, and optimization and upgrading. Therefore, a multimodal 3D modeling and virtual environment simulation system based on industrial scenarios is needed.

[0003] Existing technology, such as the invention application patent with publication number CN119917951A, discloses a digital twin system based on multimodal recognition. This system integrates devices, a digital twin, and an information transmission medium to achieve the acquisition, processing, fusion analysis, and feedback control of multimodal information. The system utilizes a multimodal signal acquisition device to collect various modal information, and performs data fusion analysis and scenario simulation through a simulator and a large multimodal model to guide the device execution entity in precise operation. The system possesses intelligent learning and adaptability, and can update its professional knowledge base through manual teaching, improving accuracy and efficiency. Simultaneously, the system uses a knowledge graph to filter uncertainties, ensuring high-precision operation. This invention provides an efficient and accurate solution for fields such as smart agriculture, intelligent manufacturing, virtual humans, and humanoid robots.

[0004] Regarding the above-mentioned solutions, the inventors of this application have discovered at least the following technical problems: 1. The existing technologies lack sufficient synergy in the acquisition and fusion of multi-source heterogeneous data, making it difficult to support accurate modeling and simulation. In current industrial scenarios, the acquisition of various types of data, such as laser point clouds, CAD models, video images, and PLC timing signals, largely relies on independent equipment and processes, lacking a unified spatiotemporal reference calibration mechanism. This results in issues such as timestamp misalignment and inconsistent spatial coordinates among data from different sources, leading to poor data compatibility. Furthermore, existing technologies for fusing heterogeneous data often remain at the level of simple splicing, failing to effectively combine industrial semantic information to achieve deep fusion. They are also susceptible to interference from the acquisition environment, resulting in data loss and signal distortion. Consequently, the fused data cannot fully and accurately reflect the true state of the industrial scenario, leading to insufficient accuracy in subsequent 3D modeling. This makes it impossible to accurately replicate core scenario information such as equipment layout and process flow, creating hidden dangers for subsequent simulation analysis and decision optimization, and failing to meet the modeling needs of complex industrial scenarios.

[0005] 2. Existing multi-domain simulation technologies lack sufficient linkage and accuracy, and lack a unified constraint and collaboration mechanism. Current industrial simulation technologies mostly adopt a single-domain independent simulation mode. Multi-domain simulations such as robot motion simulation, assembly process simulation, and logistics scheduling simulation are independent of each other, lacking a unified physical engine kernel. Simulation data from different domains cannot achieve real-time interoperability and synchronous linkage, making it difficult to simulate the real-world collaborative operation of multiple links in industrial scenarios. Furthermore, the lack of real-time constraints and guidance from industrial ontology knowledge graphs during the simulation process makes it impossible to accurately define cross-domain linkage rules. This leads to a disconnect between the simulation process and actual industrial processes, logistics rules, and safety regulations. It not only makes it difficult to accurately determine threshold conflicts of core indicators such as production line cycle time and equipment utilization, but also fails to effectively identify potential problems such as dynamic assembly interference and logistics path congestion. The practicality and reliability of the simulation results are greatly reduced, making it difficult to provide accurate support for production optimization.

[0006] 3. Existing virtual-real mapping technologies lack real-time performance and coordination, and their decision optimization outputs lack specificity and feasibility. Most existing virtual-real synchronization technologies can only achieve one-way feedback from physical equipment status to the virtual model, making it difficult to achieve bidirectional real-time mapping between the virtual model and physical equipment. Parameter adjustments in the virtual model cannot be synchronized to the physical equipment in real time, and changes in the operating status of the physical equipment cannot be promptly fed back to the virtual model for dynamic updates, leading to a disconnect between virtual simulation and actual production. Furthermore, existing decision optimization modules often output optimization solutions based on single simulation results, failing to incorporate precise traceability results regarding production bottlenecks, logistical conflicts, and assembly risks. The output layout schemes and process paths lack clear parameter definitions and implementation standards, and are not tightly integrated with the linkage constraints of the industrial ontology knowledge graph. The solutions lack specificity and are difficult to directly apply to industrial scenarios, failing to effectively address core pain points in actual production. Summary of the Invention

[0007] To address the aforementioned technical shortcomings, the present invention aims to provide a multimodal 3D modeling and virtual environment simulation system based on industrial scenarios.

[0008] To solve the above technical problems, the present invention adopts the following technical solution: The present invention provides a multimodal 3D modeling and virtual environment simulation system based on industrial scenarios, including the following modules: Multimodal spatiotemporal alignment data input module: used to deploy multi-source data acquisition devices in the target industrial scenario, acquire laser point clouds, CAD models, video images, process text, PLC timing signals, IoT real-time data and production logistics trajectories, and perform unified spatiotemporal benchmark calibration and heterogeneous data fusion.

[0009] The knowledge graph-driven industrial semantic modeling module is used to construct an industrial ontology knowledge graph containing equipment attributes, process constraints, logistics rules, safety specifications, and assembly logic based on fused heterogeneous data, thereby automatically generating a parametric 3D model with semantic constraints and automatically completing static and dynamic interference checks.

[0010] Cross-domain unified physics engine simulation module: It is used to synchronously execute robot motion simulation, assembly process simulation, human-machine engineering simulation, logistics scheduling simulation and production line cycle simulation under the same physics engine kernel. Cross-domain linkage constraints are defined in real time by industrial ontology knowledge graph, which completes the real-time interoperability of multi-domain simulation data and the strong linkage closed loop exclusive to industrial scenarios, and judges the index thresholds and linkage conflicts in real time.

[0011] Virtual-Real Synchronous Digital Twin Mapping Module: Used to establish a two-way real-time mapping relationship between virtual models and physical equipment, supporting online virtual debugging of PLC programs, real-time correction of process parameters, and synchronous mirroring of equipment status.

[0012] Autonomous Decision Optimization Output Module: Based on simulation results and twin data, it identifies production bottlenecks, logistics conflicts and assembly risks, and combines knowledge graphs to complete rule tracing, outputting the optimal layout scheme, process path, logistics strategy and equipment configuration list.

[0013] The beneficial effects of this invention are as follows: 1. This embodiment of the solution, through deep integration of multimodal spatiotemporal alignment and industrial ontology knowledge graph, unifies seven heterogeneous data sources—laser point cloud, CAD, video, process text, PLC timing, IoT data, and logistics trajectory—into a parametric 3D model with semantic constraints. This fundamentally solves the problems of data silos and semantic gaps in traditional simulations, enabling static and dynamic interference checks to automatically and comprehensively cover all spatial conflicts of equipment, tooling, personnel, and logistics paths. Secondly, based on the same physics engine kernel, it synchronously executes five-domain simulations: robot motion, assembly process, ergonomics, logistics scheduling, and production line cycle time. Furthermore, the knowledge graph defines cross-domain linkage constraints in real time, breaking the limitations of previous simulations. Overcoming the limitations of independent module operation and script-based linkage, this solution achieves real-time data exchange and a strong closed-loop linkage across the entire domain, significantly improving the detection rate and response speed of multi-domain conflicts. Furthermore, the virtual-physical bidirectional mapping and PLC online virtual debugging function enable real-time synchronization of the virtual model and physical equipment status, supporting online correction of process parameters and synchronous mirroring of equipment faults, significantly shortening the on-site debugging cycle and reducing the risk of production interruption. Finally, the system can automatically identify production bottlenecks, logistics conflicts, and assembly risks, and perform rule tracing based on knowledge graphs. It then outputs optimal layout schemes, process paths, logistics strategies, and equipment configuration lists differentiated according to the root causes of problems, achieving an end-to-end closed loop from data acquisition to simulation verification and autonomous optimization decision-making. Compared to existing technologies, this solution significantly improves the planning efficiency of complex industrial scenarios, reduces trial-and-error costs, and ensures the safety and stable operation of the production system in terms of simulation collaboration, semantic-driven capabilities, closed-loop decision-making efficiency, and industrial adaptability.

[0014] 2. This solution overcomes the limitations of existing technologies in terms of the insufficient linkage and accuracy of multi-domain simulations, constructing a multi-domain collaborative simulation system under unified constraints, thus improving the practicality and reliability of simulation results. This solution, through a cross-domain unified physics engine simulation module, supported by a single physics engine kernel, enables the synchronous execution of multi-domain simulations such as robot motion, assembly processes, and logistics scheduling. This breaks down the barriers of independent simulation in a single domain and achieves real-time interoperability of multi-domain simulation data. Simultaneously, relying on an industrial ontology knowledge graph to define cross-domain linkage constraints in real time, the simulation process strictly adheres to actual industrial processes, logistics rules, and safety regulations. It can not only accurately determine threshold conflicts of core indicators such as production line cycle time and equipment utilization, but also effectively identify potential problems such as dynamic assembly interference and logistics path congestion. Through a three-level conflict marking mechanism, it achieves accurate source tracing of conflicts, significantly improving the practicality and reliability of simulation results and providing precise technical support for production optimization.

[0015] 3. This solution overcomes the shortcomings of existing technologies, such as insufficient synergy in virtual-physical mapping and weak targeting of decision optimization schemes. It achieves deep integration of virtual simulation and actual production, and enables the practical implementation of optimization schemes. Through a virtual-physical synchronous digital twin mapping module, this solution establishes a bidirectional real-time mapping relationship between the virtual model and physical equipment, enabling instruction communication and state synchronization between the virtual model and physical equipment. This solves the problem of disconnect between virtual simulation and actual production caused by the one-way feedback of existing technologies. It supports online virtual debugging of PLC programs and real-time correction of process parameters, improving the efficiency of production operation and maintenance. Simultaneously, the autonomous decision optimization output module generates named and differentiated optimization schemes based on simulation results, twin data, and precise traceability results of bottlenecks, conflicts, and risks, targeting different traceability scenarios. The output layout schemes and process paths all have clear parameter definitions and implementation standards, closely integrated with the industrial ontology knowledge graph. The solutions are highly targeted and can be directly applied to industrial scenarios, effectively solving core pain points in actual production and helping enterprises achieve simultaneous improvements in production efficiency, product quality, and safety levels. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the system structure connection of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Examples of embodiments of the present invention Figure 1 As shown, a multimodal 3D modeling and virtual environment simulation system based on industrial scenarios includes the following modules: a multimodal spatiotemporal aligned data input module, a knowledge graph-driven industrial semantic modeling module, a cross-domain unified physics engine simulation module, a virtual-real synchronous digital twin mapping module, and an autonomous decision-making optimization output module.

[0020] The knowledge graph-driven industrial semantic modeling module is connected to the multimodal spatiotemporal aligned data input module and the cross-domain unified physics engine simulation module, respectively. The virtual-real synchronous digital twin mapping module is connected to the cross-domain unified physics engine simulation module and the autonomous decision optimization output module, respectively.

[0021] Multimodal spatiotemporal alignment data input module: used to deploy multi-source data acquisition devices in target industrial scenarios to collect laser point clouds, CAD models, video images, process text, PLC timing signals, IoT real-time data and production logistics trajectories, and perform unified spatiotemporal benchmark calibration and heterogeneous data fusion.

[0022] In a specific embodiment, the acquisition of laser point clouds, CAD models, video images, process texts, PLC timing signals, IoT real-time data, and production logistics trajectories is carried out as follows: Three-dimensional laser point cloud data of the target industrial scene is acquired using a LiDAR scanner; on-site video image data is acquired using an industrial camera; process documents and work instructions are read using a text parsing tool to extract process flows, process parameters, and quality standards; PLC timing signals are acquired via industrial Ethernet to obtain time-series data on equipment start-up and shutdown, action execution, and process connections; real-time data on equipment operating status, ambient temperature and humidity, and material location are acquired using IoT sensors; the operating trajectory, speed, load capacity, and delivery cycle data of AGV / AMR, conveyor lines, and forklifts are acquired using logistics trajectory tracking equipment; all acquired data is accessed to the data processing center via an industrial communication network to establish a unified scene coordinate system and timestamp benchmark, completing the spatiotemporal alignment and normalization processing of heterogeneous data.

[0023] The knowledge graph-driven industrial semantic modeling module is used to construct an industrial ontology knowledge graph containing equipment attributes, process constraints, logistics rules, safety specifications, and assembly logic based on fused heterogeneous data, thereby automatically generating a parametric 3D model with semantic constraints and automatically completing static and dynamic interference checks.

[0024] In a specific embodiment, the construction of an industrial ontology knowledge graph that includes equipment attributes, process constraints, logistics rules, safety specifications, and assembly logic is carried out as follows: Five core entity layers are constructed: equipment, process, logistics, safety, and personnel. The physical attributes, state attributes, motion attributes, spatial attributes, and interrelationships of each entity are obtained. Process documents are automatically parsed using NLP to extract entity features and association conditions. Entity alignment and attribute completion are completed by combining point cloud, CAD, and video data. This forms an industrial-specific knowledge graph that is dynamically iterative, includes assembly linkage constraints, logistics trigger conditions, and safety threshold constraints.

[0025] It should be noted that the process involves acquiring the physical attributes, state attributes, motion attributes, spatial attributes, and interrelationships of each entity; automatically parsing process documents using NLP to extract entity features and association conditions; combining point cloud, CAD, and video data to complete entity alignment and attribute completion; and forming an industrial-specific knowledge graph that can be dynamically iterated, with assembly linkage constraints, logistics triggering conditions, and safety threshold constraints.

[0026] Among them, equipment entity attributes include model, size, power, range of motion, installation coordinates, connection ports, and operating status; process entity attributes include process number, assembly sequence, work cycle, process parameter range, process connection sequence, and quality indicators; logistics entity attributes include material model, storage location, carrying capacity, delivery route nodes, conveying speed, and delivery cycle; safety entity attributes include safety distance threshold, protected area coordinates, restricted areas, and interference judgment conditions; personnel entity attributes include operating station coordinates, reachable range of movements, working posture range, and human-machine interaction space.

[0027] Inter-entity relationships include the execution relationship between equipment and process, the supply relationship between process and material, the matching relationship between material and equipment, the spatial relationship between equipment and safety area, and the operational relationship between personnel and workstation.

[0028] In a specific embodiment, the automatic generation of a parametric 3D model with semantic constraints is carried out as follows: the equipment attributes, process constraints, logistics constraints, safety specifications and assembly logic in the industrial ontology knowledge graph are mapped to the parametric modeling engine.

[0029] Based on the size, range of motion, and installation coordinate parameters of the equipment entity, automatically generate a 3D model of the equipment and configure kinematic pairs and motion constraints.

[0030] It should be noted that the automatic generation of 3D models for equipment is as follows: If the length of the equipment entity is between 500mm and 1000mm, the width is between 400mm and 800mm, the height is between 400mm and 1000mm, the range of motion is within ±90°, and the installation coordinate parameters are within ±0.5mm, then a small equipment body 3D model, a small motion mechanism 3D model, and a small mounting base 3D model will be automatically generated, and 1-2 sets of revolute joints, 1 set of prismatic joints, and corresponding travel limit constraints will be configured for the equipment model; if the length of the equipment entity is between 1000mm and 1800mm, the width is between 800mm and 1200mm, the height is between 1000mm and 1600mm, and the range of motion is within ±1... If the 80° range and the installation coordinate parameters are within ±0.5mm, a medium-sized equipment body 3D model, a medium-sized motion mechanism 3D model, and a medium-sized installation base 3D model will be automatically generated, and 3-4 sets of revolute joints, 2 sets of prismatic joints, and corresponding travel limit constraints will be configured for the equipment model. If the length of the equipment entity is between 1800mm and 3000mm, the width is between 1200mm and 2000mm, the height is between 1600mm and 2500mm, the motion range is within ±360°, and the installation coordinate parameters are within ±0.5mm, a large equipment body 3D model, a large motion mechanism 3D model, and a large installation base 3D model will be automatically generated, and 5-6 sets of revolute joints, 3 sets of prismatic joints, and corresponding travel limit constraints will be configured for the equipment model.

[0031] Based on the process sequence number, assembly order, and operation cycle parameters of the process entity, the workstation layout, tooling position, and operation space model are automatically generated.

[0032] It should be noted that the process model is automatically generated as follows: If the process entity's operation sequence number is between 1 and 10, the operation cycle time is between 20 and 60 seconds, the operation connection interval is between 0.2 and 1 second, the operating space height is between 600 and 800 mm, and the operating space width is between 500 and 800 mm, then a compact workstation layout model, a standard tooling positioning posture model, a small-sized personnel operating space model, and a short-distance operation connection path model will be automatically generated; if the process entity's operation sequence number is between 11 and 30, the operation cycle time is between 60 and 180 seconds, the operation connection interval is between 1 and 3 seconds, and the operating space height is between 800 and 1000 mm, then... If the process entity's process number is between 31 and 50, the work cycle time is between 180s and 360s, the process connection interval is between 3s and 10s, the operating space height is between 1000mm and 1500mm, and the operating space width is between 1200mm and 2000mm, then the extended workstation layout model, the customized tooling positioning posture model, the large-size personnel operating space model, and the long-distance process connection path model will be automatically generated.

[0033] Based on the storage location, carrying capacity, and delivery route node parameters of logistics entities, the model of the storage area, buffer area, and conveyor line is automatically generated.

[0034] It should be noted that the logistics model is automatically generated as follows: If the storage location coordinate accuracy of the logistics entity is within ±0.5mm, the storage location capacity is within 20kg-500kg, the distance between delivery route nodes is within 0.5m-2m, the conveyor line length is within 2m-20m, and the conveyor line width is within 100mm-300mm, then a light-duty warehouse rack model, a small buffer area workstation model, a short-distance conveyor line body model, and a compact logistics path guidance model will be automatically generated; if the storage location coordinate accuracy of the logistics entity is within ±0.5mm, the storage location capacity is within 500kg-2000kg, the distance between delivery route nodes is within 2m-6m, and the conveyor line width is within 100mm-300mm, then a light-duty warehouse rack model, a small buffer area workstation model, a short-distance conveyor line body model, and a compact logistics path guidance model will be automatically generated. If the length is between 20m and 60m and the conveyor line width is between 300mm and 500mm, a medium-sized warehouse rack model, a regular buffer workstation model, a medium-distance conveyor line body model, and a standard logistics path guidance model will be automatically generated. If the storage location coordinate accuracy of logistics entities is within ±0.5mm, the storage location carrying capacity is within 2000kg to 10000kg, the distribution path node spacing is within 6m to 15m, the conveyor line length is within 60m to 150m, and the conveyor line width is within 500mm to 800mm, a heavy-duty warehouse rack model, a large buffer workstation model, a long-distance conveyor line body model, and an extended logistics path guidance model will be automatically generated.

[0035] Based on the safety distance threshold, protected area coordinates, and restricted area parameters of safety entities, the safe area is automatically marked in the 3D scene.

[0036] It should be noted that automatic labeling of safety-related areas is as follows: If the safety distance threshold for a safety-related entity is between 300mm and 500mm, the coordinates of the protected area are within a small range, the boundary of the restricted area is within a small size range, and the interference judgment spacing is between 50mm and 100mm, then the small-range safety distance boundary area, small protective warning area, small-sized personnel restricted area, and precision equipment interference warning area will be automatically labeled in the 3D scene; if the safety distance threshold for a safety-related entity is between 500mm and 800mm, the coordinates of the protected area are within a medium range, the boundary of the restricted area is within a medium size range, and the interference judgment spacing is between 50mm and 100mm, then the small-range safety distance boundary area, small protective warning area, small-sized personnel restricted area, and precision equipment interference warning area will be automatically labeled in the 3D scene. If the spacing is between 100mm and 150mm, the system will automatically mark medium-range safety distance boundary areas, medium-sized protective warning areas, medium-sized personnel restricted areas, and conventional equipment interference warning areas in the 3D scene. If the safety distance threshold for safety entities is between 800mm and 1000mm, the coordinates of the protected area are in a large range, the boundaries of the restricted area are in a large size range, and the interference judgment spacing is between 150mm and 200mm, the system will automatically mark large-range safety distance boundary areas, large protective warning areas, large-sized personnel restricted areas, and wide-area equipment interference warning areas in the 3D scene.

[0037] The geometric, motion, and spatial parameters of the above model are bound in real time to the industrial ontology knowledge graph to form a simulateable 3D model with industrial linkage constraints.

[0038] In a specific embodiment, the automatic completion of static and dynamic interference checks is carried out as follows: Static interference check: Traverse the spatial coordinates and boundary dimensions of equipment, buildings, columns, and workstations in the 3D model, and calculate the minimum distance between adjacent entities one by one; compare the calculated minimum distance with a preset safety threshold, and determine static interference when the minimum distance is less than the safety threshold; mark the position coordinates, interference entity name, and distance difference of the static interference respectively.

[0039] It should be noted that, firstly, based on the knowledge graph-driven industrial semantic modeling module, a parametric 3D model with semantic constraints is generated. Spatial coordinate data and boundary dimension parameters of all entities to be inspected are extracted from the model, with the spatial coordinates based on a uniformly established scene coordinate system. Secondly, a bounding box algorithm is used to construct a minimum bounding box for each entity to be inspected, simplifying the spatial contours of irregular entities into regular bounding box structures. Then, all entities to be inspected are traversed, and the spatial position of the bounding boxes of each pair of adjacent entities is calculated using a pairwise pairing method. The spatial relative position of the two entities is determined by calculating the projection overlap of the two bounding boxes along the X, Y, and Z axes of the scene coordinate system. If there is no projection overlap between the two bounding boxes along a certain coordinate axis, the vertical distance between the edges of the two bounding boxes in that direction is calculated as the spacing in that direction. If projection overlap exists, the spacing in that direction is recorded as a negative value, indicating that there is overlap interference. Finally, the calculation results along the three coordinate axes are combined, and the minimum value is extracted as the minimum spacing between the adjacent entities in that group.

[0040] Dynamic interference check: Based on the motion constraints in the industrial ontology knowledge graph, the entire process of robot joint movement, AGV travel along the path, and personnel assembly in the workstation is simulated in sequence; the real-time position and posture of each entity are sampled at fixed time steps, and the instantaneous distance between moving entities and between moving entities and static entities is calculated; when the instantaneous distance is less than the collision threshold, it is judged as dynamic interference, and the time of interference, interference position, intersection of motion trajectory, and overlapping area of ​​action are recorded; the position, type and degree of all static and dynamic interferences are automatically marked.

[0041] Cross-domain unified physics engine simulation module: It is used to synchronously execute robot motion simulation, assembly process simulation, human-machine engineering simulation, logistics scheduling simulation and production line cycle simulation under the same physics engine kernel. Cross-domain linkage constraints are defined in real time by industrial ontology knowledge graph, which completes the real-time interoperability of multi-domain simulation data and the strong linkage closed loop exclusive to industrial scenarios, and judges the index thresholds and linkage conflicts in real time.

[0042] In a specific embodiment, the real-time interconnection and behavioral linkage of multi-domain simulation data is achieved as follows: a unified simulation clock and data interaction interface are established under the same physical engine kernel; cross-domain linkage constraints are defined in real time by the industrial ontology knowledge graph; the robot's motion state drives the assembly process in real time, the assembly state triggers logistics scheduling in real time, and the logistics state provides real-time feedback for production line cycle calculation, forming a closed loop of strong linkage across the entire industrial scenario, thereby realizing real-time interconnection and behavioral linkage of multi-domain simulation data.

[0043] In a specific embodiment, the real-time judgment of the indicator threshold and the linkage conflict is specifically judged as follows: real-time collection of simulation operation data, obtaining the actual production line cycle time, actual equipment utilization rate, actual logistics efficiency, actual assembly success rate and actual safety compliance rate, and comparing each actual indicator with the corresponding preset indicator threshold one by one.

[0044] If the actual production line cycle time exceeds the preset production line cycle time threshold, it is determined that the production line cycle time has exceeded the time limit.

[0045] If the actual equipment utilization rate is lower than the preset equipment utilization rate threshold, it is determined that the equipment utilization rate is insufficient.

[0046] If the actual logistics efficiency is lower than the preset logistics efficiency threshold, it is judged as low logistics efficiency.

[0047] If the actual assembly success rate is lower than the preset assembly success rate threshold, it is determined that the assembly success rate is not up to standard.

[0048] If the actual safety compliance rate is lower than the preset safety compliance rate threshold, it is judged as a failure to meet safety standards. The conflict is traced and located by combining the linkage constraints in the industrial ontology knowledge graph, identifying the equipment, workstation, timing node, and root cause of the conflict, and then marked in three levels according to the following criteria: General conflict: Actual indicators are lower than the threshold but the deviation is ≤10%.

[0049] Significant conflict: Actual indicators are below the threshold and the deviation is between 10% and 30%.

[0050] Emergency Conflict: Actual indicators are below the threshold and the deviation is >30%.

[0051] Virtual-Real Synchronous Digital Twin Mapping Module: Used to establish a two-way real-time mapping relationship between virtual models and physical equipment, supporting online virtual debugging of PLC programs, real-time correction of process parameters, and synchronous mirroring of equipment status.

[0052] In a specific embodiment, the establishment of the bidirectional real-time mapping relationship between the virtual model and the physical device is specifically carried out as follows: constructing an I / O interface mapping table between the virtual model and the physical device, and mapping the device action commands and status signals in the virtual model to the I / O points of the physical PLC one by one.

[0053] By using industrial communication protocols, a real-time communication connection is established between the virtual simulation system and the physical PLC, enabling the virtual model to send control commands to the physical device and the physical device to feed back operating status data to the virtual model.

[0054] When PLC program debugging is performed in a virtual environment, the actions of the devices in the virtual model are synchronized to the physical devices in real time, and the operating status and action feedback of the physical devices are mirrored to the virtual model in real time.

[0055] During the equipment status synchronization mirroring process, the operating status of the physical equipment is displayed in real time, including start / stop status, action progress, fault information, and alarm signals. When a physical equipment malfunctions, the virtual model synchronously displays the fault location and fault type.

[0056] Autonomous Decision Optimization Output Module: Based on simulation results and twin data, it identifies production bottlenecks, logistics conflicts and assembly risks, and combines knowledge graphs to complete rule tracing, outputting the optimal layout scheme, process path, logistics strategy and equipment configuration list.

[0057] In a specific embodiment, the automatic identification of production bottlenecks, logistics conflicts, and assembly risks is carried out as follows: Based on the results of multi-domain simulation analysis and the results of indicator threshold judgment, production bottlenecks, logistics conflicts, and assembly risks are automatically identified.

[0058] If the actual production line cycle time consistently exceeds the preset production line cycle time threshold, and the equipment utilization rate of the corresponding workstation is consistently lower than the preset equipment utilization rate threshold, and there is a continuous delay in the process connection at that workstation, then that workstation is determined to be a production bottleneck.

[0059] If the actual logistics efficiency is lower than the preset logistics efficiency threshold, and the AGV delivery path experiences repeated congestion, material arrival delays, or warehouse overflows or material shortages causing process waiting, and the deviation between the logistics path node spacing and the preset parameters exceeds the allowable range, then it is determined to be a logistics conflict.

[0060] If the assembly process sequence is disordered, the tooling positioning posture is mismatched, or dynamic assembly interference occurs and the interference level reaches important or above, or the actual assembly success rate is lower than the preset assembly success rate threshold and the deviation exceeds 10%, then it is judged as an assembly risk.

[0061] In a specific embodiment, the output of the optimal layout scheme, process path, logistics strategy and equipment configuration list is specifically output as follows: if the traceability results clearly indicate that the production bottleneck is insufficient utilization of workstation equipment and delay in process connection, a workstation load-balanced layout scheme is generated; at the same time, a process path with high efficiency of process collaboration, a material local delivery logistics strategy and a load-adaptive equipment configuration list are output accordingly.

[0062] It should be noted that the workstation load balancing solution involves: optimizing the layout of bottleneck workstations and surrounding workstations, reducing upstream and downstream distances, and adjusting the locations of storage and buffer areas to achieve nearby material supply; splitting the bottleneck process into 2-3 parallel processes and adding auxiliary workstations; replanning the execution sequence of the process path, clarifying the cycle time and connection standards of parallel processes, optimizing the connection process, marking time nodes and linking them to the industrial ontology knowledge graph; prioritizing the planning of AGV delivery routes for bottleneck and auxiliary workstations in the logistics strategy, and dynamically adjusting delivery cycles and warehouse allocation; and clearly defining the models and coordinates of equipment for newly added auxiliary workstations, upgrading equipment at bottleneck workstations, and configuring suitable tooling and material handling equipment.

[0063] If the tracing results clearly indicate that the logistics conflict is due to unreasonable route planning, insufficient storage capacity, or imbalanced delivery cycle, a route-storage location collaborative optimization layout scheme will be generated. At the same time, the corresponding output will include a logistics-efficient process route, a route-storage location-adaptive logistics strategy, and a logistics-collaborative equipment configuration list.

[0064] It should be noted that the route-based warehouse location optimization solution involves: In terms of layout, replanning the warehouse, buffer areas, and storage locations; expanding capacity and adding temporary buffer areas; optimizing the connection between AGV channels and conveyor lines; and clarifying the coordinates of each area. In terms of process routing, adjusting the sequence of processes to match the logistics cycle; establishing a process-logistics collaboration mechanism; optimizing handover processes; and linking to a knowledge graph. In terms of logistics strategy, replanning AGV routes; dynamically adjusting delivery cycles and warehouse location allocation; and clarifying delivery parameters. The equipment list is configured to adapt to AGVs and warehouse equipment, optimizing conveyor line parameters, and adding logistics status monitoring equipment.

[0065] If the tracing results clearly indicate that the assembly risk is tooling positioning deviation, assembly interference, or process sequence disorder, then a precise and collaborative assembly layout plan will be generated; the corresponding outputs will be a precise connection process path, an assembly-adaptive logistics strategy, and a precise assembly equipment configuration list.

[0066] It should be noted that the assembly precision collaborative solution involves: optimizing assembly station space in terms of layout, adjusting tooling and robot positions, correcting tooling positioning deviations, and clarifying equipment placement boundaries; optimizing the assembly process sequence and cycle time in terms of process path, standardizing operating procedures, optimizing robot motion trajectories, clarifying quality inspection nodes, and linking them to a knowledge graph; adjusting material delivery routes and timing in terms of logistics strategy to match assembly cycle time and ensure delivery accuracy; and upgrading and replacing deviation tooling in the equipment list, optimizing robot parameters, supplementing assembly accuracy testing equipment, and clarifying the key parameters of all equipment.

[0067] The examples described in this invention are not limited to the specific embodiments listed above. The examples are merely illustrative to facilitate understanding of the invention and do not constitute a limitation on the scope of protection of this invention. Any modifications, equivalent substitutions, etc., made within the spirit and principles of this invention should be included within the scope of protection.

[0068] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in this specification, they should all fall within the protection scope of the present invention.

Claims

1. A multimodal 3D modeling and virtual environment simulation system based on industrial scenarios, characterized in that, Includes the following modules: Multimodal spatiotemporal alignment data input module: used to deploy multi-source data acquisition devices in target industrial scenarios to collect laser point clouds, CAD models, video images, process text, PLC timing signals, IoT real-time data and production logistics trajectories, and perform unified spatiotemporal benchmark calibration and heterogeneous data fusion; Knowledge graph-driven industrial semantic modeling module: It is used to construct an industrial ontology knowledge graph containing equipment attributes, process constraints, logistics rules, safety specifications and assembly logic based on the fused heterogeneous data, thereby automatically generating a parametric 3D model with semantic constraints and automatically completing static and dynamic interference checks. Cross-domain unified physics engine simulation module: It is used to synchronously execute robot motion simulation, assembly process simulation, human-machine engineering simulation, logistics scheduling simulation and production line cycle simulation under the same physics engine kernel. Cross-domain linkage constraints are defined in real time by industrial ontology knowledge graph, and the real-time interconnection of multi-domain simulation data and the strong linkage closed loop exclusive to industrial scenarios are completed. It also judges the index threshold and linkage conflict in real time. Virtual-Real Synchronous Digital Twin Mapping Module: Used to establish a two-way real-time mapping relationship between virtual models and physical equipment, supporting online virtual debugging of PLC programs, real-time correction of process parameters, and synchronous mirroring of equipment status; Autonomous Decision Optimization Output Module: Based on simulation results and twin data, it identifies production bottlenecks, logistics conflicts and assembly risks, and combines knowledge graphs to complete rule tracing, outputting the optimal layout scheme, process path, logistics strategy and equipment configuration list.

2. The multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 1, characterized in that, The specific acquisition process for laser point clouds, CAD models, video images, process texts, PLC timing signals, IoT real-time data, and production logistics trajectories is as follows: The system collects 3D laser point cloud data of the target industrial scene using a LiDAR scanner; captures on-site video image data using industrial cameras; reads process documents and work instructions using text parsing tools to extract process flow, process parameters, and quality standards; acquires PLC timing signals via industrial Ethernet to obtain time-series data on equipment start-up, shutdown, action execution, and process connection; collects real-time data on equipment operating status, ambient temperature and humidity, and material location using IoT sensors; and collects data on the operating trajectory, speed, load capacity, and delivery cycle of AGVs / AMRs, conveyors, and forklifts using logistics trajectory tracking equipment. All collected data is connected to the data processing center via an industrial communication network to establish a unified scene coordinate system and timestamp benchmark, completing the spatiotemporal alignment and normalization of heterogeneous data.

3. The multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 2, characterized in that, The construction of the industrial ontology knowledge graph, which includes equipment attributes, process constraints, logistics rules, safety specifications, and assembly logic, is carried out as follows: Five core entity layers are constructed: equipment, process, logistics, safety, and personnel. The physical attributes, state attributes, motion attributes, spatial attributes, and interrelationships of each entity are obtained. The process documents are automatically parsed using NLP to extract entity features and association conditions. Entity alignment and attribute completion are completed by combining point cloud, CAD, and video data. This forms an industrial-specific knowledge graph that can be dynamically iterated and has assembly linkage constraints, logistics trigger conditions, and safety threshold constraints. Inter-entity relationships include the execution relationship between equipment and process, the supply relationship between process and material, the matching relationship between material and equipment, the spatial relationship between equipment and safety area, and the operational relationship between personnel and workstation.

4. The multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 3, characterized in that, The automatic generation of parametric 3D models with semantic constraints is carried out as follows: Map the equipment attributes, process constraints, logistics constraints, safety specifications, and assembly logic in the industrial ontology knowledge graph to the parametric modeling engine; Based on the size, range of motion, and installation coordinate parameters of the equipment entity, automatically generate a 3D model of the equipment and configure kinematic pairs and motion constraints; Based on the process sequence number, assembly sequence, and operation cycle parameters of the process entity, automatically generate the workstation layout, tooling pose, and operation space model; Based on the storage location, carrying capacity, and delivery route node parameters of logistics entities, automatically generate models of storage areas, buffer areas, and conveyor lines; Based on the safety distance threshold, protected area coordinates, and restricted area parameters of safety entities, the safe area is automatically marked in the 3D scene; The geometric, motion, and spatial parameters of the above model are bound to the industrial ontology knowledge graph in real time to form a simulateable 3D model with industrial linkage constraints.

5. The multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 4, characterized in that, The automatic static and dynamic interference checks are performed as follows: Static interference check: Traverse the spatial coordinates and boundary dimensions of equipment, buildings, columns, and workstations in the 3D model, and calculate the minimum distance between adjacent entities one by one; compare the calculated minimum distance with a preset safety threshold, and determine static interference when the minimum distance is less than the safety threshold; Mark the position coordinates, names of interfering entities, and spacing differences for static interference respectively; Dynamic interference check: Based on the motion constraints in the industrial ontology knowledge graph, the entire process of robot joint movement, AGV travel along the path, and personnel assembly in the workstation is simulated in sequence; the real-time position and posture of each entity are sampled at fixed time steps, and the instantaneous distance between moving entities and between moving entities and static entities is calculated; when the instantaneous distance is less than the collision threshold, it is judged as dynamic interference, and the time of interference, interference position, intersection of motion trajectory, and overlapping area of ​​action are recorded; the position, type and degree of all static and dynamic interferences are automatically marked.

6. The multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 5, characterized in that, The specific process for achieving real-time interoperability and behavioral linkage of multi-domain simulation data is as follows: A unified simulation clock and data interaction interface are established under the same physical engine kernel; cross-domain linkage constraints are defined in real time by industrial ontology knowledge graph; the robot motion state drives the assembly process in real time, the assembly state triggers logistics scheduling in real time, and the logistics state provides real-time feedback for production line cycle calculation, forming a closed loop of strong linkage across the entire industrial scenario, realizing real-time interoperability and behavioral linkage of multi-domain simulation data.

7. The multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 6, characterized in that, The real-time judgment indicator threshold and linkage conflict are judged in the following specific process: Real-time collection of simulation operation data, obtaining actual production line cycle time, actual equipment utilization rate, actual logistics efficiency, actual assembly success rate and actual safety compliance rate, and comparing each actual indicator with the corresponding preset indicator threshold one by one; If the actual production line cycle time exceeds the preset production line cycle time threshold, it is determined that the production line cycle time has exceeded the time limit. If the actual equipment utilization rate is lower than the preset equipment utilization rate threshold, it is determined that the equipment utilization rate is insufficient. If the actual logistics efficiency is lower than the preset logistics efficiency threshold, it is judged as low logistics efficiency. If the actual assembly success rate is lower than the preset assembly success rate threshold, it is determined that the assembly success rate is not up to standard. If the actual safety compliance rate is lower than the preset safety compliance rate threshold, it is judged as a failure to meet safety standards. The conflict is traced and located by combining the linkage constraints in the industrial ontology knowledge graph, identifying the equipment, workstation, timing node, and root cause of the conflict, and then marked at three levels according to the following criteria: General conflict: Actual indicators are below the threshold but the deviation is ≤10%; Significant conflict: Actual indicators are below the threshold and the deviation is between 10% and 30%; Emergency Conflict: Actual indicators are below the threshold and the deviation is >30%.

8. The multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 7, characterized in that, The specific process for establishing the bidirectional real-time mapping relationship between the virtual model and the physical device is as follows: Construct a mapping table between the virtual model and the physical device's I / O interface, and map the device action commands and status signals in the virtual model to the I / O points of the physical PLC one by one; By using industrial communication protocols, a real-time communication connection is established between the virtual simulation system and the physical PLC, enabling the virtual model to send control commands to the physical device and the physical device to feed back operating status data to the virtual model. When PLC program debugging is performed in a virtual environment, the actions of the devices in the virtual model are synchronized to the physical devices in real time, and the operating status and action feedback of the physical devices are mirrored to the virtual model in real time. During the equipment status synchronization mirroring process, the operating status of the physical equipment is displayed in real time, including start / stop status, action progress, fault information, and alarm signals. When a physical equipment malfunctions, the virtual model synchronously displays the fault location and fault type.

9. A multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 8, characterized in that, The automatic identification process for production bottlenecks, logistical conflicts, and assembly risks is as follows: Based on the results of multi-domain simulation analysis and the judgment results of index thresholds, production bottlenecks, logistics conflicts and assembly risks are automatically identified. If the actual production line cycle time continuously exceeds the preset production line cycle time threshold, and the equipment utilization rate of the corresponding workstation is consistently lower than the preset equipment utilization rate threshold, and there is a continuous delay in the process connection at that workstation, then that workstation is determined to be a production bottleneck. If the actual logistics efficiency is lower than the preset logistics efficiency threshold, and the AGV delivery path experiences repeated congestion, material arrival delays, or warehouse overflows or material shortages causing process waiting, and the deviation between the logistics path node spacing and the preset parameters exceeds the allowable range, then it is determined to be a logistics conflict. If the assembly process sequence is disordered, the tooling positioning posture is mismatched, or dynamic assembly interference occurs and the interference level reaches important or above, or the actual assembly success rate is lower than the preset assembly success rate threshold and the deviation exceeds 10%, then it is judged as an assembly risk.

10. A multimodal 3D modeling and virtual environment simulation system based on industrial scenarios according to claim 9, characterized in that, The output process for the optimal layout scheme, process route, logistics strategy, and equipment configuration list is as follows: If the source tracing results clearly indicate that the production bottleneck is insufficient utilization of workstation equipment or delay in process connection, a workstation load balancing layout scheme will be generated; at the same time, a process path with high efficiency in process collaboration, a material delivery-nearby logistics strategy, and a load-adaptive equipment configuration list will be output accordingly. If the traceability results clearly indicate that the logistics conflict is due to unreasonable route planning, insufficient storage capacity, or imbalanced delivery cycle, a route-storage location collaborative optimization layout scheme will be generated. At the same time, the corresponding output will include a logistics-efficient process route, a route-storage location-adaptive logistics strategy, and a logistics-collaborative equipment configuration list. If the tracing results clearly indicate that the assembly risk is tooling positioning deviation, assembly interference, or process sequence disorder, then a precise and collaborative assembly layout plan will be generated; the corresponding outputs will be a precise connection process path, an assembly-adaptive logistics strategy, and a precise assembly equipment configuration list.

Images