A virtual training and interactive system for safety production in underground mining scenarios

Abstract

This invention discloses a virtual training interactive system for safety production in underground mining scenarios, relating to the field of mining production technology. It includes a visibility clipping and hit detection module, used to occlude and clip the visibility layer of the initial packet of the scene graph to obtain the visible set of the current frame. Hit detection and interactivity verification are performed on the visible set of the current frame to filter out interactive actions and hit objects, which are then encapsulated into a constraint candidate packet. A gating deviation guidance module is used to perform gating detection and deviation calculation based on the interactive actions and hit objects of the constraint candidate packet, generating a detection result. The detection result is mapped to a set of visual guidance parameters and summarized with the scene graph into a special effects situation frame. This invention achieves refined quantitative representation and real-time visual feedback of dynamic risks during operation, improving the ability to perceive and guide progressive safety situations in virtual training.

Description

Technical Field

[0001] This invention relates to the field of mining production technology, and in particular to a virtual training and interactive system for safety production in underground mining scenarios. Background Technology

[0002] In the field of mine safety production, virtual reality (VR) and augmented reality (AR) technologies are increasingly being applied to the practical training of underground workers. Existing virtual training systems typically construct underground environment models based on multimodal data, simulating actual work processes through scene rendering and interactive logic. These methods rely on predefined 3D scene libraries and action rule libraries, combined with visual rendering engines to visualize tunnel layouts, equipment operations, and risk volumes. This technology aims to reduce on-site training costs through digital means and enhance the understanding and training of mine-specific risk factors.

[0003] However, existing virtual training methods still have room for improvement in terms of interactive accuracy and scenario adaptability. On the one hand, conventional systems rely heavily on static pre-calculation to handle dynamic occlusion relationships, making it difficult to respond in real time to visibility fluctuations caused by changes in viewpoint, which may affect the continuity of object detection. On the other hand, the deviation assessment of interactive actions is mostly limited to discrete state comparisons, lacking quantitative guidance on continuous parameters such as volume risk diffusion and visibility deviation magnitude, resulting in insufficiently nuanced feedback on the gradual evolution of risks during the training process. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a virtual training interactive system for safety production in underground mining scenarios to solve the problems of poor interactive accuracy and lack of nuanced risk feedback.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a virtual training and interactive system for safety production in underground mining scenarios, which includes a data scene encapsulation module, a visible clipping hit module, a gating deviation guidance module, an interactive execution response module, and a safety assessment module.

[0008] The data scene encapsulation module is used to collect and aggregate downhole multimodal datasets, generate scene graphs and visibility layers, and encapsulate the scene graphs and visibility layers into a unified scene graph initial package.

[0009] The visible clipping hit module is used to perform occlusion clipping on the visibility layer of the initial package of the scene graph to obtain the visible set of the current frame. Hit determination and interactivity verification are performed on the visible set of the current frame to filter out interactive actions and hit objects, and encapsulate them into a constraint candidate package.

[0010] The gating deviation guidance module is used to perform gating judgment and deviation calculation based on the interactive actions and hit objects of the constraint candidate package, generate judgment results, map the judgment results into a set of visual guidance parameters, and summarize them with the scene graph into a special effects situation frame.

[0011] The interactive execution response module is used to perform interactive actions on the hit object through special effects situation frames, generate process dataset, apply a situation response strategy to the process dataset, and generate a situation state package.

[0012] The security assessment module is used to extract the process dataset and special effects situation frames of the scenario state package, calculate the security assessment index set, and generate a security assessment report.

[0013] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the gate control deviation guidance module includes a gate control deviation unit and a guidance mapping unit.

[0014] The gating deviation unit is used to receive constraint candidate packets, perform gating judgment on interactive actions and hit objects, and perform deviation calculation.

[0015] The guidance mapping unit is used to map the judgment result into a set of visual guidance parameters and summarize it with the scene graph.

[0016] As a preferred embodiment of the virtual training interactive system for safety production in underground mining scenarios described in this invention, the interactive execution response module includes an interactive execution unit and a scenario response unit.

[0017] The interactive execution unit is used to receive special effects situation frames, perform corresponding interactive actions on the hit objects, and output process datasets;

[0018] The context response unit is used to parse the correspondence between interactive actions and hit objects according to the context response strategy and make a judgment.

[0019] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the underground multimodal dataset includes roadway entries, mining face entries, equipment semantic component entries, volume risk basic data, particle materials, tool library model list, standard operating procedure clause records, VR binocular camera configuration, and AR video synthesis channel configuration.

[0020] The standard operating procedure (SOP) clause records include a list of interactive actions, the start status of the task, and the completion status of the task.

[0021] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the encapsulation of the system into an initial scene graph package involves the following steps:

[0022] Write incremental numbers to the underground multimodal dataset according to the collection order, register the mutual pointing relationships between the underground multimodal dataset entries, and obtain the subordinate association between equipment semantic component entries and roadway entries, the mutual pointing relationship between volume roadways, the mutual pointing relationship between volume equipment and the mutual pointing relationship between working equipment;

[0023] VR binocular camera configuration is used as presentation pointing value, AR video synthesis channel configuration is used as synthesis pointing value and written into roadway entries, mining face entries and equipment semantic component entries, forming a scene graph by combining the underground multimodal dataset and mutual pointing relationships;

[0024] Based on the VR binocular camera configuration and AR video synthesis channel configuration, mark the occlusion items, and establish occlusion mutual pointing relationship with the roadway items, mining face items and equipment semantic component items. At the same time, register the field association in the occlusion items, and merge the occlusion items and field association into a visibility layer.

[0025] The scene graph and visibility layer are uniformly encapsulated and synchronously written into the tool library model list, standard operating procedure clause record, and the numbering relationship of VR stereo camera configuration and AR video compositing channel configuration to generate the initial scene graph package.

[0026] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the encapsulation of the system into a constraint candidate package is as follows:

[0027] The visibility layer is clipped based on the occlusion entries, and the unoccluded roadway entries, mining face entries, and equipment semantic component entries are retained as the visible set of this frame;

[0028] The equipment semantic component entries are compared with the standard operating procedure clause records and the tool library model list for consistency, and the equipment semantic component entries that are registered as consistent are the hit objects.

[0029] Read the list of interactive actions and match them with the semantic component entries of the equipment according to the mutual pointing relationship of the working equipment to form a set of interactions to be filtered. Within the scope of the hit objects, compare the set of interactions to be filtered with the list of interactive actions one by one to filter out the interactive actions.

[0030] Interactive actions, hit objects, and the visible set of this frame are uniformly encapsulated into a constraint candidate package.

[0031] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the steps for generating the judgment result are as follows:

[0032] The matched objects are paired with interactive actions selected based on the matched objects to form candidate interaction pairs;

[0033] Determine whether the interactive actions of the candidate interaction pair are included in the list of interactive actions, and compare the hit objects according to the mutual pointing relationship of the working equipment to obtain the gating judgment result;

[0034] Based on the gating judgment results, the visibility deviation type, visibility deviation magnitude, volume risk deviation type, and volume risk deviation magnitude of the hit object are calculated and summarized into the judgment results.

[0035] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the step of summarizing the data into special effects frames is as follows:

[0036] By selecting volumetric rendering transmission parameters, volumetric light intensity, fog thickness, and particle emissivity based on volumetric risk deviation type and magnitude, and combining them with visibility deviation type and magnitude, contour hints and path hints are generated at the location of the hit object.

[0037] The volume rendering transmission parameters, volumetric light intensity and fog thickness, particle emissivity, contour hints and path hints are summarized into a set of visualization guidance parameters;

[0038] The set of visual guidance parameters and scene graphs are combined to generate special effects situation frames.

[0039] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the steps for generating the scenario state package are as follows:

[0040] Based on the special effects situation frames, an execution queue is established for interactive actions and hit objects. Based on the mutual pointing relationship of the working equipment, the alleyway entries are locked in the scene graph as the operating environment. Interactive actions are executed under the view domain association according to the order of the execution queue, generating equipment status snapshots and interaction event records. The equipment status snapshots and interaction event records are then merged into a process dataset.

[0041] Based on the work step number, work start status and work completion status recorded in the standard operating procedure clauses, the consistency of status is judged, and emergency judgment is made in combination with the visibility deviation type, visibility deviation magnitude and volume risk deviation type, volume risk deviation magnitude, to obtain the progress result and emergency status indicator.

[0042] The process dataset, progress results, emergency status identifiers, and field-of-view associations are uniformly summarized to generate a scenario status package.

[0043] As a preferred embodiment of the virtual training and interactive system for safety production in underground mining scenarios described in this invention, the steps for generating a safety assessment report are as follows:

[0044] Expand equipment status snapshots and interaction event records, statistically analyze the distribution of visibility deviation and volume risk deviation, and based on the progress results, perform work step numbering and progress statistics, and merge them with the distribution of visibility deviation and volume risk deviation into a set of safety assessment indicators;

[0045] The safety assessment indicator set, progress results, and emergency status indicators are compiled into a unified safety assessment report.

[0046] The beneficial effects of this invention are as follows: the visible clipping hit module enables accurate and continuous identification and filtering of interactive objects from a dynamic perspective, effectively ensuring the real-time performance and reliability of interactive operations; the gating deviation guidance module enables refined quantitative representation and real-time visual feedback of dynamic risks during operation, improving the ability to perceive and guide progressive safety situations in virtual training. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a virtual training and interactive system for safety production in underground mining scenarios.

[0049] Figure 2 A flowchart for generating constraint candidate packages.

[0050] Figure 3 This is a flowchart for generating special effects situation frames.

[0051] Figure 4 A flowchart for generating a scenario state package. Detailed Implementation

[0052] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0053] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0054] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0055] Reference Figures 1-4 As one embodiment of the present invention, this embodiment provides a virtual training and interactive system for safety production in underground mining scenarios, comprising the following steps:

[0056] The data scene encapsulation module collects and aggregates downhole multimodal datasets, generates scene graphs and visibility layers, and encapsulates the scene graphs and visibility layers into a unified scene graph initial package.

[0057] Furthermore, each entry in the downhole multimodal dataset is written with an incrementing number according to the acquisition order.

[0058] It should be noted that the underground multimodal dataset includes roadway entries, mining face entries, equipment semantic component entries, volumetric risk basic data, particle materials, tool library model list, standard operating procedure clause records, VR binocular camera configuration, and AR video synthesis channel configuration.

[0059] The standard operating procedure (SOP) clause records include a list of interactive actions, the start status of the operation, and the completion status of the operation.

[0060] Write incremental numbers to the underground multimodal dataset according to the collection order, register the mutual pointing relationships between the underground multimodal dataset entries, and obtain the subordinate association between equipment semantic component entries and roadway entries, the mutual pointing relationship between volume roadways, the mutual pointing relationship between volume equipment and the mutual pointing relationship between operating equipment.

[0061] Furthermore, the following steps are taken: First, the mining face entry number is registered in the roadway entry; second, the roadway entry number is registered in the mining face entry; third, the equipment semantic component entry number and roadway entry number are mapped one-to-one and registered as subordinate association records; fourth, the roadway entry number is written into the equipment semantic component entry, and the equipment semantic component entry number is written into the roadway entry, establishing a subordinate association between the equipment semantic component entry and the roadway entry; fifth, the volume risk basic data number is read and written into the roadway entry, and the roadway entry number is written into the volume risk basic data, forming a mutual pointing relationship between volume roadways; sixth, the volume risk basic data number is written into the equipment semantic component entry. The process involves: writing the equipment semantic component entry number into the volume risk basic data to establish a mutual pointing relationship between the volume risk basic data and the equipment semantic component entry; writing the particle material number into the volume risk basic data and the volume risk basic data number into the particle material; writing the model number from the tool library model list into the equipment semantic component entry and the equipment semantic component entry number into the tool library model list; and writing the operation step number recorded in the standard operating procedure clause into the equipment semantic component entry and the equipment semantic component entry number into the standard operating procedure clause record to establish a mutual pointing relationship between the operation step and the equipment semantic component entry.

[0062] The VR binocular camera configuration is used as the presentation pointing value, and the AR video synthesis channel configuration is used as the synthesis pointing value. These are written into the roadway entries, mining face entries, and equipment semantic component entries, forming a scene graph by combining the underground multimodal dataset with their mutual pointing relationships.

[0063] Furthermore, the VR binocular camera configuration is written into the roadway entries, mining face entries, and equipment semantic component entries as presentation pointer values. At the same time, the roadway entry number, mining face entry number, and equipment semantic component entry number are written into the VR binocular camera configuration for field of view alignment. The AR video compositing channel configuration is written into the roadway entries, mining face entries, and equipment semantic component entries as compositing pointer values. At the same time, the roadway entry number, mining face entry number, and equipment semantic component entry number are written into the AR video compositing channel configuration. The roadway entries, mining face entries, equipment semantic component entries, volumetric risk basic data, particle materials, tool library model list, standard operating procedure clause records, VR binocular camera configuration, and AR video compositing channel configuration, together with the registered numbering relationships, constitute a scene graph.

[0064] Based on the VR binocular camera configuration and AR video synthesis channel configuration, occlusion entries are marked, and occlusion inter-pointing relationships are established with roadway entries, mining face entries, and equipment semantic component entries. At the same time, visual field associations are registered in the occlusion entries, and the occlusion entries and visual field associations are merged into a visibility layer.

[0065] Furthermore, within the observation range of the VR binocular camera configuration and the AR video synthesis channel configuration, the roadway entries, mining face entries, and equipment semantic component entries are compared sequentially according to the acquisition order. Entries with smaller incrementing numbers will cover entries with larger incrementing numbers during presentation. That is, the entries with smaller incrementing numbers are marked as occlusion entries, and the incrementing numbers are retained as occlusion pointing values. The occlusion incrementing numbers are written to the occlusion entries according to the marking order. The occlusion incrementing numbers are then written to the roadway entries, mining face entries, and equipment semantic component entries according to their pointing values, establishing occlusion mutual pointing relationships between occlusion entries and roadway entries, and between mining face entries and equipment semantic component entries. The VR binocular camera configuration number and the AR video synthesis channel configuration number are written to the occlusion entries as a field of view association. The occlusion entry numbers are written to the VR binocular camera configuration and the AR video synthesis channel configuration as pointing values, completing bidirectional registration. The occlusion entries and field of view associations are merged and registered into a single visibility layer.

[0066] The scene graph and visibility layer are uniformly encapsulated and synchronously written into the tool library model list, standard operating procedure clause record, and the numbering relationship of VR binocular camera configuration and AR video compositing channel configuration to generate the initial scene graph package.

[0067] The visible clipping hit module performs occlusion clipping on the visibility layer of the initial package of the scene graph to obtain the visible set of the current frame. Hit determination and interactivity verification are performed on the visible set of the current frame to filter out interactive actions and hit objects, and encapsulate them into a constraint candidate package.

[0068] The visibility layer is clipped based on the occlusion entries, and the unoccluded roadway entries, mining face entries, and equipment semantic component entries are retained as the visible set of this frame.

[0069] Furthermore, the occlusion entries in the visibility layer are read, and the observation range is determined based on the presentation pointing value configured by the VR binocular camera and the field of view configured by the AR video synthesis channel. The occlusion entries are traversed according to their numbers, and the roadway entries, mining face entries, and equipment semantic component entries that have established mutual pointing relationships with the occlusion entries are removed. The remaining roadway entries, mining face entries, and equipment semantic component entries are retained as the visible set of this frame.

[0070] The semantic component entries of the equipment are compared with the standard operating procedure clause records and the tool library model list for consistency, and the equipment semantic component entries that are registered as consistent are the hit objects.

[0071] Furthermore, the system performs a hit determination on the equipment semantic component entries within the visible set of this frame. It reads the operation step number recorded in the standard operating procedure clause and locates the equipment semantic component entries within the visible set of this frame according to the number relationship. Based on the equipment semantic component entry number, it compares each equipment semantic component entry with the equipment semantic component entry corresponding to the operation step recorded in the standard operating procedure clause, and compares each equipment semantic component entry with the equipment semantic component entry corresponding to the model in the tool library model list. Equipment semantic component entries that match in both comparison processes are registered as hit objects.

[0072] Read the list of interactive actions and match them with the semantic component entries of the equipment according to the mutual pointing relationship of the working equipment to form a set of interactions to be filtered. Within the scope of the hit objects, compare the set of interactions to be filtered with the list of interactive actions one by one to filter out the interactive actions.

[0073] Furthermore, the operation step number recorded in the standard operating procedure (SOP) clause is read, and the corresponding interactive action list is extracted based on the operation step number recorded in the SOP clause. The interactive action list is then matched with each semantic component item of each device according to the inter-device relationship to form the interactive actions to be filtered and the interactive actions to be filtered are formed. All the interactive actions to be filtered are then summarized into a set of interactive actions to be filtered.

[0074] The interactivity of the hit objects is checked. Within the scope of the hit objects, the list of interactive actions corresponding to the operation step number recorded in the standard operating procedure clause is read and compared with the set of interactive actions to be filtered one by one. When the interactive action to be filtered appears in the list of interactive actions corresponding to the operation step number, the interactive action is filtered out.

[0075] The selected interactive actions, hit objects, and the set of objects visible in the current frame are uniformly encapsulated to generate a constraint candidate package.

[0076] The gating deviation guidance module performs gating judgment and deviation calculation based on the interactive actions and hit objects of the constraint candidate package, generates judgment results, maps the judgment results to a set of visual guidance parameters, and summarizes them with the scene graph into a special effects situation frame.

[0077] The gating deviation guidance module includes a gating deviation unit and a guidance mapping unit.

[0078] The gating deviation unit is used to receive constraint candidate packets, perform gating judgment on interactive actions and hit objects, and perform deviation calculation.

[0079] The guidance mapping unit is used to map the judgment results into a set of visual guidance parameters and summarize them with the scene graph.

[0080] Furthermore, the hit object is paired with the interactive actions selected based on the hit object to form candidate interaction pairs.

[0081] The system compares whether the interactive actions of the candidate interaction pairs are included in the list of interactive actions, and compares the hit objects according to the mutual pointing relationship of the working equipment to obtain the gating judgment result.

[0082] Furthermore, in the gating deviation unit, gating judgment is performed on candidate interaction pairs. The operation step number recorded in the standard operating procedure clause is read, and the interactive actions of each candidate interaction pair are compared with those in the interaction action list. Candidate interaction pairs not included in the interaction action list are marked as non-compliant and directly eliminated. At the same time, the inconsistency in the interaction action list is recorded as a source of non-compliance. Based on the mutual pointing relationship of the operation equipment, the semantic component entries of the hit objects of the candidate interaction pairs are compared with the records in the standard operating procedure clause to see if a two-way pointing relationship has been established. Hit objects without a two-way pointing relationship are marked as non-compliant and directly eliminated. At the same time, the lack of a mutual pointing relationship between the operation equipment is recorded as a source of non-compliance.

[0083] Candidate interaction pairs that have completed both comparison processes are marked as compliant. The candidate interaction pairs marked as compliant and the candidate interaction pairs marked as non-compliant with the source of non-compliance recorded are aggregated and used as the gating judgment result.

[0084] Based on the gating judgment results, the visibility deviation type, visibility deviation magnitude, volume risk deviation type, and volume risk deviation magnitude of the hit object are calculated and summarized into the judgment results.

[0085] Furthermore, deviation calculations are performed on candidate interaction pairs marked as compliant in the gating judgment results, and the corresponding lane entries of the hit objects are located in the initial packet of the scene graph based on the mutual pointing relationship of the operating equipment.

[0086] Within the visibility layer, retrieve occluded body entries that are related to the hit object through occlusion. If the search result is empty, register the hit object as an unoccluded body entry. If the search result is not empty, register the hit object as an occluded body entry. Combine unoccluded body entries and occluded body entries into a visibility deviation type.

[0087] The system counts the number of occlusion objects that establish mutual occlusion relationships with the hit object and reads the incrementing occlusion number. Starting from the initial number of occlusion object entries, it increments the deviation level by setting the level of deviation. An occlusion object with one entry is registered as the first visibility level. Under view correlation, adjusting the observation position of the VR binocular camera configuration ensures the hit object stably enters the visible set of the current frame. An occlusion object with two entries is registered as the second visibility level. Even under view correlation, adjusting the observation position of the VR binocular camera configuration cannot guarantee the hit object stably enters the visible set of the current frame. This process increments until the entire range of entries is covered. Only at the first visibility level can the hit object stably enter the visible set of the current frame. All deviation levels are then summarized into a single visibility deviation level.

[0088] The system reads the volume risk baseline data and the mutual pointing relationships between volume equipment and volume roadways of the hit object. Based on the mutual pointing relationships between volume equipment, it collects the volume risk baseline data pointed to by the semantic component entries of the hit object's equipment, forming a device layer set. Based on the mutual pointing relationships between volume roadways, it collects the volume risk baseline data pointed to by the roadway entries corresponding to the semantic component entries of the hit object's equipment, forming a roadway layer set. The device layer set and roadway layer set are then deduplicated. Volume risk is determined for the hit object. If both the device layer set and the roadway layer set corresponding to the hit object are empty, the hit object is determined to be of the no-volume-risk type. If the device layer set corresponding to the hit object is empty, but the roadway layer set contains volume risk baseline data, the hit object is determined to be of the surrounding volume risk type. If the device layer set corresponding to the hit object contains volume risk baseline data, but the roadway layer set is empty, the hit object is determined to be of the direct volume risk type.

[0089] The number of items in the equipment layer set and the roadway layer set is counted, and the total number of items in the two sets is calculated. The volume risk deviation is reflected by classifying the total number of items. Based on the total number of items, it is divided into volume risk level 0, which means there are no volume risks and starts from 0. The level is increased by 2 at a time. When the total number of items is 1 or 2, it is recorded as volume risk level 1, which means there are 1 or 2 volume risks, until all the total number of items is covered.

[0090] The compliance marker, interactive action, hit object, visibility deviation type, visibility deviation magnitude, volume risk deviation type, and volume risk deviation magnitude are summarized into the judgment result.

[0091] By selecting volumetric rendering transmission parameters, volumetric light intensity, fog thickness, and particle emissivity based on volumetric risk deviation type and magnitude, and combining them with visibility deviation type and magnitude, contour hints and path hints are generated at the location of the hit object.

[0092] Furthermore, in the guided mapping unit, based on the volumetric risk deviation type and magnitude, volumetric rendering transmission parameters, volumetric light intensity, and fog thickness are selected from the volumetric risk base data in the equipment layer set and the alleyway layer set, and particle emissivity is selected in the particle material; based on the visibility deviation type and magnitude, contour hints and path hints are generated at the hit object location.

[0093] The parameters are merged into a visualization guidance parameter set according to the order of fields such as volume rendering transmission parameters, volumetric light intensity, fog thickness, particle emissivity, contour hints, and path hints.

[0094] The set of visual guidance parameters is written into the volume risk base data, particle material, tunnel entries, mining face entries and equipment semantic component entries of the scene map in the order of fields. Combined with the synthesis pointing value of VR binocular camera configuration and AR video synthesis channel configuration, the field of view is associated and synthesized to generate special effects situation frames.

[0095] The interactive execution response module performs interactive actions on the hit object through special effects situation frames, generates a process dataset, applies a situation response strategy to the process dataset, and generates a situation state package.

[0096] The interactive execution response module includes an interactive execution unit and a scenario response unit.

[0097] The interactive execution unit is used to receive special effects situation frames, perform corresponding interactive actions on the hit objects, and output process datasets.

[0098] The context response unit is used to parse the correspondence between interactive actions and hit objects and make a judgment based on the context response strategy.

[0099] Based on the special effects situation frames, an execution queue is established for interactive actions and hit objects. Based on the mutual pointing relationship of the working equipment, the alleyway entries are locked in the scene graph as the operating environment. Interactive actions are executed under the view domain association according to the order of the execution queue, generating equipment status snapshots and interaction event records. The equipment status snapshots and interaction event records are then merged into a process dataset.

[0100] Furthermore, in the interactive execution unit, an execution queue is established for interactive actions and hit objects based on the visualization guidance parameter set of the special effects situation frame. The semantic component entries of the hit objects are located according to the mutual pointing relationship of the working equipment, and the lane entries are locked in the scene map as the operating environment. In the operating environment, interactive actions are executed sequentially on the hit objects according to the order of the execution queue through the field of view association. The state of the hit objects before and after the execution of the interactive actions is recorded. At the same time, the visibility deviation type, visibility deviation magnitude, volume risk deviation type and volume risk deviation magnitude are recorded when the interactive actions are executed, and a device state snapshot is generated for the process of executing the interactive actions.

[0101] The interactive action and the hit object are collected at the moment of execution, and the compliance flag is written in combination with the gating judgment result. The interactive action, the hit object and the compliance flag are registered as interactive event records in the order of interactive action, hit object and compliance flag.

[0102] It should be noted that the device status snapshot includes the hit object, the hit object status before the interactive action is performed, the hit object status after the interactive action is performed, the view association, and the visibility deviation type, visibility deviation magnitude, volumetric risk deviation type, and volumetric risk deviation magnitude when the interactive action is performed.

[0103] The device status snapshots and interaction event records are merged in chronological order and summarized with the special effects status frames to generate a process dataset.

[0104] Based on the work step number, work start status and work completion status recorded in the standard operating procedure clauses, a status consistency judgment is made, and an emergency judgment is made in combination with the visibility deviation type, visibility deviation magnitude and volume risk deviation type, volume risk deviation magnitude, to obtain the progress result or emergency status indicator.

[0105] Furthermore, in the scenario response unit, the device status snapshots and interaction event records of the process dataset are read, and the last set of device status snapshots and interaction event records of the process dataset is taken as the latest record. The latest interactive action and the latest hit object are read from the latest record.

[0106] Read the work step number recorded in the standard operating procedure clause, and perform a status consistency judgment on the equipment status snapshot. Determine whether the status of the hit object before the execution of the interactive action in the equipment status snapshot is consistent with the work start status. At the same time, determine whether the status of the hit object after the execution of the interactive action in the equipment status snapshot is consistent with the work completion status. If both judgments are consistent, the status of the equipment status snapshot is determined to be consistent; otherwise, the status of the equipment status snapshot is determined to be inconsistent.

[0107] For emergency results of equipment status snapshots, if any of the following three conditions are met: visibility deviation type is obstructed object entry and visibility deviation magnitude is not lower than visibility level 2; volume risk deviation type is direct volume risk and volume risk deviation magnitude is not lower than volume risk level 1, the equipment status snapshot is determined to be an emergency result; otherwise, the equipment status snapshot is determined to be a non-emergency result.

[0108] If the status of the equipment status snapshot is determined to be consistent and the equipment status snapshot is determined to be a non-emergency result, the operation step number recorded in the standard operating procedure clause will be used as the progress result; if the status of the equipment status snapshot is determined to be inconsistent or the equipment status snapshot is determined to be an emergency result, an emergency status identifier will be generated for the latest hit object.

[0109] The latest hit objects, latest interactive actions, device status snapshots, interaction event records, operation step numbers, progress results, emergency status indicators, and view associations are summarized into a scenario status package.

[0110] The security assessment module extracts the process dataset and special effects situation frames from the scenario state package, calculates the security assessment index set, and generates a security assessment report.

[0111] Expand equipment status snapshots and interaction event records, calculate compliance ratios, statistically analyze visibility deviation distribution and volume risk deviation distribution, perform statistical analysis on work step numbering, and merge them into a set of safety assessment indicators.

[0112] Furthermore, the device status snapshot and interaction event records are expanded, and the number of compliance markers is counted as the number of compliant candidate interaction pairs. Simultaneously, the total number of all candidate interaction pairs is counted, and the ratio of the number of compliant candidate interaction pairs to the total number of all candidate interaction pairs is calculated to generate a compliance ratio. Based on the visibility deviation type and magnitude, the number of unobstructed and occluded object entries at each level is counted, and the proportion of each level of unobstructed and occluded object entries is calculated, resulting in a visibility deviation distribution map. Based on the volume risk deviation type and magnitude, the number of no volume risk, surrounding volume risk, and direct volume risk at each level is counted, and the proportion of each level of no volume risk, surrounding volume risk, and direct volume risk is calculated, resulting in a volume risk deviation distribution map. The advancement results are read, and the number of standard operating procedure (SOP) clause records covered by the advanced operation step number and the number of SOP clause records covered by the unadvanced operation step number are counted. The number of times the hit object appears corresponding to the volume rendering transmission parameters, volumetric light intensity, fog thickness, and particle emission rate are read from the visualization guidance parameter set to obtain the interaction guidance intensity statistics.

[0113] The compliance ratio, visibility deviation distribution, volume risk deviation distribution, and work step number advancement statistics are combined with the interactive guidance intensity statistics to output a set of safety assessment indicators.

[0114] The safety assessment indicator set, progress results, and emergency status indicators are compiled into a unified safety assessment report.

[0115] In summary, this invention achieves accurate and continuous identification and filtering of interactive objects from a dynamic perspective through the visible clipping hit module, effectively ensuring the real-time performance and reliability of interactive operations; and through the gating deviation guidance module, it achieves refined quantitative representation and real-time visual feedback of dynamic risks during operation, improving the ability to perceive and guide progressive safety situations in virtual training.

[0116] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A virtual training and interactive system for safety production in underground mining scenarios, characterized in that: This includes a data scenario encapsulation module, a visible clipping hit module, a gating deviation guidance module, an interactive execution response module, and a security assessment module; The data scene encapsulation module is used to collect and aggregate downhole multimodal datasets, generate scene graphs and visibility layers, and encapsulate the scene graphs and visibility layers into a unified scene graph initial package. The visible clipping hit module is used to perform occlusion clipping on the visibility layer of the initial package of the scene graph to obtain the visible set of the current frame. Hit determination and interactivity verification are performed on the visible set of the current frame to filter out interactive actions and hit objects, and encapsulate them into a constraint candidate package. The gating deviation guidance module is used to perform gating judgment and deviation calculation based on the interactive actions and hit objects of the constraint candidate package, generate judgment results, map the judgment results into a set of visual guidance parameters, and summarize them with the scene graph into a special effects situation frame. The interactive execution response module is used to perform interactive actions on the hit object through special effects situation frames, generate process dataset, apply a situation response strategy to the process dataset, and generate a situation state package. The security assessment module is used to extract the process dataset and special effects situation frames of the scenario state package, calculate the security assessment index set, and generate a security assessment report. The downhole multimodal dataset includes roadway entries, mining face entries, equipment semantic component entries, volumetric risk basic data, particle materials, tool library model list, standard operating procedure (SOP) clause records, VR binocular camera configuration, and AR video synthesis channel configuration. These roadway entries, mining face entries, equipment semantic component entries, volumetric risk basic data, particle materials, tool library model list, SOP clause records, VR binocular camera configuration, and AR video synthesis channel configuration, along with their registered numbering relationships, form a scene graph. Occlusion entries are marked according to the VR binocular camera configuration and AR video synthesis channel configuration, and occlusion inter-pointing relationships are established between these entries and the roadway entries, mining face entries, and equipment semantic component entries. Simultaneously, visual field associations are registered for each occlusion entry, and the occlusion entries and visual field associations are merged into a visibility layer. The visibility layer is clipped based on the occlusion entries, retaining the unoccluded roadway entries, mining face entries, and equipment semantic component entries as the visible set for this frame; the equipment semantic component entries are compared for consistency with the standard operating procedure clause records and the tool library model list, and the equipment semantic component entries that are registered as consistent are the hit objects; the interactive action list is read and matched with the equipment semantic component entries according to the mutual pointing relationship of the working equipment to form a set of interactions to be filtered; within the scope of the hit objects, the set of interactions to be filtered is compared with the interactive action list one by one to filter out interactive actions; The system pairs the hit object with interactive actions selected based on the hit object to form candidate interaction pairs. It then determines whether the interactive actions of the candidate interaction pairs are included in the interaction action list and compares the hit object based on the mutual pointing relationship of the working equipment to obtain the gating judgment result. Based on the gating judgment result, it calculates and summarizes the visibility deviation type, visibility deviation magnitude, volumetric risk deviation type, and volumetric risk deviation magnitude of the hit object as the judgment result. Using the volumetric risk deviation type and magnitude, it selects volume rendering transmission parameters, volumetric light intensity, fog thickness, and particle emissivity, and combines these with the visibility deviation type and magnitude to generate contour and path prompts at the hit object's location. Finally, it summarizes the volume rendering transmission parameters, volumetric light intensity and fog thickness, particle emissivity, contour prompts, and path prompts into a visual guidance parameter set. Finally, it combines the visual guidance parameter set with the scene graph to generate a special effects situation frame. Based on the volumetric risk deviation type and magnitude, volumetric rendering transmission parameters, volumetric light intensity, and fog thickness are selected from the volumetric risk base data in the equipment layer set and the alleyway layer set, and particle emissivity is selected in the particle material; based on the visibility deviation type and magnitude, contour hints and path hints are generated at the hit object location; and the field order of volumetric rendering transmission parameters, volumetric light intensity, fog thickness, particle emissivity, contour hints, and path hints is merged into a visualization guidance parameter set; The scenario response strategy refers to reading the device status snapshots and interaction event records of the process dataset, taking the last set of device status snapshots and interaction event records of the process dataset as the latest record, and reading the latest interactive actions and the latest hit objects of the latest record. Read the operation step number recorded in the standard operating procedure clauses and determine the consistency of the equipment status snapshot; determine the emergency result of the equipment status snapshot; when the status of the equipment status snapshot is determined to be consistent and the equipment status snapshot is determined to be a non-emergency result, the operation step number recorded in the standard operating procedure clauses is used as the progress result; when the status of the equipment status snapshot is determined to be inconsistent or the equipment status snapshot is determined to be an emergency result, an emergency status identifier is generated for the latest hit object.

2. The virtual training and interactive system for safety production in underground mining scenarios as described in claim 1, characterized in that: The gate control deviation guidance module includes a gate control deviation unit and a guidance mapping unit; The gating deviation unit is used to receive constraint candidate packets, perform gating judgment on interactive actions and hit objects, and perform deviation calculation. The guidance mapping unit is used to map the judgment result into a set of visual guidance parameters and summarize it with the scene graph.

3. The virtual training and interactive system for safety production in underground mining scenarios as described in claim 2, characterized in that: The interactive execution response module includes an interactive execution unit and a scenario response unit; The interactive execution unit is used to receive special effects situation frames, perform corresponding interactive actions on the hit objects, and output process datasets. The context response unit is used to parse the correspondence between interactive actions and hit objects according to the context response strategy and make a judgment.

4. The virtual training and interactive system for safety production in underground mining scenarios as described in claim 3, characterized in that: The standard operating procedure (SOP) clause records include a list of interactive actions, the start status of the task, and the completion status of the task.

5. The virtual training and interactive system for safety production in underground mining scenarios as described in claim 4, characterized in that: The encapsulation into an initial scene graph package involves the following steps. Write incremental numbers to the underground multimodal dataset according to the collection order, register the mutual pointing relationships between the underground multimodal dataset entries, and obtain the subordinate association between equipment semantic component entries and roadway entries, the mutual pointing relationship between volume roadways, the mutual pointing relationship between volume equipment and the mutual pointing relationship between working equipment; The VR binocular camera configuration and AR video synthesis channel configuration are used as presentation pointing values ​​and synthesis pointing values, respectively, and written into the roadway entries, mining face entries and equipment semantic component entries. The underground multimodal dataset and the mutual pointing relationships are used to form a scene graph. Based on the VR binocular camera configuration and AR video synthesis channel configuration, mark the occlusion items, and establish occlusion mutual pointing relationship with the roadway items, mining face items and equipment semantic component items. At the same time, register the field association in the occlusion items, and merge the occlusion items and field association into a visibility layer. The scene graph and visibility layer are uniformly encapsulated and synchronously written into the tool library model list, standard operating procedure clause record, and the numbering relationship of VR stereo camera configuration and AR video compositing channel configuration to generate the initial scene graph package.

6. The virtual training and interactive system for safety production in underground mining scenarios as described in claim 5, characterized in that: The encapsulation is called a constraint candidate package, which encapsulates interactive actions, hit objects, and the visible set of the current frame into a unified constraint candidate package.

7. The virtual training and interactive system for safety production in underground mining scenarios as described in claim 6, characterized in that: The steps for generating the scenario state package are as follows: Based on the special effects situation frames, an execution queue is established for interactive actions and hit objects. Based on the mutual pointing relationship of the working equipment, the alleyway entries are locked in the scene graph as the operating environment. Interactive actions are executed under the view domain association according to the order of the execution queue, generating equipment status snapshots and interaction event records. The equipment status snapshots and interaction event records are then merged into a process dataset. Based on the work step number, work start status and work completion status recorded in the standard operating procedure clauses, the consistency of status is judged, and emergency judgment is made in combination with the visibility deviation type, visibility deviation magnitude and volume risk deviation type, volume risk deviation magnitude, to obtain the progress result and emergency status indicator. The process dataset, progress results, emergency status identifiers, and field-of-view associations are uniformly summarized to generate a scenario status package.

8. The virtual training and interactive system for safety production in underground mining scenarios as described in claim 7, characterized in that: The steps for generating the security assessment report are as follows: Expand equipment status snapshots and interaction event records, statistically analyze the distribution of visibility deviation and volume risk deviation, and based on the progress results, perform work step numbering and progress statistics, and merge them with the distribution of visibility deviation and volume risk deviation into a set of safety assessment indicators; The safety assessment indicator set, progress results, and emergency status indicators are compiled into a unified safety assessment report.

Images