Real-time management and control method and system for downhole personnel based on multi-modal data fusion

Abstract

This application provides a method and system for real-time management of personnel going underground based on multimodal data fusion, specifically relating to the field of intelligent management technology for mine safety production. The method acquires multimodal data on personnel going underground, covering personnel location, vital signs, environmental monitoring, team communication, and pre-set work plan processes. It determines whether the team has entered a state of collaborative failure based on at least two data combinations. If this state is determined, a dynamic agent is identified based on the multimodal data and temporary command authority is granted. A first digital surrogate corresponding to the original team leader and a second digital surrogate with temporary command authority corresponding to the dynamic agent are generated. Based on the instructions recorded by the first digital surrogate and the real-time instructions received by the second digital surrogate, a unified virtual command view is generated and synchronized to the terminals of all team members. This method fills the gap in emergency command in critical situations, ensuring that a new management order can be quickly established after the team loses its original command core, buying time for self-rescue and rescue.

Description

Technical Field

[0001] This invention relates to the field of intelligent management technology for mine safety production, and more specifically, to a method and system for real-time control of personnel going down the mine based on multimodal data fusion. Background Technology

[0002] With the deepening of smart mine construction, real-time monitoring technology for underground personnel based on multimodal data fusion has become an industry trend. By integrating diverse information such as personnel positioning, environmental sensors, vital sign monitoring, and audio-visual data, this system can achieve comprehensive perception and early warning of the individual status of underground workers and the surrounding environment. However, current technological development mainly focuses on improving the accuracy of individual status monitoring and the ability to issue immediate alarms for environmental hazards. It lacks in-depth consideration of the overall and adaptive organizational control of the basic management unit of underground operations—the work team—in sudden emergency situations, resulting in blind spots in management logic.

[0003] Traditional solutions rely on a pre-defined, fixed management structure, where the team leader assumes command responsibilities, and the control system primarily functions as an information presentation and alarm tool. When extreme situations such as roof collapses or gas outbursts lead to team leader incapacitation, communication disruptions, or team members being scattered, existing systems can only issue the same alarm message to isolated individuals, failing to automatically intervene at the management level. This means that while the system can detect a "collaboration failure," it cannot automatically and promptly reconstruct the team's command chain. This results in a "management vacuum" between the occurrence of the "failure" and external rescue intervention, potentially plunging the team into chaos and severely impacting the efficiency and success rate of collective self-rescue.

[0004] In summary, the urgent problem to be solved is how to address the shortcomings of existing underground personnel management systems in automatically and promptly reconstructing the team command system to appoint temporary leaders and grant command authority when team coordination fails, due to reliance on a fixed management structure. Summary of the Invention

[0005] The main objective of this invention is to provide a real-time management method and system for downhole personnel based on multimodal data fusion. This aims to address the shortcomings of existing downhole personnel management systems, which lack the ability to automatically and instantly reconstruct the team command structure to designate temporary leaders and grant command authority when team coordination fails. This fills the gap in emergency command during critical situations, ensuring that the team can immediately form a new and effective management order after losing its original command core, thus gaining valuable time for self-rescue and rescue.

[0006] To achieve the above objectives, the present invention provides a method and system for real-time management and control of personnel going down into the well based on multimodal data fusion.

[0007] In a first aspect, the present invention provides a method for real-time management and control of personnel going down into the mine based on multimodal data fusion, the method comprising:

[0008] Acquire multimodal data of personnel going down the mine, including personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data;

[0009] Based on the combined state of at least two types of data in the multimodal data, it is determined whether the shift to which the personnel going down the well belong has entered a state of collaborative failure.

[0010] When it is determined that the collaborative failure state has been entered, a dynamic agent is identified from the team based on the multimodal data, and temporary command authority is granted to the dynamic agent.

[0011] Generate a first digital avatar corresponding to the original person in charge of the work group, and generate a second digital avatar corresponding to the dynamic agent and having the temporary command authority;

[0012] Based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, a unified virtual command view is generated and synchronized to the terminals of each member in the team.

[0013] Specifically, the acquisition of multimodal data of personnel going down the mine includes personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data, including:

[0014] The personnel location data is obtained through location beacons and identification cards worn by personnel;

[0015] The vital signs data are obtained through vital sign monitoring devices worn by personnel;

[0016] The environmental monitoring data is acquired by a sensor array deployed in the downhole operating area;

[0017] The team communication data is obtained by analyzing the communication status and content of the team's internal communication devices.

[0018] The pre-set work plan process data is retrieved from the database of the management and control system.

[0019] Specifically, determining whether the shift to which the personnel going down the mine belong has entered a state of collaborative failure based on the combined states of at least two types of data in the multimodal data includes:

[0020] Based on the personnel location data and the vital signs data, determine whether the original person in charge of the team is in a state of incapacity;

[0021] Based on the team communication data, determine whether the team's internal communication channel is in an abnormal communication state;

[0022] Based on the personnel location data, determine whether members in the work group exceeding a preset proportion are in a dispersed state;

[0023] Based on the pre-set work plan process data, determine whether the current work process of the team is in a stagnant state;

[0024] When at least two of the following states—disabled state, abnormal communication state, dispersed location state, and stagnant state—are simultaneously determined to be true, the work group is determined to have entered a collaborative failure state.

[0025] Specifically, when it is determined that the collaborative failure state has been entered, the process of determining a dynamic agent from within the work group based on the multimodal data and granting temporary command authority to the dynamic agent includes:

[0026] When it is determined that the collaborative failure state has been entered, obtain a list of other available members in the team other than those who are in a disabled state.

[0027] Based on the available member list, and according to each member's vital sign data, personnel location data, historical emergency record data, and current local environment data, the real-time agent competence index of each member is calculated.

[0028] Select the member with the highest real-time agent competence index from the list of available members, and determine the selected member as the dynamic agent;

[0029] An authorization instruction containing the scope and validity period of the authorization is sent to the terminal device worn by the dynamic agent to grant the temporary command authority.

[0030] Specifically, the generation of a first digital avatar corresponding to the original person in charge of the work group, and the generation of a second digital avatar corresponding to the dynamic agent and possessing the temporary command authority, include:

[0031] Create a first data container and store the last valid instruction issued by the original person in charge through the control system before entering the collaborative failure state into the first data container to generate the first digital surrogate;

[0032] A second data container is created, and the attribute information of the temporary command authority is associated with the identity of the dynamic agent to the second data container to generate the second digital avatar, which is configured to receive real-time instructions from the dynamic agent.

[0033] Specifically, the step of generating a unified virtual command view based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, and synchronizing the virtual command view to the terminals of each member within the team, includes:

[0034] Read the instructions recorded by the first digital double as a historical instruction stream;

[0035] Read the real-time instructions received by the dynamic agent by the second digital avatar as a real-time instruction stream;

[0036] By comparing the historical instruction stream with the real-time instruction stream, when an instruction conflict occurs, the real-time instruction stream shall prevail, and a unified instruction sequence shall be generated by merging them.

[0037] The unified instruction sequence, instruction source identifier, and real-time status overview of the work team are combined and encapsulated to generate the unified virtual command view;

[0038] The virtual command view is pushed to the terminal devices worn by each member of the team.

[0039] Specifically, determining whether members of the work group exceeding a preset proportion are geographically dispersed based on the personnel location data includes:

[0040] Based on the personnel location data, calculate the real-time distance between any two members in the work group;

[0041] Count the number of member pairs whose real-time distance exceeds a preset security collaboration threshold;

[0042] When the ratio of the number of member pairs exceeding the preset safety collaboration threshold to the total number of member pairs in the team is greater than the preset dispersion ratio threshold, the team is determined to be in a dispersed state.

[0043] Secondly, the present invention provides a real-time management and control system for personnel going down into the mine based on multimodal data fusion, wherein the management and control system applies the management and control method described in the first aspect, and the management and control system includes:

[0044] The multimodal data acquisition module is used to acquire multimodal data of personnel going down into the well. The multimodal data includes personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data.

[0045] The collaborative failure determination module is connected to the multimodal data acquisition module. The collaborative failure determination module is used to determine whether the shift to which the personnel going down the mine belongs has entered a collaborative failure state based on the combination state of at least two types of data in the multimodal data.

[0046] The dynamic agent determination and authorization module is connected to the collaborative failure determination module. When the collaborative failure state is determined, the dynamic agent determination and authorization module determines a dynamic agent from the team based on the multimodal data and grants temporary command authority to the dynamic agent.

[0047] A digital surrogate generation module is connected to the dynamic agent determination and authorization module. The digital surrogate generation module is used to generate a first digital surrogate corresponding to the original person in charge of the team, and to generate a second digital surrogate corresponding to the dynamic agent and having the temporary command authority.

[0048] The command view generation and synchronization module is connected to the digital double generation module. The command view generation and synchronization module is used to generate a unified virtual command view based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, and synchronize the virtual command view to the terminals of each member in the team.

[0049] Specifically, the multimodal data acquisition module includes:

[0050] The positioning data acquisition unit is used to acquire the personnel positioning data through positioning beacons and identification cards worn by personnel;

[0051] A vital signs data acquisition unit is used to acquire the vital signs data through a vital signs monitoring device worn by the personnel.

[0052] An environmental data acquisition unit is used to acquire the environmental monitoring data through a sensor array deployed in the downhole working area;

[0053] A communication data acquisition unit is used to acquire the team communication data through the communication status and communication content of the team's internal communication equipment.

[0054] The process data retrieval unit is used to retrieve the preset work plan process data from the database of the management and control system.

[0055] Specifically, the collaborative failure determination module includes:

[0056] The incapacity determination unit for the person in charge is used to determine whether the original person in charge of the team is incapacitated based on the personnel location data and the vital signs data.

[0057] The communication status determination unit is used to determine whether the internal communication channel of the work group is in an abnormal communication state based on the work group communication data.

[0058] The personnel dispersion determination unit is used to determine, based on the personnel positioning data, whether members in the work group exceeding a preset proportion are in a dispersed state.

[0059] The process stagnation determination unit is used to determine whether the current work process of the team is in a stagnant state based on the preset work plan process data.

[0060] The integrated status determination unit is connected to the responsible person's incompetence determination unit, the communication status determination unit, the personnel dispersion determination unit, and the process stagnation determination unit. The integrated status determination unit is used to determine that the team has entered a collaborative failure state when at least two of the incompetence state, the abnormal communication state, the location dispersion state, and the stagnation state are determined to be true at the same time.

[0061] This application provides a method and system for real-time management of personnel going underground based on multimodal data fusion. The method first acquires multimodal data on personnel going underground, including personnel location, vital signs, environmental monitoring, team communication, and pre-set work plan workflow data. Based on the combined state of at least two data types in the multimodal data, it determines whether the team to which the personnel belongs has entered a state of collaborative failure. If determined to be in a state of failure, a dynamic agent is identified from within the team based on the multimodal data and granted temporary command authority. A first digital surrogate corresponding to the original team leader and a second digital surrogate corresponding to the dynamic agent and possessing temporary command authority are then generated. Based on the instructions recorded by the first digital surrogate and the real-time instructions received by the second digital surrogate, a unified virtual command view is generated and synchronized to the terminals of all team members. This method overcomes the shortcomings of existing systems, fills the gap in emergency command, and ensures that a new management order can be quickly established after the team loses its original command core, thus buying time for self-rescue and rescue. Attached Figure Description

[0062] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0063] Figure 1 A flowchart illustrating the real-time control method for personnel going down into the well based on multimodal data fusion provided in this application;

[0064] Figure 2 A connection diagram of the real-time control system for personnel going down into the well based on multimodal data fusion provided in this application.

[0065] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0068] In this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0069] This application provides a method and system for real-time management and control of personnel going underground based on multimodal data fusion. This method acquires multimodal data on personnel going underground, covering information such as personnel location and vital signs, and uses this data to determine the team's collaborative status. Once a collaborative failure is determined, a dynamic agent is identified using the multimodal data and granted temporary command authority. Digital avatars corresponding to the original person in charge and the dynamic agent are then generated. Based on their commands, a unified virtual command view is generated and synchronized to member terminals, enabling automatic and real-time reconstruction of the team's command system and filling gaps in emergency command.

[0070] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0071] Figure 1 This is a flowchart illustrating the real-time control method for underground personnel based on multimodal data fusion provided in this application, as shown below. Figure 1 As shown in this embodiment, a real-time management method for personnel going down into the mine based on multimodal data fusion is provided. The method includes:

[0072] S101: Acquire multimodal data of personnel going down the mine, including personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data.

[0073] Specifically, the acquisition of multimodal data of personnel going down the well includes personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data. This includes: acquiring personnel positioning data through positioning beacons and identification cards worn by personnel; acquiring vital sign data through vital sign monitoring devices worn by personnel; acquiring environmental monitoring data through sensor arrays deployed in the downhole work area; acquiring team communication data through the communication status and content of internal team communication devices; and retrieving the pre-set work plan process data from the database of the control system.

[0074] The specific steps of implementation S101 include:

[0075] Multiple UWB (Ultra-Wideband) positioning beacons are deployed on the top of the underground roadway. Each UWB positioning beacon has a unique identifier and known fixed three-dimensional coordinates, and communicates with identification cards worn by personnel going down the mine. The identification cards have built-in UWB tags and periodically send wireless pulse signals to the UWB positioning beacons. The control system receives the signal arrival time information uploaded by the UWB positioning beacons, uses the Time Difference of Arrival (TDOA) algorithm to calculate the distance from the identification card to the multiple UWB positioning beacons, and then uses trilateration to calculate the real-time three-dimensional coordinates of the identification card as personnel positioning data. The personnel positioning data includes coordinate values ​​(x, y, z) and the corresponding timestamp.

[0076] Vital signs data are acquired through smart bracelets worn by personnel working underground. These bracelets integrate a heart rate sensor, a blood oxygen sensor, and a triaxial accelerometer. The heart rate sensor uses photoplethysmography (PPG) to measure heart rate and outputs a heart rate value in beats per minute. The blood oxygen sensor emits red and infrared light to illuminate the skin and calculates blood oxygen saturation (SpO2) based on the ratio of reflected light intensity, outputting a blood oxygen saturation value in percentage. The triaxial accelerometer measures the acceleration of the personnel in three axes, outputting an acceleration value in meters per second squared (m² / s), and uses a threshold method based on the acceleration value to detect falls. The smart bracelet transmits the heart rate, blood oxygen saturation, and acceleration values ​​to an underground wireless access point via Bluetooth Low Energy (BLE) protocol, which then forwards them to the control system, forming vital signs data.

[0077] Environmental monitoring data is acquired through a sensor array deployed in the downhole operating area. The sensor array includes a methane sensor, a carbon monoxide sensor, a temperature sensor, and a humidity sensor. Each sensor is connected to the control system substation via wired or Zigbee wireless protocol. The methane sensor uses the catalytic combustion principle to measure methane concentration and outputs a concentration value in %LEL (lower explosive limit percentage). The carbon monoxide sensor uses the electrochemical principle to measure carbon monoxide concentration and outputs a concentration value in ppm (parts per million). The temperature sensor uses a thermocouple to measure ambient temperature and outputs a temperature value in degrees Celsius. The humidity sensor uses a capacitive sensing element to measure relative humidity and outputs a humidity value in percentage. The control system substation periodically reads the sensor values ​​and adds timestamps to form environmental monitoring data.

[0078] Team communication data is acquired through explosion-proof walkie-talkies equipped within the work group. These walkie-talkies connect to the control system base station via a Zigbee wireless network. The control system base station monitors the network connection status of each walkie-talkie in real time, recording whether the status is "online" or "offline." The voice communication of the walkie-talkies is converted into a digital audio stream via a vocoder. The control system base station receives the digital audio stream and applies a Voice Activity Detection (VAD) algorithm to analyze the audio stream. The VAD algorithm determines whether each frame of audio is valid speech based on short-time energy and zero-crossing rate characteristics, and calculates the proportion of valid speech frames to determine whether the communication content contains valid speech. The team communication data includes the walkie-talkie connection status and the valid speech detection results.

[0079] The system retrieves pre-set work plan workflow data from the relational database of the management and control system. The relational database of the management and control system is a MySQL database, which stores the plan workflow table. The plan workflow table includes fields: team number, plan start time, plan end time, task step sequence, and current step index. The management and control system retrieves the task step sequence and current step index corresponding to the current team as the pre-set work plan workflow data using the SQL query statement "SELECT task step sequence, current step index FROM plan workflow table WHERE team number = [current team number]".

[0080] The control system temporarily stores personnel location data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data in a data buffer. All data is appended with a timestamp and data source identifier in a uniform format. The timestamp is accurate to milliseconds, and the data source identifier is used to distinguish data categories and specific equipment.

[0081] This step uses specific hardware sensors, communication protocols, and database query methods to collect and integrate multi-dimensional data on the location, physiology, environment, communication, and task progress of personnel in the mine in real time. This ensures the accuracy and real-time nature of the data source and provides comprehensive and reliable data input for intelligent judgment of collaborative failure in subsequent steps, forming the data perception foundation of the entire control method.

[0082] S102: Based on the combined state of at least two types of data in the multimodal data, determine whether the shift to which the personnel going down the well belong has entered a collaborative failure state.

[0083] Specifically, determining whether the work group to which the personnel going down the mine belongs has entered a state of collaborative failure based on the combined state of at least two types of data in the multimodal data includes: determining whether the original leader of the work group is in a state of incapacity based on the personnel positioning data and the vital signs data; determining whether the internal communication channel of the work group is in an abnormal communication state based on the work group communication data; determining whether more than a preset proportion of members in the work group are in a state of location dispersion based on the personnel positioning data; determining whether the current work process of the work group is in a state of stagnation based on the preset work plan process data; and determining that the work group has entered a state of collaborative failure when at least two of the incapacity state, the abnormal communication state, the location dispersion state, and the stagnation state are simultaneously determined to be true.

[0084] Specifically, determining whether members in the work group exceeding a preset proportion are in a dispersed state based on the personnel positioning data includes: calculating the real-time distance between any two members in the work group based on the personnel positioning data; counting the number of member pairs whose real-time distance exceeds a preset safe collaboration threshold; and determining that the work group is in a dispersed state when the ratio of the number of member pairs exceeding the preset safe collaboration threshold to the total number of member pairs in the work group is greater than a preset dispersion proportion threshold.

[0085] The specific steps of implementation S102 include:

[0086] The control system reads the latest multimodal data from the data buffer generated in step S101, within a preset time window (e.g., 30 seconds) prior to the current timestamp. For the original team leader, the system extracts the coordinate value sequence from the personnel positioning data corresponding to the original leader's identification card and calculates the standard deviation of the coordinate value sequence within the time window. If the standard deviation of the coordinate value sequence is less than a preset stationary distance threshold (e.g., 0.5 meters), the system determines that the original leader is stationary. Simultaneously, the system extracts the vital signs data corresponding to the original leader's smart bracelet: heart rate is read; if the heart rate is continuously below 50 beats / minute or continuously above 150 beats / minute, an abnormal heart rate is determined; blood oxygen saturation is read; if the blood oxygen saturation is continuously below 90%, an abnormal blood oxygen is determined; and three-axis acceleration counts are read, and a pre-trained threshold-based fall detection algorithm is applied. If the algorithm outputs "fall," a fall event is determined to have occurred. When at least two of the following four conditions are met simultaneously: "position is stationary", "abnormal heart rate", "abnormal blood oxygen", and "fall event occurs", the original leader of the team is determined to be in a disabled state, and the Boolean variable Is_Leader_Disabled is set to True; otherwise, it is set to False.

[0087] The control system reads the communication data of all explosion-proof walkie-talkies within the work group. It checks the walkie-talkie connection status in the communication data; if more than half of the walkie-talkies in the group are "offline," the communication connection is considered abnormal. Simultaneously, it analyzes the effective voice detection results in the communication data, calculating the proportion of audio frames identified as effective voice by the Voice Activity Detection (VAD) algorithm out of the total audio frames uploaded by all walkie-talkies in the past minute, obtaining the effective voice ratio V_active_ratio. If the effective voice ratio V_active_ratio is lower than a preset silence threshold (e.g., 5%), the communication content is considered abnormal. When at least one of the two conditions, "abnormal communication connection" and "abnormal communication content," is met, the internal communication channel of the work group is determined to be in an abnormal communication state, and the abnormal communication state Boolean variable Is_Comm_Abnormal is set to True; otherwise, it is set to False.

[0088] The control system uses personnel location data to determine whether members within a work group exceeding a preset proportion are geographically dispersed. First, it retrieves the location coordinates P_i(x_i, y_i, z_i) of all N members in the work group at the latest timestamp from the data buffer, where i represents the member number from 1 to N. Then, it calculates the three-dimensional Euclidean distance D_{j,k} between any two different members j and k within the work group, using the formula: D_{j,k} = sqrt((x_j - x_k)). 2+ (y_j - y_k) 2 + (z_j - z_k) 2 Next, each distance D_{j,k} is compared with a preset safety collaboration threshold D_safe (e.g., 20 meters), and the number of member pairs where D_{j,k} > D_safe is counted, denoted as M. The total number of member pairs in the team is the number of combinations C(N,2). Finally, the dispersion ratio R_disperse = M / C(N,2) is calculated. If the dispersion ratio R_disperse is greater than the preset dispersion ratio threshold R_th (e.g., 0.3), the team is determined to be in a dispersed state, and the Boolean variable Is_Team_Dispersed is set to True; otherwise, it is set to False.

[0089] The control system determines whether the current work process of a work group is stalled based on pre-set work plan process data. It reads the task step sequence and the current step index from the pre-set work plan process data. Based on the task step sequence, it obtains the expected duration T_expected and the planned start time T_start of the planned task step pointed to by the current step index. It retrieves the current timestamp T_now from the system. It calculates the planned progress time T_elapsed = T_now - T_start. If the planned progress time T_elapsed exceeds 150% of the expected duration T_expected, and during this timeout period, the control system does not record any operation logs related to advancing the task step that are confirmed by the original person in charge of the work group or the system, then the current work process of the work group is determined to be stalled, and the Boolean variable Is_Process_Stalled is set to True; otherwise, it is set to False.

[0090] The control system performs a comprehensive assessment of collaborative failure. It reads four Boolean state variables: Is_Leader_Disabled (disabled state), Is_Comm_Abnormal (abnormal communication state), Is_Team_Dispersed (dispersed location state), and Is_Process_Stalled (stagnant state). A counter, count, is set, initially to 0. If Is_Leader_Disabled is True, count is incremented by 1; if Is_Comm_Abnormal is True, count is incremented by 1; if Is_Team_Dispersed is True, count is incremented by 1; if Is_Process_Stalled is True, count is incremented by 1. If the counter value count >= 2, the team to which the personnel going down the mine belongs is determined to have entered a collaborative failure state, generating a collaborative failure assessment result Is_Synergy_Failure = True; otherwise, a collaborative failure assessment result Is_Synergy_Failure = False is generated. The control system records the collaborative failure judgment result Is_Synergy_Failure, the specific combination of state variables that triggered the judgment, and the judgment timestamp to the event log, and uses it as the decision basis for triggering the subsequent step S103.

[0091] This step defines and calculates four quantifiable states in parallel: incapacitated leader, communication anomaly, personnel dispersion, and process stagnation. It then uses a combination logic of "at least two states must be true simultaneously" for comprehensive decision-making. This transforms the abstract concept of "cooperative failure" into clear and automatically executable judgment rules, significantly improving the accuracy and reliability of the system's identification of abnormal management situations in underground work groups. It also avoids false triggers caused by false alarms from a single sensor, providing accurate and timely decision input for subsequent emergency command reconstruction.

[0092] S103: When it is determined that the collaborative failure state has been entered, a dynamic agent is determined from the team based on the multimodal data, and temporary command authority is granted to the dynamic agent.

[0093] Specifically, when it is determined that the collaborative failure state has been entered, determining a dynamic agent from within the team based on the multimodal data and granting temporary command authority to the dynamic agent includes: when it is determined that the collaborative failure state has been entered, obtaining a list of other available members within the team excluding those in a disabled state; based on the list of available members, calculating the real-time agent competence index of each member according to their vital signs data, personnel location data, historical emergency record data, and current local environmental data; selecting the member with the highest real-time agent competence index from the list of available members and determining the selected member as the dynamic agent; and sending an authorization instruction containing the authorization scope and validity period to the terminal device worn by the dynamic agent to grant the temporary command authority.

[0094] The specific steps in step S103 during implementation include:

[0095] When the control system determines that the collaborative failure judgment result Is_Synergy_Failure generated in step S102 is True, it triggers the dynamic agent determination and authorization process. The control system first removes the original person in charge who was determined to be in a disabled state (i.e., Is_Leader_Disabled is True) from the team personnel information table, and generates a list of available members L_available that contains the identity identifiers of all other members in the team.

[0096] For each candidate member m in the available member list L_available, the control system calculates its real-time agent competence index CSI_m. The calculation of CSI_m is based on a weighted scoring model that integrates real-time and historical data from four dimensions. The formula is: CSI_m = w1 * S_vital(m) + w2 * S_central(m) + w3 * S_historical(m) - w4 * R_env(m). Here, w1, w2, w3, and w4 are preset positive weight coefficients, and w1 + w2 + w3 = 1, with w4 being the environmental risk discount coefficient.

[0097] Dimension 1: Vital Sign Stability Score S_vital(m): The control system reads the vital sign data sequence of candidate member m within the most recent minute. The standard deviation of the heart rate sequence σ_HR and the standard deviation of the blood oxygen saturation sequence σ_SpO2 are calculated. Heart rate stability thresholds TH_HR and TH_SpO2 are set. The calculation rule for the vital sign stability score S_vital(m) is as follows: if σ_HR < TH_HR and σ_SpO2 < TH_SpO2, then S_vital(m) = 1.0; if only one criterion is met, then S_vital(m) = 0.5; if neither criterion is met, then S_vital(m) = 0. This score reflects whether the candidate member's current physiological state is suitable for undertaking command pressure.

[0098] Dimension Two: Location Centrality Score S_central(m): The control system reads the latest personnel location coordinates of all team members (including candidate member m). It calculates the sum of Euclidean distances D_sum(m) from candidate member m to all other members within the team (including the original leader). Then, it calculates the average D_avg and minimum D_min of all available members' D_sum(m). The location centrality score S_central(m) is calculated using a linear mapping: S_central(m) = 1.0 - (D_sum(m) - D_min) / (D_avg - D_min). When D_sum(m) is less than D_avg, the score approaches 1, indicating that the member is located in a relatively central position within the team, facilitating coordination.

[0099] Dimension 3: Historical Emergency Record Score S_historical(m): The control system queries the historical record database for candidate member m's historical emergency record data. This data includes the number of emergency drills participated in over the past year, N_drill, the average performance score in the drills, Score_avg (out of 100), and whether there are any actual emergency response records, Flag_actual (a Boolean value). Historical Emergency Record Score S_historical(m) = (N_drill / N_drill_max) * 0.3 + (Score_avg / 100) * 0.5 + (Flag_actual ? 0.2 : 0), where N_drill_max is the preset maximum number of drills as a reference value. This score is standardized to between 0 and 1.

[0100] Dimension Four: Current Local Environmental Risk Coefficient R_env(m): Based on the personnel location data coordinates of candidate member m, the control system locates the three closest environmental monitoring sensors (from environmental monitoring data in step S101). The methane concentration, carbon monoxide concentration, and temperature values ​​of these three sensors are read. The calculation rule for the local environmental risk coefficient R_env(m) is as follows: for each sensor whose methane concentration exceeds the warning value, R_env(m) increases by 0.2; for each sensor whose carbon monoxide concentration exceeds the warning value, R_env(m) increases by 0.2; for each sensor whose temperature exceeds the warning value, R_env(m) increases by 0.1. The maximum value of R_env(m) is limited to 1.0. This coefficient, as a deduction term, reflects the risk of the local environment in which the candidate member is located; the more dangerous the environment, the greater the discount on their competence index.

[0101] Based on the aforementioned formulas and rules, the control system calculates the Real-Time Agent Competency Index (CSI_m) for each candidate member m in the available member list L_available. Then, the control system selects the candidate member with the highest CSI_m value and identifies it as a dynamic agent (Agent_dynamic). If there is a tie for the highest score, the member with the higher S_historical(m) score is prioritized.

[0102] Once the dynamic agent (Agent_dynamic) is identified, the control system generates an authorization instruction. This instruction is a structured data packet containing the following fields: instruction type ("authorization granted"), target member identifier (Agent_dynamic), scope of granted temporary command authority (e.g., "can view vital signs and environmental data of the entire group", "can broadcast text and voice instructions to the entire group", "can mark dangerous areas"), validity period of the authority (e.g., 30 minutes from the date of grant), and a unique encrypted authorization code generated by the system. The control system pushes this authorization instruction data packet to the smart terminal (such as an explosion-proof smartphone or ruggedized tablet computer) worn by the dynamic agent (Agent_dynamic) through the underground industrial ring network.

[0103] Upon receiving the authorization command, the smart terminal using the Agent_dynamic agent parses and displays the authorization information, prompting the user to perform biometric verification (e.g., fingerprint verification via the terminal's fingerprint sensor). Only after successful verification does the smart terminal return a confirmation receipt to the control system, containing the authorization code. Upon receiving the receipt with the correct authorization code, the control system officially records the successful grant of temporary command permissions and binds the Agent_dynamic's identity identifier with its granted permission scope, effective time, and expiration time, storing it in the system's runtime permission list, thus completing the granting of temporary command permissions.

[0104] This step, by designing a multi-dimensional weighted scoring model that integrates real-time physiological status, spatial location, historical experience, and local environmental risks, enables an intelligent decision-making process for selecting the most suitable temporary commander from available members in emergency situations, avoiding the blindness of subjective or sequential designation. Furthermore, by combining encrypted authorization instructions with biometric verification, the security and seriousness of the authorization process are ensured, laying a legal and credible foundation for the subsequent formal transfer and exercise of command authority.

[0105] S104: Generate a first digital avatar corresponding to the original person in charge of the work group, and generate a second digital avatar corresponding to the dynamic agent and having the temporary command authority.

[0106] Specifically, generating a first digital surrogate corresponding to the original person in charge of the work group, and generating a second digital surrogate corresponding to the dynamic agent and having the temporary command authority, includes: creating a first data container and storing the last valid instruction issued by the original person in charge through the control system before entering the collaborative failure state into the first data container to generate the first digital surrogate; creating a second data container and associating the attribute information of the temporary command authority with the identity identifier of the dynamic agent to the second data container to generate the second digital surrogate, wherein the second digital surrogate is configured to receive real-time instructions from the dynamic agent.

[0107] The specific steps in step S104 during implementation include:

[0108] The control system creates the first digital surrogate in memory. First, the control system creates a new collection in a non-relational database (such as MongoDB), named DigitalTwin_Leader_[team number]_[timestamp], which serves as the first data container. Next, the control system queries the instruction log database. The instruction log database records all instructions issued through the control system. Each instruction log entry contains fields: instruction ID, issuer identity, instruction type (voice-to-text, text, map marker, etc.), instruction content, list of receiving terminals, issuance timestamp, and instruction status (sent, received, confirmed, executed). The control system constructs a query statement to retrieve instruction records from the instruction log database that meet all of the following conditions: the issuer identity is equal to the original team leader's identity; the instruction status is "sent" or "received" (indicating successful transmission); and the issuance timestamp is earlier than the time when the collaborative failure judgment result Is_Synergy_Failure = True was generated in step S102. From these records, the control system selects the instruction record with the latest timestamp, determining it to be the last valid instruction issued by the original responsible person before entering the collaborative failure state. Then, the control system serializes all fields of this instruction record into a JSON data packet. Finally, the control system writes the serialized JSON data packet as a document into a MongoDB collection created for the first digital surrogate, completing the generation of the first digital surrogate. The instructions recorded by this first digital surrogate are static and historical, and only a read interface is provided subsequently.

[0109] The control system creates a second digital avatar in memory. First, it creates a new collection in the same NoSQL database, named DigitalTwin_Agent_[Dynamic Agent Identity]_[Timestamp], which serves as the second data container. Next, the control system reads the temporary command permission attribute information bound to the identity of the dynamic agent Agent_dynamic from the runtime permission list in step S103. This attribute information is a data structure containing a list of permission scopes (e.g., [“VIEW_ALL_VITAL”, “BROADCAST_MSG”, “MARK_HAZARD”]), permission activation timestamps, and permission expiration timestamps. Then, the control system retrieves the complete identity information of the dynamic agent Agent_dynamic from the personnel information database, including name, employee ID, and work group. Next, the control system constructs an initialization document containing the following core fields: `agent_identity` (dynamic agent identity information), `granted_permissions` (temporary command permission attributes), `status` (initial status set to "ACTIVE"), `creation_time` (creation timestamp), and an empty `command_inbox` array field for receiving real-time commands. The control system writes this initialization document to a MongoDB collection created for the second digital avatar. To enable the second digital avatar to receive real-time commands from the dynamic agent, the control system provides a dedicated, authenticated, and authorized RESTful API write endpoint for the `command_inbox` field of this MongoDB collection. The client application running on the terminal held by the dynamic agent `Agent_dynamic`, after gaining authorization through biometric verification, can append the real-time commands input by the dynamic agent (formatted as a JSON object containing command type, content, and timestamp) to the `command_inbox` array by calling this API endpoint, thus submitting the real-time commands. The second digital avatar is configured to continuously monitor changes to the `command_inbox` array to receive new commands.

[0110] After generating the first and second digital avatars, the control system records their metadata in the system's digital avatar management registry. The metadata includes the set name corresponding to the digital avatar, the associated physical entity (original responsible person or dynamic agent), the type ("HISTORICAL" or "ACTIVE"), and the lifecycle status. The first and second digital avatars logically belong to the same virtual command space through their associated work group number field, providing a data source for the instruction flow fusion in step S105.

[0111] This step concretizes the abstract concept of "digital surrogate" into persistently stored and programmably accessed data entities by creating specific sets with clear data structures for the original person in charge and the dynamic agent, storing them in a non-relational database, and defining different data access and update strategies. The first digital surrogate permanently stores historical instructions to maintain management continuity, while the second digital surrogate dynamically receives new instructions and associates temporary permissions to support current command. Together, they constitute a precise and dynamic mapping of command roles and permissions in the physical world in the virtual space, providing a structured data foundation for the subsequent generation of a unified command view.

[0112] S105: Based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, a unified virtual command view is generated, and the virtual command view is synchronized to the terminals of each member in the team.

[0113] Specifically, the step of generating a unified virtual command view based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, and synchronizing the virtual command view to the terminals of each member in the team, includes: reading the instructions recorded by the first digital double as a historical instruction stream; reading the real-time instructions received by the dynamic agent by the second digital double as a real-time instruction stream; comparing the historical instruction stream and the real-time instruction stream, and when instructions conflict, using the real-time instruction stream as the standard to generate a unified instruction sequence; combining the unified instruction sequence, the instruction source identifier, and the team's real-time status overview to encapsulate and generate the unified virtual command view; and pushing the virtual command view to the terminal devices worn by each member in the team.

[0114] The specific steps in step S105 during implementation include:

[0115] The command view generation service of the control system first queries the digital stunt double management registry for the storage location information (i.e., the corresponding MongoDB collection name) of the first and second digital stunt doubles corresponding to the current shift. This service then connects to the corresponding non-relational database (MongoDB) and performs data read operations.

[0116] For the first digital surrogate, the view generation service sends a query request to a collection named DigitalTwin_Leader_[team number]_[timestamp] with an empty query condition to retrieve the unique document in that collection. From the returned document, the service extracts fields such as "instruction content," "instruction type," and "issuance timestamp," and sorts these fields according to the chronological order of the "issuance timestamps" to form an ordered list. This list is defined as the Historical Command Stream (Historical_Command_Stream). Each instruction in the Historical Command Stream is tagged with its source as "original responsible person (historical)."

[0117] For the second digital avatar, the view generation service sends a query request to a collection named DigitalTwin_Agent_[Dynamic Agent Identity]_[Timestamp], specifically querying the latest content of the command_inbox array field. The service extracts a list of command JSON objects sorted by timestamp from the command_inbox array and defines it as a Real-time Command Stream. Each command in the Real-time Command Stream is marked as originating from "Dynamic Agent (Real-time)". Simultaneously, the service verifies whether the timestamp of each real-time command is within the validity period of the temporary command permission granted in step S103, filtering out expired commands.

[0118] The view generation service then starts the command fusion engine. The engine takes the Historical Command Stream and the Real-time Command Stream as input. At its core is a rule-based and time-based conflict detection and resolution module. The engine merges the two command streams in chronological order (from earliest to latest), forming a preliminary merge list. When the engine detects a conflict in the merge list—for example, two operation commands for the same target device (such as a local ventilator, one from the historical stream requesting "on," and the other from the real-time stream requesting "off"), or two path planning commands for the same area but with opposite destinations—it activates the conflict resolution rules. The default conflict resolution rules are clear: when operation commands on the same target entity conflict, the later command takes precedence; if conflicting commands are the same time (or cannot be precisely compared), the command originating from "Dynamic Agent (Real-time)" is prioritized by default. Based on this rule, the engine retains higher-priority commands in the merge list and removes or marks lower-priority conflicting commands as "overridden." Ultimately, the engine outputs a unified command sequence (Unified_Command_Sequence) that eliminates critical operation conflicts and is ordered by timestamps.

[0119] Next, the view generation service creates a real-time team status overview, Team_Status_Overview. The service retrieves the latest multimodal data snapshot from the data buffer (generated in step S101): extracting the latest coordinates of all members from personnel positioning data and calculating the team center point; extracting each member's heart rate and blood oxygen status (normal / abnormal) from vital sign data; and extracting key sensor readings for each area from environmental monitoring data (such as the highest methane concentration and its location). The service organizes this information into a structured JSON object, which serves as the Team_Status_Overview.

[0120] Then, the view generation service encapsulates the data. The service creates a new JSON object as the data packet for the Virtual_Command_View. This data packet contains the following main fields:

[0121] 1. unified_command_sequence: The value is the unified command sequence Unified_Command_Sequence.

[0122] 2. command_source_markers: A list that records the original source ("original owner (historical)" or "dynamic agent (real-time)") of each instruction in Unified_Command_Sequence.

[0123] 3. team_status_overview: The value is Team_Status_Overview, which provides a real-time overview of the team's status.

[0124] 4. view_generation_timestamp: The timestamp when the view was generated.

[0125] 5. current_commander: The currently active commander identity (i.e., the identity information of the dynamic agent).

[0126] Finally, the view generation service pushes the encapsulated Virtual_Command_View data packet in real time to the smart terminal (such as an explosion-proof smartphone) worn by each member of the team via a WebSocket connection in the underground industrial ring network. Upon receiving the data packet, the member's terminal device parses and renders the data using a local application: clearly displaying the unified_command_sequence in a timeline or list format, and using different colors or icons to distinguish the source of instructions in the command_source_markers; simultaneously, displaying the team_status_overview graphically in another area of ​​the interface (such as a floor plan overlaid with status icons). The view refresh frequency is synchronized with the server push frequency (e.g., once per second) or with active requests from member terminals.

[0127] This step extracts the command stream from two digital avatars, applies explicit conflict resolution rules to generate an unambiguous unified command sequence, and synchronizes it with a structured data view encapsulated in a real-time team status overview. This enables intelligent fusion and unified presentation of potentially multi-source and out-of-time commands during dynamic changes in command authority, ensuring that all team members receive consistent, up-to-date command information that is integrated with the on-site situation. This constructs a clear, authoritative, and unified command interface at the virtual level, effectively preventing on-site chaos caused by inconsistent command sources or outdated information.

[0128] This embodiment provides a real-time management method for personnel going down into the mine based on multimodal data fusion. This method acquires multimodal data on personnel, covering personnel location, vital signs, environmental monitoring, team communication, and pre-set work plans and processes, providing comprehensive information support for management. Based on the combined states of at least two data types from the multimodal data, it determines whether the team to which the personnel belong has entered a state of collaborative failure, accurately identifying special situations. If this state is determined, a dynamic agent is identified from within the team based on the multimodal data and granted temporary command authority, enabling rapid adjustment of the command system. Subsequently, a first digital surrogate corresponding to the original team leader and a second digital surrogate corresponding to the dynamic agent and possessing temporary command authority are generated. A unified virtual command view is generated based on the instructions of both and synchronized to the terminals of all team members. This method overcomes the shortcomings of existing systems, fills the gap in emergency command, and ensures that a new management order can be quickly established after the team loses its original command core, buying time for self-rescue and rescue.

[0129] Figure 2 This is a connection diagram of the real-time control system for downhole personnel based on multimodal data fusion provided in this application, as shown in the image. Figure 2 As shown, this embodiment provides a real-time management and control system for personnel going down into the mine based on multimodal data fusion. This system applies... Figure 1 The real-time management and control method for personnel going down the mine based on multimodal data fusion described in the embodiment includes a management and control system comprising:

[0130] The multimodal data acquisition module is used to acquire multimodal data of personnel going down into the well. The multimodal data includes personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data.

[0131] The collaborative failure determination module is connected to the multimodal data acquisition module. The collaborative failure determination module is used to determine whether the shift to which the personnel going down the mine belongs has entered a collaborative failure state based on the combination state of at least two types of data in the multimodal data.

[0132] The dynamic agent determination and authorization module is connected to the collaborative failure determination module. When the collaborative failure state is determined, the dynamic agent determination and authorization module determines a dynamic agent from the team based on the multimodal data and grants temporary command authority to the dynamic agent.

[0133] A digital surrogate generation module is connected to the dynamic agent determination and authorization module. The digital surrogate generation module is used to generate a first digital surrogate corresponding to the original person in charge of the team, and to generate a second digital surrogate corresponding to the dynamic agent and having the temporary command authority.

[0134] The command view generation and synchronization module is connected to the digital double generation module. The command view generation and synchronization module is used to generate a unified virtual command view based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, and synchronize the virtual command view to the terminals of each member in the team.

[0135] Specifically, the multimodal data acquisition module includes:

[0136] The positioning data acquisition unit is used to acquire the personnel positioning data through positioning beacons and identification cards worn by personnel;

[0137] A vital signs data acquisition unit is used to acquire the vital signs data through a vital signs monitoring device worn by the personnel.

[0138] An environmental data acquisition unit is used to acquire the environmental monitoring data through a sensor array deployed in the downhole working area;

[0139] A communication data acquisition unit is used to acquire the team communication data through the communication status and communication content of the team's internal communication equipment.

[0140] The process data retrieval unit is used to retrieve the preset work plan process data from the database of the management and control system.

[0141] Specifically, the collaborative failure determination module includes:

[0142] The incapacity determination unit for the person in charge is used to determine whether the original person in charge of the team is incapacitated based on the personnel location data and the vital signs data.

[0143] The communication status determination unit is used to determine whether the internal communication channel of the work group is in an abnormal communication state based on the work group communication data.

[0144] The personnel dispersion determination unit is used to determine, based on the personnel positioning data, whether members in the work group exceeding a preset proportion are in a dispersed state.

[0145] The process stagnation determination unit is used to determine whether the current work process of the team is in a stagnant state based on the preset work plan process data.

[0146] The integrated status determination unit is connected to the responsible person's incompetence determination unit, the communication status determination unit, the personnel dispersion determination unit, and the process stagnation determination unit. The integrated status determination unit is used to determine that the team has entered a collaborative failure state when at least two of the incompetence state, the abnormal communication state, the location dispersion state, and the stagnation state are determined to be true at the same time.

[0147] In implementation, the real-time personnel management system based on multimodal data fusion provided in this embodiment includes a sensor network deployed in the underground area, personnel wearable devices, communication base stations, and server clusters and workstations deployed on the surface or in underground refuge chambers. Each module of the system uses the aforementioned hardware as a carrier and implements its functions through software programs. The connection between modules is achieved through software-level data interface calls, message queue transmission, and shared database access.

[0148] The multimodal data acquisition module includes a positioning data acquisition unit, a vital signs data acquisition unit, an environmental data acquisition unit, a communication data acquisition unit, and a process data retrieval unit. The positioning data acquisition unit includes multiple UWB (Ultra-Wideband) positioning beacons deployed within the tunnel and UWB identification cards worn by personnel. The UWB beacons periodically communicate wirelessly with the identification cards for ranging. The positioning data acquisition unit, running on a server, receives the ranging information, calculates the distance using the Time Difference of Arrival (TDOA) algorithm, and then calculates the three-dimensional coordinates of each identification card using trilateration, thereby continuously generating personnel positioning data. The vital signs data acquisition unit includes a smart bracelet worn by personnel, integrating a PPG (Photoplethysmography) heart rate and blood oxygen sensor and a triaxial accelerometer. The smart bracelet transmits the collected raw heart rate, blood oxygen saturation, and acceleration data via Bluetooth to the nearest wireless access point, ultimately transmitting it to the server. The vital signs data acquisition unit parses the raw data and uses a threshold judgment method to perform fall detection on the acceleration data, forming structured vital signs data. The environmental data acquisition unit comprises a sensor array consisting of a catalytic combustion methane sensor, an electrochemical carbon monoxide sensor, a thermocouple temperature sensor, and a capacitive humidity sensor, deployed in key areas such as the mining face and return airway. These sensors upload concentration, temperature, and humidity readings to the substation via an industrial bus or Zigbee network. The environmental data acquisition unit periodically polls the substation to read data, generating timestamped environmental monitoring data. The communication data acquisition unit connects to the explosion-proof digital walkie-talkies equipped in the work group and their connected Zigbee communication base stations. The communication data acquisition unit polls the base station in real time to obtain the online / offline connection status of each walkie-talkie and applies a short-time energy and zero-crossing rate-based voice activity detection (VAD) algorithm to the voice stream forwarded by the base station to analyze the effective voice proportion, thereby generating work group communication data containing connection status and communication content characteristics. The process data retrieval unit connects to the MySQL relational database in the management system backend. By executing structured query language (SQL) statements, it retrieves and returns the task step sequence and current step index corresponding to the current work group from the database table storing planned processes, generating pre-set work plan process data. The multimodal data acquisition module adds timestamps and source identifiers to the above five types of data and writes them into a shared memory data buffer or message middleware (such as Kafka), providing standardized and real-time multi-source data input for subsequent modules.

[0149] The collaborative failure determination module subscribes to the latest data written to the data buffer by the multimodal data acquisition module. The collaborative failure determination module includes a responsible person incapacity determination unit, a communication status determination unit, a personnel dispersion determination unit, a process stagnation determination unit, and a comprehensive status determination unit. The responsible person incapacity determination unit reads the latest personnel location data and vital sign data of the original responsible person from the data buffer. It calculates the standard deviation of the responsible person's coordinate movement within a short time window; if it is lower than the static threshold, it is determined to be static. Simultaneously, it checks whether the heart rate exceeds the normal range, whether blood oxygen is lower than the threshold, and whether acceleration triggers a fall judgment. When at least two of the four conditions—"static," "abnormal heart rate," "abnormal blood oxygen," and "fall"—are simultaneously met, the responsible person incapacity determination unit outputs a Boolean value "true" indicating that the responsible person is incapacitated. The communication status determination unit reads the team's communication data. It checks whether the number of online walkie-talkies is less than half of the team's total and calculates whether the percentage of effective voice frames in the past minute is lower than the silence threshold. If any condition is met, the communication status determination unit outputs a Boolean value "true" indicating that the internal communication channel is in an abnormal communication state. The personnel dispersion determination unit reads the latest personnel location data coordinates of all team members, calculates the Euclidean distance between all pairs of members, counts the number of member pairs exceeding the preset safe collaborative distance threshold, and calculates the ratio of this number to the total number of team member pairs. If this ratio is greater than the preset dispersion ratio threshold, the personnel dispersion determination unit outputs a Boolean value "true" indicating that the team is in a dispersed state. The process stagnation determination unit reads the preset work plan process data, obtains the planned duration and start time of the current task step, calculates the elapsed time since the start time, and if the elapsed time exceeds a certain percentage (e.g., 150%) of the planned duration, and there is no operation log for advancing this step during this period, the process stagnation determination unit outputs a Boolean value "true" indicating that the current work process is in a stagnant state. The comprehensive status determination unit is connected to the above four determination units through program function calls or internal message passing, and receives the Boolean status values ​​output by these four units. The pre-defined logic within the integrated status determination unit is as follows: when two or more of the four received Boolean values ​​are "true," the integrated status determination unit determines that the work group has entered a collaborative failure state and generates a collaborative failure determination event. This event is sent as a trigger signal to the dynamic agent determination and authorization module. This combined determination logic, which requires at least two states to be true, significantly improves the accuracy of identifying the complex management situation of "collaborative failure" through multi-dimensional cross-validation, avoiding single-factor misjudgments.

[0150] The dynamic agent determination and authorization module listens for events issued by the collaborative failure judgment module. When an event is triggered, the dynamic agent determination and authorization module first excludes members judged to be in a disabled state from the team information and generates a list of available members. Next, for each member in the list, the dynamic agent determination and authorization module initiates a real-time agent competency index calculation process. This process calls a weighted scoring model, with the formula CSI = w1S_vital + w2S_central + w3S_historical - w4R_env. Here, S_vital (vital sign stability score) is calculated based on the standard deviation of the member's recent heart rate and blood oxygen data; S_central (location centrality score) is obtained by calculating the relative magnitude of the sum of the distances from this member to all other members; S_historical (historical emergency record score) is calculated by querying the database for the number of drills, average scores, and actual handling records; and R_env (local environmental risk coefficient) is obtained by accumulating the exceedances of environmental sensors near the member. The dynamic agent determination and authorization module calculates the CSI value of all available members and selects the member with the highest CSI value as the dynamic agent. Subsequently, the dynamic agent determination and authorization module generates a data structure containing the scope of permissions, validity period, and unique authorization code, and sends it to the smart terminal held by the dynamic agent via the network. The terminal requests the dynamic agent to perform biometric verification such as fingerprint recognition. After the confirmation information is returned, the dynamic agent determination and authorization module records this authorization in the system runtime permission table, completing the granting of temporary command authority. This intelligent selection and secure authorization mechanism ensures that in the most critical moment, the member with the most suitable comprehensive conditions obtains command authority, and the process is non-repudiable.

[0151] The digital surrogate generation module starts after the dynamic agent is determined. The digital surrogate generation module includes a first digital surrogate creation submodule and a second digital surrogate creation submodule. The first digital surrogate creation submodule creates a collection named after the work group and the original responsible person's ID in a MongoDB NoSQL database as a first data container. The first digital surrogate creation submodule queries the instruction log database, retrieves the last instruction issued by the original responsible person before the collaboration failure event with a status of "sent," serializes all fields of this instruction into a JSON document, stores it in the first data container, and generates a static, read-only first digital surrogate. The second digital surrogate creation submodule creates another collection named after the dynamic agent's ID in MongoDB as a second data container. The second digital surrogate creation submodule reads the temporary command permission attributes granted to the dynamic agent from the runtime permission table, combines them with the dynamic agent's identity information into an initialization JSON document, and writes it to the second data container. Simultaneously, the second digital surrogate creation submodule configures a dedicated, authentication-required RESTful API write interface for the `command_inbox` field of this container. The end application of the dynamic proxy can use this API to submit real-time commands as JSON objects to the `command_inbox` array, thereby enabling the second digital avatar to receive and store real-time commands. The digital avatar generation module records the metadata and association relationship between the two digital avatars in the system registry.

[0152] The command view generation and synchronization module periodically reads data from the MongoDB collection created by the digital avatar generation module. It reads stored historical instructions from the collection corresponding to the first digital avatar as a historical instruction stream. It then queries the latest contents of the `command_inbox` array from the collection corresponding to the second digital avatar as a real-time instruction stream. The module initiates an instruction fusion engine that merges the two instruction streams chronologically and applies a core conflict resolution rule: when operation instructions targeting the same entity conflict, the instruction with the later timestamp takes precedence; if timestamps are indistinguishable or identical, the real-time instruction from the second digital avatar (dynamic agent) is prioritized. The engine then generates a conflict-free unified instruction sequence. Simultaneously, the module retrieves the latest team personnel locations, vital sign summaries, and environmental risk point information from the shared data buffer, combining them to form a real-time team status overview. Finally, the command view generation and synchronization module encapsulates the unified instruction sequence, the source identifier of each instruction, a real-time overview of the shift's status, and the current commander's information into a specific JSON format data packet, i.e., the virtual command view. This module proactively pushes this data packet to the smart terminals of every member of the shift in real time via a WebSocket long-lived connection established through the underground industrial ring network. The client application on the terminal parses and renders this view, displaying command instructions and team status in a clear and unified format. This instruction fusion and real-time synchronization based on conflict resolution rules ensures that all members receive an authoritative, consistent, and up-to-date action guide when faced with potentially conflicting historical and real-time instructions, effectively supporting orderly command and coordination in emergency situations.

[0153] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only.

[0154] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for real-time management and control of personnel going down into the mine based on multimodal data fusion, characterized in that, The method includes: Acquire multimodal data of personnel going down the mine, including personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data; Based on the combined state of at least two types of data in the multimodal data, it is determined whether the shift to which the personnel going down the well belong has entered a state of collaborative failure. When it is determined that the collaborative failure state has been entered, a dynamic agent is identified from the team based on the multimodal data, and temporary command authority is granted to the dynamic agent. Generate a first digital avatar corresponding to the original person in charge of the work group, and generate a second digital avatar corresponding to the dynamic agent and having the temporary command authority; Based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, a unified virtual command view is generated and synchronized to the terminals of each member in the team.

2. The method for real-time control of personnel going down into the mine based on multimodal data fusion according to claim 1, characterized in that, The acquisition of multimodal data of personnel going down the mine includes personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data, including: The personnel location data is obtained through location beacons and identification cards worn by personnel; The vital signs data are obtained through vital sign monitoring devices worn by personnel; The environmental monitoring data is acquired by a sensor array deployed in the downhole operating area; The team communication data is obtained by analyzing the communication status and content of the team's internal communication devices. The pre-set work plan process data is retrieved from the database of the management and control system.

3. The method for real-time control of personnel going down into the mine based on multimodal data fusion according to claim 1, characterized in that, The determination of whether the shift to which the personnel going down the mine belong has entered a state of collaborative failure based on the combined state of at least two types of data in the multimodal data includes: Based on the personnel location data and the vital signs data, determine whether the original person in charge of the team is in a state of incapacity; Based on the team communication data, determine whether the team's internal communication channel is in an abnormal communication state; Based on the personnel location data, determine whether members in the work group exceeding a preset proportion are in a dispersed state; Based on the pre-set work plan process data, determine whether the current work process of the team is in a stagnant state; When at least two of the following states—disabled state, abnormal communication state, dispersed location state, and stagnant state—are simultaneously determined to be true, the work group is determined to have entered a collaborative failure state.

4. The method for real-time control of personnel going down into the mine based on multimodal data fusion according to claim 1, characterized in that, When it is determined that the collaborative failure state has been entered, a dynamic agent is identified from within the work group based on the multimodal data, and temporary command authority is granted to the dynamic agent, including: When it is determined that the collaborative failure state has been entered, obtain a list of other available members in the team other than those who are in a disabled state. Based on the available member list, and according to each member's vital sign data, personnel location data, historical emergency record data, and current local environment data, the real-time agent competence index of each member is calculated. Select the member with the highest real-time agent competence index from the list of available members, and determine the selected member as the dynamic agent; An authorization instruction containing the scope and validity period of the authorization is sent to the terminal device worn by the dynamic agent to grant the temporary command authority.

5. The method for real-time control of personnel going down into the mine based on multimodal data fusion according to claim 1, characterized in that, The generation of a first digital avatar corresponding to the original person in charge of the work group, and the generation of a second digital avatar corresponding to the dynamic agent and having the temporary command authority, include: Create a first data container and store the last valid instruction issued by the original person in charge through the control system before entering the collaborative failure state into the first data container to generate the first digital surrogate; A second data container is created, and the attribute information of the temporary command authority is associated with the identity of the dynamic agent to the second data container to generate the second digital avatar, which is configured to receive real-time instructions from the dynamic agent.

6. The method for real-time control of personnel going down into the mine based on multimodal data fusion according to claim 1, characterized in that, The process of generating a unified virtual command view based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, and synchronizing the virtual command view to the terminals of each member within the team, includes: Read the instructions recorded by the first digital double as a historical instruction stream; Read the real-time instructions received by the dynamic agent by the second digital avatar as a real-time instruction stream; By comparing the historical instruction stream with the real-time instruction stream, when an instruction conflict occurs, the real-time instruction stream shall prevail, and a unified instruction sequence shall be generated by merging them. The unified instruction sequence, instruction source identifier, and real-time status overview of the work team are combined and encapsulated to generate the unified virtual command view; The virtual command view is pushed to the terminal devices worn by each member of the team.

7. The method for real-time control of personnel going down into the mine based on multimodal data fusion according to claim 3, characterized in that, The step of determining whether members of the work group exceeding a preset proportion are in a dispersed state based on the personnel location data includes: Based on the personnel location data, calculate the real-time distance between any two members in the work group; Count the number of member pairs whose real-time distance exceeds a preset security collaboration threshold; When the ratio of the number of member pairs exceeding the preset safety collaboration threshold to the total number of member pairs in the team is greater than the preset dispersion ratio threshold, the team is determined to be in a dispersed state.

8. A real-time control system for personnel going down into the mine based on multimodal data fusion, characterized in that, The control system applies the control method according to any one of claims 1-7, and the control system comprises: The multimodal data acquisition module is used to acquire multimodal data of personnel going down the mine. The multimodal data includes personnel positioning data, vital sign data, environmental monitoring data, team communication data, and pre-set work plan process data. The collaborative failure determination module is connected to the multimodal data acquisition module. The collaborative failure determination module is used to determine whether the shift to which the personnel going down the mine belongs has entered a collaborative failure state based on the combination state of at least two types of data in the multimodal data. The dynamic agent determination and authorization module is connected to the collaborative failure determination module. When the collaborative failure state is determined, the dynamic agent determination and authorization module determines a dynamic agent from the team based on the multimodal data and grants temporary command authority to the dynamic agent. A digital surrogate generation module is connected to the dynamic agent determination and authorization module. The digital surrogate generation module is used to generate a first digital surrogate corresponding to the original person in charge of the team, and to generate a second digital surrogate corresponding to the dynamic agent and having the temporary command authority. The command view generation and synchronization module is connected to the digital double generation module. The command view generation and synchronization module is used to generate a unified virtual command view based on the instructions recorded by the first digital double and the real-time instructions received by the second digital double, and synchronize the virtual command view to the terminals of each member in the team.

9. The real-time control system for personnel going down into the mine based on multimodal data fusion according to claim 8, characterized in that, The multimodal data acquisition module includes: The positioning data acquisition unit is used to acquire the personnel positioning data through positioning beacons and identification cards worn by personnel; A vital signs data acquisition unit is used to acquire the vital signs data through a vital signs monitoring device worn by the personnel. An environmental data acquisition unit is used to acquire the environmental monitoring data through a sensor array deployed in the downhole working area; A communication data acquisition unit is used to acquire the team communication data through the communication status and communication content of the team's internal communication equipment. The process data retrieval unit is used to retrieve the preset work plan process data from the database of the management and control system.

10. The real-time control system for personnel going down into the mine based on multimodal data fusion according to claim 8, characterized in that, The collaborative failure determination module includes: The incapacity determination unit for the person in charge is used to determine whether the original person in charge of the team is incapacitated based on the personnel location data and the vital signs data. The communication status determination unit is used to determine whether the internal communication channel of the work group is in an abnormal communication state based on the work group communication data. The personnel dispersion determination unit is used to determine, based on the personnel positioning data, whether members in the work group exceeding a preset proportion are in a dispersed state. The process stagnation determination unit is used to determine whether the current work process of the team is in a stagnant state based on the preset work plan process data. The integrated status determination unit is connected to the responsible person's incompetence determination unit, the communication status determination unit, the personnel dispersion determination unit, and the process stagnation determination unit. The integrated status determination unit is used to determine that the team has entered a collaborative failure state when at least two of the incompetence state, the abnormal communication state, the location dispersion state, and the stagnation state are determined to be true at the same time.

Images