Excavation equipment state prediction method based on digital twinning and multi-modal fusion

Abstract

The invention discloses a mining equipment state prediction method based on digital twinning and multi-modal fusion, and relates to the technical field of intelligent operation and maintenance, and the method comprises the steps: inputting a multi-modal alignment working condition window sequence into a digital twinning mechanism link, carrying out the same-window ideal response deduction, outputting an ideal multi-modal response, and constructing a twinning dynamic residual error; meanwhile, segmenting a residual structure event to generate a residual event token set; positioning a multi-modal data fragment based on the residual event token set, executing cross-modal attention alignment, and performing topology propagation aggregation in combination with a mechanical topology observation mapping table to generate a topology constraint fusion table set; and inputting the topological constraint fusion table set into the graph neural network, carrying out message passing in combination with a mechanical topological graph, outputting a component state and a complete machine state, and packaging the component state and the complete machine state into a mining equipment state prediction set. According to the method, structured analysis of the twinborn dynamic residual error is realized, and the method is used for accurately positioning an abnormal event and improving the sensitivity and timeliness of state prediction.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology, and in particular to a method for predicting the status of mining equipment based on digital twin and multimodal fusion. Background Technology

[0002] Against the backdrop of smart mines and the digital transformation of heavy equipment, condition prediction technology for mining equipment has become a key support for ensuring operational safety and improving maintenance efficiency. In recent years, with the rapid development of industrial IoT, digital twins, and artificial intelligence technologies, equipment condition monitoring has gradually evolved from single-sensor threshold alarms to multi-source information fusion and mechanism-data dual-driven approaches. Existing research largely focuses on fault diagnosis models based on single-mode signals such as vibration, temperature, or current. Some solutions attempt to introduce shallow multi-mode fusion strategies, such as feature splicing or weighted averaging, to enhance condition characterization capabilities. Meanwhile, digital twin technology, by constructing virtual mappings of physical equipment, has shown potential in simulation and anomaly detection.

[0003] Existing technologies have two shortcomings: First, the multimodal data acquisition and fusion process generally ignores the internal mechanical topology of the equipment and its dynamic transmission path, resulting in rough modeling of the correlation between different modes and failing to effectively capture the fault propagation mechanism across components and modes; Second, traditional digital twin models mostly adopt static or quasi-static response mechanisms and lack the ability to structure and analyze the dynamic residuals of "actual observation - ideal deduction", resulting in vague location of abnormal events and delayed prediction. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a method for predicting the state of mining equipment based on digital twin and multimodal fusion to solve the problems of lack of topological constraints in multimodal fusion and the inability to structurally analyze the dynamic residuals of digital twins.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for predicting the state of mining equipment based on digital twin and multimodal fusion. The method includes: binding multimodal acquisition points to topological nodes and edges to construct a mechanical topology graph; registering component coupling and transmission relationships to obtain the digital twin mechanism link; simultaneously performing topological edge weight calibration to generate a mechanical topology observation mapping table; collecting multimodal data according to the mechanical topology observation mapping table; performing sampling rate alignment and clock drift correction according to a unified time calibrator to generate a multimodal aligned operating condition window sequence; inputting the multimodal aligned operating condition window sequence into the digital twin mechanism link; performing ideal response deduction within the window; outputting an ideal multimodal response; constructing a twin dynamic residual; simultaneously segmenting residual structural events to generate a residual event token set; locating multimodal data segments based on the residual event token set; performing cross-modal attention alignment; and combining the mechanical topology observation mapping table to perform topology propagation aggregation to generate a topology constraint fusion representation set; inputting the topology constraint fusion representation set into a graph neural network; combining the mechanical topology graph for message passing; outputting component states and overall machine states; and encapsulating this into a mining equipment state prediction set.

[0007] As a preferred embodiment of the mining equipment status prediction method based on digital twin and multimodal fusion described in this invention, the steps for constructing the mechanical topology map are as follows: Read the list of mining equipment components and the list of multimodal acquisition points, organize them into a set of bindable items, align the binding fields and parse the installation pose, and generate a point binding preparation list; Based on the point binding preparation list, port alignment is performed to form edges, topology nodes and topology edges are established, and data collection point identifiers are bound to generate a mechanical topology map.

[0008] As a preferred embodiment of the mining equipment status prediction method based on digital twin and multimodal fusion described in this invention, the steps for generating the mechanical topology observation mapping table are as follows: The mechanical topology diagram is sorted by edge type and the transmission direction is fixed and arranged to obtain the coupling and transmission relationship of components, and then linked into callable chain entries to generate a digital twin mechanism link. The digital twin mechanism link is converted into edge weight calibration values, and the edge weight calibration is unified and standardized. At the same time, it is merged and linked with the collection point identifier to generate a mechanical topology observation mapping table.

[0009] As a preferred embodiment of the mining equipment status prediction method based on digital twin and multimodal fusion described in this invention, the step of collecting multimodal data according to the mechanical topology observation mapping table is as follows: Based on the mechanical topology observation mapping table, the acquisition points are identified and grouped, and a multimodal acquisition orchestration list is obtained by arranging the sampling rate configuration and the channel clock source description. Based on the multimodal acquisition orchestration list, multimodal data is acquired, and time stamp and frame sequence number fields are encapsulated. At the same time, abnormal frames are removed to generate a set of original multimodal acquisition streams.

[0010] As a preferred embodiment of the mining equipment condition prediction method based on digital twin and multimodal fusion described in this invention, the steps for generating the multimodal aligned working condition window sequence are as follows: Based on the multimodal raw acquisition stream set, a unified time scale sequence is established, and time scale aggregation slicing and time scale interpolation are performed to generate a multimodal time-aligned segment set; The adjacent time marker difference sequence is extracted from the multimodal time-aligned segment set, and clock drift back-off correction is performed. At the same time, the multimodal alignment segment is encapsulated to generate a multimodal alignment condition window sequence.

[0011] As a preferred embodiment of the mining equipment status prediction method based on digital twin and multimodal fusion described in this invention, the steps for performing ideal response deduction are as follows: Extract time window identifiers and unified time scale index ranges from the multimodal aligned working condition window sequence, and combine them with the mechanical topology observation mapping table to map the acquisition point identifiers and generate the same window inference input package; Based on the input package from the peer simulation, the digital twin mechanism link is driven to perform chain-like item scheduling simulation to generate an ideal multimodal response; A correspondence is established between the ideal multimodal response and the multimodal time-aligned fragment set, and then the fragments are clipped and aligned to generate a peer ideal response comparison set.

[0012] As a preferred embodiment of the mining equipment status prediction method based on digital twin and multimodal fusion described in this invention, the steps for generating the residual event token set are as follows: Based on the ideal response comparison set of classmates, multi-form segments of residuals are extracted, twin dynamic residuals are constructed, and residual structure events are segmented along a unified time scale index range to generate a set of residual event entries; The residual event entry set is encapsulated into a token payload field and a topology prior field according to the time window identifier and the topology node identifier, and written into the event sequence index to generate a residual event token set.

[0013] As a preferred embodiment of the mining equipment status prediction method based on digital twin and multimodal fusion described in this invention, the steps for performing cross-modal attention alignment are as follows: Based on the residual event token set, extract the segment playback range and map the acquisition point identifier. At the same time, trim the multimodal alignment segments within the multimodal alignment working condition window sequence to generate an event segment location list. Extract event anchor trajectories from the multimodal aligned fragments in the event fragment location list, and perform cross-modal attention alignment to generate a cross-modal aligned event representation set.

[0014] As a preferred embodiment of the mining equipment state prediction method based on digital twin and multimodal fusion described in this invention, the steps for generating the topological constraint fusion representation set are as follows: The cross-modal aligned event representation set is attached to the mechanical topology observation mapping table according to the topology node identifier and topology edge identifier, and the propagation message is fed back to update, generating the topology propagation aggregate representation set; Organize the node and edge representation entries of the topology propagation aggregation representation set, and add time window identifiers and event sequence indexes to generate a topology constraint fusion representation set.

[0015] As a preferred embodiment of the mining equipment status prediction method based on digital twin and multimodal fusion described in this invention, the encapsulation into a mining equipment status prediction set comprises the following steps: The topological constraint fusion representation set is subjected to graph structure batch orchestration, which is organized into node input sequences and edge input sequences. A topological index table is established, cross-modal aligned weight summaries are merged, and a topological input orchestration package is generated. The topology input orchestration package is input into the graph neural network, aligned with the connection relationships of the mechanical topology graph, and directional message passing is initiated. Simultaneously, intensity modulation and type gating are performed, and the topology inference update representation set is output. Based on the updated representation set of topological reasoning, the component status is aggregated and output according to the mapping relationship between component identifier and topological node identifier, and the overall machine status is aggregated and output according to the topological hierarchy relationship of the mechanical topology graph, generating a mining equipment status prediction set.

[0016] The beneficial effects of this invention are as follows: by constructing a weighted and calibrated mechanical topology observation mapping table, a structured binding of multimodal data and equipment mechanisms is realized, which provides topological constraints for fusion and improves the physical consistency of cross-modal alignment; by generating residual event token sets, a structured parsing of twin dynamic residuals is realized, which accurately locates abnormal events and improves the sensitivity and timeliness of state prediction. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for predicting the state of mining equipment based on digital twins and multimodal fusion.

[0019] Figure 2 This is a schematic diagram illustrating the evolution of twin dynamic residual energy under operating conditions and the triggering of residual event tokens.

[0020] Figure 3 A comparison chart showing how cross-modal event anchor point consistency deviation changes with clock drift.

[0021] Figure 4 This is a schematic diagram comparing the boundary alignment effects of twin dynamic residual events. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a method for predicting the state of mining equipment based on digital twin and multimodal fusion, including the following steps: S1: Bind multimodal acquisition points to topology nodes and topology edges to construct a mechanical topology graph, register component coupling and transmission relationships to obtain digital twin mechanism links, and simultaneously perform topology edge weight calibration to generate a mechanical topology observation mapping table; S1.1: Read the list of mining equipment components and the list of multimodal acquisition points, organize them into a set of bindable items, align the binding fields and parse the installation pose, and generate a point binding preparation list; Furthermore, the process involves reading the list of mining equipment components and merging duplicate entries according to component identifiers, verifying the pairwise integrity of connection port identifiers and marking incomplete entries as those to be completed, reading the list of multimodal acquisition points and merging the acquisition channel configuration field set and installation location description according to acquisition point identifiers, extracting the coordinate reference and orientation reference from the installation location description, and performing installation pose parsing to obtain pose expression entries. The process also involves aligning the binding fields of component identifiers and connection port identifiers in the mining equipment component list with the acquisition point identifiers and installation pose parsing records in the multimodal acquisition point list, establishing a bindable mapping relationship between acquisition point identifiers and candidate component identifiers and candidate connection port identifiers, and registering the alignment status. Finally, the process involves aggregating the bindable mapping relationships, alignment statuses, and pose expression entries, sorting them according to acquisition point identifiers, and generating a point binding preparation list.

[0026] It should be noted that the coordinate reference and orientation reference are extracted and fixed from the mounting surface reference and port pointing reference bound to the component identification in the installation position description. The coordinate reference is determined by the mounting surface reference to determine the origin and axis, and the orientation reference is determined by the port pointing reference to determine the positive direction and rotation direction.

[0027] S1.2: Based on the point binding preparation list, align ports to form edges, establish topology nodes and topology edges, bind the collection point identifiers, and generate a mechanical topology map; Furthermore, candidate component identifiers and candidate connection port identifiers are extracted from the bindable mapping relationship of the point binding preparation list and registered as candidate topology node entries. Duplicates are removed from the candidate component identifiers, and topology node entries are generated and the node index is backfilled. Candidate connection port identifiers are traversed to align execution ports into edges to determine the starting and ending topology node identifiers, and the transmission direction description is registered to generate topology edge entries. For port alignment conflict entries, priority adjudication is performed based on alignment status and pose expression entries, and closable connection relationships are preserved. The point binding preparation list is traversed according to the acquisition point identifiers, and acquisition point identifiers are attached to the corresponding topology node entries or corresponding topology edge entries based on the bindable mapping relationship. Pose expression entries are preserved as point spatial positioning information. Acquisition point identifiers with no matching connection relationships are registered as suspended point entries, and candidate component identifiers are preserved for subsequent completion, generating a mechanical topology map.

[0028] S1.3: Perform edge type sorting and transmission direction solidification and arrangement on the mechanical topology diagram, obtain the component coupling and transmission relationship, and connect them into callable chain entries to generate digital twin mechanism links; Furthermore, the topology edges are traversed along the mechanical topology diagram, and topology edge type entries, starting topology node identifiers, and ending topology node identifiers are extracted. Edge type sorting is performed according to the topology edge type entries to form a sorting edge list. The transfer direction description is solidified by pointing from the starting topology node identifier to the ending topology node identifier, thus completing the solidification and arrangement of the transfer direction. The set of coupling constraint points corresponding to the topology node identifiers and topology edge type entries at both ends of the sorting edge list is organized one by one. Adjacent topology edges are connected into transfer links according to the transfer direction description and merged into component coupling transfer relationships. The chain entry identifier, chain entry input field set, and chain entry output field set are encapsulated one by one for the component coupling transfer relationship, and the callable order mark of the chain entry is registered. The chain entries are arranged into digital twin mechanism links according to the callable order mark.

[0029] It should be noted that the set of coupling constraint key points is a summary of the transmission constraints and boundary constraints that must be kept consistent in the component coupling transmission relationship corresponding to the topology edge type entry. It is used to limit the input-output matching aperture and directional transmission conditions when the transmission link is connected.

[0030] S1.4: Convert the digital twin mechanism link into edge weight calibration values, unify the edge weight calibration standard, and merge and link it with the collection point identifier to generate a mechanical topology observation mapping table. Furthermore, the chain entries of the digital twin mechanism link are traversed, and the topological edge identifiers and transmission strength descriptions associated with the chain entries are extracted. The transmission strength descriptions are converted into edge weight calibration values ​​according to a unified weight scale, and the source tags of the edge weight calibration values ​​are registered. The edge weight calibration values ​​of different edge type sets are unified to eliminate differences in dimensions and magnitudes, and comparable sorting relationships are preserved. The values ​​are then backfilled into the corresponding topological edge identifier entries and, together with the transmission direction description, form a set of topological constraint fields. The acquisition point identifiers and attached topological node identifiers or attached topological edge identifiers are extracted according to the observation anchor point entries of the digital twin mechanism link. The acquisition point identifiers and the set of topological constraint fields are merged and attached, and a traceable mapping key is registered. The topological node identifiers, topological edge identifiers, acquisition point identifiers, and edge weight calibration values ​​are aggregated and sorted according to the topological node identifiers and topological edge identifiers to generate a mechanical topology observation mapping table.

[0031] It should be noted that the edge weight calibration values ​​are obtained by interval mapping of the transfer strength description according to the transfer strength index corresponding to the edge type, and the edge weight calibration is unified by quantile scaling within the same edge type set. The dimensions of the transfer strength index are given by the stiffness index, damping index and energy transfer coefficient index in the set of coupling constraint points and written into the edge weight calibration value source label.

[0032] S2: Based on the mechanical topology observation mapping table, collect multimodal data, and perform sampling rate alignment and clock drift correction according to a unified time caliber to generate a multimodal aligned working condition window sequence; S2.1: Based on the mechanical topology observation mapping table, the acquisition point identification group is performed, and the multimodal acquisition orchestration list is obtained by arranging the sampling rate configuration and the channel clock source description; Furthermore, the acquisition point identifiers are traversed according to the mechanical topology observation mapping table, and the topology node identifiers and topology edge identifiers associated with the acquisition point identifiers are extracted. Acquisition point identifiers are grouped according to the topology node identifiers and topology edge identifiers, and grouping keys are registered so that acquisition point identifiers corresponding to the same topology node identifier or the same topology edge identifier are grouped into the same group. For each group, the sampling rate configuration and channel clock source description are aggregated and consistency checks are performed. For groups with inconsistent sampling rate configurations, sampling rate alignment prompts are registered, and for groups with inconsistent channel clock source descriptions, clock anchoring candidate entries are registered. At the same time, the grouping key, acquisition point identifiers, sampling rate configurations, and channel clock source descriptions are combined into acquisition orchestration entries, and the acquisition orchestration order is determined by sorting according to the transmission direction description of the topology node identifiers and topology edge identifiers. The acquisition orchestration entries and prompt entries are aggregated and organized into an ordered list to generate a multimodal acquisition orchestration list.

[0033] S2.2: Based on the multimodal acquisition orchestration list, acquire multimodal data, encapsulate time stamp and frame sequence number fields, remove abnormal frames, and generate a set of original multimodal acquisition streams; Furthermore, the acquisition channels corresponding to the acquisition point identifiers are sequentially accessed according to the acquisition arrangement order of the multimodal acquisition orchestration list, and acquisition sessions are started. For each acquisition channel, the effective payload field set is pulled according to the sampling rate configuration, and a time stamp is generated by taking the value from the channel clock source description. At the same time, an incremental frame sequence number field is established for each acquisition channel, and the acquisition point identifier, time stamp, frame sequence number field, and effective payload field set are encapsulated into an acquisition entry. Frame format constraint items are checked for acquisition entries and frames with format mismatch are marked. Time stamp continuity is checked for acquisition entries and time jump frames are marked. Frame sequence number continuity is checked for acquisition entries and duplicate frames and gap frames are marked. Frames with format mismatch, time jump frames, duplicate frames, and gap frames are merged into abnormal frames and removed from the acquisition channel output sequence. At the same time, the continuous replayable frame sequence number range is registered to maintain traceable playback capability. The acquisition entry sequences of each acquisition channel are aggregated and archived according to the acquisition point identifier to generate a multimodal raw acquisition stream set.

[0034] It should be noted that the channel clock source description selects a unified time source according to the time source priority (e.g., PTP is preferred, followed by GPS, and then NTP) to generate time stamps. For acquisition channels with reduced sampling rate, low-pass anti-aliasing filtering is performed before resampling. For acquisition channels with increased sampling rate, linear interpolation is performed to supplement points according to a unified time scale sequence.

[0035] S2.3: Based on the multimodal raw acquisition stream set, establish a unified time scale sequence, and perform time scale aggregation slicing and time scale interpolation to generate a multimodal time-aligned segment set; Furthermore, the system iterates through the acquisition point identifiers according to the multimodal original acquisition stream set and extracts the time stamp sequence and frame number field. Based on the integrity of the time stamps and the stability of the continuous replayable frame number interval, a unified time caliber anchoring channel is selected, and the time stamps of the unified time caliber anchoring channel are extracted as the reference scale. A unified time scale sequence covering the acquisition session range is generated according to the reference scale. The acquisition item sequences corresponding to each acquisition point identifier in the multimodal original acquisition stream set are mapped to the unified time scale sequence. For acquisition item sequences with a sampling rate higher than the unified time scale sequence, time scale aggregation slicing is performed according to the scale boundary, and a set of effective payload fields is aggregated for each scale. For acquisition item sequences with a sampling rate lower than the unified time scale sequence, time scale interpolation is performed according to the scale gap, the interpolation source segment boundary is registered, and time stamps and scale indices are uniformly backfilled so that each acquisition point identifier forms a one-to-one corresponding item under the unified time scale sequence. Continuous aligned segments are trimmed according to the acquisition point identifier and scale index range, and the segment boundaries are registered to generate a multimodal time-aligned segment set.

[0036] S2.4: Extract the adjacent time marker difference sequence from the multimodal time-aligned segment set, perform clock drift back-off correction, and encapsulate the multimodal alignment segment to generate a multimodal alignment condition window sequence; Furthermore, the multimodal time-aligned segment set is traversed according to the acquisition point identifier and the time stamp sequence is extracted. The difference between adjacent time stamps is calculated based on the time stamp sequence, and the statistical summary of the difference sequence is registered. The drift backoff amount is generated according to the linear offset trend of the difference sequence, and clock drift backoff correction is performed on the time stamp sequence. The corrected time stamp sequence is re-aligned to a unified time scale sequence and the scale index mapping relationship is updated. At the same time, drift anomaly prompt entries are registered for abnormal jump sections of the difference sequence before and after correction. Time window identifiers and unified time scale index ranges are generated on the unified time scale sequence according to the predetermined working condition window length and sliding step size. The multimodal time-aligned segment set entries covered by each unified time scale index range are trimmed into multimodal aligned segments and merged according to the acquisition point identifier. At the same time, a set of working condition summary fields is added to the multimodal aligned segments and the working condition window coverage range and playback index are registered, so that working condition window entries that can be called in parallel are formed under the same time window identifier. The working condition window entries are arranged in order of time window identifier to generate a multimodal aligned working condition window sequence.

[0037] It should be noted that the multimodal alignment condition window sequence is a sequence in which a unified time scale index range is fixed for each time window identifier on a unified time scale sequence, and the multimodal alignment segments covered by the index range are merged and encapsulated into condition window entries according to the acquisition point identifier. Its function is to provide a playback and alignment calibrator that can be called in parallel for multimodal alignment segments within the same time window, and to serve as a time organization carrier for subsequent same-window inference input packages and event segment location lists.

[0038] Based on the statistical summary of the difference sequence, a robust linear fit is performed on the difference sequence according to the uniform time scale index to obtain the linear offset slope and linear offset intercept and convert them into drift back amount; quantile limit screening is performed on the residual sequence of the difference sequence and the linear fit result and consecutive out-of-bounds residual segments are registered as abnormal jump segments (e.g., three consecutive scales exceeding the 95% quantile limit of the residual).

[0039] The formula for calculating the difference between adjacent time stamps is: ; in, Indicates the first Article and No. Difference between adjacent time markers between bars Represents the first in the time-stamped sequence Each time stamp value can be obtained. Represents the first in the time-stamped sequence Each time stamp value can be obtained. Indicates and The corresponding frame sequence number field value is used to represent the frame interval spanned between two adjacent acquisition entries. Indicates and The corresponding frame sequence number field value is obtained. This represents the frame interval normalization factor. When there are duplicate frame numbers or the frame number has not advanced, it is treated as 1 to avoid division by zero and to keep the difference calculable.

[0040] It should be noted that the operating condition window length (example range: 1s-20s) is directly used as the length of the uniform time scale index range covered by each time window identifier when dividing the uniform time scale sequence according to the time window identifier.

[0041] Sliding step size (example range: 0.1s-5s): When generating a uniform time scale sequence by sliding according to the time window identifier, the index increment of the starting point of the adjacent time window identifier on the uniform time scale sequence is directly used as the sliding step size.

[0042] S3: Input the multimodal aligned working condition window sequence into the digital twin mechanism link, perform the ideal response deduction of the same window, output the ideal multimodal response, construct the twin dynamic residual, and at the same time segment the residual structure event to generate the residual event token set; S3.1: Extract the time window identifier and unified time scale index range from the multimodal aligned working condition window sequence, and combine it with the mechanical topology observation mapping table to map the acquisition point identifiers and generate the same window inference input package; Furthermore, the multimodal aligned working condition window sequence is traversed in order of time window identifier, and the time window identifier and unified time scale index range are extracted. The multimodal aligned segments corresponding to the unified time scale index range are merged according to the acquisition point identifier and the segment playback index is registered. According to the mechanical topology observation mapping table, the acquisition point identifier is mapped and retrieved to obtain the topology node identifier, topology edge identifier, edge weight calibration value and transmission direction description associated with the acquisition point identifier. These are then merged with the time window identifier and unified time scale index range to form a simulation positioning entry. The simulation positioning entries are sorted according to the topology node identifier and topology edge identifier, and a chain entry retrieval key is generated according to the transmission direction description to form the topology location list required for chain entry scheduling. At the same time, the working condition summary field set is organized into a driving field set and encapsulated with the topology location list to generate the same-window simulation input package.

[0043] S3.2: Based on the input package of the peer simulation, drive the digital twin mechanism link to perform chain-style item scheduling simulation to generate an ideal multimodal response; Furthermore, the chain entries of the digital twin mechanism link are retrieved one by one according to the topology location list in the peer simulation input package, and the chain entry scheduling order is determined according to the transmission direction description. The driving field set is matched and organized according to the input fields required by the chain entry to generate the chain entry driving load. The chain entries are executed sequentially according to the chain entry scheduling order. The intermediate response output by the upstream chain entry is backfilled as the input of the downstream chain entry according to the topology node identifier and topology edge identifier, and the time window identifier is kept consistent with the unified time scale index range until the chain entry sequence is fully covered. The ideal response output by each chain entry is mapped to the ideal current response, ideal vibration response and ideal acoustic emission response according to the acquisition point identifier, and clipped to the unified time scale index range. At the same time, the missing link segment is registered as a chain entry gap prompt entry. The ideal response fragments corresponding to each acquisition point identifier are collected and archived according to the time window identifier to generate the ideal multimodal response.

[0044] It should be noted that the ideal multimodal response refers to the set of ideal current response, ideal vibration response, and ideal acoustic emission response segments output by the digital twin mechanism link along the chain item scheduling order under the constraint of the input package of the same window inference. Its function is to serve as the ideal side benchmark in the same window ideal response comparison set, to be used to construct the twin dynamic residual by comparing with the measured segments of the multimodal time-aligned segment set and to trigger the residual structure event segmentation.

[0045] The chain entries of the digital twin mechanism link are labeled according to the chain entry type and subjected to linear transfer processing or table lookup interpolation processing based on the transfer strength description constraint. The linear transfer processing performs gain mapping and frequency band filtering on the input field set of the chain entry to obtain the output field set of the chain entry. The table lookup interpolation processing uses the transfer strength description index mapping table and interpolates the input field set of the chain entry to obtain the output field set of the chain entry. The output field set of the chain entry is then mapped to ideal current response, ideal vibration response and ideal acoustic emission response according to the acquisition point identifier to form an ideal multimodal response.

[0046] S3.3: Establish a correspondence between the ideal multimodal response and the multimodal time-aligned segment set, and perform peer clipping and alignment to generate a peer ideal response comparison set; Furthermore, the ideal multimodal response is traversed according to the time window identifier, and the acquisition point identifier, time window identifier, and unified time scale index range are extracted. For the multimodal time-aligned segment set, the corresponding measured segment is retrieved according to the acquisition point identifier, and the consistency of the unified time scale index range is checked. A one-to-one correspondence is established between the ideal response segments with consistent acquisition point identifiers and matching index ranges and the measured segments, and a reference hooking entry is generated. For the reference hooking entries with index range boundary offsets, window-based trimming and alignment are performed. The intersection of the unified time scale index range is taken to trim the ideal response segment and the measured segment, and the trimming offset summary is recorded. For the reference hooking entries with gaps, gap prompt entries are recorded and the alignable segment segments are retained. The trimmed and aligned ideal response segments and the trimmed and aligned measured segments, together with the acquisition point identifier, time window identifier, and topology location field set, are merged and encapsulated into reference entries and sorted and aggregated according to the time window identifier to generate a window-based ideal response reference set.

[0047] S3.4: Based on the ideal response comparison set of classmates, perform residual multi-form segment extraction, construct twin dynamic residuals, and segment residual structure events along a unified time scale index range to generate a residual event entry set; Furthermore, the ideal response comparison set is traversed according to the time window identifier, and the acquisition point identifier, unified time scale index range, ideal response segment and measured segment are extracted. The ideal response segment and measured segment are aligned scale by scale within the unified time scale index range to generate a difference sequence. At the same time, the difference sequence is bound and registered as a twin dynamic residual with the acquisition point identifier, the topology location field set and the time window identifier. On the twin dynamic residual, multi-form residual segment extraction is performed along the unified time scale index range. The segment boundaries are located and the event start time scale and event end time scale are registered for sudden jump segment, continuous side band enhancement segment, envelope pulse density segment and acoustic emission cluster segment respectively. At the same time, residual intensity summary is extracted for each segment. Event clipping is performed on the twin dynamic residual according to the segment boundary, and the clipped event segments are merged and packaged with the event type label, event boundary, acquisition point identifier, topology location field set and time window identifier into residual event entries. The residual event entries are sorted and aggregated according to the time window identifier and topology node identifier to generate a residual event entry set.

[0048] It should be noted that twin dynamic residuals are defined as differential sequences after the measured segment and the ideal response segment are aligned scale by scale within a unified time scale index range. The triggering condition for residual structure events is limited to the residual energy summary or residual change rate summary of the differential sequence continuously exceeding the quantile limit segment obtained based on the residual statistical summary within the same time window, which is then registered as the event segment boundary.

[0049] S3.5: Encapsulate the residual event entry set into a token payload field and a topology prior field according to the time window identifier and the topology node identifier, and write them into the event sequence index to generate a residual event token set; Furthermore, the residual event entries are grouped and sorted according to the time window identifier and topology node identifier, and the event type label, event boundary, and event fragment of each residual event entry are extracted. The event fragments are compressed and organized with token payloads, and the time structure and amplitude structure are preserved and aggregated into a token payload field. For residual event entries in the same group, the topology node identifier, topology edge identifier, edge weight scaling value, and propagation direction description are extracted and organized into a topology prior field. The token payload field and the topology prior field are merged and encapsulated into a residual event token entry. The residual event token entries are sorted according to the start time scale of the event boundary within the time window identifier, and an event sequence index is generated. The event sequence index is backfilled into the residual event token entries to maintain the consistency of the sequence position across the acquisition point identifiers. The residual event token entries with event sequence indexes are aggregated and output according to the time window identifier to generate a residual event token set.

[0050] It should be noted that, Figure 2 This diagram illustrates the temporal evolution of the twin dynamic residual energy and the triggering positions of residual event tokens under continuous operating window numbers. The horizontal axis represents the operating window number, and the vertical axis represents the twin dynamic residual energy, used to characterize the dynamic deviation between the actual multimodal response and the digital twin mechanism deduction results. As shown in the figure, when the equipment is in a stable operating phase, the twin dynamic residual energy remains in a low and controlled fluctuation range; when the operating state gradually deviates from the mechanism deduction trend, the residual energy continuously rises and forms a peak segment. By marking the residual event token triggering points on the residual energy curve, continuous residual changes can be structured into locatable abnormal event starting points, transforming the anomaly from a "continuous numerical deviation" into a "discrete event token." Compared to methods that only rely on residual amplitude changes for judgment, this mechanism can complete the event starting point marking in advance during the residual rise phase, providing a more sufficient time response window for subsequent state prediction and handling.

[0051] S4: Locate multimodal data segments based on residual event token sets and perform cross-modal attention alignment. At the same time, combine the mechanical topology observation map table to perform topology propagation aggregation and generate a topology constraint fusion representation set. S4.1: Based on the residual event token set, extract the segment playback range and map the acquisition point identifier. At the same time, trim the multimodal alignment segments within the multimodal alignment working condition window sequence to generate an event segment location list. Furthermore, time window identifiers, acquisition point identifiers, event boundaries, and event sequence indexes are extracted one by one according to the residual event token set. The event boundaries are converted into a unified time scale index range and registered as the segment playback range. According to the mechanical topology observation mapping table, the acquisition point identifiers are mapped to topology node identifiers, topology edge identifiers, and channel positioning information. Matching condition window entries are retrieved in the multimodal alignment condition window sequence according to the time window identifier. Within the matching condition window entries, the current alignment segment, vibration alignment segment, acoustic emission alignment segment, and acoustic emission alignment segment are indexed and truncated according to the segment playback range. The truncated segments are bound and encapsulated with the acquisition point identifier, topology node identifier, topology edge identifier, event sequence index, and event boundary and written into the playback range consistency check record to generate an event segment positioning list.

[0052] S4.2: Extract event anchor point trajectories from the multimodal aligned fragments in the event fragment location list, and perform cross-modal attention alignment to generate a cross-modal aligned event representation set; Furthermore, current-aligned segments, vibration-aligned segments, acoustic emission-aligned segments, and acoustic emission-aligned segments are located one by one according to the event segment location list, and the segment playback range and event boundaries are extracted. Within the segment playback range, event anchor point trajectories are extracted around the event boundaries. The event anchor point trajectories include the amplitude change trajectory of the current-aligned segment, the energy rise trajectory of the vibration-aligned segment, the pulse cluster trajectory of the acoustic emission-aligned segment, and the impact envelope trajectory of the acoustic emission-aligned segment. The event anchor point trajectories are aligned and organized into event anchor point trajectory groups according to a unified time scale index range. Time-domain statistical features and frequency-domain statistical features are extracted from the current-aligned segment, vibration-aligned segment, and acoustic emission-aligned segment respectively and converged into event segment representation vectors. Based on the similarity score of the event segment representation vectors, normalization mapping is performed to obtain cross-modal alignment weights. The edge weight calibration values ​​in the topological prior field set and the transmission direction description are superimposed as weight bias constraints. The cross-modal alignment weight summary and event segment representation are encapsulated into an entry set according to the acquisition point identifier, topological node identifier, topological edge identifier, event sequence index, and event type label to generate a cross-modal alignment event representation set.

[0053] It should be noted that the event anchor point trajectory refers to the set of boundary response trajectories extracted from the current-aligned segment, vibration-aligned segment, and acoustic emission-aligned segment around the event boundary within the segment playback range and aligned and organized according to a unified time scale index range. It is used to provide alignment guidance and anchor point reference for cross-modal attention alignment.

[0054] S4.3: Connect the cross-modal aligned event representation set to the mechanical topology observation mapping table according to the topology node identifier and topology edge identifier, and perform propagation message backfeed update to generate the topology propagation aggregate representation set; Furthermore, the topology node identifier, topology edge identifier, cross-modal alignment weight summary, and event fragment representation are extracted from each cross-modal aligned event representation set, and a hooking index is established according to the topology node identifier and topology edge identifier. Based on the hooking index, the event fragment representation is merged and hooked to the corresponding topology node identifier and topology edge identifier entries in the mechanical topology observation map table, and the edge weight scaling value and propagation direction description are hooked simultaneously to form a topology location event representation entry. The topology edge identifier entries in the mechanical topology observation map table are traversed according to the propagation direction description. Adjacent topology location event representations are aggregated based on the topology node identifiers at both ends of the topology edge identifier, and weighted aggregation is performed according to the edge weight scaling value to generate a propagation message. At the same time, the cross-modal alignment weight summary is used to perform gating screening to remove cross-modal inconsistent propagation contributions. The propagation message obtained by the topology edge identifier aggregation is written back to the node representation corresponding to the terminating topology node identifier and the node representation update summary is registered. At the same time, the edge representation summary corresponding to the adjacent topology edge identifier is updated synchronously according to the node representation update summary. Multiple rounds of propagation message backfeeding and updating are repeated according to the time window identifier, and the update records of each round are summarized to generate a topology propagation aggregate representation set.

[0055] It should be noted that the topology propagation aggregation representation set refers to the set of cross-modal aligned event representations that are attached to the mechanical topology observation mapping table according to the topology node identifier and topology edge identifier. Then, based on the propagation direction description and edge weight scaling value, the adjacent topology positions are propagated and aggregated, and the updated set of node representation entries and edge representation entries is written back. This is used to expand the representation of local event fragments under topology constraints into a globally consistent representation input that can be used for subsequent topology constraint fusion representation set organization and graph neural network inference.

[0056] S4.4: Organize the node representation entries and edge representation entries of the topology propagation aggregation representation set, and add time window identifiers and event sequence indexes to generate the topology constraint fusion representation set; Furthermore, the topology propagation aggregation representation set is traversed according to the time window identifier, and the updated topology location event representation corresponding to the topology node identifier and the propagation constraint information corresponding to the topology edge identifier are extracted. Based on the topology node identifier, the updated topology location event representation is organized into node representation entries and the topology node identifier is registered. Based on the topology edge identifier, the propagation constraint information is organized into edge representation entries and the topology edge identifier, edge weight scaling value, and propagation direction description are registered. Time window identifiers are added to the node representation entries and edge representation entries, and an index alignment relationship is established according to the event sequence index of the residual event token set. The event sequence index is written into the node representation entries and edge representation entries to maintain consistency of the retrieval caliber across steps. Entry-level integrity verification is performed on the node representation entries and edge representation entries, and abnormal entries that lack time window identifiers or topology identifiers are removed. The remaining entries are sorted according to the time window identifier and deduplicated and merged according to the topology node identifier and topology edge identifier to generate a topology constraint fusion representation set.

[0057] It should be noted that the propagation constraint information is a summary of the edge weight scaling values ​​and propagation direction descriptions that play a constraining role in the process of propagating message aggregation by topological edge identifiers. It is used to limit the weighted convergence strength and propagation direction of node representation entries between adjacent topological positions.

[0058] Figure 3 This paper presents a comparison of the changes in cross-modal event anchor consistency deviation under different clock drift amplitudes, highlighting key difference intervals through magnified local plots. The horizontal axis represents the clock drift amplitude, and the vertical axis represents the cross-modal event anchor consistency deviation, reflecting the alignment degree between multimodal acquired data and mechanistic event anchors in the time dimension. The overall trend shows that the consistency deviation of all schemes increases with the increase of clock drift amplitude; among them, the scheme combining the mechanical topology observation mapping table and introducing residual event tokens has a lower deviation growth rate and can maintain a relatively stable alignment level within the medium drift range. The magnified local area further illustrates the separation degree of different schemes within this range, and the difference value is marked at the drift amplitude with the largest difference, to visually illustrate the supporting role of residual event tokens in cross-modal anchor alignment. This demonstrates that residual event tokens provide a stable time anchor basis for cross-modal alignment.

[0059] Figure 4 This paper compares and illustrates the localization results of the twin dynamic residual event boundaries under different schemes, demonstrating the role of residual event tokens in determining the anomaly initiation time. The horizontal axis represents the operating condition window number, and the vertical axis represents the twin dynamic residual event score. The rising segment of the event score is used to characterize the boundary initiation point of the anomaly event. The figure marks the boundary initiation positions under the baseline scheme, the scheme with mechanical topology observation mapping table enabled, the scheme with residual event token generated, and the scheme with mapping table and token superposition. The comparison shows that without the introduction of residual event tokens, the boundary initiation point is further back, usually falling after the residual energy peak section; after generating residual event tokens, the boundary initiation point moves forward and maintains a more consistent localization result after superimposing topology constraints. By comparing the boundary initiation advance of different schemes, the effect of this invention on early anomaly event localization can be intuitively presented, thus providing a more sufficient time margin for state prediction.

[0060] S5: Input the topological constraint fusion representation set into the graph neural network, combine it with the mechanical topology graph for message passing, output the component state and the whole machine state, and encapsulate it into a mining equipment state prediction set; S5.1: Perform graph-structured intra-batch orchestration on the topological constraint fusion representation set, organize it into node input sequences and edge input sequences, establish a topological index table, merge cross-modal aligned weight summaries, and generate a topological input orchestration package; Furthermore, the topological constraint fusion representation set is traversed according to the time window identifier, and node representation entries and edge representation entries are extracted. Node representation entries are deduplicated and merged based on the topological node identifier, and sorted by the time window identifier and event sequence index to form the node input sequence. Similarly, edge representation entries are deduplicated and merged based on the topological edge identifier, and sorted by the time window identifier and event sequence index to form the edge input sequence. The corresponding sequence positions in the node input sequence are located based on the start and end topological node identifiers in the edge representation entries, and the association pairs between edge input sequence positions and node input sequence positions are recorded to form a topological index table. The cross-modal alignment weight summary is aligned according to the event sequence index, and the cross-modal alignment weight summary is merged into the corresponding entries of the node input sequence and edge input sequence, with missing item filling and conflict item removal performed to generate the topological input orchestration package.

[0061] S5.2: Input the topology input orchestration package into the graph neural network, align it with the connection relationships of the mechanical topology graph, initiate directional message passing, and simultaneously perform intensity modulation and type gating to output the topology inference update representation set; Furthermore, the topology input orchestration package is input into the graph neural network. Based on the connection relationship of the mechanical topology graph, the topology index table is aligned for consistency and a removal list is formed. For each topology edge identifier, a directional message is initiated from the starting topology node identifier to the ending topology node identifier according to the propagation direction description. The node input sequence representation and the edge input sequence representation are combined into a propagation payload and intensity modulation is performed according to the edge weight scaling value. At the same time, type gating and gating filtering are performed according to the topology edge type entries and cross-modal alignment weight summary. The propagation payload is aggregated according to the ending topology node identifier to update the node representation and the update summary is sent back to update the edge representation. The directional message is repeated according to the predetermined propagation rounds. The propagation rounds are selected based on the number of hops of the topology diameter of the mechanical topology graph, with a minimum coverage round and an upper limit set (e.g., rounds 2 to 5 in Example) to control the inference latency. The topology inference update representation set is output.

[0062] It should be noted that the topology reasoning update representation set refers to the set of updated node representations and updated edge representations obtained by the graph neural network after performing directional message passing, intensity modulation and type gating on the topology input orchestration package under the constraints of the mechanical topology graph. It is used as the direct input for the mapping relationship between component identifiers and topology node identifiers to aggregate the output component state and further aggregate the output whole machine state.

[0063] S5.3: Update the representation set based on topological reasoning, aggregate and output the component status according to the mapping relationship between component identifier and topological node identifier, and aggregate and output the whole machine status according to the topological hierarchy relationship of the mechanical topology diagram, generating a mining equipment status prediction set; Furthermore, the topology inference update representation set is traversed according to the time window identifier, and the updated node representation corresponding to the topology node identifier is extracted. The updated node representations are then grouped into component-level representation entries according to the mapping relationship between component identifiers and topology node identifiers. State decoding is performed on the component-level representation entries to output the component status. The component status is output using a set of health levels (e.g., normal, warning, fault), and health scores can be output in parallel (e.g., Example 0 to Example 1). Simultaneously, a confidence label consistent with the time window identifier is attached to the component status. The confidence label uses the normalized confidence distribution of the health level set or the uncertainty of the health score. Define the degree interval and register the confidence caliber marker; aggregate all component-level characterization items hierarchically according to the topological hierarchy of the mechanical topology diagram to form whole-machine-level characterization items, and perform state decoding on the whole-machine-level characterization items to output the whole-machine status. The whole-machine status is output using the whole-machine health level set and can output the whole-machine health score in parallel. The whole-machine health level is obtained by consistent aggregation of component status according to the topological hierarchy of the mechanical topology diagram; align and organize the component status and whole-machine status according to the time window identifier, and encapsulate the traceable index field set of component identifier, time window identifier and topology node identifier to generate the mining equipment status prediction set.

[0064] It should be noted that the mining equipment status prediction set refers to the result set formed by uniformly encapsulating the component status and the overall machine status, as well as the corresponding confidence labels and traceable index fields according to the time window identifier. It is used to provide a locationable and traceable status output carrier for subsequent operation and maintenance, alarm linkage and status playback.

[0065] In summary, this invention achieves structured binding of multimodal data and equipment mechanisms by constructing a weighted and calibrated mechanical topology observation mapping table, which provides topological constraints for fusion and improves the physical consistency of cross-modal alignment; and by generating residual event token sets, it achieves structured parsing of twin dynamic residuals, which accurately locates abnormal events and improves the sensitivity and timeliness of state prediction.

[0066] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for predicting the condition of mining equipment based on digital twin and multimodal fusion, characterized in that: include, By binding multimodal acquisition points to topological nodes and topological edges, a mechanical topology graph is constructed, and the coupling and transmission relationships of components are registered to obtain the digital twin mechanism link. At the same time, the topological edge weights are calibrated to generate a mechanical topology observation mapping table. Based on the mechanical topology observation mapping table, multimodal data are collected, and sampling rate alignment and clock drift correction are performed according to a unified time caliber to generate a multimodal aligned operating condition window sequence; The multimodal aligned working condition window sequence is input into the digital twin mechanism link to perform the ideal response deduction of the window, output the ideal multimodal response, construct the twin dynamic residual, and at the same time segment the residual structure event to generate the residual event token set; Multimodal data fragments are located based on residual event token sets, and cross-modal attention alignment is performed. At the same time, topology propagation aggregation is performed by combining mechanical topology observation mapping tables to generate a topology constraint fusion representation set. The topological constraint fusion representation set is input into the graph neural network, combined with the mechanical topology graph for message passing, and outputs the component state and the overall machine state, which are then encapsulated into a mining equipment state prediction set.

2. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 1, characterized in that: The steps for constructing the mechanical topology diagram are as follows: Read the list of mining equipment components and the list of multimodal acquisition points, organize them into a set of bindable items, align the binding fields and parse the installation pose, and generate a point binding preparation list; Based on the point binding preparation list, port alignment is performed to form edges, topology nodes and topology edges are established, and data collection point identifiers are bound to generate a mechanical topology map.

3. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 2, characterized in that: The steps for generating the mechanical topology observation mapping table are as follows: The mechanical topology diagram is sorted by edge type and the transmission direction is fixed and arranged to obtain the coupling and transmission relationship of components, and then linked into callable chain entries to generate a digital twin mechanism link. The digital twin mechanism link is converted into edge weight calibration values, and the edge weight calibration is unified and standardized. At the same time, it is merged and linked with the collection point identifier to generate a mechanical topology observation mapping table.

4. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 3, characterized in that: The steps for collecting multimodal data based on the mechanical topology observation mapping table are as follows: Based on the mechanical topology observation mapping table, the acquisition points are identified and grouped, and a multimodal acquisition orchestration list is obtained by arranging the sampling rate configuration and the channel clock source description. Based on the multimodal acquisition orchestration list, multimodal data is acquired, and time stamp and frame sequence number fields are encapsulated. At the same time, abnormal frames are removed to generate a set of original multimodal acquisition streams.

5. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 4, characterized in that: The steps for generating the multimodal aligned working condition window sequence are as follows: Based on the multimodal raw acquisition stream set, a unified time scale sequence is established, and time scale aggregation slicing and time scale interpolation are performed to generate a multimodal time-aligned segment set; The adjacent time marker difference sequence is extracted from the multimodal time-aligned segment set, and clock drift back-off correction is performed. At the same time, the multimodal alignment segment is encapsulated to generate a multimodal alignment condition window sequence.

6. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 5, characterized in that: The steps for conducting the ideal response simulation among classmates are as follows. Extract time window identifiers and unified time scale index ranges from the multimodal aligned working condition window sequence, and combine them with the mechanical topology observation mapping table to map the acquisition point identifiers and generate the same window inference input package; Based on the input package from the peer simulation, the digital twin mechanism link is driven to perform chain-like item scheduling simulation to generate an ideal multimodal response; A correspondence is established between the ideal multimodal response and the multimodal time-aligned fragment set, and then the fragments are clipped and aligned to generate a peer ideal response comparison set.

7. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 6, characterized in that: The steps for generating the residual event token set are as follows: Based on the ideal response comparison set of classmates, multi-form segments of residuals are extracted, twin dynamic residuals are constructed, and residual structure events are segmented along a unified time scale index range to generate a set of residual event entries; The residual event entry set is encapsulated into a token payload field and a topology prior field according to the time window identifier and the topology node identifier, and written into the event sequence index to generate a residual event token set.

8. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 7, characterized in that: The steps for performing cross-modal attention alignment are as follows: Based on the residual event token set, extract the segment playback range and map the acquisition point identifier. At the same time, trim the multimodal alignment segments within the multimodal alignment working condition window sequence to generate an event segment location list. Extract event anchor trajectories from the multimodal aligned fragments in the event fragment location list, and perform cross-modal attention alignment to generate a cross-modal aligned event representation set.

9. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 8, characterized in that: The steps for generating the topological constraint fusion representation set are as follows: The cross-modal aligned event representation set is attached to the mechanical topology observation mapping table according to the topology node identifier and topology edge identifier, and the propagation message is fed back to update, generating the topology propagation aggregate representation set; Organize the node and edge representation entries of the topology propagation aggregation representation set, and add time window identifiers and event sequence indexes to generate a topology constraint fusion representation set.

10. The method for predicting the state of mining equipment based on digital twin and multimodal fusion as described in claim 9, characterized in that: The encapsulation is a mining equipment status prediction set, and the steps are as follows. The topological constraint fusion representation set is subjected to graph structure batch orchestration, which is organized into node input sequences and edge input sequences. A topological index table is established, cross-modal aligned weight summaries are merged, and a topological input orchestration package is generated. The topology input orchestration package is input into the graph neural network, aligned with the connection relationships of the mechanical topology graph, and directional message passing is initiated. Simultaneously, intensity modulation and type gating are performed, and the topology inference update representation set is output. Based on the updated representation set of topological reasoning, the component status is aggregated and output according to the mapping relationship between component identifier and topological node identifier, and the overall machine status is aggregated and output according to the topological hierarchy relationship of the mechanical topology graph, generating a mining equipment status prediction set.

Images