An artificial intelligence driven multi-modal data adaptive processing system

Abstract

The application discloses an artificial intelligence driven multi-modal data adaptive processing system. A space-time phase space mapping module receives a multi-modal asynchronous data stream and reconstructs into a high-dimensional phase space trajectory vector; an adaptive space-time coding module performs nonlinear mapping to output a hidden layer feature vector; a transient mutual information calculation module calculates the mutual information between different hidden layer feature vectors; a dynamic hypergraph construction unit compares a dynamic coupling threshold to generate a hypergraph topological correlation matrix; an adaptive routing scheduler distributes hidden layer feature vectors within the same hyperedge to a high-order joint network and distributes isolated hidden layer feature vectors to a single modal processing sub-network; and a closed loop parameter adjustment module extracts an overall comprehensive loss and a reverse gradient flow to calculate an update amount, and overwrite the current delay time parameter and embedding dimension parameter. The application realizes closed loop control of data distribution path and reconstruction parameters.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to an artificial intelligence-driven multimodal data adaptive processing system based on deep learning neural networks, belonging to the field of deep learning technology. Background Technology

[0002] With the development of sensor technology, in scenarios such as industrial automation monitoring, equipment error prevention and early warning, and collaborative control, systems often need to process data from multiple heterogeneous information sources simultaneously. Multimodal data processing technology aims to extract features and jointly analyze different types of asynchronously acquired data to obtain a more comprehensive understanding of the system's operating status.

[0003] In existing technologies, multimodal data processing is typically divided into two fixed stages: independent data preprocessing and deep neural network model inference. In the preprocessing stage, the system mainly performs forced temporal interpolation or resampling on heterogeneous data of different modalities to unify all data to the same sampling frequency, so as to meet the fixed dimension requirements of deep learning neural networks for input tensors. In the inference stage, the system adopts a pre-set multi-layer deep neural network structure to directly concatenate the aligned multimodal features in the channel dimension, and then performs forward propagation through a fixed computation graph to output the joint processing result.

[0004] However, existing technologies suffer from a disconnect between forced temporal alignment at the front end and fixed network topology fusion at the back end when processing multimodal asynchronous data streams. Due to the differences in the underlying physical mechanisms of different sensors, forced interpolation alignment of the front-end time axis directly smooths out the high-frequency nonlinear dynamic features in the original signal and introduces redundant noise, resulting in distorted input data. At the same time, the data reconstruction process at the front end is separated from the model computation at the back end, and the deep learning network cannot adjust the feature mapping parameters of the front end in reverse according to the actual physical correlation between the multimodalities. This causes the system to maintain a fixed network path and perform forced data splicing when facing weakly coupled or unrelated modalities, triggering a negative transfer effect in feature computation and ultimately reducing the reliability of the system output results. Summary of the Invention

[0005] To address the problems mentioned in the background, this invention provides an artificial intelligence-driven multimodal data adaptive processing system, comprising: a spatiotemporal phase space mapping module, an adaptive spatiotemporal coding module, a transient mutual information calculation module, a dynamic hypergraph construction unit, an adaptive routing scheduler, a deep learning processing array, and a closed-loop parameter adjustment module. The spatiotemporal phase space mapping module receives multimodal asynchronous data streams collected by front-end sensors of the industrial equipment monitoring system. This data stream contains heterogeneous sensor signals characterizing the physical state of the industrial equipment. It extracts the time series of each mode from the multimodal asynchronous data stream and reconstructs the time series of each mode into high-dimensional phase space trajectory vectors along the time axis based on set initial delay time parameters and initial embedding dimension parameters. The adaptive spatiotemporal coding module performs nonlinear mapping calculations based on a recurrent neural network structure on the high-dimensional phase space trajectory vectors, outputting fixed-dimensional hidden layer feature vectors to align the tensor dimensions of each modality's data before inputting it into the deep learning processing array, thereby characterizing the spatial topological features of different physical sensor signals. The transient mutual information calculation module calculates the mutual information between different hidden layer feature vectors within a set time window to measure the underlying physical coupling relationship between different industrial sensing modalities. The dynamic hypergraph construction unit compares the mutual information with a set dynamic coupling threshold. When the mutual information corresponding to a set of hidden layer feature vectors is greater than the dynamic coupling threshold, it generates a hyperedge containing the corresponding hidden layer feature vector node. Based on the relationship between the node and the hyperedge... The connection relationship outputs a hypergraph topology correlation matrix; the deep learning processing array contains a single-modal processing sub-network and a high-order joint network, which are used to perform forward propagation calculations and output multimodal processing results for warning signals or equipment collaborative control commands indicating abnormal states of industrial equipment; the adaptive routing scheduler is used to identify the network node connection status of each hidden layer feature vector in the hypergraph topology correlation matrix, uniformly distribute hidden layer feature vectors inside the same hyperedge to the high-order joint network in the deep learning processing array, and distribute isolated hidden layer feature vectors to the corresponding single-modal processing sub-network in the deep learning processing array; the closed-loop parameter adjustment module is used to extract the target task processing result containing the warning signal or collaborative control command, calculate the overall comprehensive loss based on the target task processing result and the real label data, and extract the back gradient flow generated by the high-order joint network, calculate the update amount of the delay time parameter and the update amount of the embedding dimension parameter according to the overall comprehensive loss and the back gradient flow; the spatiotemporal phase space mapping module is also used to overwrite the current delay time parameter and embedding dimension parameter with the update amount, and enter the data phase space reconstruction of the next time window based on the updated parameters.

[0006] Preferably, the spatiotemporal phase space mapping module reconstructs the time series of each mode into a high-dimensional phase space trajectory vector along the time axis as follows: A time axis unfolding operation is performed using the delay embedding theorem; the current time series observation value is concatenated with multiple historical observation values ​​shifted backward by an integer multiple of the delay time parameter. The number of concatenated historical observation values ​​is determined by the embedding dimension parameter, thereby generating a high-dimensional phase space trajectory vector; based on the delay time parameter and the embedding dimension parameter, the state space of each mode's time series is reconstructed according to a set sampling step size, maintaining the original sampling rate of each mode's data during the reconstruction process, and extracting the nonlinear dynamic features in the time series.

[0007] Preferably, the specific configuration for the adaptive spatiotemporal coding module to perform nonlinear mapping calculations based on a recurrent neural network structure on high-dimensional phase space trajectory vectors is as follows: The system-preset hidden layer dimension constant is read as the target dimension of the output features; the high-dimensional phase space trajectory vector is decomposed into discrete sequence elements according to the chronological order, an initial hidden state vector is constructed, and all values ​​are initialized to zero; the recurrent nonlinear mapping calculation is performed step by step along the feature sequence direction. The calculation logic for a single-step mapping is as follows: the product of the input layer weight matrix and the current sequence element is calculated, plus the product of the hidden layer state weight matrix and the previous step's long hidden state vector, and the bias parameter is superimposed. Finally, the current hidden state vector is obtained through processing with a nonlinear activation function; after completing the calculation of the last sequence element, the corresponding recurrent neural network hidden state vector is extracted as the final hidden layer feature vector.

[0008] Preferably, the specific configuration for the transient mutual information calculation module to calculate the mutual information between different hidden layer feature vectors is as follows: within the time range of the current sliding time window, extract time-series data segments corresponding to each hidden layer feature vector; establish a multi-dimensional state space for the extracted time-series data segments; perform probability distribution statistics on the time-series data segments in the multi-dimensional state space based on the kernel density estimation method; calculate the joint probability distribution corresponding to the extracted time-series data segments of the two hidden layer feature vectors, and calculate the marginal probability distributions independently; the calculation logic of mutual information is as follows: calculate the logarithmic ratio between the product of the joint probability distribution and the two marginal probability distributions, and sum the product of the logarithmic ratio and the joint probability distribution over all discrete values ​​to obtain the mutual information.

[0009] Preferably, the specific configuration for generating hyperedges by the dynamic hypergraph construction unit is as follows: Each independent modality in the multimodal data system is mapped to a graph structure node; all graph structure nodes are combined to establish a global node set, and an initially empty hyperedge set is constructed; the global node set is traversed, and all nodes containing at least two graph structure nodes are extracted and combined to form multiple candidate node subsets; the transient mutual information between any two graph structure nodes within a candidate node subset is extracted and compared numerically with a dynamic coupling threshold; when the transient mutual information between any two graph structure nodes within a candidate node subset is greater than the dynamic coupling threshold, a hyperedge is generated, all graph structure nodes within the candidate node subset are assigned to the same hyperedge, and the generated hyperedge is added to the hyperedge set; if there is a transient mutual information less than or equal to the dynamic coupling threshold within a candidate node subset, no hyperedge is generated for the candidate node subset.

[0010] Preferably, the specific configuration for the adaptive routing scheduler to distribute hidden layer feature vectors is as follows: For multiple hidden layer feature vectors belonging to the same hyperedge, a joint routing distribution instruction is generated, and the multiple hidden layer feature vectors are synchronously imported into the higher-order joint network according to the joint routing distribution instruction; the higher-order joint network receives the multiple hidden layer feature vectors and performs a concatenation operation on the tensor dimension, and performs multimodal feature fusion calculation based on the concatenated tensor; For isolated hidden layer feature vectors not belonging to any hyperedge, an independent routing distribution instruction is generated, and the isolated hidden layer feature vectors are imported into the corresponding single-modal processing sub-network according to the independent routing distribution instruction, and the tensor computation graph resources of the higher-order joint network are not allocated to the isolated hidden layer feature vectors according to the independent routing distribution instruction, so as to block the data path of the isolated hidden layer feature vectors input into the higher-order joint network.

[0011] Preferably, after identifying the network node connection status, the adaptive routing scheduler is further configured to: compare the hypergraph topology association matrix of the current time period with the historical hypergraph topology association matrix of the previous time period to identify the differences in node connection status between the hidden layer feature vectors in the current time period and the previous time period; when the hidden layer feature vector changes from an internal node of a hyperedge to an isolated node, generate a transient inheritance instruction for network weights; extract the network weight tensor of the higher-order joint network in the previous time period, and locate the corresponding local network channel inside the higher-order joint network according to the tensor index corresponding to the hidden layer feature vector that has changed to an isolated node; extract the activation state weight data corresponding to the local network channel from the network weight tensor, read the preset dimension projection matrix, multiply the activation state weight data with the dimension projection matrix to complete the affine transformation operation, and output the initial state weights with the same spatial dimension as the input of the single-modality processing sub-network; write the initial state weights into the corresponding single-modality processing sub-network, overwrite the original historical static weights with the initial state weights, and receive the isolated hidden layer feature vector based on the initial state weights.

[0012] Preferably, the specific configuration for the closed-loop parameter adjustment module to extract the overall comprehensive loss is as follows: compare the target task processing results with the system's preset real label data, perform error measurement through cross-entropy calculation or mean square error calculation, and output the main task network loss; extract the high-dimensional hidden layer feature matrix generated by the high-order joint network, and analyze and separate the high-dimensional hidden layer feature vectors corresponding to different independent modes; calculate the Euclidean distance between any two high-dimensional hidden layer feature vectors in the high-dimensional hidden space output by the high-order joint network; read the node connection states recorded inside the hypergraph topology association matrix, and extract the topology connection weight values ​​of the corresponding mode nodes; construct a first empirical distribution based on the Euclidean distance values ​​of all node pairs, construct a second empirical distribution based on the topology connection weight values, and calculate the distribution divergence between the first empirical distribution and the second empirical distribution using relative entropy; traverse all mode node pair combinations, sum and aggregate the distribution divergences of all node pairs to generate the topology maintenance regularization loss; perform a multiplication operation on the topology maintenance regularization loss and the balance coefficient to obtain a regularization penalty term, and sum the regularization penalty term with the main task network loss to output the overall comprehensive loss.

[0013] Preferably, the specific configuration for the closed-loop parameter adjustment module to calculate the update amounts of the delay time parameter and the embedded dimension parameter is as follows: extract the weight gradient flow generated during the backpropagation stage from the computation graph interface of the high-order joint network; identify the delay time parameter and the embedded dimension parameter as non-differentiable discrete variables, call the preset pass-through estimator logic, and merge the overall loss and the weight gradient flow as the input variables of the pass-through estimator logic; execute the pass-through estimator logic, skip the non-differentiable breakpoints of the discrete variables during backpropagation calculation, copy and map the continuous spatial gradient of the high-order joint network to discrete spatial gradient; and output the adjustment step values ​​of the delay time parameter and the embedded dimension parameter corresponding to each independent mode.

[0014] Preferably, the specific configuration for the spatiotemporal phase space mapping module to overwrite the current parameters is as follows: The currently recorded delay time parameter and the delay time parameter adjustment step value are algebraically added together, and the result is rounded down to obtain the updated delay time parameter; the currently recorded embedding dimension parameter and the embedding dimension parameter adjustment step value are algebraically added together, and the result is rounded down to obtain the updated embedding dimension parameter; the updated delay time parameter and the updated embedding dimension parameter are written to the configuration register, and the updated parameters are read to trigger the multimodal asynchronous time series high-dimensional phase space feature reconstruction step for the next time period.

[0015] The present invention, by adopting the above technical solution, can bring the following beneficial effects:

[0016] 1. This invention reconstructs the time series of each modality in a multimodal asynchronous data stream into a high-dimensional phase space trajectory vector by utilizing the delayed embedding theorem. The vector is then used to perform nonlinear mapping calculations based on a recurrent neural network structure via an adaptive spatiotemporal coding module to output a hidden layer feature vector of fixed dimension. This achieves the extraction of nonlinear dynamic features of the sequence while maintaining the original sampling frequency, avoiding the high-frequency signal smoothing problem and redundant noise caused by traditional forced interpolation alignment. At the same time, it effectively avoids tensor dimension conflicts when inputting into the deep learning processing array during the physical reconstruction stage.

[0017] 2. This invention calculates the transient mutual information between hidden layer feature vectors and constructs a hyperedge structure containing modal nodes by combining dynamic thresholds. The scheduler distributes the feature vectors within the hyperedge to a higher-order joint network for fusion calculation and distributes isolated feature vectors to a single-modal processing subnetwork. This achieves data flow distribution based on physical correlation and directly cuts off the data path of weakly correlated modalities before inputting into the neural network. It significantly suppresses the negative migration effect caused by the forced splicing of irrelevant data at the network topology level, while avoiding the ineffective use of hardware computing power.

[0018] 3. This invention extracts the overall integrated loss and backward gradient flow of the high-order joint network, calls the pass-through estimator logic to calculate parameters and adjust the step size by bypassing the non-differentiable breakpoints of discrete variables, and reverses the update quantity to overwrite the front-end delay time and embedding dimension parameters. This achieves closed-loop control of the back-end computation state on the front-end physical mapping process, effectively alleviating the problem of the separation between data preprocessing and network model operation. It enables deep learning networks to actively reshape the phase space observation dimension of multimodal data and establishes an adaptive feedback architecture with complete mathematical connectivity. Attached Figure Description

[0019] Figure 1 This is an architectural block diagram of the multimodal data adaptive processing system based on dynamic hypergraph routing according to the present invention;

[0020] Figure 2 This is a flowchart of the multimodal data adaptive processing method of the present invention;

[0021] Figure 3 This is a flowchart of the front-end data reconstruction and spatiotemporal coding process of the present invention;

[0022] Figure 4 This is a flowchart of the dynamic hypergraph routing scheduling and weight inheritance process of this invention;

[0023] Figure 5 This is a flowchart of the physical layer closed-loop feedback and discrete parameter reshaping process of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Example 1

[0026] See attached document Figure 1 , Figure 1 This is an architectural block diagram of an AI-driven multimodal data adaptive processing system according to an embodiment of the present invention. The present invention provides a multimodal data adaptive processing system based on dynamic hypergraph routing, which may include:

[0027] The spatiotemporal phase space mapping module 100 is used to receive multimodal asynchronous data streams collected by front-end sensors of industrial equipment monitoring systems, and to map the one-dimensional time series of each mode in the multimodal asynchronous data stream into a high-dimensional phase space trajectory vector according to the set delay time and embedding dimension.

[0028] The adaptive spatiotemporal coding module 200 is connected to the spatiotemporal phase space mapping module 100. It is used to receive the high-dimensional phase space trajectory vector output by the spatiotemporal phase space mapping module 100, unify the dimension of the high-dimensional phase space trajectory vector through nonlinear mapping operation, and output a hidden layer feature vector of fixed dimension.

[0029] The transient mutual information calculation module 300, connected to the adaptive spatiotemporal coding module 200, is used to calculate the mutual information between hidden layer feature vectors within a sliding continuous time window and generate a mutual information matrix.

[0030] The dynamic hypergraph construction unit 400 is connected to the transient mutual information calculation module 300. It is used to read the mutual information matrix, determine the node set and hyperedge set according to the set dynamic coupling threshold, and generate the hypergraph topological association matrix.

[0031] The adaptive route scheduler 500, connected to the dynamic hypergraph construction unit 400, is used to parse the hypergraph topology association matrix and generate routing control instructions based on the node information contained in the hyperedges to allocate data distribution paths for hidden layer feature vectors.

[0032] The deep learning processing array 600 is connected to the adaptive routing scheduler 500. It contains several single-modal processing sub-networks 610 and a high-order joint network 620. The single-modal processing sub-networks 610 are used to process independent modal features, and the high-order joint network 620 is used to perform high-dimensional fusion and inference of multimodal features.

[0033] The closed-loop parameter adjustment module 700 is connected between the deep learning processing array 600 and the spatiotemporal phase space mapping module 100. It is used to extract the comprehensive loss and weight gradient of the deep learning processing array 600, calculate the discrete adjustment amount of the mapping parameters, and transmit the discrete adjustment amount to the spatiotemporal phase space mapping module 100.

[0034] Example 2

[0035] See attached document Figure 2 , Figure 2 This is a flowchart of an AI-driven adaptive multimodal data processing method according to an embodiment of the present invention. Based on the system architecture of Embodiment 1, the present invention provides an AI-driven adaptive multimodal data processing method, which includes the following steps:

[0036] Step S100: The spatiotemporal phase space mapping module 100 receives the multimodal asynchronous data stream collected by the front-end sensor of the industrial equipment monitoring system. The spatiotemporal phase space mapping module 100 extracts the time series of each mode in the multimodal asynchronous data stream. According to the set initial delay time parameter and initial embedding dimension parameter, the spatiotemporal phase space mapping module 100 reconstructs the time series of each mode into a high-dimensional phase space trajectory vector along the time axis.

[0037] Step S200: The adaptive spatiotemporal coding module 200 obtains the high-dimensional phase space trajectory vector output by the spatiotemporal phase space mapping module 100. The adaptive spatiotemporal coding module 200 performs nonlinear mapping calculation based on the recurrent neural network structure on the high-dimensional phase space trajectory vector and outputs a hidden layer feature vector of fixed dimension to complete the tensor dimension alignment of each modality data in front of the input deep learning processing array 600.

[0038] Step S300: The transient mutual information calculation module 300 receives the hidden layer feature vectors output by the adaptive spatiotemporal coding module 200, and the transient mutual information calculation module 300 calculates the mutual information between different hidden layer feature vectors within a set time window.

[0039] Step S400: The dynamic hypergraph construction unit 400 obtains the mutual information output by the transient mutual information calculation module 300 and compares the mutual information with the set dynamic coupling threshold. When the mutual information corresponding to a set of hidden layer feature vectors is greater than the dynamic coupling threshold, the dynamic hypergraph construction unit 400 generates a hyperedge containing the corresponding hidden layer feature vector node. The dynamic hypergraph construction unit 400 outputs the hypergraph topology association matrix according to the connection relationship between the node and the hyperedge.

[0040] In step S500, the adaptive routing scheduler 500 reads the hypergraph topology association matrix output by the dynamic hypergraph construction unit 400. The adaptive routing scheduler 500 identifies the network node connection status of each hidden layer feature vector in the hypergraph topology association matrix. For hidden layer feature vectors belonging to the same hyperedge, the adaptive routing scheduler 500 distributes the hidden layer feature vectors belonging to the same hyperedge to the high-order joint network 620 in the deep learning processing array 600. For isolated hidden layer feature vectors without hyperedge connections, the adaptive routing scheduler 500 distributes the isolated hidden layer feature vectors to the corresponding single-modality processing sub-network 610 in the deep learning processing array 600.

[0041] In step S600, the deep learning processing array 600 performs forward propagation calculations and outputs multimodal processing results for warning signals or equipment collaborative control commands used to indicate abnormal states of industrial equipment. The closed-loop parameter adjustment module 700 extracts the target task processing results output by the deep learning processing array 600, calculates the overall comprehensive loss based on the target task processing results and the system's preset real label data, and extracts the back gradient flow generated by the high-order joint network 620. The closed-loop parameter adjustment module 700 calculates the update amount of the delay time parameter and the update amount of the embedded dimension parameter based on the overall comprehensive loss and the back gradient flow. The closed-loop parameter adjustment module 700 sends the parameter update amount to the spatiotemporal phase space mapping module 100. The spatiotemporal phase space mapping module 100 uses the parameter update amount to overwrite the current delay time parameter and embedded dimension parameter, and enters the data phase space reconstruction of the next time window based on the updated parameters.

[0042] Example 3

[0043] Based on the main process of Embodiment 2, in order to further clarify the specific execution details of steps S100 to S300, this embodiment combines the appendix. Figure 3 To be continued Figure 4 The paper provides a detailed explanation of the front-end data reconstruction, adaptive spatiotemporal coding, and transient mutual information calculation process.

[0044] In particular, the spatiotemporal phase space mapping process in step S100 of Embodiment 2 includes the following steps in the front-end data reconstruction stage:

[0045] Step S101: The spatiotemporal phase space mapping module 100 receives the multimodal asynchronous data stream collected by the front-end sensor of the industrial equipment monitoring system. The spatiotemporal phase space mapping module 100 extracts the corresponding time series for each independent mode in the multimodal asynchronous data stream.

[0046] Step S102: During the system initialization phase, the spatiotemporal phase space mapping module 100 reads the set initial delay time parameter and initial embedding dimension parameter. During the non-system initialization phase, the spatiotemporal phase space mapping module 100 reads the delay time parameter and embedding dimension parameter transmitted in reverse by the closed-loop parameter adjustment module 700. The spatiotemporal phase space mapping module 100 uses the delay embedding theorem to perform time axis expansion operation on the time series of each mode.

[0047] Step S103: The spatiotemporal phase space mapping module 100 reconstructs the time series into a high-dimensional phase space trajectory vector. For the time series of the i-th mode, the high-dimensional phase space mapping calculation logic of the spatiotemporal phase space mapping module 100 is expressed by the following formula:

[0048]

[0049] In the formula, Indicates the first The modality at time... The generated high-dimensional phase space trajectory vector; Indicates the first The modality at time... Time series observations; Indicates the first The current delay time parameter for each mode; Indicates the first The current embedding dimension parameter of each modality determines the length of the trajectory vector in the high-dimensional phase space.

[0050] Step S104: The spatiotemporal phase space mapping module 100 reconstructs the state space of the time series of each mode according to the delay time parameter and the embedding dimension parameter and the set sampling step size. During the reconstruction process, the spatiotemporal phase space mapping module 100 maintains the original sampling rate of each mode data and extracts the nonlinear dynamic features in the time series.

[0051] Step S105: After completing the mapping calculation, the spatiotemporal phase space mapping module 100 outputs the high-dimensional phase space trajectory vectors corresponding to each mode. The spatiotemporal phase space mapping module 100 transmits the high-dimensional phase space trajectory vectors to the adaptive spatiotemporal coding module 200. The adaptive spatiotemporal coding module 200 receives the high-dimensional phase space trajectory vectors and uses them as the input basis data for subsequent unified operation of feature dimensions.

[0052] The adaptive spatiotemporal coding process in step S200 of Example 2 specifically includes the following steps in the feature dimension unification stage:

[0053] Step S201: The adaptive spatiotemporal coding module 200 receives the high-dimensional phase space trajectory vector transmitted by the spatiotemporal phase space mapping module 100. The dimension length of the high-dimensional phase space trajectory vector changes with the adjustment of the embedded dimension parameter.

[0054] Step S202: The adaptive spatiotemporal coding module 200 reads the hidden layer dimension constant preset by the system. The adaptive spatiotemporal coding module 200 uses the hidden layer dimension constant as the target dimension of the output feature and performs a dimension alignment operation on the high-dimensional phase space trajectory vector to generate a hidden layer feature vector with a fixed dimension value.

[0055] Step S203: The adaptive spatiotemporal coding module 200 decomposes the high-dimensional phase space trajectory vector into discrete sequence elements according to the time sequence. The adaptive spatiotemporal coding module 200 constructs an initial hidden state vector and initializes all values ​​of the initial hidden state vector to zero. The adaptive spatiotemporal coding module 200 performs nonlinear mapping calculations based on a recurrent neural network structure step by step along the feature sequence direction. The logic of the single-step recurrent mapping calculation is expressed by the following formula:

[0056]

[0057] In the formula, Indicates the first The mode in the th ... Hidden state vectors with a mapping step size; Represents a nonlinear activation function; Represents the input layer weight matrix; The first element in the trajectory vector of the high-dimensional phase space represents the first element. 1 sequence element, and The maximum value is equal to the current embedding dimension parameter; This represents the hidden layer state weight matrix; This represents the hidden state vector of the previous step length; Indicates the bias parameter;

[0058] Step S204: The adaptive spatiotemporal coding module 200 traverses all sequence elements inside the high-dimensional phase space trajectory vector. After completing the calculation of the last sequence element, the adaptive spatiotemporal coding module 200 extracts the hidden state vector of the recurrent neural network corresponding to the last sequence element as the final hidden layer feature vector. The dimension of the hidden layer feature vector is equal to the matrix dimension set by the hidden layer state weight matrix.

[0059] Step S205: After the adaptive spatiotemporal coding module 200 completes the mapping calculation, it outputs the hidden layer feature vectors corresponding to each modality. The adaptive spatiotemporal coding module 200 transmits each hidden layer feature vector to the transient mutual information calculation module 300. The transient mutual information calculation module 300 receives the hidden layer feature vectors and uses them as the basic input for subsequent calculation of the data coupling metric between multiple modalities.

[0060] The transient mutual information calculation process in step S300 of Example 2 specifically includes the following steps in the transient physical correlation measurement stage within a continuous time window:

[0061] Step S301: The transient mutual information calculation module 300 receives the hidden layer feature vectors output by the adaptive spatiotemporal coding module 200. The transient mutual information calculation module 300 establishes a sliding time window on the system time axis and reads the set time step and window width values.

[0062] Step S302: Within the time range of the current sliding time window, the transient mutual information calculation module 300 extracts the time-series data segments corresponding to the hidden layer feature vectors. The transient mutual information calculation module 300 establishes a multi-dimensional state space for the extracted time-series data segments. Based on the kernel density estimation method, the transient mutual information calculation module 300 performs probability distribution statistics on the time-series data segments in the multi-dimensional state space.

[0063] Step S303: The transient mutual information calculation module 300 extracts time-series data segments of any two hidden layer feature vectors, calculates the joint probability distribution corresponding to the extracted time-series data segments of the two hidden layer feature vectors, and independently calculates the marginal probability distribution corresponding to the extracted time-series data segments of the two hidden layer feature vectors.

[0064] Step S304: The transient mutual information calculation module 300 calculates the mutual information using the joint probability distribution and the marginal probability distribution. For the hidden layer feature vector corresponding to the i-th mode and the hidden layer feature vector corresponding to the j-th mode, the mutual information calculation logic of the transient mutual information calculation module 300 is expressed by the following formula:

[0065]

[0066] In the formula, Indicates the first The hidden layer feature vector corresponding to the mode and the th mode The mutual information between the hidden layer feature vectors corresponding to each modality; Indicates the first Temporal data segments of hidden layer feature vectors corresponding to each modality; Indicates the first Temporal data segments of hidden layer feature vectors corresponding to each modality; Represents time-series data segments Discrete values ​​in; Represents time-series data segments Discrete values ​​in; Represents the joint probability distribution; and These represent the corresponding marginal probability distributions. The transient mutual information calculation module 300 traverses all hidden layer feature vector pairings and performs the above probability density and mutual information calculation steps to obtain the transient mutual information between all combinations.

[0067] Step S305: The transient mutual information calculation module 300 maps all transient mutual information quantities acquired within the time window into matrix form according to the modal correspondence relationship, generates a mutual information matrix, and transmits the mutual information matrix to the dynamic hypergraph construction unit 400. The dynamic hypergraph construction unit 400 receives the mutual information matrix and uses it as the numerical basis for generating the subsequent topological association structure. The transient mutual information calculation module 300 slides the time window forward according to the time step and triggers the mutual information quantity calculation of the next time period.

[0068] Through the aforementioned front-end data reconstruction, adaptive spatiotemporal coding, and transient mutual information calculation, a mutual information matrix characterizing the dynamic coupling relationship between multiple modalities is obtained, providing a numerical basis for subsequent hypergraph topology construction and adaptive routing scheduling. The following embodiments further elaborate on the specific execution details of steps S400 to S600 and closed-loop feedback.

[0069] Example 4

[0070] Based on the main process of Embodiment 2, in order to further clarify the specific execution details of steps S400 to S600 and the closed-loop feedback, this embodiment combines the appendix. Figure 4 To be continued Figure 5 This paper provides a detailed explanation of the construction of the hypergraph structure with dynamic coupling threshold, adaptive route scheduling, inheritance of route switching weights, calculation of topology maintenance regularization loss, and reverse propagation of discrete mapping parameters.

[0071] The dynamic hypergraph construction process in step S400 of Example 2 specifically includes the following steps in the hypergraph topology generation stage:

[0072] Step S401: The dynamic hypergraph construction unit 400 receives the mutual information matrix transmitted by the transient mutual information calculation module 300. The dynamic hypergraph construction unit 400 maps each independent mode in the multimodal data system to a graph structure node. The dynamic hypergraph construction unit 400 combines all the graph structure nodes to establish a global node set and constructs a hyperedge set with an initial state of emptiness.

[0073] Step S402: The dynamic hypergraph construction unit 400 reads the system's preset dynamic coupling threshold, and the dynamic hypergraph construction unit 400 traverses the global node set to extract all node combinations that contain at least two graph structure nodes from the global node set, forming multiple candidate node subsets.

[0074] Step S403: The dynamic hypergraph construction unit 400 extracts the transient mutual information between any two graph structure nodes within the candidate node subset from the mutual information matrix. The dynamic hypergraph construction unit 400 then compares all the extracted transient mutual information with the dynamic coupling threshold one by one.

[0075] Step S404: When the transient mutual information between any two graph structure nodes within the candidate node subset is greater than the dynamic coupling threshold, the dynamic hypergraph construction unit 400 generates a hyperedge. The dynamic hypergraph construction unit 400 assigns all graph structure nodes within the candidate node subset to the same hyperedge and adds the generated hyperedge to the hyperedge set. If there is a transient mutual information within the candidate node subset that is less than or equal to the dynamic coupling threshold, the dynamic hypergraph construction unit 400 does not generate a hyperedge for the candidate node subset.

[0076] Step S405: The dynamic hypergraph construction unit 400 completes the traversal comparison calculation of the global node set, summarizes the data mapping relationship between the global node set and the hyperedge set, generates the hypergraph topology association matrix, transmits the hypergraph topology association matrix to the adaptive routing scheduler 500, and uses the hypergraph topology association matrix as the structured instruction parameter for the adaptive routing scheduler 500 to allocate data flow direction.

[0077] The adaptive route scheduling process in step S500 of Example 2 includes a hypergraph-driven hard route allocation process for data flows during the dynamic route allocation phase, specifically including the following steps:

[0078] Step S501: The adaptive routing scheduler 500 receives the hypergraph topology association matrix transmitted by the dynamic hypergraph construction unit 400, and the adaptive routing scheduler 500 synchronously receives the hidden layer feature vector transmitted by the adaptive spatiotemporal coding module 200.

[0079] Step S502: The adaptive route scheduler 500 parses the hypergraph topology association matrix, extracts the node connection parameters recorded in the hypergraph topology association matrix, and identifies whether the graph structure nodes corresponding to each hidden layer feature vector belong to a specific hyperedge in the hyperedge set based on the node connection parameters.

[0080] Step S503: For multiple hidden layer feature vectors belonging to the same hyperedge, the adaptive route scheduler 500 generates a joint route distribution instruction. The adaptive route scheduler 500 synchronously imports the multiple hidden layer feature vectors belonging to the same hyperedge into the high-order joint network 620 in the deep learning processing array 600 according to the joint route distribution instruction. The high-order joint network 620 receives multiple hidden layer feature vectors and performs a concatenation operation in the tensor dimension. The high-order joint network 620 performs multimodal feature fusion calculation based on the concatenated tensor.

[0081] Step S504: For isolated hidden layer feature vectors that do not belong to any hyperedge, the adaptive route scheduler 500 generates independent route distribution instructions. The adaptive route scheduler 500 imports the isolated hidden layer feature vectors into the corresponding single-modality processing sub-network 610 in the deep learning processing array 600 according to the independent route distribution instructions. The adaptive route scheduler 500 does not allocate tensor computation graph resources of the high-order joint network 620 to the isolated hidden layer feature vectors according to the independent route distribution instructions, so as to block the data path of the isolated hidden layer feature vectors input to the high-order joint network 620.

[0082] Step S505: The single-modal processing sub-network 610 receives isolated hidden layer feature vectors and performs single-modal feature calculation. The deep learning processing array 600 summarizes the calculation data of the high-order joint network 620 and the single-modal processing sub-network 610. The deep learning processing array 600 outputs the target task processing result containing the warning signal or cooperative control instruction based on the summarized calculation data.

[0083] Furthermore, the adaptive route scheduling process in step S500 of Embodiment 2 also includes a transient inheritance process of network weights during route switching in the dynamic route allocation phase, specifically including the following steps:

[0084] Step S511: The adaptive routing scheduler 500 monitors the structural change status of the hypergraph topology association matrix. The adaptive routing scheduler 500 compares the hypergraph topology association matrix of the current time period with the historical hypergraph topology association matrix of the previous time period. The adaptive routing scheduler 500 identifies the differences in node connection status between the hidden layer feature vectors in the current time period and the previous time period.

[0085] Step S512: When the hidden layer feature vector changes from the internal node of the hyperedge in the historical hypergraph topology association matrix to the isolated node in the current hypergraph topology association matrix, the adaptive routing scheduler 500 generates a transient inheritance instruction for network weights and sends the transient inheritance instruction for network weights to the deep learning processing array 600.

[0086] Step S513: The deep learning processing array 600 receives the transient inheritance instruction of network weights, the deep learning processing array 600 extracts the network weight tensor of the high-order joint network 620 in the previous time period, and the deep learning processing array 600 locates the corresponding local network channel inside the high-order joint network 620 according to the tensor index corresponding to the hidden layer feature vector that has been changed to an isolated node.

[0087] Step S514: The deep learning processing array 600 extracts the activation state weight data corresponding to the local network channel from the network weight tensor. The deep learning processing array 600 reads the preset dimension projection matrix. The deep learning processing array 600 multiplies the activation state weight data with the dimension projection matrix to complete the affine transformation operation and outputs the initial state weights with the same spatial dimension as the input of the single-modal processing sub-network 610.

[0088] In step S515, the deep learning processing array 600 writes the initial state weights into the corresponding single-modal processing sub-network 610. The single-modal processing sub-network 610 uses the initial state weights to overwrite the original historical static weights. The single-modal processing sub-network 610 receives the isolated hidden layer feature vector based on the initial state weights and performs subsequent single-modal calculation steps.

[0089] In Example 2, the closed-loop parameter adjustment process in step S600 includes a joint network topology regularization loss calculation process in the physical layer closed-loop feedback stage, specifically including the following steps:

[0090] Step S601: The closed-loop parameter adjustment module 700 receives the target task processing result containing the warning signal or cooperative control instruction output by the deep learning processing array 600. The closed-loop parameter adjustment module 700 synchronously reads the real label data preset by the system. The closed-loop parameter adjustment module 700 compares the target task processing result containing the warning signal or cooperative control instruction with the real label data, performs error measurement by cross-entropy calculation or mean square error calculation, and outputs the main task network loss.

[0091] Step S602: The closed-loop parameter adjustment module 700 extracts the high-dimensional hidden layer feature matrix generated by the high-order joint network 620 in the multimodal feature fusion calculation stage. The closed-loop parameter adjustment module 700 is connected to the dynamic hypergraph construction unit 400 to obtain the hypergraph topology association matrix currently output by the dynamic hypergraph construction unit 400.

[0092] Step S603: The closed-loop parameter adjustment module 700 parses the high-dimensional hidden layer feature matrix and separates the high-dimensional hidden layer feature vectors corresponding to different independent modes. In the high-dimensional hidden space output by the high-order joint network 620, the closed-loop parameter adjustment module 700 calculates the Euclidean distance between any two high-dimensional hidden layer feature vectors.

[0093] Step S604: The closed-loop parameter adjustment module 700 reads the node connection status recorded inside the hypergraph topology association matrix, extracts the topology connection weight values ​​of the corresponding modal nodes, constructs a first empirical distribution based on the Euclidean distance values ​​of all node pairs, constructs a second empirical distribution based on the topology connection weight values ​​of all node pairs, calculates the distribution divergence between the first and second empirical distributions using relative entropy, and iterates through all modal node pair combinations. The closed-loop parameter adjustment module 700 sums and aggregates the distribution divergences of all node pairs to generate the topology maintenance regularization loss.

[0094] Step S605: The closed-loop parameter adjustment module 700 reads the balance coefficient set by the system. The closed-loop parameter adjustment module 700 performs a multiplication operation on the topology maintenance regularization loss and the balance coefficient to obtain the regularization penalty term. The closed-loop parameter adjustment module 700 adds the regularization penalty term to the main task network loss and outputs the overall comprehensive loss. The closed-loop parameter adjustment module 700 stores the overall comprehensive loss in memory as the numerical basis for triggering the reverse update of the front-end mapping parameters.

[0095] Furthermore, the closed-loop parameter adjustment process in step S600 of Embodiment 2 also includes a discrete mapping parameter inverse gradient transfer and reshaping process in the physical layer closed-loop feedback stage, specifically including the following steps:

[0096] Step S611: The closed-loop parameter adjustment module 700 reads the overall loss from the memory space, connects to the computation graph interface of the high-order joint network 620, and extracts the weight gradient flow generated during the backpropagation stage from the computation graph interface of the high-order joint network 620.

[0097] Step S612: The closed-loop parameter adjustment module 700 identifies the delay time parameter and the embedded dimension parameter required for the operation of the spatiotemporal phase space mapping module 100 as non-differentiable discrete variables. The closed-loop parameter adjustment module 700 calls the preset pass-through estimator logic and merges the overall integrated loss and the weight gradient flow as the input variables of the pass-through estimator logic.

[0098] Step S613: The closed-loop parameter adjustment module 700 executes the pass-through estimator logic. The closed-loop parameter adjustment module 700 uses the pass-through estimator logic to skip the non-differentiable breakpoints of discrete variables during backpropagation calculation. The closed-loop parameter adjustment module 700 uses the pass-through estimator logic to copy and map the continuous spatial gradient of the high-order joint network 620 to discrete spatial gradient. The closed-loop parameter adjustment module 700 outputs the step size values ​​of the delay time parameter adjustment and the step size values ​​of the embedding dimension parameter adjustment for each independent mode.

[0099] Step S614: The closed-loop parameter adjustment module 700 sends the delay time parameter adjustment step value and the embedded dimension parameter adjustment step value to the spatiotemporal phase space mapping module 100 through the reverse communication link at the bottom layer of the system. The spatiotemporal phase space mapping module 100 receives the delay time parameter adjustment step value and the embedded dimension parameter adjustment step value.

[0100] Step S615: The spatiotemporal phase space mapping module 100 performs an algebraic addition operation on the currently recorded delay time parameter and the delay time parameter adjustment step value, and performs an integer operation on the addition result to obtain the updated delay time parameter. The spatiotemporal phase space mapping module 100 performs an algebraic addition operation on the currently recorded embedding dimension parameter and the embedding dimension parameter adjustment step value, and performs an integer operation on the addition result to obtain the updated embedding dimension parameter. The spatiotemporal phase space mapping module 100 writes the updated delay time parameter and the updated embedding dimension parameter into the configuration register. The spatiotemporal phase space mapping module 100 reads the updated delay time parameter and the updated embedding dimension parameter, triggering the high-dimensional phase space feature reconstruction step for the multimodal asynchronous time series of the next time period.

[0101] Through the construction of the hypergraph structure with the aforementioned dynamic coupling threshold, dynamic route allocation, route switching weight inheritance, and closed-loop feedback adjustment, high-order fusion inference of multimodal data and reverse optimization of front-end mapping parameters are realized, thereby completing the closed-loop operation of the entire multimodal data adaptive processing system.

Claims

1. An artificial intelligence-driven multimodal data adaptive processing system, characterized in that, include: The spatiotemporal phase space mapping module (100) is used to receive multimodal asynchronous data streams collected by front-end sensors of an industrial equipment monitoring system. The multimodal asynchronous data streams contain heterogeneous sensor signals characterizing the physical state of the industrial equipment. The module extracts the time series of each mode in the multimodal asynchronous data streams and reconstructs the time series of each mode into a high-dimensional phase space trajectory vector along the time axis according to the set initial delay time parameters and initial embedding dimension parameters. An adaptive spatiotemporal coding module (200) is connected to the spatiotemporal phase space mapping module (100) and is used to perform nonlinear mapping calculation based on a recurrent neural network structure on the high-dimensional phase space trajectory vector, and output a hidden layer feature vector of fixed dimension to complete the tensor dimension alignment of each modality data before inputting into the deep learning processing array (600), thereby characterizing the spatial topological features of different physical sensor signals. The transient mutual information calculation module (300) is connected to the adaptive spatiotemporal coding module (200) and is used to calculate the mutual information between different hidden layer feature vectors within a set time window in order to measure the underlying physical coupling relationship between different industrial sensing modes. The dynamic hypergraph construction unit (400) is connected to the transient mutual information calculation module (300) and is used to compare the mutual information with the set dynamic coupling threshold. When the mutual information corresponding to a set of hidden layer feature vectors is greater than the dynamic coupling threshold, a hyperedge containing the corresponding hidden layer feature vector node is generated, and the hypergraph topology association matrix is ​​output according to the connection relationship between the node and the hyperedge. A deep learning processing array (600) contains a single-modal processing sub-network (610) and a high-order joint network (620). The deep learning processing array (600) is used to perform forward propagation calculations and output multimodal processing results for indicating abnormal states of industrial equipment or equipment collaborative control commands. An adaptive routing scheduler (500) is connected to the dynamic hypergraph construction unit (400) and the deep learning processing array (600) respectively. It is used to identify the network node connection status of each hidden layer feature vector in the hypergraph topology association matrix, distribute the hidden layer feature vectors inside the same hyperedge to the higher-order joint network (620) in the deep learning processing array (600), and distribute isolated hidden layer feature vectors to the corresponding single-modality processing sub-network (610) in the deep learning processing array (600). A closed-loop parameter adjustment module (700) is connected between the deep learning processing array (600) and the spatiotemporal phase space mapping module (100). It is used to extract the target task processing result output by the deep learning processing array (600) containing the warning signal or cooperative control instruction, calculate the overall comprehensive loss based on the target task processing result and the real label data preset by the system, extract the inverse gradient flow generated by the high-order joint network (620), calculate the update amount of the delay time parameter and the update amount of the embedded dimension parameter according to the overall comprehensive loss and the inverse gradient flow, and send the update amount of the delay time parameter and the update amount of the embedded dimension parameter to the spatiotemporal phase space mapping module (100). The spatiotemporal phase space mapping module (100) is also used to overwrite the current delay time parameter and the embedded dimension parameter using the update amount of the delay time parameter and the update amount of the embedded dimension parameter, and enter the data phase space reconstruction of the next time window based on the updated parameters.

2. The AI-driven multimodal data adaptive processing system according to claim 1, characterized in that, The spatiotemporal phase space mapping module (100) reconstructs the time series of each mode into a high-dimensional phase space trajectory vector along the time axis, specifically configured as follows: Regarding the first The time series of each modality is subjected to time axis expansion using the delayed embedding theorem. The high-dimensional phase space mapping calculation logic is expressed by the following formula: In the formula, Indicates the first The modality at time... The generated high-dimensional phase space trajectory vector; Indicates the first The modality at time... Time series observations; Indicates the first The current delay time parameter for each mode; Indicates the first The current embedding dimension parameter of each modality determines the length of the high-dimensional phase space trajectory vector; Based on the delay time parameter and the embedding dimension parameter, the state space of the time series of each mode is reconstructed according to the set sampling step size. During the reconstruction process, the original sampling rate of each mode data is maintained, and the nonlinear dynamic features in the time series are extracted.

3. The AI-driven multimodal data adaptive processing system according to claim 1, characterized in that, The adaptive spatiotemporal coding module (200) performs nonlinear mapping calculations based on a recurrent neural network structure on the high-dimensional phase space trajectory vector, and outputs a hidden layer feature vector of fixed dimension, specifically configured as follows: Read the hidden layer dimension constant preset by the system, and use the hidden layer dimension constant as the target dimension of the output feature; The high-dimensional phase space trajectory vector is decomposed into discrete sequence elements according to the chronological order, an initial hidden state vector is constructed and all values ​​are initialized to zero, and a nonlinear mapping calculation based on a recurrent neural network structure is performed step by step along the feature sequence direction. The logic of the single-step recurrent mapping calculation is expressed by the following formula: In the formula, Indicates the first The mode in the th ... Hidden state vectors with mapping step size; Represents a nonlinear activation function; Represents the input layer weight matrix; The first element in the high-dimensional phase space trajectory vector represents the... 1 sequence element, and The maximum value is equal to the current embedding dimension parameter; This represents the hidden layer state weight matrix; This represents the hidden state vector of the previous step length; Indicates the bias parameter; After the calculation of the last sequence element is completed, the hidden state vector of the recurrent neural network corresponding to the last sequence element is extracted as the final hidden layer feature vector. The dimension of the hidden layer feature vector is equal to the matrix dimension set by the hidden layer state weight matrix.

4. The AI-driven multimodal data adaptive processing system according to claim 1, characterized in that, The transient mutual information calculation module (300) calculates the mutual information between different hidden layer feature vectors within a set time window, specifically configured as follows: Within the current sliding time window, time-series data segments corresponding to each hidden layer feature vector are extracted. A multi-dimensional state space is established for the extracted time-series data segments. The probability distribution statistics of the time-series data segments in the multi-dimensional state space are performed based on the kernel density estimation method. Calculate the joint probability distribution corresponding to the two extracted hidden layer feature vector time-series data segments, and independently calculate the marginal probability distribution corresponding to the two extracted hidden layer feature vector time-series data segments respectively; calculate the mutual information using the joint probability distribution and the marginal probability distribution, the mutual information calculation logic being expressed by the following formula: In the formula, Indicates the first The modality and the first The mutual information between the hidden layer feature vectors corresponding to each modality; and Each represents a corresponding time-series data segment; and These represent the discrete values ​​in the time-series data segment, respectively. Represents the joint probability distribution; and These represent the corresponding marginal probability distributions.

5. The AI-driven multimodal data adaptive processing system according to claim 1, characterized in that, The dynamic hypergraph construction unit (400) compares the mutual information with a set dynamic coupling threshold. When the mutual information corresponding to a set of hidden layer feature vectors is greater than the dynamic coupling threshold, a hyperedge containing the corresponding hidden layer feature vector node is generated. The specific configuration is as follows: Each independent modality in the multimodal data system is mapped to a graph structure node, all graph structure nodes are combined to establish a global node set, and an initial set of hyperedges is constructed. Traverse the global node set and extract all node combinations that contain at least two graph structure nodes from the global node set to form multiple candidate node subsets; Extract the transient mutual information between any two graph structure nodes within the candidate node subset, and compare all extracted transient mutual information with the dynamic coupling threshold one by one. When the transient mutual information between any two graph structure nodes within the candidate node subset is greater than the dynamic coupling threshold, a hyperedge is generated, all graph structure nodes within the candidate node subset are assigned to the same hyperedge, and the generated hyperedge is added to the hyperedge set. If the candidate node subset contains transient mutual information less than or equal to the dynamic coupling threshold, no hyperedge is generated for the candidate node subset.

6. The AI-driven multimodal data adaptive processing system according to claim 1, characterized in that, The adaptive routing scheduler (500) uniformly distributes hidden layer feature vectors within the same hyperedge to the higher-order joint network (620) within the deep learning processing array (600), and distributes isolated hidden layer feature vectors to the corresponding single-modality processing sub-network (610) within the deep learning processing array (600). The specific configuration is as follows: For multiple hidden layer feature vectors belonging to the same hyperedge, a joint routing distribution instruction is generated, and the multiple hidden layer feature vectors are synchronously imported into the higher-order joint network (620) according to the joint routing distribution instruction. The high-order joint network (620) receives multiple hidden layer feature vectors and performs a concatenation operation in the tensor dimension, and performs multimodal feature fusion calculation based on the concatenated tensor; For isolated hidden layer feature vectors that do not belong to any hyperedge, an independent routing distribution instruction is generated. The isolated hidden layer feature vector is then imported into the corresponding single-modality processing sub-network (610) according to the independent routing distribution instruction. The tensor computation graph resources of the higher-order joint network (620) are not allocated to the isolated hidden layer feature vector according to the independent routing distribution instruction, so as to block the data path of the isolated hidden layer feature vector into the higher-order joint network (620).

7. The AI-driven multimodal data adaptive processing system according to claim 1, characterized in that, After identifying the network node connection status of each of the hidden layer feature vectors in the hypergraph topology association matrix, the adaptive routing scheduler (500) is further configured as follows: By comparing the hypergraph topological correlation matrix of the current time period with the historical hypergraph topological correlation matrix of the previous time period, the differences in node connection states of hidden layer feature vectors between the current time period and the previous time period can be identified. When the hidden layer feature vector changes from an internal node of a hyperedge in the historical hypergraph topology association matrix to an isolated node in the current hypergraph topology association matrix, a network weight transient inheritance instruction is generated. Extract the network weight tensor of the higher-order joint network (620) in the previous time period, and locate the corresponding local network channel inside the higher-order joint network (620) according to the tensor index corresponding to the hidden layer feature vector of the isolated node. Extract the activation state weight data corresponding to the local network channel from the network weight tensor, read the preset dimension projection matrix, multiply the activation state weight data with the dimension projection matrix to complete the affine transformation operation, and output the initial state weights whose spatial dimension is consistent with the input of the single-modal processing sub-network (610). The initial state weights are written into the corresponding single-modal processing sub-network (610), and the original historical static weights are overwritten using the initial state weights. The isolated hidden layer feature vector is received based on the initial state weights.

8. The AI-driven multimodal data adaptive processing system according to claim 1, characterized in that, The closed-loop parameter adjustment module (700) extracts the target task processing result output by the deep learning processing array (600), which includes the warning signal or cooperative control command. Based on the target task processing result and the system's preset real label data, it calculates the overall comprehensive loss and extracts the inverse gradient flow generated by the high-order joint network (620). The specific configuration is as follows: Compare the target task processing results containing the warning signal or collaborative control instruction with the real label data preset by the system, perform error measurement by cross-entropy calculation or mean square error calculation, and output the main task network loss. Extract the high-dimensional hidden layer feature matrix generated by the high-order joint network (620) in the multimodal feature fusion calculation stage, analyze the high-dimensional hidden layer feature matrix, and separate the high-dimensional hidden layer feature vectors corresponding to different independent modes; Within the high-dimensional latent space output by the high-order joint network (620), calculate the Euclidean distance between any two high-dimensional latent layer feature vectors; read the node connection states recorded inside the hypergraph topological association matrix, extract the topological connection weight values ​​of the corresponding modal nodes, construct a first empirical distribution based on the Euclidean distance values ​​of all node pairs, construct a second empirical distribution based on the topological connection weight values ​​of all node pairs, and calculate the distribution divergence between the first empirical distribution and the second empirical distribution using relative entropy; Traverse all modal node pairings, sum and aggregate the distribution divergence of all node pairs to generate topology maintenance regularization loss; multiply the topology maintenance regularization loss with the system-defined balance coefficient to obtain a regularization penalty term, sum the regularization penalty term with the main task network loss, and output the overall comprehensive loss.

9. The AI-driven multimodal data adaptive processing system according to claim 8, characterized in that, The closed-loop parameter adjustment module (700) calculates the update amount of the delay time parameter and the update amount of the embedded dimension parameter based on the overall comprehensive loss and the reverse gradient flow, and is specifically configured as follows: Extract the weight gradient flow generated during the backpropagation phase from the computation graph interface of the higher-order joint network (620); The delay time parameter and the embedding dimension parameter are identified as non-differentiable discrete variables. A preset pass-through estimator logic is invoked, and the overall comprehensive loss and the weight gradient flow are merged as the input variables of the pass-through estimator logic. The pass-through estimator logic is executed, and the non-differentiable breakpoints of discrete variables are skipped during backpropagation computation using the pass-through estimator logic. The continuous spatial gradient of the higher-order joint network (620) is copied and mapped to the discrete spatial gradient. Output the step size values ​​for adjusting the delay time parameter and the step size values ​​for adjusting the embedding dimension parameter for each independent mode.

10. The AI-driven multimodal data adaptive processing system according to claim 9, characterized in that, The spatiotemporal phase space mapping module (100) overwrites the current delay time parameter and embedding dimension parameter using the update amount of the delay time parameter and the update amount of the embedding dimension parameter, specifically configured as follows: The currently recorded delay time parameter is algebraically added to the delay time parameter adjustment step value, and the result is rounded down to obtain the updated delay time parameter. The currently recorded embedding dimension parameter is algebraically added to the embedding dimension parameter adjustment step value, and the result is rounded down to obtain the updated embedding dimension parameter. Write the updated delay time parameter and the updated embedding dimension parameter to the configuration register, read the updated delay time parameter and the updated embedding dimension parameter, and trigger the high-dimensional phase space feature reconstruction step for the multimodal asynchronous time series of the next time period.

Images