A machine learning-based multi-source data processing method and system

Abstract

This invention discloses a multi-source data processing method and system based on machine learning. The invention relates to the field of machine learning technology. The method extracts partial differential equation boundary conditions and geometric constraints from equipment CAD geometric models and CAE simulation data, constructs a graph topology carrying physical constraints, and inputs it into a physical graph neural network to obtain physical features. Multimodal features are extracted after asynchronous alignment of visual, temporal sensor, and knowledge base data. Gating weights are calculated based on the physical features. When a modality is missing, cross-modal knowledge distillation branches are activated to generate substitute features, which are then weighted and fused. Training is performed based on a joint loss function consisting of data-driven loss, feature distribution alignment loss, and physical residual penalty term. This invention is applicable to the fields of industrial IoT digital twins and intelligent manufacturing, and can maintain stable inference when modalities are missing, with output results conforming to physical laws.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, specifically to a multi-source data processing method and system based on machine learning. Background Technology

[0002] In scenarios such as industrial IoT digital twins, intelligent manufacturing, and high-end equipment condition monitoring, it is often necessary to fuse heterogeneous data from multiple sources, such as visual image data, high-frequency time-series sensor data, and historical knowledge base text data.

[0003] Existing multimodal fusion schemes mainly include feature layer concatenation fusion method and decision layer weighted fusion method: the former extracts features from each modal data and concatenates them at the feature layer, then inputs them into a unified classifier or regressor for decision output; the latter trains sub-models for each modal data separately and weights the decision results of each sub-model according to a pre-set fixed weight. Both of these schemes require that the modal data be strictly synchronized in time and complete in quantity, and that the internal structure of the model is not related to the physical constraints of the equipment such as geometry, mechanics, and thermal, relying only on a large amount of labeled data for statistical fitting.

[0004] Furthermore, graph neural networks can model the connection relationships between nodes and edges in geometric models, and physical information neural networks can make the output satisfy physical constraints by introducing the residuals of partial differential equations into the loss function. However, the above basic methods have not yet been combined with multimodal dynamic fusion mechanisms.

[0005] When a certain modality of data is dynamically missing or subjected to severe noise interference, the overall performance of the fusion model is prone to a significant decline. When the model faces working conditions that have not appeared in the training data, the output results may also violate objective physical laws, making it difficult to meet the actual needs of industrial sites for the robustness and reliability of the fusion model. Summary of the Invention

[0006] The technical problem this invention aims to solve is: in complex production environments such as industrial IoT digital twins, intelligent manufacturing, and high-end equipment, how to fuse multi-source heterogeneous data such as visual images, high-frequency time-series sensor signals, and historical knowledge base text, so that the fusion model can still maintain stable reasoning ability when a certain modality of data is dynamically missing or subject to severe noise interference, and ensure that the output results conform to objective physical laws, thereby avoiding the erroneous conclusions that violate physical laws produced by existing pure data-driven models under unknown working conditions, while also taking into account processing efficiency and avoiding the problem that pure physical simulation calculations are too time-consuming and cannot meet real-time processing requirements.

[0007] To address the aforementioned technical problems, this invention provides a multi-source data processing method based on machine learning, comprising:

[0008] The CAD geometric model data and CAE simulation result data of the target device are acquired, and the boundary conditions and geometric constraints of the partial differential equations are extracted from them. Based on the boundary conditions of the partial differential equations and the geometric constraints, a physical graph topology carrying physical constraint information is constructed. The physical graph topology includes a node feature matrix and an edge feature matrix.

[0009] The physical graph topology is input into the physical graph neural network to obtain physical features;

[0010] The visual data, temporal sensor data, and knowledge base retrieval data of the target device are acquired. Asynchronous spatiotemporal alignment processing is performed on the visual data, the temporal sensor data, and the knowledge base retrieval data to obtain aligned multimodal data. The aligned multimodal data are then input into the corresponding modal feature extraction subnetwork to obtain multiple modal features.

[0011] The physical features and the multiple modal features are input into the dynamic routing gating network, so that the calculation of the gating weight depends on both the physical features and the corresponding modal features, thus obtaining the gating weight corresponding to each modal feature.

[0012] If it is determined that there is missing modal data at the current moment, the gating weight corresponding to the missing modality is reset to zero, and the cross-modal knowledge distillation branch is activated. The non-missing modal features are used as input to the student model to generate the substitute features corresponding to the missing modality.

[0013] If it is determined that there is no missing modal data at the current moment, the corresponding modal features will be used as the features to participate in the fusion.

[0014] The features participating in the fusion are weighted and summed according to the gating weights to obtain the fused feature vector;

[0015] During the training phase, a joint optimization loss function is calculated based on the fused feature vector. The joint optimization loss function includes data-driven loss, feature distribution alignment loss, and residual penalty term of physical law partial differential equation. Backpropagation is then performed based on the joint optimization loss function to update the network parameters.

[0016] During the inference phase, the prediction result is output based on the fused feature vector.

[0017] The present invention provides a multi-source data processing system based on machine learning, including a physical graph topology construction module, a multimodal feature extraction module, a dynamic gating fusion module, and a joint optimization training and inference module, wherein each module works together to implement the processing described above.

[0018] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows:

[0019] Firstly, by directly transforming the boundary conditions and geometric constraints of partial differential equations in the CAD geometric model and CAE simulation results into the underlying topology of the physical graph neural network, the model output is naturally constrained by physical laws. This can improve the model's generalization ability under conditions not covered by training data and avoid erroneous conclusions with high confidence but that violate physical laws.

[0020] Secondly, by introducing physical features into the weight calculation process of the dynamic routing gating network, the allocation of gating weights reflects both the reliability and physical rationality of the modal data. This differs from existing technologies that calculate weights solely based on the modal features themselves. Furthermore, by combining cross-modal knowledge distillation branches, when a modality is dynamically missing, the teacher model branch guides the student model branch to generate alternative features with physical consistency to participate in the fusion. This collaborative mechanism of "physical feature-driven gating + knowledge distillation alternatives" is not a simple superposition of existing single technical means. It overcomes the technical bias of existing pure data-driven fusion schemes that cannot balance robustness and physical consistency in modality-missing scenarios, and avoids the significant drop in fusion performance caused by the failure of a certain modality in existing schemes.

[0021] Third, by constructing a joint optimization loss function that includes data-driven loss, feature distribution alignment loss, and residual penalty term of physical law partial differential equations, both data fitting accuracy and physical consistency can be taken into account during the network forward inference stage. Compared with pure physical simulation schemes, this avoids the large amount of computation time consumed by iterative discrete solution of partial differential equations, and can meet the needs of real-time data processing and decision-making in production sites. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall system architecture of the present invention;

[0023] Figure 2 This is a flowchart of the multi-source data processing method of the present invention;

[0024] Figure 3 This is a schematic diagram of the multimodal asynchronous spatiotemporal alignment and dynamic gating fusion module of the present invention. Detailed Implementation

[0025] The following is in conjunction with the appendix Figure 1-3 The specific embodiments of the present invention will be further described below. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0026] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] I. System Overall Architecture

[0028] The multi-source data processing system provided in this embodiment includes, from bottom to top, a data acquisition layer, a physical graph topology construction module, a multimodal feature extraction layer, an asynchronous spatiotemporal alignment and dynamic gating fusion module, a joint optimization loss calculation and training module, and a decision output layer. The data acquisition layer includes industrial cameras or other visual sensing devices, a high-frequency time-series sensor array, a historical knowledge base retrieval interface, and a device CAD / CAE simulation data interface, used to acquire visual data, time-series sensor data, knowledge base retrieval data, CAD geometric model data, and CAE simulation result data of the target device, respectively. The physical graph topology construction module receives CAD / CAE data and outputs a physical graph topology structure carrying physical constraint information. The multimodal feature extraction layer comprises a visual feature extraction subnetwork, a temporal feature extraction subnetwork, a knowledge retrieval embedding subnetwork, and a physical graph neural network feature extraction subnetwork, used to extract features from the above-mentioned data types respectively. The asynchronous spatiotemporal alignment and dynamic gating fusion module receives the features of each modality and outputs a fused feature vector. The joint optimization loss calculation and training module receives the fused feature vector and labeled tags during the training phase, calculates the joint optimization loss, and updates all network parameters in reverse. The decision output layer outputs the prediction result or control decision based on the fused feature vector during the inference phase. The data flow between the above modules is that the output of the previous module is used as the input of the next module, forming a series-connected and collaborative overall processing link.

[0029] In this embodiment, the above-mentioned functional modules can be implemented by a processor and a memory: the memory stores a computer program, and when the processor executes the computer program, it implements the processing steps corresponding to the above-mentioned modules. The above-mentioned modules can be integrated in the same computing device or distributed in multiple computing nodes connected by a communication network. This invention does not limit this.

[0030] II. Physical Graph Topology Construction Module

[0031] The physical graph topology construction module is used to extract physical constraint information from equipment CAD geometric model data and CAE simulation result data, and transform it into the underlying topology of physical graph neural network. The specific implementation process is as follows.

[0032] First, obtain the CAD geometric model file and CAE simulation result file of the target device, analyze the geometric elements (including nodes, surfaces, and volumes) and their adjacency relationships, and generate an initial geometric topology diagram, denoted as . ,in Represents the initial set of nodes. This represents the initial set of edges.

[0033] Secondly, the boundary conditions of the partial differential equations defined in the CAE simulation results are extracted. The boundary conditions include at least one of temperature boundary conditions, stress boundary conditions, and fluid flow boundary conditions. The extracted boundary condition parameters are labeled on the corresponding geometric nodes or edges in the initial geometric topology graph to form a physical constraint graph with boundary attributes.

[0034] Then, the physical constraint graph is simplified and key nodes are sampled: based on at least one of the curvature change rate and stress concentration degree, the importance score of each node is obtained by weighted summation according to preset weights. After sorting all nodes from high to low score, nodes whose score ranking is within a preset ranking range (e.g., the top 30%) and the edges connected to these nodes are retained, resulting in a simplified topology graph with the number of nodes controlled within a preset threshold, denoted as . .

[0035] Next, the physical attributes of each node are encoded into an initial node feature vector, denoted as . The physical properties include material parameters, geometric dimensions, initial conditions, and boundary conditions. The physical connectivity represented by each edge is encoded as an edge feature, denoted as... The physical connection relationship includes heat transfer coefficient, contact stiffness, and fluid resistance coefficient, wherein... , This represents the two nodes connected by the edge.

[0036] Finally, output the complete physical graph topology, denoted as... ,in This represents the node feature matrix, which is composed of the feature vectors of all nodes. This represents the edge feature matrix, which consists of all edge features and serves as the input to the downstream physical graph neural network.

[0037] The physical graph topology is input into the physical graph neural network for forward propagation to obtain physical features, while retaining the intermediate outputs of the network. The intermediate output must satisfy the following partial differential equation constraints:

[0038] ;

[0039] in, Representing the spatial location of a physical graph neural network ,time The predicted physical quantities output, such as the values ​​of the temperature field or stress field, are... The learnable parameters of a physical graph neural network are represented. This represents the partial differential equation operator determined by the physical problem being modeled. This represents a differential operator related to the spatial derivative, the specific form of which is determined by the physical process being modeled (heat conduction, structural mechanics, fluid mechanics). This constraint is reflected in the subsequent joint optimization loss function as a physical residual penalty term, used to guide the network output to conform to physical laws.

[0040] III. Multimodal Asynchronous Spatiotemporal Alignment and Dynamic Gating Fusion Module

[0041] This module is used to implement the joint mapping of visual data, time-series sensor data, and knowledge base retrieval data, and to automatically activate the cross-modal knowledge distillation branch to perform feature replacement when a certain modality of data is dynamically missing. The specific implementation process is as follows.

[0042] First, timestamps are applied to the visual data stream, the time-series sensor data stream, and the knowledge base retrieval result stream to construct a unified timeline.

[0043] Secondly, taking the sampling period of one of the modal data as the reference period, a sliding time window mechanism is used to aggregate modal data with sampling periods shorter than the reference period within the time window. Modal data without valid acquisition values ​​within the time window are marked as missing, without forced interpolation to fill in the missing data, so as to avoid introducing false data.

[0044] Then, the modal data after asynchronous spatiotemporal alignment are input into the corresponding modal feature extraction subnetworks: In this embodiment, the vision branch uses a convolutional neural network structure to extract features from the visual data, the temporal branch uses a gated recurrent unit structure to extract features from the temporal sensor data, and the knowledge retrieval branch uses a graph embedding retrieval structure to extract features from the knowledge base retrieval data, thus obtaining modal feature representations respectively. ,in , The number represents the total number of modalities participating in the fusion. It should be understood that the specific network structure described above is only an example. In other embodiments, each modality feature extraction subnetwork may also be implemented using other network structures that can achieve the corresponding modality feature extraction function.

[0045] Next, the modal features are represented. The physical features output by the physical graph neural network are input together with the dynamic routing gating network to calculate the gating weights of each mode at the current time step. The calculation method is as follows:

[0046] ;

[0047] in, Indicates the first The gating weights corresponding to each modality range from 0 to 1, and the sum of all modality weights is 1. Indicates the first The feature vectors output by the feature extraction subnetwork for each modality Indicates the relationship between the first and second gated networks. The learnable weight vector corresponding to each modality Indicates the relationship with the first The learnable bias term corresponding to each modality This represents the total number of modes participating in the fusion. The summation operation in the denominator is used to normalize the weights of all modes.

[0048] Furthermore, determine whether a modality is missing at the current moment: if it is determined that the first... If a modality is missing at the current moment, then its corresponding gating weights are... Directly set to zero and activate the cross-modal knowledge distillation branch—using the teacher model branch pre-trained on historical complete multimodal data, and taking existing modal features as input to the student model branch, guides the student model branch to generate the substitute features corresponding to the missing modality, denoted as... If the judgment is made If no mode is missing at the current moment, then Take the original features obtained in the previous step directly. .

[0049] Finally, based on the obtained gating weights and substitute features, the modal features are weighted and summed to obtain the fused feature vector, calculated as follows:

[0050] ;

[0051] in, This represents the final output multimodal fusion feature vector. Indicates the first The gating weights of each modality Indicates the first The features obtained after processing each modality in the previous step include both original features and substitute features. The summation symbol represents the summation over all modalities. The weighted features of each modality are accumulated, and the resulting fused feature vector is used as the input to the subsequent decision network and joint optimization loss calculation module.

[0052] IV. Hybrid-driven joint optimization training and inference module

[0053] This module constructs a joint optimization objective function that includes data-driven loss, feature distribution alignment loss, and residual penalty terms of partial differential equations of physical laws. This function enables the training and updating of model parameters and the output of results during the inference phase. The specific implementation process is as follows.

[0054] The data-driven loss is calculated as follows:

[0055] ;

[0056] in, Indicates data-driven loss. This represents the total number of samples in the training batch. This represents the total number of categories in a classification task. For a regression task, this loss can be replaced with the mean squared error loss, and the principle is the same. Indicates the first The sample at the th The actual label value on the class can be either 0 or 1. The model represents the first The sample at the th Predicted probabilities on a class.

[0057] The feature distribution alignment loss is calculated as follows:

[0058] ;

[0059] in, This represents the feature distribution alignment loss, used to ensure that the feature distribution derived from historical simulation data remains consistent with the feature distribution derived from real-time sensing data. , These represent the number of samples in the source domain and the target domain, respectively. , Representing the source domain, respectively The sample and the target domain Feature representation of a sample This represents the mapping function that maps features to the reproducing kernel Hilbert space. In this Hilbert space The norm squared in.

[0060] The method for calculating the residual penalty term in the partial differential equations of physical laws is as follows:

[0061] ;

[0062] in, This represents the physical residual penalty term. This indicates the number of sampling points used to calculate the physical residual. , They represent the first Spatial and temporal coordinates of each sampling point This represents the physical quantity predicted by the physical graph neural network at this sampling point. This represents the partial differential equation operator defined above. The smaller the value of this term, the better the prediction result conforms to the constraints of physical laws.

[0063] The joint optimization objective function is calculated as follows:

[0064] ;

[0065] in, This represents the joint optimization objective ultimately used for backpropagation. , , These represent the preset weight coefficients corresponding to the data-driven loss, feature distribution alignment loss, and physical residual penalty term, respectively, used to balance the relative contributions of the three losses during the training process. Their specific values ​​can be preset according to the actual application scenario.

[0066] During the training phase, the parameters of the physical graph neural network, multimodal feature extraction network, dynamic routing gating network, and downstream decision network are first initialized. A batch of multi-source data samples is sampled from the training dataset. Each sample contains equipment CAD / CAE physical topology data, corresponding time-series visual image data, sensor time-series data, and labels. The corresponding processing steps of physical graph topology construction, multimodal asynchronous alignment, and dynamic gating fusion are executed sequentially, and the fused feature vector is obtained by forward computation. The prediction results are output by the decision network. Meanwhile, the intermediate outputs of the physical graph neural network are preserved. This is for subsequent calculation of physical residuals, whereby data-driven losses are calculated according to the formulas described above. Feature distribution alignment loss and physical residual penalty item The joint optimization objective is obtained by weighting and summing the results according to preset weighting coefficients. Calculated using the backpropagation algorithm The gradients of the physical graph neural network parameters, multimodal feature extraction network parameters, gating network parameters, and decision network parameters are calculated, and an adaptive optimizer is used to update all parameters. The above sampling and updating process is repeated until the joint optimization objective converges or the preset maximum number of iterations is reached, and the trained model parameters are output.

[0067] During the inference phase, the trained model parameters are loaded, and the same data organization method and forward computation process as in the training phase are performed on the real-time collected multi-source data, but backpropagation is not performed. The final output is a prediction result that combines data fitting accuracy and physical consistency for subsequent control decisions.

[0068] V. Overall Methodology and Process

[0069] The overall execution flow of the multi-source data processing method of this invention includes the following steps: acquiring equipment CAD / CAE data and constructing a physical graph topology; collecting real-time multimodal data from vision, time series, and knowledge base and performing asynchronous spatiotemporal alignment; determining whether a modality is missing; if missing, activating the cross-modal knowledge distillation branch to generate substitute features; if not missing, directly using the original features; performing weighted fusion according to gating weights to obtain a fused feature vector; determining whether the current stage is training or inference; if training, calculating the joint optimization loss and updating the parameters in reverse; if inference, directly outputting the prediction result based on the fused feature vector, thus forming a complete closed-loop process from data input to result output.

[0070] VI. Other Implementation Methods

[0071] In other implementations of the above-mentioned multimodal gating fusion process, in addition to using the dynamic routing gating network based on softmax normalization as described above, a multi-head self-attention mechanism can also be used to fuse the features of each modality: each modality feature is mapped to a query vector, a key vector, and a value vector, respectively. The attention score matrix between modalities is calculated to weight and aggregate the features of each modality to obtain the fused features. In the case of modality missing, the attention score of the corresponding modality can be set to a preset minimum value. After normalization, the weight of the corresponding modality approaches zero, achieving a missing feature masking effect similar to that of the gating network.

[0072] In other implementations of the physical graph topology construction process described above, in addition to directly encoding the boundary conditions of the partial differential equations as attributes of graph nodes and edges, soft constraint regularization can also be used: without changing the original network topology, only an additional physical residual penalty term calculated based on automatic differentiation technology is introduced into the joint optimization loss function. Through the constraints at the loss function level, the network parameters are guided to learn a mapping relationship that conforms to physical laws. This implementation can be used as a simplified implementation in scenarios with limited computing resources.

[0073] As can be seen from the above specific implementation methods, the present invention directly embeds physical constraints into the topology of the graph neural network, so that the model output naturally satisfies the constraints of physical laws. At the same time, it achieves stable inference under the condition of dynamic missing modalities through dynamic routing gating network and cross-modal knowledge distillation branch. Furthermore, by jointly optimizing the loss function, it balances data fitting accuracy and physical consistency while ensuring processing efficiency. Thus, it solves the technical problems of insufficient robustness, representation gap and lack of real-time performance in the prior art.

[0074] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-source data processing method based on machine learning, characterized in that, include: The CAD geometric model data and CAE simulation result data of the target device are acquired, and the boundary conditions and geometric constraints of the partial differential equations are extracted from them. Based on the boundary conditions of the partial differential equations and the geometric constraints, a physical graph topology carrying physical constraint information is constructed. The physical graph topology includes a node feature matrix and an edge feature matrix. The physical graph topology is input into the physical graph neural network to obtain physical features; The visual data, temporal sensor data, and knowledge base retrieval data of the target device are acquired. Asynchronous spatiotemporal alignment processing is performed on the visual data, the temporal sensor data, and the knowledge base retrieval data to obtain aligned multimodal data. The aligned multimodal data are then input into the corresponding modal feature extraction subnetwork to obtain multiple modal features. The physical features and the multiple modal features are input into the dynamic routing gating network, so that the calculation of the gating weight depends on both the physical features and the corresponding modal features, thus obtaining the gating weight corresponding to each modal feature. If it is determined that there is missing modal data at the current moment, the gate weight corresponding to the missing modality is reset to zero, and the cross-modal knowledge distillation branch is activated. The non-missing modal features are used as input to the student model to generate the substitute features corresponding to the missing modality. If it is determined that there is no missing modal data at the current moment, the corresponding modal features are used as features to participate in the fusion. The features participating in the fusion are weighted and summed according to the gating weights to obtain the fused feature vector; During the training phase, a joint optimization loss function is calculated based on the fused feature vector. The joint optimization loss function includes data-driven loss, feature distribution alignment loss, and residual penalty term of physical law partial differential equation. Backpropagation is performed based on the joint optimization loss function to update the network parameters. During the inference phase, the prediction result is output based on the fused feature vector.

2. The method according to claim 1, characterized in that, The steps for constructing a physical graph topology carrying physical constraint information include: The geometric units and their adjacency relationships in the CAD geometric model data are analyzed to generate an initial geometric topology graph; The boundary conditions of the partial differential equations are labeled on the corresponding geometric nodes or edges in the initial geometric topology graph to obtain a physical constraint graph with boundary attributes. Based on at least one of the curvature change rate and stress concentration degree, the nodes in the physical constraint diagram are scored for importance, and the nodes whose scores are within a preset sorting range and the edges connected to the nodes are retained to obtain a simplified topology diagram. The physical attributes of each node are encoded as node features, and the physical connection relationships represented by each edge are encoded as edge features, thus obtaining the node feature matrix and the edge feature matrix.

3. The method according to claim 1, characterized in that, The gating weight is calculated in the following way: ； in, Indicates the first The gating weights corresponding to each mode Indicates the first The features output by the modality feature extraction subnetwork are obtained from each modality. Indicates the relationship with the first The learnable weight vector corresponding to each modality Indicates the relationship with the first The learnable bias term corresponding to each modality This represents the total number of modes participating in the fusion.

4. The method according to claim 1, characterized in that, The cross-modal knowledge distillation branch includes a teacher model branch and a student model branch. The teacher model branch is pre-trained based on historical complete multimodal data. The step of generating substitute features corresponding to missing modalities includes: inputting the non-missing modal features into the student model branch, and having the teacher model branch guide the student model branch to output the substitute features.

5. The method according to claim 1, characterized in that, The joint optimization loss function is calculated in the following way: ； in, This indicates the value of the joint optimization loss function. This represents the data-driven loss. This represents the feature distribution alignment loss, used to constrain modal feature distributions from different sources to maintain consistency. The residual penalty term of the partial differential equation representing the physical law is obtained by substituting the intermediate output of the physical graph neural network into the partial differential equation operator. , , These represent the preset weight coefficients corresponding to the data-driven loss, the feature distribution alignment loss, and the residual penalty term of the physical law partial differential equation, respectively.

6. The method according to claim 1, characterized in that, The steps for asynchronous spatiotemporal alignment processing of the visual data, the temporal sensor data, and the knowledge base retrieval data include: The visual data, the temporal sensor data, and the knowledge base retrieval data are timestamped respectively to construct a unified timeline; Using the sampling period of one of the modal data as the reference period, a sliding time window mechanism is adopted to aggregate modal data with sampling periods shorter than the reference period within the time window, and to mark modal data without valid acquisition values ​​within the time window as missing.

7. A multi-source data processing system based on machine learning, characterized in that, include: The physical graph topology construction module is used to acquire CAD geometric model data and CAE simulation result data of the target device, extract partial differential equation boundary conditions and geometric constraint relationships from them, construct a physical graph topology structure carrying physical constraint information based on the partial differential equation boundary conditions and geometric constraint relationships, and input the physical graph topology structure into the physical graph neural network to obtain physical features; A multimodal feature extraction module is used to acquire visual data, temporal sensor data and knowledge base retrieval data of the target device, perform asynchronous spatiotemporal alignment processing on the visual data, the temporal sensor data and the knowledge base retrieval data to obtain aligned multimodal data, and input the aligned multimodal data into the corresponding modal feature extraction subnetwork to obtain multiple modal features; The dynamic gating fusion module is used to input the physical features and the multiple modal features into the dynamic routing gating network, so that the calculation of the gating weights depends on both the physical features and the corresponding modal features, to obtain the gating weights corresponding to each modal feature. If it is determined that there is a lack of modal data at the current time, the gating weights corresponding to the missing modal are reset to zero, and the cross-modal knowledge distillation branch is activated. The non-missing modal features are used as the input of the student model to generate the substitute features corresponding to the missing modal. If it is determined that there is no lack of modal data at the current time, the corresponding modal features are used as the features participating in the fusion. The features participating in the fusion are weighted and summed according to the gating weights to obtain the fused feature vector. The joint optimization training and inference module is used to calculate a joint optimization loss function based on the fused feature vector during the training phase. The joint optimization loss function includes data-driven loss, feature distribution alignment loss, and residual penalty term of physical law partial differential equations. Backpropagation is performed based on the joint optimization loss function to update network parameters. During the inference phase, the prediction result is output based on the fused feature vector.

8. The system according to claim 7, characterized in that, The physical graph topology construction module is specifically used for: The geometric units and their adjacency relationships in the CAD geometric model data are analyzed to generate an initial geometric topology graph; The boundary conditions of the partial differential equations are labeled on the corresponding geometric nodes or edges in the initial geometric topology graph to obtain a physical constraint graph with boundary attributes. Based on at least one of the curvature change rate and stress concentration degree, the nodes in the physical constraint diagram are scored for importance, and the nodes whose scores are within a preset sorting range and the edges connected to the nodes are retained to obtain a simplified topology diagram. The physical attributes of each node are encoded as node features, and the physical connection relationships represented by each edge are encoded as edge features, resulting in a node feature matrix and an edge feature matrix.

9. The system according to claim 7, characterized in that, The cross-modal knowledge distillation branch includes a teacher model branch and a student model branch. The teacher model branch is pre-trained based on historical complete multimodal data. When the dynamic gating fusion module generates the substitute features corresponding to the missing modalities, it inputs the non-missing modal features into the student model branch, and the teacher model branch guides the student model branch to output the substitute features.

10. The system according to claim 7, characterized in that, The joint optimization loss function is calculated in the following way: ； in, This indicates the value of the joint optimization loss function. This represents the data-driven loss. This represents the feature distribution alignment loss, used to constrain modal feature distributions from different sources to maintain consistency. The residual penalty term of the partial differential equation representing the physical law is obtained by substituting the intermediate output of the physical graph neural network into the partial differential equation operator. , , These represent the preset weight coefficients corresponding to the data-driven loss, the feature distribution alignment loss, and the residual penalty term of the physical law partial differential equation, respectively.

Images