Cross-domain multimodal industrial data extraction and fusion method based on transfer learning
Through the method of transfer learning, multimodal industrial data extraction and fusion are performed, the problem of cross-domain data application is solved, efficient and accurate data processing and fusion is achieved, and the adaptability and performance of the model is improved.
Patent Information
- Application Number
- CN202411235791.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The existing technology is difficult to achieve cross-domain multimodal industrial data extraction and fusion, and traditional machine learning methods are difficult to apply between different fields, and data isolation makes it difficult to share.
Using a transfer learning-based method, the standard data of multi-domain data sets is transferred to the target database through the data transfer model, and a multi-modal fusion model is constructed for data fusion. Standard data is generated using feature similarity and differential characteristics, and resource allocation and neural Architecture search technology to improve model performance.
It significantly improves the efficiency and accuracy of industrial data extraction and fusion, reduces data labeling costs, improves the generalization capabilities of the model, can run at high performance in real-time environments, and provides model interpretability.
Smart Images

Figure CN118733660B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data processing, and specifically relates to a cross-domain multimodal industrial data extraction and fusion method based on transfer learning. Background Art
[0002] In the current field of industrial data analysis, data extraction and fusion technologies in various industries have their own characteristics, and technical means in different fields are difficult to reuse directly. Traditional machine learning methods rely on data from a large number of fields for training, but in actual industrial applications, the differences between fields are huge, making it difficult for a single model to be universal. In addition, due to data availability and privacy policies, data from different industries are isolated and difficult to share. Therefore, a data processing technology that can be applied across fields and quickly adapt to new environments is needed.
[0003] Similar prior art includes a Chinese patent application with publication number CN117150031A, which discloses a processing method and system for multimodal data, a multimodal knowledge extraction method based on knowledge transfer, which adopts the idea of knowledge transfer and is based on the results of text modality extraction and integrates image and video modalities to jointly complete the extraction of entities and relationships. A cross-modal knowledge fusion method based on unified semantic representation maps the knowledge elements of each modality to a potential semantic subspace through a unified semantic representation learning method, and then links the multimodal knowledge to a multimodal knowledge graph through a cross-modal knowledge linking method. An analysis and reasoning method based on a multimodal knowledge graph uses multimodal knowledge as a driver to perform deep semantic analysis and understanding of multimodal data. There is also a Chinese patent application with publication number CN114692624A, which discloses an information extraction method, device and electronic device based on multi-task migration. The method includes: obtaining texts of different tasks, encoding the texts, and generating a semantic representation sequence of the texts; inputting the semantic representation sequence into a common feature extraction network, and outputting the common features of different tasks; inputting the semantic representation sequence into a private feature extraction network, and outputting the private features of different tasks; fusing the common features and private features, and obtaining the category information of each position in the sequence through sequence decoding to complete information extraction. This embodiment of the invention reduces the excessive reliance of the algorithm model on the amount of data in a specific information extraction task; reduces the manpower and time cost of manually annotating a large amount of data; and improves the efficiency of information extraction.
[0004] However, the above-mentioned prior art only analyzes and fuses semantic texts, and in actual situations, multimodal data processing is required. Summary of the invention
[0005] To solve the above problems, the present invention provides a cross-domain multimodal industrial data extraction and fusion method based on transfer learning to solve the problems in the prior art.
[0006] In order to achieve the above-mentioned invention object, the present invention proposes a cross-domain multimodal industrial data extraction and fusion method based on transfer learning, including:
[0007] S1: Collect industrial data from multiple fields, generate multiple data sets, and build a data extraction model, wherein the data extraction model obtains a feature set in the data set, extracts key information from the data set based on the feature set, and generates standard data based on the key information;
[0008] S2: setting the standard data corresponding to any data set as a training set, building a data migration model based on the training set, setting the key information of the corresponding data set in the target database as a target label, and using the data migration model to migrate the standard data corresponding to the source database to the target database based on the target label to generate migration data;
[0009] S3: Compare the data similarity between the source database and the same standard data in the target database to generate a first accuracy rate of the data migration model, correct the key information corresponding to the standard data in the remaining data set based on the first accuracy rate, and generate a data label;
[0010] S4: Construct a multimodal fusion model, and fuse the corresponding standard data in the target database with the migration data based on the multimodal fusion model to generate fused data. Repeat this step until the standard data corresponding to all data sets are fused into the target database to generate a multi-domain fused data set.
[0011] Furthermore, the generating of standard data based on the key information comprises the following steps:
[0012] Randomly extract two data sets from multiple data sets, set them as the first data set and the second data set respectively, obtain a first feature set corresponding to the first data set and a second feature set corresponding to the second data set, set both the first feature set and the second feature set as feature sets, compare feature similarities between all feature sets, extract similar features from the first feature set and the second feature set based on the feature similarities, set the remaining feature sets as difference features, input the similar features in the same feature set into a preset classification prediction model, output prediction labels corresponding to the same feature set, and generate restored data based on the prediction labels and the difference features in the same feature set by decoding calculation;
[0013] Repeat this step until all data sets are extracted, set all the predicted labels as the key information of the corresponding data sets, and set all the restored data as the standard data of the corresponding data sets in turn.
[0014] Furthermore, calculating the feature similarity comprises the following steps:
[0015] Establishing a training data set, the training data set includes target modality data and learning modality data, extracting data features of the training data set, the data features include target modality features and learning modality features, mapping the data features and multiple standard features to the same standard feature space, generating mapping features corresponding to the data features, calculating mapping errors based on the mapping features, establishing a mapping model based on the mapping errors, the mapping model maps the data features to a shared space, repeating this step until the mapping model maps the target modality features and the learning modality features to a minimum distance in the shared space;
[0016] All feature sets are mapped to the mapping model to generate multiple mapping points, a mapping distance between mapping points corresponding to any first feature set and any second feature set is calculated, and the reciprocal of the minimum mapping distance is set as the feature similarity between the feature sets.
[0017] Furthermore, the constructing of the data extraction model includes:
[0018] The data extraction model learns the classification errors between all the key information and the standard data based on the error back propagation method, generates adjustment parameters, adjusts the data extraction model based on the adjustment parameters, and repeats this step until the classification error is less than a first preset value to generate a new data extraction model, wherein the data correlation between the standard data and the corresponding data set is set as the classification error.
[0019] Furthermore, the construction of the multimodal fusion model includes the following steps:
[0020] A technical model of a multi-task learning framework is set up to simultaneously process data of multiple modalities based on the data generation type and optimize resource allocation. The optimal technical model configuration is automatically set through the deconstruction and dispersion strategy and neural architecture search technology of the technical model, and the attention mechanism and graph convolutional network are introduced. A meta-learning technology is set up to obtain the convergence index of the technical model. Based on the convergence index simulation and synthetic data generation technology, the technical model is trained, and the trained technical model is set as the multimodal fusion model.
[0021] Furthermore, generating the fused data comprises the following steps:
[0022] The standard data and the migration data contained in the target database are both set as data to be fused, the data generation type and time information of the data to be fused are extracted based on the multimodal fusion model, and the data to be fused is classified into a first type of data and a second type of data based on the time information. If the fused data has the time information, it is classified as the first type of data, otherwise, it is classified as the second type of data;
[0023] Generate latent variable distributions corresponding to the time information contained in the same data generation type in the first category of data based on a variational approximation method, mark the hidden state of the multimodal fusion model, update the hidden state based on the latent variable distribution, input the updated hidden state and the latent variable distribution into the multimodal fusion model, output result data corresponding to the latent variable distribution based on the neural architecture search technology, and set all the result data as the first fusion data corresponding to the first category of data;
[0024] The second category data is sequentially aggregated based on the data generation type and the key information to generate second fused data, and the first fused data and the second fused data are combined to be set as the fused data.
[0025] Furthermore, based on the pre-trained Transformer model as the starting point of transfer learning, the training set is input into the Transformer model, and the training generates an adapted neural network model, and the neural network model is set as the data migration model.
[0026] Furthermore, the generating of data labels comprises the following steps:
[0027] Based on database operation commands, a data difference value between the standard data in the source database and the migration data in the target database is generated and set as the data similarity. The data similarity is set to the first accuracy rate. If the first accuracy rate is greater than a second preset value, the data migration model generates the data label based on the target label prediction. Otherwise, the parameters of the data migration model are adjusted so that the first accuracy rate is greater than the second preset value.
[0028] Compared with the prior art, the beneficial effects of the present invention are at least as follows:
[0029] The technical solution of the present invention can significantly improve the efficiency and accuracy of industrial data extraction and fusion. The data migration model can quickly adapt to new fields on a small number of samples, which not only saves data annotation costs, but also improves the generalization ability of the model. Multimodal data processing and information fusion enable the multimodal fusion model to integrate multiple data characteristics and more accurately identify the characteristics of industrial processes. The optimization mechanism ensures the high performance of the model in a real-time environment, while the model interpretability tool enhances the interaction between the model and the industrial field. Overall, this solution provides a new and efficient technical approach for industrial data analysis, which has broad application prospects. The present invention can improve the accuracy of transfer learning of multiple data sets with different characteristics, while improving the integrity of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a flowchart of the steps of the cross-domain multimodal industrial data extraction and fusion method based on transfer learning of the present invention. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.
[0032] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.
[0033] like Figure 1 As shown in the figure, the cross-domain multimodal industrial data extraction and fusion method based on transfer learning includes:
[0034] S1: Collect industrial data from multiple fields, generate multiple data sets, build a data extraction model, the data extraction model obtains the feature set in the data set, extracts key information in the data set based on the feature set, and generates standard data based on the key information.
[0035] Specifically, in this embodiment, the data set is composed of industrial data corresponding to each field, wherein industrial data refers to various types of data generated, collected and used in the industrial field, including but not limited to sensor data, generated data, process parameters and design files. The data extraction model is a deep learning model that is used to automatically extract standard data from a data set to adapt to data input and output of multiple modalities. A feature set refers to a set of features extracted from a data set that is used to describe the properties of the data set. Key information is the class label generated by the feature set in the data set, and standard data refers to key data corresponding to the key information extracted from the data set. The extraction of standard data can improve the accuracy of subsequent transfer learning using multiple data sets with different features.
[0036] S2: Set the standard data corresponding to any data set as the training set, build a data migration model based on the training set, set the key information of the corresponding data set in the target database as the target label, and the data migration model migrates the standard data corresponding to the source database to the target database based on the target label to generate migration data.
[0037] Specifically, in this embodiment, the training set refers to the training data composed of standard data, and the data migration model is a neural network model generated by deep learning of the training set. The target database refers to the database corresponding to the destination of data migration, and the target label refers to the class label contained in the target database. Among them, data migration refers to the process of transferring data from one system or storage location to another system or storage location, which can be simply understood as transferring data from one database to another. The source database refers to the database corresponding to the data to be migrated, and the migration data refers to the data of the standard data to be migrated in the source database after it is migrated to the target database. Since different databases have different data structures, there may be data changes after data migration based on the data migration model.
[0038] S3: Compare the data similarity between the same standard data in the source database and the target database, generate a first accuracy rate of the data migration model, correct the key information corresponding to the standard data in the remaining data set based on the first accuracy rate, and generate a data label.
[0039] Specifically, comparing the data similarity between the same standard data refers to comparing the data similarity before and after data migration, where data similarity refers to the data difference value, which can be calculated according to the database operation instruction. The first accuracy rate is a numerical value corresponding to the migration effect of the data migration model used to quantify the first accuracy rate, which indicates that the effect of the data migration model is better. The data label refers to the class label generated after the key information of the standard data in the remaining data set is corrected by the data migration model.
[0040] S4: Construct a multimodal fusion model. Based on the multimodal fusion model, fuse the corresponding standard data in the target database with the migration data to generate fused data. Repeat this step until the standard data corresponding to all data sets are fused into the target database to generate a multi-domain fused data set.
[0041] Specifically, in this embodiment, the multimodal fusion model is a model that integrates information processing and analysis from different data sources or modalities (such as text, image, sound, video, etc.). The target database contains the original standard data and the migrated data after data migration. Therefore, data fusion can be performed based on the multimodal fusion model to generate fused data. Repeat steps S1 to S4 so that the standard data corresponding to all data sets are fused into the target database, and all fused data are set as fused data sets.
[0042] As a preferred technical solution of the present invention, generating standard data based on key information includes the following steps:
[0043] Two data sets are randomly selected from multiple data sets, and are set as the first data set and the second data set respectively. A first feature set corresponding to the first data set and a second feature set corresponding to the second data set are obtained, and the first feature set and the second feature set are both set as feature sets. The feature similarities between all feature sets are compared, and similar features are extracted from the first feature set and the second feature set based on the feature similarities, and the remaining feature sets are set as difference features. Similar features in the same feature set are input into a preset classification prediction model, and prediction labels corresponding to the same feature set are output. The restored data is generated based on the prediction labels and the difference features in the same feature set.
[0044] Repeat this step until all data sets are extracted, set all predicted labels as the key information of the corresponding data sets, and set all restored data as the standard data of the corresponding data sets in turn.
[0045] Specifically, in this embodiment, the first feature set refers to the distribution of eigenvalues contained in the first data set. First, the feature extraction of the data set is performed to generate feature quantities, and then the PCA principal component analysis method is used to perform dimensionality reduction processing on the feature quantities to generate the distribution of multiple feature quantities, namely the first feature set and the second feature set. Feature similarity refers to the measure of the similarity between two or more features (such as vectors, sets or objects) in data analysis and machine learning. Since the data set contains data of different modalities, feature similarity needs to be analyzed in combination with the feature quantities corresponding to the multimodal data. The calculation of feature similarity is described later. Similar features refer to feature quantities in the feature set whose feature similarity is greater than a preset similarity threshold. For example, the first feature set is W1, W2, W3 and W4, and the second feature set is M1, M2, M3 and M4, where W1 is similar to M2, W3 is similar to M1, and M2 is similar to M4. In the feature set, W1, W3, M1, M2 and M4 are all similar features, and the remaining W2, W4 and M3 are all difference features. The classification prediction model is a feature classification model used for label prediction. It belongs to a machine learning model and can also restore and generate data through label information and feature quantities, that is, restored data. The restored data refers to the data restored and generated corresponding to the difference features. For example, the difference features in the first feature set are W2 and W4, and the predicted labels corresponding to the similar features W1 and W3 are F1 and F2, respectively. The difference feature W2 and the predicted labels F1 and F2 are decoded and calculated by the classification prediction model to generate restored data D1, and the difference feature W4 and the predicted labels F1 and F2 are decoded and calculated by the classification prediction model to generate restored data D2. The restored data corresponding to the first feature set is D1 and D2.
[0046] This step can calculate the restored data corresponding to all feature sets, and set all the prediction labels contained in the feature sets as the key information of the data set to which the feature sets belong, and set all the restored data corresponding to the feature sets as the standard data of the data set to which the feature sets belong. This step can accurately extract the key data of multiple data sets with different properties.
[0047] As a preferred technical solution of the present invention, calculating feature similarity includes the following steps:
[0048] A training data set is established, the training data set includes target modal data and learning modal data, data features of the training data set are extracted, the data features include target modal features and learning modal features, the data features and multiple standard features are mapped to the same standard feature space, mapping features corresponding to the data features are generated, mapping errors are calculated based on the mapping features, a mapping model is established based on the mapping errors, the mapping model maps the data features to a shared space, and this step is repeated until the mapping model maps the target modal features and the learning modal features respectively to a minimum distance in the shared space.
[0049] All feature sets are mapped to a mapping model to generate multiple mapping points, a mapping distance between mapping points corresponding to any first feature set and any second feature set is calculated, and the inverse of the minimum mapping distance is set as the feature similarity between the feature sets.
[0050] Specifically, in this embodiment, the training data set is a data sample containing different modal types, including but not limited to image samples, character samples, sound samples, and video samples, etc., and it is necessary to establish multimodal feature recognition and feature similarity calculation through the training data set. The training data set is split into a validation set and a learning training set, that is, target modal data and learning modal data, and data features are extracted from the target modal data and the learning modal data respectively. The standard feature refers to the standard feature set for each modal data, and the standard feature space refers to the feature space framework constructed by the standard feature, in which the features of different modalities are converted or mapped so that they can be compared and analyzed on the same dimension and scale. The mapped feature refers to the feature formed in the standard feature space after the data feature is projected and mapped. Feature error refers to the accuracy of mapping features generated by data feature mapping. The mapping parameters are adjusted by the mapping error to generate a mapping model, wherein the mapping model is a mapping recognition model of data features generated by mapping parameters, which can effectively identify the modal type of data features and map the data features to a shared space, wherein the shared space is a feature space framework for comparing data features of any identical data modality, and the mapping parameters of the mapping model are continuously adjusted until the distance between the mapped target modal features and the learning modal features in the shared space is minimized. The distance calculation method can use the Chebyshev distance calculation method. Thus, the mapping model training is completed.
[0051] After the feature set is input into the mapping model, multiple mapping points are generated in the shared space, and the mapping distances between all mapping points corresponding to the first feature set and the second feature set are compared. The mapping distance can be obtained by the Chebyshev distance calculation method. The smaller the mapping distance, the more similar the corresponding first feature set and the second feature set are. Therefore, the minimum mapping distance is set to the feature similarity between the feature sets corresponding to two different data sets.
[0052] As a preferred technical solution of the present invention, constructing a data extraction model includes:
[0053] The data extraction model learns the classification error between all key information and standard data based on the error back propagation method, generates adjustment parameters, adjusts the data extraction model based on the adjustment parameters, and repeats this step until the classification error is less than a first preset value to generate a new data extraction model, wherein the data correlation between the standard data and the corresponding data set is set as the classification error.
[0054] Specifically, in this embodiment, the error back propagation method is a supervised learning algorithm used when training a multi-layer feedforward neural network, especially for training a multi-layer perceptron with at least one hidden layer. This method updates the network weights by calculating the gradient of the loss function with respect to the network parameters, thereby minimizing the prediction error of the network. Classification error refers to the error between the predicted label and the actual label that may exist when the data extraction model extracts standard data and corresponding key information. The classification error is generated by error back propagation, and an adjustment parameter is generated based on the classification error, that is, the adjustment parameter of the data extraction model. The parameters of the data extraction model are repeatedly adjusted until the classification error is less than the first preset value. Data correlation is used to measure the strength of the linear relationship between two or more data variables, and can be obtained through the calculation formula of the Pearson correlation coefficient.
[0055] As a preferred technical solution of the present invention, constructing a multimodal fusion model includes the following steps:
[0056] Set up a technical model of a multi-task learning framework, process data of multiple modalities simultaneously based on the data generation type, and optimize resource allocation. Automatically set the optimal technical model configuration through the deconstruction and dispersion strategy of the technical model and the neural architecture search technology. Introduce the attention mechanism and graph convolutional network, and set up meta-learning technology to obtain the convergence index of the technical model. Based on the convergence index simulation and synthetic data generation technology, train the technical model and set the trained technical model as a multimodal fusion model.
[0057] Specifically, in this embodiment, the data generation type includes sensor data and non-sensor data, wherein sensor data refers to data generated by sensor collection, and non-sensor data is collected by other means. The technical model is a learning framework for building a multimodal fusion model. Deconstructing decentralized strategies refers to in-depth analysis of existing structures or strategies to identify their core components and potential weaknesses. Neural Architecture Search (NAS) technology is an automated machine learning process for designing and optimizing the structure of artificial neural networks. NAS aims to automate the process of selecting network layers, number of neurons, connection patterns, activation functions, and other hyperparameters to improve the performance and efficiency of the model. Technical model configuration refers to the parameters contained in the technical model. The attention mechanism is a technology that simulates human attention in a deep learning model, which allows the model to focus on specific parts of the input data when processing information. Graph convolutional network is a deep learning model specifically designed to process graph structured data. Meta-learning technology is a technology in the field of machine learning that enables technical models to extract knowledge from past learning experiences, thereby accelerating the learning process and improving performance on new and related tasks. Convergence index is usually used to describe the speed at which an optimization algorithm or iterative algorithm converges to an optimal solution or stable state in the process of finding a solution to a problem. Simulation and synthetic data generation techniques are methods to create artificial datasets that can be used to train technical models to produce multimodal fusion models.
[0058] As a preferred technical solution of the present invention, generating fusion data includes the following steps:
[0059] The standard data and migration data contained in the target database are set as the data to be fused. The data generation type and time information of the data to be fused are extracted based on the multimodal fusion model. The data to be fused is classified into the first category of data and the second category of data based on the time information. If the fused data has time information, it is classified as the first category of data, otherwise, it is classified as the second category of data.
[0060] In the first category of data, based on the variational approximation method, latent variable distributions corresponding to the time information contained in the same data generation type are generated respectively, the hidden states of the multimodal fusion model are marked, and the hidden states are updated based on the latent variable distributions. The updated hidden states and latent variable distributions are input into the multimodal fusion model, and the result data corresponding to the latent variable distributions are output based on the neural architecture search technology, and all result data are set as the first fusion data corresponding to the first category of data.
[0061] The second type of data is sequentially aggregated based on the data generation type and the key information to generate second fused data, and the first fused data and the second fused data are combined to be set as fused data.
[0062] Specifically, in this embodiment, different data generation types will contain different time information, for example, sensor data will be associated with a time series, and time information refers to time features, for example, the time point of data collection. Therefore, the first type of data refers to data with time features in the data to be fused, and the second type of data refers to data without time features in the data to be fused.
[0063] Variational approximation methods are a type of technology used in statistics, machine learning, and signal processing to approximate complex probability distributions or solve optimization problems. They can simplify the calculation process by finding a simpler distribution to approximate the target distribution, that is, to generate a latent variable distribution. Hidden state refers to the internal state of the recursive neural network contained in the multimodal fusion model that is used to represent the time dependency or contextual information in the time series data. When processing the time series data corresponding to the first type of data, it will be iteratively updated, that is, the hidden state is updated. The multimodal fusion model uses the updated hidden state and latent variable distribution, and the neural architecture search technology fuses all the first type of data to output the result data, which is the first fused data.
[0064] The second type of data can be aggregated and integrated with the same key information to generate second fused data. The fused data includes the first fused data and the second fused data, and the integrity of the generated fused data can be improved through the data generation type.
[0065] As a preferred technical solution of the present invention, a pre-trained Transformer model is used as the starting point of transfer learning, the training set is input into the Transformer model, the training generates an adapted neural network model, and the neural network model is set as a data migration model.
[0066] Specifically, in this embodiment, the Transformer model is a deep learning architecture model based on the self-attention mechanism. A neural network model adapted to the Transformer model is generated from a training set, thereby generating a data migration model.
[0067] As a preferred technical solution of the present invention, generating data tags includes the following steps:
[0068] Based on the database operation command, the data difference value between the standard data in the source database and the migrated data in the target database is generated and set as the data similarity. The data similarity is set to a first accuracy rate. If the first accuracy rate is greater than a second preset value, the data migration model generates a data label based on the target label prediction. Otherwise, the parameters of the data migration model are adjusted so that the first accuracy rate is greater than the second preset value.
[0069] Specifically, in this embodiment, the database operation command refers to the use of structured query language (SQL) to perform various operations, including but not limited to querying, inserting, updating and deleting data. The data difference value refers to the difference between two sets of data, which can be calculated by a comparison formula, and the comparison formula is: , where K is the data difference value, g is the number of identical data between the migration data and the standard data, and G is the total number of data contained in the standard data. Thus, data similarity and a first accuracy rate can be generated. If the first accuracy rate is less than or equal to a second preset value, it means that data omissions exist when the data migration model performs data migration. Therefore, the parameters of the data migration model need to be adjusted until the first accuracy rate is greater than the second preset value.
[0070] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.
[0071] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0072] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0073] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
[0074] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. Cross-domain multimodal industrial data extraction and fusion method based on transfer learning, It is characterized in that The method comprises the following steps: S1: Collect industrial data from multiple fields, generate multiple data sets, and build a data extraction model, wherein the data extraction model obtains a feature set in the data set, extracts key information from the data set based on the feature set, and generates standard data based on the key information; S2: The standard data corresponding to any data set is set as a training set, a data migration model is constructed based on the training set, the key information of the corresponding data set in the target database is set as a target label, and the data migration model migrates the standard data corresponding to the source database to the target database based on the target label to generate migration data, wherein the pre-trained Transformer model is used as the starting point of migration learning, the training set is input into the Transformer model, an adapted neural network model is generated through training, and the neural network model is set as the data migration model; S3: Compare the data similarity between the source database and the same standard data in the target database to generate a first accuracy rate of the data migration model, correct the key information corresponding to the standard data in the remaining data set based on the first accuracy rate, and generate a data label; S4: construct a multimodal fusion model, and fuse the corresponding standard data in the target database with the migration data based on the multimodal fusion model to generate fused data, and repeat this step until the standard data corresponding to all data sets are fused into the target database to generate a fused data set in multiple fields; Wherein, the construction of the multimodal fusion model includes the following steps: A technical model of a multi-task learning framework is set up to simultaneously process data of multiple modalities based on the data generation type and optimize resource allocation. The optimal technical model configuration is automatically set through the deconstruction and dispersion strategy of the technical model and the neural architecture search technology, and the attention mechanism and graph convolution network are introduced. A meta-learning technology is set up to obtain the convergence index of the technical model. Based on the convergence index simulation and synthetic data generation technology, the technical model is trained, and the trained technical model is set as the multimodal fusion model. Generating the fused data comprises the following steps: The standard data and the migration data contained in the target database are both set as data to be fused, the data generation type and time information of the data to be fused are extracted based on the multimodal fusion model, and the data to be fused is classified into a first type of data and a second type of data based on the time information. If the fused data has the time information, it is classified as the first type of data, otherwise, it is classified as the second type of data; Generate latent variable distributions corresponding to the time information contained in the same data generation type in the first category of data based on a variational approximation method, mark the hidden state of the multimodal fusion model, update the hidden state based on the latent variable distribution, input the updated hidden state and the latent variable distribution into the multimodal fusion model, output result data corresponding to the latent variable distribution based on the neural architecture search technology, and set all the result data as the first fusion data corresponding to the first category of data; The second category data is sequentially aggregated based on the data generation type and the key information to generate second fused data, and the first fused data and the second fused data are combined to be set as the fused data.
2. The method according to claim 1, It is characterized in that Generating standard data based on the key information comprises the following steps: Randomly extract two data sets from multiple data sets, set them as the first data set and the second data set respectively, obtain a first feature set corresponding to the first data set and a second feature set corresponding to the second data set, set both the first feature set and the second feature set as feature sets, compare feature similarities between all feature sets, extract similar features from the first feature set and the second feature set based on the feature similarities, set the remaining feature sets as difference features, input the similar features in the same feature set into a preset classification prediction model, output prediction labels corresponding to the same feature set, and generate restored data based on the prediction labels and the difference features in the same feature set by decoding calculation; Repeat this step until all data sets are extracted, set all the predicted labels as the key information of the corresponding data sets, and set all the restored data as the standard data of the corresponding data sets in turn.
3. The method according to claim 2, It is characterized in that Calculating the feature similarity includes the following steps: Establishing a training data set, the training data set includes target modality data and learning modality data, extracting data features of the training data set, the data features include target modality features and learning modality features, mapping the data features and multiple standard features to the same standard feature space, generating mapping features corresponding to the data features, calculating mapping errors based on the mapping features, establishing a mapping model based on the mapping errors, the mapping model maps the data features to a shared space, repeating this step until the mapping model maps the target modality features and the learning modality features to a minimum distance in the shared space; All feature sets are mapped to the mapping model to generate multiple mapping points, a mapping distance between mapping points corresponding to any first feature set and any second feature set is calculated, and the reciprocal of the minimum mapping distance is set as the feature similarity between the feature sets.
4. The method according to claim 2, It is characterized in that The constructing of the data extraction model comprises: The data extraction model learns the classification errors between all the key information and the standard data based on the error back propagation method, generates adjustment parameters, adjusts the data extraction model based on the adjustment parameters, and repeats this step until the classification error is less than a first preset value to generate a new data extraction model, wherein the data correlation between the standard data and the corresponding data set is set as the classification error.
5. The method according to claim 1, It is characterized in that Generating data tags comprises the following steps: Based on database operation commands, a data difference value between the standard data in the source database and the migration data in the target database is generated and set as the data similarity. The data similarity is set to the first accuracy rate. If the first accuracy rate is greater than a second preset value, the data migration model generates the data label based on the target label prediction. Otherwise, the parameters of the data migration model are adjusted so that the first accuracy rate is greater than the second preset value.
Citation Information
Patent Citations
Information extraction method and device based on multi-task migration and electronic equipment
CN114692624A
Processing method and system for multi-modal data
CN117150031A
Multi-mode bearing fault intelligent diagnosis method based on transfer learning
CN116401603A
Big data cross-domain and cross-platform application method based on transfer learning
CN117668458A
Training method and system of small classification model based on cross-modal migration knowledge data
CN118035794A
