A multimodal industrial data fusion method and system for discrete manufacturing

Abstract

The present invention discloses a multimodal industrial data fusion method and system for discrete manufacturing. The method collects multimodal data of discrete manufacturing industry in real time and extracts features to obtain feature data, inputs the feature data into a differentiable computer network to obtain data fusion results, and the differentiable computer is also used to complete the prediction target according to the data fusion results; the differentiable computer network includes a data fusion module and a memory system, the data fusion module fuses the feature data in an intelligent body, and the memory system is used to store and read and write the data after the intelligent body is fused; the data in the memory system is input into an evaluation system to calculate a reward value, and the data fusion of the intelligent body is guided according to the reward value and the memory system is updated. The present invention does not increase the training parameters while increasing the capacity of the memory system, and fuses complex multimodal industrial data to facilitate decision-making in the discrete manufacturing industry.

Description

Technical Field

[0001] The present invention relates to a data fusion method and system, in particular to a multi-modal industrial data fusion method and system for discrete manufacturing. Background Art

[0002] The sensor technology widely used in discrete manufacturing has generated a large amount of data. These data sources cover all aspects of the production environment, including equipment operating status, temperature, pressure, humidity and other information. This data diversity and large scale provide a rich source of information for data fusion. With the rapid development of Internet of Things technology, the correlation between devices and systems has been greatly improved. This means that data from various devices and systems can be collected, transmitted and integrated more easily. This correlation improves the monitoring, optimization and control of the production process, thereby improving production efficiency and quality levels.

[0003] The digital transformation of factories is becoming a major trend in the discrete manufacturing industry. This means that the amount of data involved in the production environment is increasing. From data obtained by sensors to real-time monitoring data of production processes and information related to the supply chain, data is everywhere, and data fusion has become a key way to integrate, analyze and apply these multi-source data.

[0004] Multimodal data fusion technology This technology involves integrating multimodal data from different sensors or data sources. These data may include different types of data such as images, videos, sounds, texts, etc., and more comprehensive information is provided by fusing these data sources. Since there may be quality differences among the various data sources, including data inconsistency, missing values, errors or noise, data fusion technology may be affected by these data quality issues, resulting in inaccurate or unreliable fusion results. At the same time, existing data fusion technology cannot meet the needs of real-time processing and decision-making. Especially in industrial applications that require fast response, processing and fusing large amounts of data may cause delays, affecting the effectiveness of real-time decision-making. In addition, traditional data fusion technology cannot adapt to the ever-changing industrial environment and is not flexible enough for situations where data changes frequently and is highly complex. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a multimodal industrial data fusion method and system for discrete manufacturing, which can dynamically handle complex environmental problems through interactive learning of the environment and a large amount of data, and maximize the optimization of data structure and dimension through the reward form of a memory-adaptive discrete manufacturing environment, obtain data fusion results, and then guide discrete manufacturing decision makers.

[0006] Technical solution: The multimodal industrial data fusion method for discrete manufacturing described in the present invention comprises the following steps:

[0007] Collect discrete manufacturing industrial data in real time and extract features to obtain feature data, input the feature data into a differentiable computer network to obtain a data fusion result, and the differentiable computer is also used to complete the prediction target according to the data fusion result;

[0008] The discrete manufacturing industry data includes video, text, image and sound data;

[0009] The differentiable computer network includes a data fusion module and a memory system. The data fusion module fuses the feature data in the intelligent body, and the memory system is used to store and read and write the fused data of the intelligent body; the data in the memory system is input into the evaluation system to calculate the reward value, and the data fusion of the intelligent body is guided according to the reward value and the memory system is updated.

[0010] Furthermore, the real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data includes: extracting semantic features of text data, extracting image features of image data, extracting frame features of video data, and extracting spectral features of sound data.

[0011] Further, the semantic features of the text data are extracted, the text is converted into a set of words, a weight is assigned to each word, and the weight is set according to the frequency of the word in the text and the distribution in the corpus; the words are mapped to a high-dimensional space to obtain the semantic features of the text data;

[0012] Extract image features from image data, use the Canny algorithm to extract image edges, and use convolutional neural networks to extract image features;

[0013] Extract frame features of video data, extract key frames from the video, and use convolutional neural networks to extract frame features;

[0014] Extract the spectral features of sound data, extract the spectrum of sound data, and use convolutional neural network to extract spectral features.

[0015] Furthermore, the data fusion module fuses the feature data in the agent including:

[0016] At time t, the controller receives the feature data input vector x t , in the memory system's storage matrix M t-1 Get R read vectors in After being processed by the agent network, the output vector y is obtained t ;

[0017] At each moment, the agent network calculates a network output vector v t and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, is the output of the Lth layer of the agent at time t, w y and w ξ is the tap coefficient;

[0018] The agent passes information back by creating a loop in the calculation, thereby obtaining v t and t ; The output vector of the agent is: w t Read weights.

[0019] Furthermore, the controller is a long short-term memory network.

[0020] Furthermore, in the evaluation system, the error between the prediction of the predicted target by the calculation model and the actual target is calculated, and the prediction error is used as a reward value to guide the training of the differentiable computer network.

[0021] Furthermore, in the evaluation system, the game combination weight of the reward value is calculated, a weighted sum is performed according to the game combination weight to obtain a weighted reward value, and the parameters of the differentiable computer network are adjusted according to the weighted reward value.

[0022] The multimodal industrial data fusion system for discrete manufacturing described in the present invention includes:

[0023] A feature extraction unit, used for real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data; the discrete manufacturing industrial data includes video, text, image and sound data;

[0024] A data fusion unit is used to input the feature data into a differentiable computer network to obtain a data fusion result. The differentiable computer is also used to complete the prediction target according to the data fusion result. The differentiable computer network includes a data fusion module and a memory system. The data fusion module fuses the feature data in the intelligent body. The memory system is used to store and read and write the fused data of the intelligent body.

[0025] The evaluation unit is used to calculate the reward value according to the data in the memory system, and guide the data fusion of the intelligent agent and update the memory system according to the reward value.

[0026] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and is characterized in that when the computer program is loaded into the processor, the multimodal industrial data fusion method for discrete manufacturing is implemented.

[0027] The computer-readable storage medium of the present invention stores a computer program, and is characterized in that when the computer program is executed by a processor, the multimodal industrial data fusion method for discrete manufacturing is implemented.

[0028] Beneficial effect: Compared with the prior art, the advantages of the present invention are: the present invention fuses complex multimodal data collected in real time, and the memory-enhanced neural network of the present invention separates the grid training parameters from the memory system capacity, thereby increasing the capacity of the memory system without causing an increase in the training parameters, thus reducing a large amount of training time and facilitating the discrete manufacturing industry to save more time for decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flow chart of the method of the present invention.

[0030] Figure 2 This is a structural diagram of the DNC model of the present invention.

[0031] Figure 3 Schematic diagram of the intelligent agent computing process of the present invention.

[0032] Figure 4 It is a schematic diagram of the read and write operation flow of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0034] Common memory systems include long short-term memory networks (LSTM) and gated recurrent units (GRU). In order to enhance the ability of intelligent agents to cope with complex environments, existing technologies often increase the memory capacity by increasing the number of LSTM and GRU. However, the number of network training parameters will increase sharply as the capacity of the memory system increases, making training difficult.

[0035] The present invention separates the memory system from the parameter training process and proposes the use of a memory-enhanced neural network for multimodal data fusion. The agent reads or writes the required information from the memory system and executes the decision. At the same time, the multimodal data fusion system gives feedback to the actions performed by the agent in the form of rewards. This process is performed multiple times to ultimately maximize the reward.

[0036] Differentiable Computer (DNC) is a neural network architecture with external memory, which combines the learning ability of neural network with the storage capacity of external memory. Data fusion using DNC can be performed in the following steps: Collect the remaining featurized data sets in the multimodal data module that need to be fused, ensure that the features of the data set are available and clean, and data preprocessing is a key step to ensure data quality; Use DNC implementation to determine the input and output of the model. The input will be the data to be fused, and the output is the result of the integration of the data. Train the DNC model using the prepared data set. During training, the model will learn how to effectively fuse data from different sources. Evaluate the performance of the model using the validation data set. This helps to check whether the model can fuse the data correctly and whether it has good generalization ability. Debug and improve the model based on the validation results.

[0037] like Figure 1 As shown, the multimodal industrial data fusion method for discrete manufacturing described in the present invention specifically includes the following steps.

[0038] S1, the discrete manufacturing industrial data collected in real time from discrete manufacturing workshops such as machining workshops is transmitted to the multimodal data module for characterization.

[0039] S2, transmits the characterized data in the multimodal data module to the DNC. The DNC includes a discrete manufacturing industry data fusion module and a memory system, which respectively perform discrete manufacturing industry data fusion and memory reading and writing, and output the data fusion result.

[0040] S2-1, input the data of the memory system into the evaluation system, evaluate it according to traditional evaluation indicators, and feed the data of the memory system back to the multimodal data module;

[0041] S2-2, based on the reward feedback from the evaluation, the industrial data fusion module of the discrete machinery manufacturing industrial workshop is fed back to guide the data fusion, so as to continuously update the memory system.

[0042] In step S1, various types of industrial data of discrete manufacturing industry in discrete manufacturing machinery workshops collected in real time (including such as mechanical processing workshops: text data recorded in real-time workshops, video data under monitoring, audio data recording machine status, and picture data generated during workshop production) are characterized. The characterization steps are as follows:

[0043] ①Convert the text into a collection of words, regardless of the order of the words, and give each word a weight that takes into account the frequency of the word in the document and its distribution in the corpus. Map the words to vectors in a high-dimensional space, preserving the semantic and contextual information of the words.

[0044] ② Describe the distribution of colors in the image, use the Canny algorithm to extract image edges, and use the convolutional neural network (CNN) to automatically extract image features.

[0045] ③ Extract key frames from the video to describe the movement of pixels between different frames. Combine time and space information and use the CNN deep learning model to extract features.

[0046] ④Describe the spectral characteristics of audio signals, which are widely used in speech recognition. Use CNN to extract audio features.

[0047] In step S2, since the traditional artificial neural network (ANN) integrates calculation and storage when processing data, DNC can be regarded as a combination of ANN and external storage matrix, consisting of a controller and a memory system, and DNC can selectively read and write to the storage, and iteratively modify the stored data. Using the storage matrix of the memory system, DNC can obtain reasoning information, store important data in the reasoning process, and improve reasoning efficiency by simulating the reasoning process of the human brain.

[0048] The core of DNC is the controller, which is responsible for processing task-related calculations. After memory reading, the controller network reads information from external memory. The external memory of DNC is readable and writable, similar to computer memory. This step enables DNC to access previously stored information and thus fuse historical information. The neural network then generates an output, including a prediction or decision for the task. This output is guided by the controller network to make full use of the information in memory. Then, based on the output of the task, the controller network may write some information to the external memory. This process allows DNC to store key information of the task for use in future tasks. Finally, backpropagation is performed to adjust the parameters of the neural network through the output of DNC and the actual results. This ensures that DNC ​​learns how to use the information in external memory more effectively. Through this cycle, DNC can continuously extract key information from the input data, integrate it into the memory system, and then output it according to the task requirements. This method of combining memory and neural networks allows DNC to perform better than traditional neural networks when dealing with tasks with long-term dependencies.

[0049] The DNC of the present invention includes a data fusion module and a memory system, such as Figure 2 As shown, the data fusion module obtains the fusion result by continuously updating the agent. At time t, the controller receives the input vector x t , in the memory system's storage matrix M t-1 Get R read vectors in After being processed by the agent network, the output vector y is obtained t .like Figure 3 Shown is a schematic diagram of the agent's calculation process.

[0050] The present invention uses the LSTM network as the controller network. At time t, the output value of the agent layer l is

[0051] At each moment, the agent network calculates a network output vector ν t and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, w y and w ξ is the tap coefficient:

[0052]

[0053] The agent passes information back by creating a loop in the computational graph, thereby obtaining v t etc.; finally, the output vector of the agent is w t Read weights.

[0054] This designed agent regulates its output decisions by strengthening its reliance on the memory system's storage matrix.

[0055] The agent operates the data in the memory system through the read-write head, such as Figure 4 The following is a schematic diagram of the read / write head operation; the read and write positions are determined by the corresponding weights, and the set of weights allowed at N positions is R N The nonnegative quadrant of the standard simplex in:

[0056]

[0057] In read operations, multiple read weights, It is used to calculate the weighted average of the content, so the read vector is defined as:

[0058]

[0059] In a write operation, write weight Combined with the erase vector e t and write vector v t , modify the memory system storage matrix:

[0060]

[0061] in: It represents the Hadamard product, which is the product of corresponding elements of two matrices of the same order; E is an N×M matrix of all ones.

[0062] The addressing mechanism of DNC is a combination of multiple methods: when writing data in the memory system, content-based addressing and dynamic addressing are used; when reading data in the memory system, content-based addressing and sequential memory links are used to obtain the location.

[0063] In steps S2-1 and S2-2, a prediction target needs to be determined first, which can be part of the task that the model needs to learn. This target is usually related to the main goal of the task, but may be easier to obtain from the data. Use the existing data to train the DNC model to predict the prediction target as accurately as possible. This can be done through supervised learning, where the labels of the existing data or other available information are used to guide the learning of the model.

[0064] During training, the error between the model’s prediction of the target and the actual target is calculated. This can be achieved using various loss functions, such as w m Minimize deviation. Use the prediction error as a reward value to guide the training of the model. Usually, lower prediction errors are regarded as higher rewards, while higher errors are regarded as lower rewards or penalties. This reward value can be mapped to a specific reward range through some transformation or normalization to ensure the stability and controllability of the reward value. Use the obtained reward value to optimize the model parameters to minimize the prediction error and improve the accuracy of the prediction target. This usually involves using gradient descent or other optimization algorithms to update the model parameters and adjust the parameters according to the direction of the reward value.

[0065] In order to eliminate the difference in the final evaluation results caused by different dimensions, the positive and negative reward values ​​should be processed separately when conducting data evaluation:

[0066] Any linear combination of weight vectors is:

[0067]

[0068] Among them, β m is the linear combination coefficient And β m >0.

[0069] Optimize the linear combination coefficients, that is, w and each w m Minimize the deviation, that is:

[0070]

[0071] According to the properties of matrix differential, the first-order derivative condition of the above formula is obtained as follows:

[0072]

[0073] Its linear equation system is:

[0074]

[0075] β m The values ​​are normalized:

[0076]

[0077] The final game combination weight is a method of combining multiple evaluation criteria or factors in decision analysis. It is used in multi-criteria decision-making problems, especially when there are conflicting goals or criteria. This method borrows ideas from game theory and finds the best compromise by assigning weights to different criteria or factors. The game combination weights are as follows:

[0078]

[0079] Compare the weighted total scores of each solution and select the solution with the highest score as the best choice. Finally, the required evaluation feedback is given to the fusion module, and key features are extracted from the data collected by the evaluation system. These features should be able to represent the core indicators of the evaluation system, and the extracted features are integrated with the existing manufacturing data. Through continuous evaluation, data collection, analysis and feedback, a closed-loop optimization system is formed to continuously improve the manufacturing process.

[0080] Taking the automobile parts manufacturing workshop as an example, the production process on the production line is monitored by monitoring cameras, sensors and sound collection equipment. The data generated by these devices include different types of data such as images, videos, sounds and texts.

[0081] Surveillance cameras: Real-time video of the parts assembly process on the production line.

[0082] Sensors: record physical parameters such as temperature, humidity, pressure, vibration, etc. on the production line.

[0083] Sound collection equipment: records abnormal sounds during the production process, machine operation sounds, etc.

[0084] Text data: such as workers’ log records, equipment maintenance reports, etc.

[0085] The data fusion method of the workshop includes the following steps.

[0086] (1) Data integration: Integrate data from different devices into a unified data platform, ensure uniform data format, and handle missing values ​​and outliers.

[0087] (2) Feature extraction: Extract features from the integrated data to extract features related to production efficiency, product quality, and safety. For image and video data, computer vision technology can be used to extract information such as objects in the image and the status of the production line. For sound data, information such as sound spectrum features and frequency distribution can be extracted. For text data, natural language processing can be performed to extract keywords, sentiment analysis, and other information.

[0088] (3) Model building: Building a DNC model. DNC is a neural network architecture with external memory, which combines the learning ability of neural networks with the storage capacity of external memory. Data fusion using DNC can be performed through the following steps: Data preparation: Collect the data sets that need to be fused. These data can come from different sources and have different features or formats. Ensure that the features of the data set are available and clean. Data preprocessing is a key step to ensure data quality. The model can predict abnormal conditions on the production line, optimize production plans, or perform real-time monitoring and early warning based on the status of the production line.

[0089] (4) Model evaluation and optimization: Use historical data to evaluate and tune the established model to ensure that the model has good generalization capabilities. Techniques such as cross-validation can be used to evaluate the performance of the model and adjust and optimize it as needed.

[0090] (5) Real-time monitoring and feedback: Based on the monitoring results, timely adjust the production plan or equipment parameters to achieve optimization and control of the production process. Deploy the established model to the real-time production environment, and establish a real-time monitoring system to monitor the status of the production line. Through such a data fusion solution, automotive parts manufacturing workshops can better utilize multi-source data to optimize production processes, improve product quality and safety, thereby reducing production costs and enhancing competitiveness.

[0091] The multimodal industrial data fusion system for discrete manufacturing described in the present invention includes:

[0092] A feature extraction unit, used for real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data; the discrete manufacturing industrial data includes video, text, image and sound data;

[0093] A data fusion unit is used to input the feature data into a differentiable computer network to obtain a data fusion result. The differentiable computer is also used to complete the prediction target according to the data fusion result. The differentiable computer network includes a data fusion module and a memory system. The data fusion module fuses the feature data in the intelligent body. The memory system is used to store and read and write the fused data of the intelligent body.

[0094] The evaluation unit is used to calculate the reward value according to the data in the memory system, and guide the data fusion of the intelligent agent and update the memory system according to the reward value.

[0095] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and is characterized in that when the computer program is loaded into the processor, the multimodal industrial data fusion method for discrete manufacturing is implemented.

[0096] The computer-readable storage medium of the present invention stores a computer program, and is characterized in that when the computer program is executed by a processor, the multimodal industrial data fusion method for discrete manufacturing is implemented.

[0097] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0098] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A multimodal industrial data fusion method for discrete manufacturing, characterized in that: The steps include: Collect discrete manufacturing industrial data in real time and extract features to obtain feature data, input the feature data into a differentiable computer network to obtain a data fusion result, and the differentiable computer is also used to complete the prediction target according to the data fusion result; The discrete manufacturing industry data includes video, text, image and sound data; The differentiable computer network includes a data fusion module and a memory system. The data fusion module fuses the feature data in the intelligent body, and the memory system is used to store and read and write the fused data of the intelligent body; the data in the memory system is input into the evaluation system to calculate the reward value, and the data fusion of the intelligent body is guided according to the reward value and the memory system is updated; The data fusion module fuses the feature data in the agent including: At time t, the controller receives the feature data input vector x t , in the memory system's storage matrix M t-1 Get R read vectors in After being processed by the agent network, the output vector y is obtained t ; At each moment, the agent network calculates a network output vector v t and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, is the output of the Lth layer of the agent at time t, w y and w ξ is the tap coefficient; The agent passes information back by creating a loop in the calculation, thereby obtaining v t and t ; The output vector of the agent is: w t Read weights.

2. The multimodal industrial data fusion method for discrete manufacturing according to claim 1 is characterized in that: The real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data includes: extracting semantic features of text data, extracting image features of image data, extracting frame features of video data, and extracting spectral features of sound data.

3. The multimodal industrial data fusion method for discrete manufacturing according to claim 2 is characterized in that: Extracting semantic features of text data, converting the text into a set of words, assigning a weight to each word, wherein the weight is set according to the frequency of the word in the text and the distribution in the corpus; mapping the words to a high-dimensional space to obtain semantic features of the text data; Extract image features from image data, use the Canny algorithm to extract image edges, and use convolutional neural networks to extract image features; Extract frame features of video data, extract key frames from the video, and use convolutional neural networks to extract frame features; Extract the spectral features of sound data, extract the spectrum of sound data, and use convolutional neural network to extract spectral features.

4. The multimodal industrial data fusion method for discrete manufacturing according to claim 1 is characterized in that: The controller is a long short-term memory network.

5. The multimodal industrial data fusion method for discrete manufacturing according to claim 1, characterized in that: In the evaluation system, the error between the model's prediction of the predicted target and the actual target is calculated, and the prediction error is used as a reward value to guide the training of the differentiable computer network.

6. The multimodal industrial data fusion method for discrete manufacturing according to claim 1, characterized in that: In the evaluation system, the game combination weight of the reward value is calculated, a weighted sum is performed according to the game combination weight to obtain a weighted reward value, and the parameters of the differentiable computer network are adjusted according to the weighted reward value.

7. A multimodal industrial data fusion system for discrete manufacturing, characterized in that: include: A feature extraction unit, used for real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data; The discrete manufacturing industry data includes video, text, image and sound data; A data fusion unit is used to input the feature data into a differentiable computer network to obtain a data fusion result. The differentiable computer is also used to complete the prediction target according to the data fusion result. The differentiable computer network includes a data fusion module and a memory system. The data fusion module fuses the feature data in the intelligent body. The memory system is used to store and read and write the fused data of the intelligent body. An evaluation unit, used to calculate a reward value based on the data in the memory system, and guide the data fusion of the agent and update the memory system according to the reward value; In the data fusion unit, the data fusion module fuses the feature data in the intelligent agent including: At time t, the controller receives the feature data input vector x t , in the memory system's storage matrix M t-1 Get R read vectors in After being processed by the agent network, the output vector y is obtained t ; At each moment, the agent network calculates a network output vector v t and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, is the output of the Lth layer of the agent at time t, w y and w ξ is the tap coefficient; The agent passes information back by creating a loop in the calculation, thereby obtaining v t and t ; The output vector of the agent is: w t Read weights.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the multimodal industrial data fusion method for discrete manufacturing according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal industrial data fusion method for discrete manufacturing according to any one of claims 1 to 6 is implemented.

Images