Multimodal industrial data fusion method and system for discrete manufacturing

Abstract

Disclosed in the present invention are a multimodal industrial data fusion method and system for discrete manufacturing. The method comprises: collecting discrete manufacturing industrial multimodal data in real time, performing feature extraction to obtain feature data, and inputting the feature data into a differentiable computer network to obtain a data fusion result, wherein a differentiable computer is further used for accomplishing a prediction objective on the basis of the data fusion result, and the differentiable computer network comprises a data fusion module and a memory system, the data fusion module fusing the feature data in an agent, and the memory system being used for storing and reading / writing refused data in the agent; and inputting the data in the memory system into an evaluation system to calculate a reward value, guiding the agent in data fusion on the basis of the reward value, and updating the memory system. The present invention increases the capacity of a memory system without leading to an increase in training parameters, and fuses complex multimodal industrial data, thereby facilitating decision-making in the discrete manufacturing industry.

Description

A multimodal industrial data fusion method and system for discrete manufacturing Technical Field

[0001] The present invention relates to a data fusion method and system, in particular to a multi-modal industrial data fusion method and system for discrete manufacturing. Background Art

[0002] The widespread use of sensor technology in discrete manufacturing has generated massive amounts of data. These data sources cover every aspect of the production environment, including equipment operating status, temperature, pressure, humidity, and other information. This diverse and massive data volume provides a rich source of information for data fusion. With the rapid development of the Internet of Things (IoT), the connectivity between devices and systems has significantly increased. This means that data from various devices and systems can be more easily collected, transmitted, and integrated. This connectivity improves the monitoring, optimization, and control of production processes, thereby increasing production efficiency and quality.

[0003] The digital transformation of factories is becoming a major trend in the discrete manufacturing industry. This means the amount of data involved in production environments is constantly increasing. From sensor data to real-time monitoring data of production processes and information related to the supply chain, data is everywhere. Data fusion has become a key method for integrating, analyzing, and applying this multi-source data.

[0004] Multimodal data fusion technology involves integrating multimodal data from different sensors or data sources. This data may include different types of data such as images, video, sound, and text. By fusing these data sources, more comprehensive information is provided. Because individual data sources may have quality differences, including data inconsistencies, missing values, errors, or noise, data fusion technology may be affected by these data quality issues, resulting in inaccurate or unreliable fusion results. At the same time, existing data fusion technology cannot meet the needs of real-time processing and decision-making. Especially in industrial applications that require rapid response, processing and fusing large amounts of data can cause delays, affecting the effectiveness of real-time decision-making. In addition, traditional data fusion technology cannot adapt to the ever-changing industrial environment and is not flexible enough for situations where data changes frequently and is highly complex. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a multimodal industrial data fusion method and system for discrete manufacturing, which dynamically handles complex environmental problems through interactive learning of the environment and a large amount of data, and maximizes the optimization of data structure and dimension through the reward form of a memory-adaptive discrete manufacturing environment to obtain data fusion results, thereby guiding discrete manufacturing decision makers.

[0006] Technical solution: The multimodal industrial data fusion method for discrete manufacturing described in the present invention includes the following steps:

[0007] Real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data, the feature data is input into a differentiable computer network to obtain a data fusion result, and the differentiable computer is further used to complete a prediction target based on the data fusion result;

[0008] The discrete manufacturing industry data includes video, text, image and sound data;

[0009] The differentiable computer network includes a data fusion module and a memory system. The data fusion module fuses feature data in the intelligent body, and the memory system is used to store and read and write the fused data of the intelligent body; the data in the memory system is input into the evaluation system to calculate the reward value, and the data fusion of the intelligent body is guided and the memory system is updated according to the reward value.

[0010] Furthermore, the real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data includes: extracting semantic features of text data, extracting image features of image data, extracting frame features of video data, and extracting spectral features of sound data.

[0011] Furthermore, the semantic features of the text data are extracted, the text is converted into a set of words, a weight is assigned to each word, and the weight is set according to the frequency of the word in the text and the distribution in the corpus; the words are mapped to a high-dimensional space to obtain the semantic features of the text data;

[0012] Extract image features from image data, use the Canny algorithm to extract image edges, and use convolutional neural networks to extract image features;

[0013] Extract frame features of video data, extract key frames from the video, and use convolutional neural networks to extract frame features;

[0014] Extract the spectral features of sound data, extract the spectrum of sound data, and use convolutional neural network to extract spectral features.

[0015] Furthermore, the data fusion module fuses the feature data in the agent including:

[0016] At time t, the controller receives the feature data input vector x t , in the memory system's storage matrix M t-1 Get R read vectors After processing by the agent network, the output vector y is obtained t ;

[0017] At each moment, the agent network calculates a network output vector νt and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, is the output of the Lth layer of the agent at time t, w y and w ξ is the tap coefficient;

[0018] The agent passes information back by creating a loop in the calculation, thereby obtaining v t and ξ t ; The output vector of the agent is: w t Read weights.

[0019] Furthermore, the controller is a long short-term memory network.

[0020] Furthermore, in the evaluation system, the error between the model's prediction of the predicted target and the actual target is calculated, and the prediction error is used as a reward value to guide the training of the differentiable computer network.

[0021] Furthermore, in the evaluation system, the game combination weight of the reward value is calculated, and a weighted sum is performed according to the game combination weight to obtain a weighted reward value, and the parameters of the differentiable computer network are adjusted according to the weighted reward value.

[0022] The multimodal industrial data fusion system for discrete manufacturing according to the present invention comprises:

[0023] A feature extraction unit is used to collect discrete manufacturing industry data in real time and perform feature extraction to obtain feature data; the discrete manufacturing industry data includes video, text, image and sound data;

[0024] a data fusion unit, configured to input the feature data into a differentiable computer network to obtain a data fusion result, wherein the differentiable computer is further configured to complete a prediction target based on the data fusion result; the differentiable computer network includes a data fusion module and a memory system, wherein the data fusion module fuses the feature data in the intelligent agent, and the memory system is configured to store and read and write the fused data of the intelligent agent;

[0025] The evaluation unit is used to calculate the reward value based on the data in the memory system, guide the data fusion of the intelligent agent and update the memory system according to the reward value.

[0026] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. It is characterized in that when the computer program is loaded into the processor, it implements the multimodal industrial data fusion method for discrete manufacturing.

[0027] The computer-readable storage medium of the present invention stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the multimodal industrial data fusion method for discrete manufacturing.

[0028] Beneficial effects: Compared with the existing technology, the advantages of the present invention are: the present invention fuses complex multimodal data collected in real time, and the memory-enhanced neural network of the present invention separates the grid training parameters from the memory system capacity, thereby increasing the capacity of the memory system without causing an increase in the training parameters, reducing a large amount of training time, and saving more time for decision-making in the discrete manufacturing industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] FIG1 is a flow chart of the method of the present invention.

[0030] FIG2 is a structural diagram of a DNC model of the present invention.

[0031] FIG3 is a schematic diagram of the intelligent agent calculation process of the present invention.

[0032] FIG4 is a flow chart of the read and write operations of the present invention. DETAILED DESCRIPTION

[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0034] Common memory systems include long short-term memory (LSTM) networks and gated recurrent units (GRU). To enhance the ability of intelligent agents to cope with complex environments, existing technologies often increase memory capacity by increasing the number of LSTMs and GRUs. However, the number of network training parameters increases dramatically as the memory system capacity increases, making training difficult.

[0035] This invention separates the memory system from the parameter training process and proposes the use of a memory-enhanced neural network for multimodal data fusion. The agent reads or writes the required information from the memory system to execute decisions. Simultaneously, the multimodal data fusion system provides feedback in the form of rewards for the agent's actions. This process is repeated repeatedly to ultimately maximize the reward.

[0036] A differentiable computer (DNC) is a neural network architecture with external memory that combines the learning capabilities of neural networks with the storage capacity of external memory. Data fusion using DNC can be performed through the following steps: Collect the remaining featurized datasets from the multimodal data module to be fused, ensuring that the features of the dataset are available and clean. Data preprocessing is a key step in ensuring data quality. Using the DNC implementation, determine the input and output of the model. The input will be the data to be fused, and the output will be the result of the data integration. The DNC model is trained using the prepared dataset. During training, the model learns how to effectively fuse data from different sources. Evaluate the model's performance using a validation dataset. This helps verify that the model can correctly fuse the data and has good generalization capabilities. Debug and improve the model based on the validation results.

[0037] As shown in FIG1 , the multimodal industrial data fusion method for discrete manufacturing according to the present invention specifically includes the following steps.

[0038] S1, transmits the discrete manufacturing industrial data collected in real time from discrete manufacturing workshops such as machining workshops to the multimodal data module for characterization.

[0039] S2, transmits the characterized data in the multimodal data module to the DNC. The DNC includes a discrete manufacturing industry data fusion module and a memory system, which respectively perform discrete manufacturing industry data fusion and memory reading and writing, and output the data fusion results.

[0040] S2-1, input the data of the memory system into the evaluation system, evaluate it according to traditional evaluation indicators, and feed the data of the memory system back to the multimodal data module;

[0041] S2-2, based on the reward feedback from the evaluation, the industrial data fusion module of the discrete machinery manufacturing industrial workshop is fed back to guide the data fusion, so as to continuously update the memory system.

[0042] In step S1, various types of industrial data collected in real time from discrete manufacturing machinery workshops (including, for example, text data recorded in real-time workshops, video data under monitoring, audio data recording machine status, and image data generated during workshop production) are characterized. The characterization steps are as follows:

[0043] ① Convert the text into a collection of words, regardless of their order, and assign each word a weight that takes into account its frequency in the document and its distribution in the corpus. Map the words to vectors in a high-dimensional space, preserving their semantic and contextual information.

[0044] ② Describe the distribution of colors in the image, use the Canny algorithm to extract image edges, and use convolutional neural networks (CNN) to automatically extract image features.

[0045] ③ Extract key frames from the video to describe the movement of pixels between different frames. Combine temporal and spatial information and use a CNN deep learning model to extract features.

[0046] ④ Describes the spectral characteristics of audio signals, which is widely used in speech recognition. Use CNN to extract audio features.

[0047] In step S2, while traditional artificial neural networks (ANNs) integrate computation and storage when processing data, the DNC can be viewed as a combination of the ANN and an external memory matrix, consisting of a controller and a memory system. The DNC can selectively read and write to the memory and iteratively modify stored data. By utilizing the memory system's storage matrix, the DNC can acquire reasoning information and store important data during the reasoning process, thereby improving reasoning efficiency by simulating the human brain's reasoning process.

[0048] The core of the DNC is the controller, responsible for handling task-related calculations. Then, through memory readout, the controller network reads information from external memory. The DNC's external memory is readable and writable, similar to computer memory. This step enables the DNC to access previously stored information and integrate historical information. The neural network then generates an output, including a prediction or decision for the task. This output is guided by the controller network to fully utilize the information in memory. Based on the task's outcome, the controller network may then write some information to external memory. This process allows the DNC to store key task information for future use. Finally, backpropagation is performed using the DNC's output and actual results to adjust the neural network parameters. This ensures that the DNC learns how to more effectively utilize the information in external memory. Through this cycle, the DNC continuously extracts key information from the input data, integrates it into its memory system, and then produces output based on task requirements. This combined memory and neural network approach enables the DNC to outperform traditional neural networks in handling tasks with long-term dependencies.

[0049] The DNC of the present invention includes a data fusion module and a memory system. As shown in FIG2 , the data fusion module obtains the fusion result by continuously updating the intelligent agent. At time t, the controller receives the input vector x t , in the memory system's storage matrix M t-1 Get R read vectors After processing by the agent network, the output vector y is obtained t Figure 3 is a schematic diagram of the agent’s calculation process.

[0050] The present invention uses LSTM network as the controller network. At time t, the output value of the agent layer 1 is

[0051] At each moment, the agent network calculates a network output vector ν t and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, w y and w ξ is the tap coefficient:

[0052] The agent passes information back by creating a loop in the computational graph, thereby obtaining v t etc.; finally, the output vector of the agent is w t Read weights.

[0053] This designed agent regulates its output decisions by strengthening its reliance on the memory system's storage matrix.

[0054] The agent operates the data in the memory system through the read-write head. Figure 4 shows the schematic diagram of the read-write head operation. The reading and writing positions are determined by the corresponding weights. The set of weights allowed at N positions is R N The nonnegative quadrant of the standard simplex in:

[0055] In read operations, multiple read weights, It is used to calculate the weighted average of the content, so the read vector is defined as:

[0056] In write operations, write weight Combined with the erase vector e t and write vector v t , modify the memory system storage matrix:

[0057] Where: ° represents the Hadamard product, which is the product of corresponding elements of two matrices of the same order; E is an N×M matrix of all ones.

[0058] The addressing mechanism of DNC is a combination of multiple methods: when writing data in the memory system, content-based addressing and dynamic addressing are used; when reading data in the memory system, content-based addressing and temporal memory links are used to obtain the location.

[0059] In steps S2-1 and S2-2, a prediction target must first be determined. This can be part of the task the model is trying to learn. This target is often related to the primary objective of the task but may be more easily derived from the data. The DNC model is trained using existing data to predict the target as accurately as possible. This can be accomplished through supervised learning, where labels from existing data or other available information are used to guide the model's learning.

[0060] During training, the error between the model’s prediction of the target and the actual target is calculated. This can be achieved using various loss functions, such as w m Minimize deviation. Use prediction error as a reward to guide model training. Typically, lower prediction errors are considered higher rewards, while higher errors are considered lower rewards or penalties. This reward value can be mapped to a specific reward range through some transformation or normalization to ensure the stability and controllability of the reward value. Use the obtained reward value to optimize the model parameters to minimize the prediction error and improve the accuracy of the predicted target. This usually involves using gradient descent or other optimization algorithms to update the model parameters and adjust the parameters based on the direction of the reward value.

[0061] In order to eliminate the influence of different dimensions on the final evaluation results, the positive and negative reward values ​​should be processed separately when conducting data evaluation:

[0062] Any linear combination of weight vectors is:

[0063] Among them, β m is the linear combination coefficient And β m >0.

[0064] Optimize the linear combination coefficients, even if w is different from each w m Minimize the deviation, that is:

[0065] According to the properties of matrix differentials, the first-order derivative condition of the above formula is:

[0066] Its linear equations are:

[0067] β m The values ​​are normalized:

[0068] The final game combination weighting is a method for combining multiple evaluation criteria or factors in decision analysis. It is used in multi-criteria decision-making problems, especially when there are conflicting objectives or criteria. This method draws on the ideas of game theory to find the best compromise by assigning weights to different criteria or factors. The game combination weighting is as follows:

[0069] The weighted total scores of each solution are compared, and the one with the highest score is selected as the optimal choice. The required evaluation feedback is then fed into the fusion module, which extracts key features from the data collected by the evaluation system. These features should represent the core indicators of the evaluation system. The extracted features are then integrated with existing manufacturing data. Through continuous evaluation, data collection, analysis, and feedback, a closed-loop optimization system is formed to continuously improve the manufacturing process.

[0070] Taking the automotive parts manufacturing workshop as an example, the production process on the production line is monitored by surveillance cameras, sensors, and sound collection equipment. The data generated by these devices includes different types of data such as images, videos, sounds, and text.

[0071] Surveillance cameras: Real-time recording of the parts assembly process on the production line.

[0072] Sensors: Record physical parameters such as temperature, humidity, pressure, vibration, etc. on the production line.

[0073] Sound collection equipment: records abnormal sounds during the production process, machine operation sounds, etc.

[0074] Text data: such as workers’ log records, equipment maintenance reports, etc.

[0075] The data fusion method of this workshop includes the following steps.

[0076] (1) Data integration: Integrate data from different devices into a unified data platform, ensure the uniformity of data format, and handle missing values ​​and outliers.

[0077] (2) Feature extraction: Feature extraction is performed on the integrated data to extract features related to production efficiency, product quality, and safety. For image and video data, computer vision technology can be used to extract information such as objects in the image and the status of the production line. For sound data, information such as sound spectrum characteristics and frequency distribution can be extracted. For text data, natural language processing can be performed to extract keywords, sentiment analysis, and other information.

[0078] (3) Model building: Build a DNC model. DNC is a neural network architecture with external memory that combines the learning ability of neural networks with the storage capacity of external memory. Data fusion using DNC can be performed through the following steps: Data preparation: Collect the data sets that need to be fused. These data can come from different sources and have different characteristics or formats. Ensure that the characteristics of the data set are available and clean. Data preprocessing is a key step in ensuring data quality. The model can predict abnormal conditions on the production line, optimize production plans, or perform real-time monitoring and early warning based on the status of the production line.

[0079] (4) Model evaluation and optimization: Use historical data to evaluate and tune the established model to ensure that the model has good generalization capabilities. Techniques such as cross-validation can be used to evaluate the performance of the model and adjust and optimize it as needed.

[0080] (5) Real-time monitoring and feedback: Based on the monitoring results, timely adjustments are made to production plans or equipment parameters to optimize and control the production process. The established model is deployed in a real-time production environment, and a real-time monitoring system is established to monitor the status of the production line. Through such a data fusion solution, automotive parts manufacturing workshops can better utilize multi-source data to optimize production processes, improve product quality and safety, thereby reducing production costs and enhancing competitiveness.

[0081] The multimodal industrial data fusion system for discrete manufacturing according to the present invention comprises:

[0082] A feature extraction unit is used to collect discrete manufacturing industry data in real time and perform feature extraction to obtain feature data; the discrete manufacturing industry data includes video, text, image and sound data;

[0083] a data fusion unit, configured to input the feature data into a differentiable computer network to obtain a data fusion result, wherein the differentiable computer is further configured to complete a prediction target based on the data fusion result; the differentiable computer network includes a data fusion module and a memory system, wherein the data fusion module fuses the feature data in the intelligent agent, and the memory system is configured to store and read and write the fused data of the intelligent agent;

[0084] The evaluation unit is used to calculate the reward value based on the data in the memory system, guide the data fusion of the intelligent agent and update the memory system according to the reward value.

[0085] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. It is characterized in that when the computer program is loaded into the processor, it implements the multimodal industrial data fusion method for discrete manufacturing.

[0086] The computer-readable storage medium of the present invention stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements the multimodal industrial data fusion method for discrete manufacturing.

[0087] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0088] The processor is configured to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment.

Claims

1. A multimodal industrial data fusion method for discrete manufacturing, characterized by: The steps include: Real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data, the feature data is input into a differentiable computer network to obtain a data fusion result, and the differentiable computer is further used to complete a prediction target based on the data fusion result; The discrete manufacturing industry data includes video, text, image and sound data; The differentiable computer network includes a data fusion module and a memory system. The data fusion module fuses feature data in the intelligent agent, and the memory system is used to store and read and write the fused data of the intelligent agent. The data in the memory system is input into the evaluation system to calculate the reward value, and the reward value is used to guide the data fusion of the intelligent agent and update the memory system. The data fusion module fuses the feature data in the agent including: At time t, the controller receives the feature data input vector x t , in the memory system's storage matrix M t-1 Get R read vectors After processing by the agent network, the output vector y is obtained t ; At each moment, the agent network calculates a network output vector ν t and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, is the output of the Lth layer of the agent at time t, w y and w ξ is the tap coefficient; The agent passes information back by creating a loop in the calculation, thereby obtaining v t and ξ t ; The output vector of the agent is: w t Read weights.

2. The multimodal industrial data fusion method for discrete manufacturing according to claim 1, characterized in that: The real-time collection of discrete manufacturing industrial data and feature extraction to obtain feature data includes: extracting semantic features of text data, extracting image features of image data, extracting frame features of video data, and extracting spectral features of sound data.

3. The multimodal industrial data fusion method for discrete manufacturing according to claim 2, characterized in that: Extracting semantic features of text data, converting the text into a set of words, assigning a weight to each word based on the frequency of the word in the text and its distribution in the corpus; mapping the words into a high-dimensional space to obtain semantic features of the text data; Extract image features from image data, use the Canny algorithm to extract image edges, and use convolutional neural networks to extract image features; Extract frame features of video data, extract key frames from the video, and use convolutional neural networks to extract frame features; Extract the spectral features of sound data, extract the spectrum of sound data, and use convolutional neural network to extract spectral features.

4. The multimodal industrial data fusion method for discrete manufacturing according to claim 1, characterized in that: The controller is a long short-term memory network.

5. The multimodal industrial data fusion method for discrete manufacturing according to claim 1, characterized in that: In the evaluation system, the error between the model's prediction of the predicted target and the actual target is calculated, and the prediction error is used as a reward value to guide the training of the differentiable computer network.

6. The multimodal industrial data fusion method for discrete manufacturing according to claim 1, characterized in that: In the evaluation system, the game combination weight of the reward value is calculated, a weighted sum is performed according to the game combination weight to obtain a weighted reward value, and the parameters of the differentiable computer network are adjusted according to the weighted reward value.

7. A multimodal industrial data fusion system for discrete manufacturing, characterized by: include: Feature extraction unit, used to collect discrete manufacturing industrial data in real time and perform feature extraction to obtain feature data; The discrete manufacturing industry data includes video, text, image and sound data; a data fusion unit, configured to input the feature data into a differentiable computer network to obtain a data fusion result, wherein the differentiable computer is further configured to complete a prediction target based on the data fusion result; the differentiable computer network includes a data fusion module and a memory system, wherein the data fusion module fuses the feature data in the intelligent agent, and the memory system is configured to store and read and write the fused data of the intelligent agent; An evaluation unit, configured to calculate a reward value based on the data in the memory system, and guide the agent's data fusion and update the memory system based on the reward value; In the data fusion unit, the data fusion module fuses the feature data in the agent, including: At time t, the controller receives the feature data input vector x t , in the memory system's storage matrix M t-1 Get R read vectors After processing by the agent network, the output vector y is obtained t ; At each moment, the agent network calculates a network output vector ν t and an interaction vector ξ t , where the interaction vector is used to parameterize the interaction between the agent and the memory system at time t, is the output of the Lth layer of the agent at time t, w y and w ξ is the tap coefficient; The agent passes information back by creating a loop in the calculation, thereby obtaining v t and ξ t ; The output vector of the agent is: w t Read weights.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is loaded into a processor, the multimodal industrial data fusion method for discrete manufacturing according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal industrial data fusion method for discrete manufacturing according to any one of claims 1 to 6 is implemented.

Images