Intelligent factory process optimization method and system based on multi-modal data fusion

Abstract

The invention relates to the technical field of industrial data processing, and provides an intelligent factory process optimization method and system based on multi-modal data fusion. Production data in the working process of a factory are obtained through a sensor; preprocessing the production data to obtain first data, and generating a multi-modal tensor according to a data source of the first data and an association relationship; decoupling the multi-modal tensor, inputting the obtained image sub-modal data and acoustic sub-modal data into a defect identification network, and outputting defect features; constructing a Bayesian causal graph based on the defect features, the physical features representing the process physical state in the production data and the environmental features, and outputting adjustment parameters of the production process through the Bayesian causal graph; and generating a process adjustment scheme according to the adjustment parameters and the production data. Closed-loop optimization from data perception to decision execution is formed, so that the process parameters have dynamic adaptive ability, and the accuracy and yield of process optimization are improved.

Description

Technical Field

[0001] This application relates to the technical field of industrial data processing. Specifically, it relates to an intelligent factory process optimization method and system based on multi-modal data fusion. Background Art

[0002] With the rapid development of Industry 4.0 and intelligent manufacturing, the production process of modern factories has become increasingly complex, involving real-time acquisition and analysis of multi-source heterogeneous data, such as sensor time-series data, visual images, acoustic signals, and process logs. By integrating multi-modal data fusion technology, multiple information such as equipment status, production process, and environmental parameters can be integrated to achieve real-time monitoring and intelligent control of the production process. For example, in the field of intelligent manufacturing, by fusing sensor data, image data, and video data, production efficiency and product quality can be improved.

[0003] In practical applications, manual parameter adjustment depends on expert experience and is difficult to adapt to dynamic working conditions. Moreover, the sampling frequencies, dimensions, and semantic differences of different modal data are large, making it difficult to directly perform correlation analysis. As a result, the accuracy of process optimization in industrial production processes is relatively low. Summary of the Invention

[0004] This application provides an intelligent factory process optimization method and system based on multi-modal data fusion, which can at least to some extent solve the problem of relatively low accuracy of process optimization in industrial production processes.

[0005] Other features and advantages of this application will become apparent through the following detailed description, or will be partially learned through the practice of this application.

[0006] According to one aspect of this application, an intelligent factory process optimization method based on multi-modal data fusion is provided, including: acquiring production data during the operation of the factory through sensors; preprocessing the production data to obtain first data, and generating a multi-modal tensor according to the data source and association relationship of the first data; decoupling the multi-modal tensor, inputting the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and outputting defect features; constructing a Bayesian causal graph based on the defect features, physical features representing the process physical state, and environmental features in the production data, and outputting adjustment parameters of the production process through the Bayesian causal graph; generating a process adjustment plan according to the adjustment parameters and the production data.

[0007] In this application, based on the foregoing solution, the acquiring production data during the operation of the factory through sensors includes: acquiring temperature data, image streams, waveform data, and production logs during the operation of the factory through sensors as the production data.

[0008] In this application, based on the foregoing solution, the preprocessing of the production data to obtain the first data and the generation of a multimodal tensor according to the data source and association relationship of the first data include: cleaning the production data to obtain the first data, and determining the association relationship and corresponding time vectors among the first data; generating tensor nodes based on the data sources corresponding to the first data, and generating tensor hyperedges according to the association relationship and the time vectors among the first data; and generating a multimodal tensor according to the tensor nodes and the tensor hyperedges.

[0009] In this application, based on the foregoing solution, the decoupling of the multimodal tensor and the input of the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network to output defect features include: decoupling the multimodal tensor to separate the image sub-modal data and the acoustic sub-modal data therefrom; constructing a defect recognition network based on an encoder, a generator, and a discriminator, and training the defect recognition network with a mixed data composed of normal samples and defect samples; and respectively inputting the image sub-modal data and the acoustic sub-modal data into the defect recognition network, and separating abnormal features through a reconstruction loss and a distribution alignment loss to output defect features.

[0010] In this application, based on the foregoing solution, the construction of a Bayesian causal graph based on defect features, physical features representing the process physical state in production data, and environmental features, and the output of adjustment parameters for the production process through the Bayesian causal graph include: constructing a Bayesian causal graph based on defect features, physical features representing the process physical state in production data, and environmental features; simulating and adjusting the parameters in the Bayesian causal graph to obtain the influence parameters of the simulated and adjusted parameters on the relationship chain in the Bayesian causal graph; and adjusting the parameters in the Bayesian causal graph based on the influence parameters to obtain the adjustment parameters for the production process.

[0011] In this application, based on the foregoing solution, the generation of a process adjustment plan according to the adjustment parameters and the production data includes: generating candidate parameter combinations based on the adjustment parameters and the production data, and determining an objective function and constraint conditions according to actual production requirements; deriving diversified initial parameter combinations based on the candidate parameter combinations, and searching for the optimal parameter combination from the initial parameter combinations according to the objective function and the constraint conditions; and generating a process adjustment plan based on the optimal parameter combination.

[0012] In this application, based on the foregoing solution, after generating a process adjustment plan according to the adjustment parameters and the production data, it further includes: forwarding the process adjustment plan to a control terminal and displaying it on the interface of the control terminal.

[0013] According to one aspect of the present application, there is provided an intelligent factory process optimization system based on multimodal data fusion, including:

[0014] An acquisition unit, configured to acquire production data during the operation of the factory through sensors;

[0015] A tensor unit, configured to preprocess the production data to obtain first data, and generate a multimodal tensor according to the data source and association relationship of the first data;

[0016] A detection unit, configured to decouple the multimodal tensor, input the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and output defect features;

[0017] An adjustment unit, configured to construct a Bayesian causal graph based on the defect features, physical features representing the process physical state in the production data, and environmental features, and output adjustment parameters of the production process through the Bayesian causal graph;

[0018] A solution unit, configured to generate a process adjustment solution according to the adjustment parameters and the production data.

[0019] According to one aspect of the present application, there is provided a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the intelligent factory process optimization method based on multimodal data fusion as described in the above embodiments.

[0020] According to one aspect of the present application, there is provided an electronic device, including: one or more processors; a storage device, configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the intelligent factory process optimization method based on multimodal data fusion as described in the above embodiments.

[0021] According to one aspect of the present application, there is provided a computer program product or a computer program, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the intelligent factory process optimization method based on multimodal data fusion provided in the above various alternative implementation manners.

[0022] The technical solution of this application obtains production data during the operation of the factory through sensors; preprocesses the production data to obtain first data, and generates a multi-modal tensor according to the data source and association relationship of the first data; decouples the multi-modal tensor, inputs the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and outputs defect features; constructs a Bayesian causal graph based on the defect features, physical features representing the process physical state, and environmental features in the production data, and outputs adjustment parameters of the production process through the Bayesian causal graph; generates a process adjustment plan according to the adjustment parameters and the production data. By constructing a spatio-temporally aligned multi-modal tensor through multi-modal data fusion, efficient fusion of heterogeneous data is achieved. Combining the Bayesian causal graph to reveal the causal relationship between process parameters and quality indicators, the pseudo-association defect of correlation analysis is avoided. Finally, a process adjustment plan is generated through multi-objective optimization, forming a closed-loop optimization from data perception to decision execution, enabling process parameters to have dynamic adaptability and improving the accuracy and yield of process optimization.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit this application. Brief Description of the Drawings

[0024] The drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application. Obviously, the drawings in the following description are only some embodiments of this application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0025] Figure 1 Schematically shows the flowchart of an intelligent factory process optimization method based on multi-modal data fusion in an embodiment of this application.

[0026] Figure 2 Schematically shows the flowchart of generating a multi-modal tensor in an embodiment of this application.

[0027] Figure 3 Schematically shows the schematic diagram of an intelligent factory process optimization system based on multi-modal data fusion in an embodiment of this application.

[0028] Figure 4 Shows the structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of this application. Detailed Description of the Embodiments

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art.

[0030] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of this application. However, those skilled in the art will realize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be used. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of this application.

[0031] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the drawings are only illustrative and not necessarily include all the content and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0033] The implementation details of the technical solutions of this application are elaborated in detail below:

[0034] Figure 1 A flowchart of an intelligent factory process optimization method based on multimodal data fusion according to an embodiment of this application is shown. Referring to Figure 1 as shown, the intelligent factory process optimization method based on multimodal data fusion at least includes steps S110 to step S150, which are introduced in detail as follows:

[0035] In step S110, production data during the factory's working process is obtained through sensors.

[0036] In an embodiment of this application, temperature data, image streams, waveform data, and production logs during the factory's working process are obtained through sensors as the production data. In addition, the production data can also include temperature sensor time series, pressure, industrial camera image streams, acoustic vibration signals, and process parameter log texts, etc.

[0037] In this embodiment, temperature, pressure and other data during the operation of the factory are obtained through sensors to obtain sensor data based on a time series, and the sampling frequency is at the millisecond level. High-resolution video frames are collected through the industrial camera image stream, and the sampling frequency is relatively low, such as 10 frames per second. Acoustic vibration signals are collected through sensors such as oscilloscopes, such as high-frequency vibration waveform data with a sampling rate at the kHz level. Process parameter log texts are obtained through the collected data stream, such as discrete event records of parameter adjustment time points, operation instructions, etc.

[0038] In step S120, the production data is preprocessed to obtain first data, and a multi-modal tensor is generated according to the data source and the association relationship of the first data.

[0039] In an embodiment of the present application, the production data is subjected to data cleaning and standardization processing to obtain first data, and a multi-modal tensor is generated according to the data source for generating or collecting the first data and the association relationship between the first data.

[0040] As Figure 2 shown, in an embodiment of the present application, preprocessing the production data to obtain first data, and generating a multi-modal tensor according to the data source and the association relationship of the first data, includes:

[0041] S210, perform data cleaning on the production data to obtain first data, and determine the association relationship and the corresponding time vector between the first data;

[0042] S220, generate tensor nodes based on the data source corresponding to the first data, and generate tensor hyperedges according to the association relationship between the first data and the time vector;

[0043] S230, generate a multi-modal tensor according to the tensor nodes and the tensor hyperedges.

[0044] In this embodiment, data cleaning is performed on the production data, a comprehensive check is performed on the production data, missing values are identified and processed, and according to the data distribution characteristics and business logic, filling is performed by mean, median, mode, or prediction filling based on an algorithm model to ensure data integrity. At the same time, outliers are accurately located, thresholds are set based on statistical methods or combined with business rules, and outliers are corrected, removed, or retained and marked for subsequent analysis.

[0045] In addition, the preprocessing process can also unify the data format, standardize the representation forms of fields such as dates and times, and eliminate data misunderstandings and calculation errors caused by format differences; remove duplicate records, judge duplicates based on the combination of key fields, and retain valid information; correct data entry deviations such as spelling mistakes and inconsistent expressions to ensure the consistency of data in semantics and expression. After this series of rigorous data cleaning operations, accurate, complete, consistent and analysis-required first data is finally obtained, providing a solid and reliable data foundation for subsequent in-depth mining and decision-making.

[0046] After obtaining the first data, generate tensor nodes based on the data source corresponding to the first data. The tensor nodes include: v1 represents the temperature node corresponding to the temperature sensor time series, v2 represents the image node corresponding to the industrial camera image stream, v3 represents the vibration node corresponding to the acoustic vibration signal, and v4 represents the log node corresponding to the process parameter log text.

[0047] According to the attributes of the first data, determine the association relationships between the first data. In this embodiment, the dependency relationships between multi-modal data are defined by the following rules, including the first time alignment constraint rule. For example, if an image frame corresponds to temperature data in a specific time period, then construct a hyperedge to connect v1 (temperature node) and v2 (image node); the spatial association constraint rule. For example, if the vibration signal is related to a certain device position, then the hyperedge connects v3 (vibration node) and the sensor node at the corresponding position. The semantic association constraint rule. For example, v4 corresponding to the process parameter adjustment event may be associated with the responses of multiple sensors (v1, v2, and v3).

[0048] Generate a vector representation of the original timestamp based on the original timestamp of the first data. According to the original timestamp and the vector representation, generate the time vector t' mapped to the hyperbolic space as:

[0049]

[0050] where t represents the original timestamp, x represents the vector representation of the original timestamp, ||x|| represents the Euclidean norm of the vector x. By projecting the timestamp into a non-linear space, the long-period alignment error is reduced, and the hierarchy and periodicity of the time series are modeled more naturally.

[0051] After generating the association relationships and the time vector, generate a tensor hyperedge according to the association relationships between the first data and the timestamp as:

[0052] e j = ({v j1 , v j2 , v jn}, w j , t')

[0053] After generating tensor nodes and tensor hyperedges, the tensors are combined to generate multimodal tensors.

[0054] The above process obtains multi-source heterogeneous data such as temperature data, image streams, waveform data, and production logs through sensors. After data cleaning and time alignment processing, a multimodal tensor is constructed. This structured representation method can retain the spatio-temporal correlation relationships between different modal data, providing a unified data basis for subsequent analysis. Compared with traditional single-modal analysis, multimodal tensors can more comprehensively reflect the production status and avoid information loss.

[0055] In step S130, the multimodal tensor is decoupled, and the obtained image sub-modal data and acoustic sub-modal data are input into a defect recognition network to output defect features.

[0056] After generating the multimodal tensor, the multimodal tensor is decoupled to obtain image sub-modal data and acoustic sub-modal data. A defect recognition network is pre-constructed, and the image sub-modal data and acoustic sub-modal data are input into the defect recognition network to output defect features. In this embodiment, the defect features are used to represent the defect conditions existing in the current process during industrial production.

[0057] In an embodiment of the present application, decoupling the multimodal tensor and inputting the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network to output defect features includes:

[0058] Decouple the multimodal tensor and separate the image sub-modal data and acoustic sub-modal data therefrom;

[0059] Construct a defect recognition network based on an encoder, a generator, and a discriminator, and train the defect recognition network with a mixed data composed of normal samples and defect samples;

[0060] Respectively input the image sub-modal data and acoustic sub-modal data into the defect recognition network, and separate abnormal features through reconstruction loss and distribution alignment loss to output defect features.

[0061] Separate the image sub-modal data and acoustic sub-modal data from the multimodal tensor. Specifically, separate the image and acoustic signal sub-data from the spatio-temporally aligned tensor and perform normalization processing. The image data is adjusted to a unified size, and the pixel values are normalized to the interval [0, 1]. The acoustic signal is denoised, such as wavelet denoising, and segmented into fixed-length time series segments. Through this process, it can be ensured that the input formats of different modal data are compatible, facilitating subsequent network processing.

[0062] Construct a recognition network consisting of three parts: an encoder, a generator, and a discriminator. Among them, the encoder is used to compress the original data (image or acoustic signal) into a low-dimensional latent variable; the generator is used to reconstruct the original data from the latent variable and attempts to restore the input information; the discriminator is used to judge whether the latent variable conforms to a preset prior distribution, such as a standard normal distribution. Optionally, the generator and the discriminator share the weight parameters of some intermediate layers, such as convolutional kernels or fully connected layers. By sharing parameters, the model complexity is reduced, and at the same time, the generator and the discriminator are forced to learn consistent feature representations to avoid feature conflicts.

[0063] Train the defect recognition network with a mixed data composed of normal samples and defect samples. During this process, the encoder maps the input data to the latent space and generates a latent variable distribution. The latent variables of normal samples will closely cluster in the central region of the prior distribution; the latent variables of abnormal samples will deviate from the central region and the distribution will be marginalized.

[0064] The generator generates reconstructed data based on the latent variables. The reconstruction error of normal data is small, while the reconstruction error of abnormal data increases significantly. If the input is a defective image, the generator may not be able to restore the defect details, resulting in an obvious difference between the reconstructed image and the original image. The abnormality is indirectly identified by the size of the reconstruction error, and at the same time, the distribution shift of the latent variables directly reflects the abnormal pattern.

[0065] The discriminator learns to distinguish the latent variables generated by the encoder from the samples randomly sampled from the prior distribution, so that the latent variable distribution output by the encoder approximates the prior distribution, ensuring the structural consistency of the latent space. The generator reconstructs data during the adversarial training process and retains key features, such as defects, while ignoring irrelevant noise during reconstruction. The latent variables not only retain the core information of the input data, such as defect features, but also conform to the standardized distribution, facilitating subsequent module processing. Through adversarial training, the sensitivity of the model to subtle defects (such as small scratches, low-frequency abnormal vibrations) is significantly improved.

[0066] In this embodiment, intermediate features are generated by the discriminator, and reconstructed data is obtained by the generator based on the intermediate features. Then, a reconstruction loss is generated based on the reconstructed data and the input data; the latent variable distribution is defined by the encoder, and the KL divergence between the latent variable distribution and the prior distribution is used to measure the difference between the two distributions as the distribution alignment loss. Then, the loss function is determined by the reconstruction loss and the distribution alignment loss as:

[0067] L AAE =||G(D(x))-x‖ 2 +KL(q(z|x)||p(z))

[0068] Among them, \(x\) represents the input data, such as an image or an acoustic signal; \(G(D(x))\) represents the reconstructed data generated by the generator based on the intermediate features of the discriminator, \(q(z|x)\) represents the latent variable distribution defined by the encoder, usually assumed to be a Gaussian distribution, \(p(z)\) represents the prior distribution, such as the standard Gaussian distribution; \(KL(\cdot||\cdot)\) represents the KL divergence, which is used to measure the difference between two distributions.

[0069] After that, based on the comparison result between the value of the loss function and the preset threshold, abnormal features are separated to obtain defect features. The latent variable distributions of the image and acoustic signal branches are cross-modally fused through the reconstruction loss and the distribution alignment loss, and the attention mechanism is used to weight and fuse the latent variable distributions of the two modalities to highlight the common features related to the defects. For example, if the image shows a surface crack and the acoustic signal shows high-frequency vibration during a certain period, the fused features will strengthen this abnormal association, and in this way, unified defect features are output. Through the joint training of images and acoustic signals, complex defects that are difficult to detect by a single modality can be identified, such as latent faults with normal vision but abnormal vibration.

[0070] In the above process, a defect recognition network with an encoder, a generator, and a discriminator architecture is adopted. Through the training of normal samples and defect samples, the feature representation for distinguishing abnormal patterns is learned. The reconstruction loss ensures that the network can accurately reconstruct normal samples while generating a large error for defect samples; the distribution alignment loss makes the latent features conform to a predetermined distribution, facilitating anomaly detection. This adversarial training mechanism can discover more concealed defect patterns and is more robust to noise compared to the traditional threshold method.

[0071] In step S140, based on the defect features, physical features representing the physical state of the process in the production data, and environmental features, a Bayesian causal graph is constructed, and the adjustment parameters of the production process are output through the Bayesian causal graph.

[0072] In an embodiment of the present application, a Bayesian causal graph is constructed based on the defect features, physical features representing the physical state of the process in the production data, and environmental features; among them, the defect features are the features characterizing product anomalies, such as surface defect image features, abnormal vibration waveforms; the physical features are the features reflecting the physical state of the process, such as temperature distribution, pressure change, etc. The environmental features represent the external environment and the long-term operating state features of the equipment, such as environmental temperature and humidity, equipment aging trend. After that, the adjustment parameters of the production process are output through the Bayesian causal graph.

[0073] In an embodiment of the present application, a Bayesian causal graph is constructed based on the defect features, physical features representing the physical state of the process in the production data, and environmental features, and the adjustment parameters of the production process are output through the Bayesian causal graph, including:

[0074] Construct a Bayesian causal graph based on defect characteristics, physical characteristics representing the physical state of the process in production data, and environmental characteristics;

[0075] Simulate and adjust the parameters in the Bayesian causal graph to obtain the influence parameters of the simulated and adjusted parameters on the relationship chain in the Bayesian causal graph;

[0076] Based on the influence parameters, adjust the parameters in the Bayesian causal graph to obtain the adjustment parameters of the production process.

[0077] Construct a Bayesian causal graph based on defect characteristics, physical characteristics, and environmental characteristics. Specifically, the Bayesian causal graph consists of nodes, edges, and parameters. Among them, the nodes include all feature dimensions (such as temperature sensor data, defect types) and process parameters (such as heating power, cooling rate); the edges represent causal relationships, such as an increase in heating power leading to an increase in temperature, which in turn leads to a decrease in defects; the parameters define the conditional probability distribution, such as the defect probability decreasing when the temperature increases.

[0078] After that, verify whether the causal relationship between the features and parameters holds. For example, when the device temperature is known, test whether the environmental humidity is independent of the defect characteristics. If new latent variables are found, such as the unmonitored internal wear state of the device, dynamically add nodes and update the edge structure to ensure that the Bayesian causal graph covers the explicit and implicit factors in real production and avoid missing key causal relationships.

[0079] Simulate and adjust the parameters in the Bayesian causal graph to obtain the influence parameters of the simulated and adjusted parameters on the relationship chain in the Bayesian causal graph. Assume a virtual adjustment is made to a certain parameter, such as increasing the heating power by 5% while keeping other parameters unchanged. Based on the structural equation of the Bayesian causal graph, how the simulated and adjusted parameters affect the quality index, such as the yield rate, through the causal relationship chain, and obtain the influence parameters of the adjusted parameters on the causal relationship chain, such as the numerical change of the yield rate. Through the above process, isolate the causal effect of a single parameter adjustment and exclude the interference of confounding factors such as environmental noise.

[0080] Based on the influence parameters, calculate the marginal influence of each parameter on the quality index. Adjust the parameters in the Bayesian causal graph to obtain the adjustment parameter Δp of the production process i as:

[0081]

[0082] where represents the sensitivity of the preset index Y to the change of the i-th parameter p i Pa(Y) represents the set of causal parent nodes directly related to the preset index Y in the Bayesian causal graph, and ∫·dY represents the integral operation of Y.

[0083] Generate the parameter adjustment direction and amplitude that meet the actual production conditions in this way to ensure the safety and feasibility of the optimization plan. For example, when the heating power increases by 1%, the yield is expected to increase by 0.3%. If the parameter adjustment has a positive impact on the quality index, such as an increase in the yield, it is recommended to increase this parameter; if it is negative, such as a sharp increase in energy consumption, it is recommended to reduce this parameter.

[0084] The Bayesian causal graph constructed through the above process explicitly models the causal relationship between defect characteristics, physical characteristics, and environmental characteristics. By analyzing the impact of parameter adjustment on the quality index through virtual intervention, key process parameters can be identified. Compared with the correlation-based method, this causal reasoning can avoid spurious associations and ensure the scientific nature of the adjustment recommendations.

[0085] In step S150, generate a process adjustment plan according to the adjustment parameters and the production data. <-

[0086] After obtaining the adjustment parameters and production data, generate a process adjustment plan based on the current production status and in combination with the status to be adjusted.

[0087] In an embodiment of the present application, generating a process adjustment plan according to the adjustment parameters and the production data includes:

[0088] Generate candidate parameter combinations based on the adjustment parameters and the production data, and determine the objective function and constraint conditions according to the actual production requirements;

[0089] Derive diverse initial parameter combinations based on the candidate parameter combinations, and search for the optimal parameter combination from the initial parameter combinations according to the objective function and the constraint conditions;

[0090] Generate a process adjustment plan based on the optimal parameter combination.

[0091] Based on the adjustment parameters and real-time production data, first analyze the correlation and adjustment range between parameters, and combine equipment limitations (such as maximum temperature, pressure threshold) and resource constraints (such as energy consumption budget) to generate candidate parameter combinations covering different adjustment directions. These combinations are injected with diversity through algorithms to avoid over-concentration of the initial solutions. At the same time, define multi-objective functions (such as reducing energy consumption, increasing the yield, extending the equipment life) and hard constraint conditions (such as safety thresholds) according to production requirements, and construct a mathematical model of the optimization problem to delimit the solution space and evaluation criteria for subsequent searches.

[0092] Encode the candidate parameters into a quantum superposition state and use the quantum annealing algorithm to explore a vast number of parameter combinations in parallel. By dynamically adjusting the coupling relationship between parameters, such as the synergistic effect of temperature and pressure, preferentially search for regions sensitive to the objective function. During the search process, eliminate combinations that violate constraints in real time, such as over-temperature scenarios, and perform non-dominated sorting on the remaining solutions to retain the Pareto-optimal solution set that is balanced among multiple objectives.

[0093] Decode the optimal parameter combination to obtain specific process instructions, such as setting the temperature to 480 °C, adjusting the pressure gradient, and sort them by priority, such as giving priority to ensuring the product qualification rate. Simulate the execution effect of the solution through the digital twin module to verify its stability and feasibility in the virtual environment. The solution that passes the verification is converted into control instructions and sent to the production line for execution. At the same time, the actual operation data is fed back to the optimization system to form a closed loop of optimization-verification-execution-feedback, continuously improving the adaptability and reliability of the process parameters.

[0094] The above process searches for the optimal parameter combination considering real-time working condition constraints. This process balances multiple conflicting objectives and can provide more comprehensive decision-making support compared to single-objective optimization. For example, it can give multiple feasible solutions such as high quality and high consumption and low quality and energy saving at the same time.

[0095] In an embodiment of the present application, after generating a process adjustment plan according to the adjusted parameters and the production data, it further includes: forwarding the process adjustment plan to a control terminal and displaying it on the interface of the control terminal.

[0096] In an embodiment of the present application, the finally generated process adjustment plan, such as the temperature setting value, pressure parameters, etc., is transmitted to the control terminal of the production line in real time through an industrial communication protocol and dynamically displayed in the form of a structured table and a trend chart on the human-machine interface of the terminal.

[0097] Optionally, the interface will highlight the adjustment range of key parameters, such as the heating power +5%. At the same time, an operation confirmation button and an explanation of the adjustment reason are provided, such as to improve the product qualification rate.

[0098] Optionally, the control terminal automatically checks whether the parameters are within the safe range of the equipment. If they exceed the limit, an alarm is triggered, and the parameter update is executed after manual confirmation to ensure the safety and controllability of the adjustment process.

[0099] Optionally, after execution, the actual effect data of the new parameters, such as the real-time temperature curve, is synchronously fed back to the interface to form a closed-loop monitoring.

[0100] The final process adjustment plan is transmitted to the control terminal through a standardized interface, and the adjustment content and basis are displayed in a visual form. The system continuously monitors the implementation effect to form a closed loop of optimization - execution - feedback. This dynamic adjustment mechanism enables process parameters to adapt to changes in the production environment, improving the system's adaptability compared to fixed parameter settings.

[0101] In the technical solution of this application, production data during the operation of the factory is obtained through sensors; the production data is pre - processed to obtain first data, and a multi - modal tensor is generated according to the data source and correlation relationship of the first data; the multi - modal tensor is decoupled, and the obtained image sub - modal data and acoustic sub - modal data are input into a defect recognition network to output defect features; based on the defect features, physical features representing the process physical state and environmental features in the production data, a Bayesian causal graph is constructed, and adjustment parameters of the production process are output through the Bayesian causal graph; according to the adjustment parameters and the production data, a process adjustment plan is generated. By constructing a spatio - temporally aligned multi - modal tensor through multi - modal data fusion, efficient fusion of heterogeneous data is achieved. Combining the Bayesian causal graph to reveal the causal relationship between process parameters and quality indicators, the pseudo - association defect of correlation analysis is avoided. Finally, a process adjustment plan is generated through multi - objective optimization, forming a closed - loop optimization from data perception to decision execution, enabling process parameters to have dynamic adaptive capabilities and improving the accuracy of process optimization and the yield rate.

[0102] The following introduces the device embodiments of this application, which can be used to execute the intelligent factory process optimization method based on multi - modal data fusion in the above - mentioned embodiments of this application. It can be understood that the device can be a computer program (including program code) running in a computer device, for example, the device is an application software; the device can be used to execute the corresponding steps in the method provided by the embodiments of this application. For details not disclosed in the device embodiments of this application, please refer to the embodiments of the intelligent factory process optimization method based on multi - modal data fusion in the above - mentioned of this application.

[0103] Figure 3 The block diagram of an intelligent factory process optimization system based on multi - modal data fusion according to an embodiment of this application is shown.

[0104] Refer to Figure 3 As shown, an intelligent factory process optimization system based on multi - modal data fusion according to an embodiment of this application includes:

[0105] An acquisition unit 310, configured to obtain production data during the operation of the factory through sensors;

[0106] A tensor unit 320, configured to pre - process the production data to obtain first data, and generate a multi - modal tensor according to the data source and correlation relationship of the first data;

[0107] A detection unit 330, configured to decouple the multimodal tensor, input the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and output defect features;

[0108] An adjustment unit 340, configured to construct a Bayesian causal graph based on the defect features, physical features representing the process physical state in the production data, and environmental features, and output adjustment parameters of the production process through the Bayesian causal graph;

[0109] A solution unit 350, configured to generate a process adjustment solution according to the adjustment parameters and the production data.

[0110] In this application, based on the foregoing solution, the production data obtained by the sensor during the operation of the factory includes: temperature data, image streams, waveform data, and production logs obtained by the sensor during the operation of the factory, as the production data.

[0111] In this application, based on the foregoing solution, the production data is preprocessed to obtain first data, and a multimodal tensor is generated according to the data source and association relationship of the first data, including: cleaning the production data to obtain first data, and determining the association relationship and corresponding time vectors between the first data; generating tensor nodes based on the data sources corresponding to the first data, and generating tensor hyperedges according to the association relationship and the time vectors between the first data; generating a multimodal tensor according to the tensor nodes and the tensor hyperedges.

[0112] In this application, based on the foregoing solution, the multimodal tensor is decoupled, and the obtained image sub-modal data and acoustic sub-modal data are input into a defect recognition network to output defect features, including: decoupling the multimodal tensor to separate the image sub-modal data and acoustic sub-modal data therefrom; constructing a defect recognition network based on an encoder, a generator, and a discriminator, and training the defect recognition network with a mixed data composed of normal samples and defect samples; inputting the image sub-modal data and acoustic sub-modal data into the defect recognition network respectively, separating abnormal features through a reconstruction loss and a distribution alignment loss, and outputting defect features.

[0113] In this application, based on the foregoing solution, a Bayesian causal graph is constructed based on defect features, physical features representing the physical state of the process in production data, and environmental features, and adjustment parameters for the production process are output through the Bayesian causal graph, including: constructing a Bayesian causal graph based on defect features, physical features representing the physical state of the process in production data, and environmental features; performing simulation adjustment on the parameters in the Bayesian causal graph to obtain the influence parameters of the parameters after simulation adjustment on the relationship chain in the Bayesian causal graph; and adjusting the parameters in the Bayesian causal graph based on the influence parameters to obtain the adjustment parameters for the production process.

[0114] In this application, based on the foregoing solution, generating a process adjustment plan according to the adjustment parameters and the production data includes: generating candidate parameter combinations based on the adjustment parameters and the production data, and determining an objective function and constraint conditions according to actual production requirements; deriving diversified initial parameter combinations based on the candidate parameter combinations, and searching for the optimal parameter combination from the initial parameter combinations according to the objective function and the constraint conditions; and generating a process adjustment plan based on the optimal parameter combination.

[0115] In this application, based on the foregoing solution, after generating a process adjustment plan according to the adjustment parameters and the production data, it further includes: forwarding the process adjustment plan to a control terminal and displaying it on the interface of the control terminal.

[0116] The technical solution of this application obtains production data during the operation of the factory through sensors; preprocesses the production data to obtain first data, and generates a multimodal tensor according to the data source and association relationship of the first data; decouples the multimodal tensor, inputs the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network to output defect features; constructs a Bayesian causal graph based on defect features, physical features representing the physical state of the process in production data, and environmental features, and outputs adjustment parameters for the production process through the Bayesian causal graph; and generates a process adjustment plan according to the adjustment parameters and the production data. By constructing a spatio-temporally aligned multimodal tensor through multimodal data fusion, efficient fusion of heterogeneous data is achieved. Combining the Bayesian causal graph reveals the causal relationship between process parameters and quality indicators, avoiding the pseudo-association defect of correlation analysis. Finally, a process adjustment plan is generated through multi-objective optimization, forming a closed-loop optimization from data perception to decision execution, enabling process parameters to have dynamic adaptability and improving the accuracy and yield of process optimization.

[0117] Figure 4 The structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of this application is shown.

[0118] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0119] The computer system in this embodiment includes a central processing unit 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory 402 or the program loaded from the storage section 408 into the random access memory 403, such as executing the intelligent factory process optimization method based on multi-modal data fusion described in the above embodiments. In the random access memory 403, various programs and data required for system operation are also stored. The central processing unit 401, the read-only memory 402, and the random access memory 403 are connected to each other via a bus 404. The input / output interface 405 is also connected to the bus 404.

[0120] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that the computer program read from it can be installed into the storage section 408 as needed.

[0121] Specifically, according to the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication section 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the system of the present application are executed.

[0122] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of a computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0124] The units involved in the embodiments described in this application can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation to the unit itself in some cases.

[0125] According to one aspect of the present application, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above various alternative implementation manners.

[0126] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist alone without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the intelligent factory process optimization method based on multi-modal data fusion described in the above embodiments.

[0127] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0128] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented in software or in the form of software combined with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (such as a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0129] After considering the specification and practicing the disclosed embodiments herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application.

[0130] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. An intelligent factory process optimization method based on multi-modal data fusion, characterized in that Including: Obtaining production data during the factory working process through sensors; Preprocessing the production data to obtain first data, and generating a multimodal tensor according to the data source and correlation relationship of the first data; Decoupling the multimodal tensor, inputting the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and outputting defect features; Based on the defect features, physical features representing the process physical state in the production data, and environmental features, constructing a Bayesian causal graph, and outputting adjustment parameters of the production process through the Bayesian causal graph; Generating a process adjustment plan according to the adjustment parameters and the production data.

2. The intelligent factory process optimization method based on multi-modal data fusion according to claim 1, characterized in that The obtaining production data during the factory working process through sensors includes: Obtaining temperature data, image streams, waveform data, and production logs during the factory working process through sensors as the production data.

3. The intelligent factory process optimization method based on multi-modal data fusion according to claim 1, wherein, The preprocessing the production data to obtain first data, and generating a multimodal tensor according to the data source and correlation relationship of the first data includes: Performing data cleaning on the production data to obtain first data, and determining the correlation relationship and corresponding time vectors between the first data; Generating tensor nodes based on the data sources corresponding to the first data, and generating tensor hyperedges according to the correlation relationship and the time vectors between the first data; Generating a multimodal tensor according to the tensor nodes and the tensor hyperedges.

4. The intelligent factory process optimization method based on multi-modal data fusion according to claim 1, characterized in that The decoupling the multimodal tensor, inputting the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and outputting defect features includes: Decoupling the multimodal tensor to separate out image sub-modal data and acoustic sub-modal data; Constructing a defect recognition network based on an encoder, a generator, and a discriminator, and training the defect recognition network with mixed data composed of normal samples and defect samples; Inputting the image sub-modal data and the acoustic sub-modal data into the defect recognition network respectively, separating abnormal features through reconstruction loss and distribution alignment loss, and outputting defect features.

5. The intelligent factory process optimization method based on multi-modal data fusion according to claim 1, wherein, The constructing a Bayesian causal graph based on the defect features, physical features representing the process physical state in the production data, and environmental features, and outputting adjustment parameters of the production process through the Bayesian causal graph includes: Constructing a Bayesian causal graph based on the defect features, physical features representing the process physical state in the production data, and environmental features; Performing simulation adjustment on the parameters in the Bayesian causal graph, and obtaining influence parameters of the relationship chain in the Bayesian causal graph after the simulation adjustment; Adjusting the parameters in the Bayesian causal graph based on the influence parameters to obtain adjustment parameters of the production process.

6. The intelligent factory process optimization method based on multi-modal data fusion according to claim 1, characterized in that The generating a process adjustment plan according to the adjustment parameters and the production data includes: Generating candidate parameter combinations based on the adjustment parameters and the production data, and determining an objective function and constraint conditions according to actual production requirements; Deriving diversified initial parameter combinations based on the candidate parameter combinations, and searching for an optimal parameter combination from the initial parameter combinations according to the objective function and the constraint conditions; Generating a process adjustment plan based on the optimal parameter combination.

7. The intelligent factory process optimization method based on multi-modal data fusion according to claim 1, wherein After generating the process adjustment plan according to the adjustment parameters and the production data, it further includes: Forwarding the process adjustment plan to the control terminal and displaying it on the interface of the control terminal.

8. An intelligent factory process optimization system based on multi-modal data fusion, characterized in that, It includes: An acquisition unit for acquiring production data during the operation of the factory through sensors; A tensor unit for preprocessing the production data to obtain first data and generating a multimodal tensor according to the data source and the association relationship of the first data; A detection unit for decoupling the multimodal tensor, inputting the obtained image sub-modal data and acoustic sub-modal data into a defect recognition network, and outputting defect features; An adjustment unit for constructing a Bayesian causal graph based on the defect features, physical features representing the process physical state in the production data, and environmental features, and outputting adjustment parameters of the production process through the Bayesian causal graph; A plan unit for generating a process adjustment plan according to the adjustment parameters and the production data.

9. The intelligent factory process optimization system based on multi-modal data fusion according to claim 8, characterized in that The acquisition of production data during the operation of the factory through sensors includes: Acquiring temperature data, image streams, waveform data, and production logs during the operation of the factory through sensors as the production data.

10. The intelligent factory process optimization system based on multi-modal data fusion according to claim 8, characterized in that, The preprocessing of the production data to obtain first data and the generation of a multimodal tensor according to the data source and the association relationship of the first data include: Performing data cleaning on the production data to obtain first data, and determining the association relationship between the first data and the corresponding time vectors; Generating tensor nodes based on the data sources corresponding to the first data, and generating tensor hyperedges according to the association relationship between the first data and the time vectors; Generating a multimodal tensor according to the tensor nodes and the tensor hyperedges.

Images