Manufacturing process perception and quality control method and device based on multiple modes

Abstract

The invention discloses a manufacturing process sensing and quality control method and device based on multiple modes, and the method comprises the steps: collecting the multi-mode sensing data in the manufacturing process, and the multi-mode sensing data comprise image data, sound or vibration data and thermal imaging data; extracting process features corresponding to each mode based on the multi-mode perception data, and performing adaptive fusion on the process features of each mode based on an attention mechanism to generate a fusion feature vector; inputting the fusion feature vector and the current process parameter into a quality prediction model to obtain a prediction result of a preset quality index, and determining an influence relationship of the process parameter on the preset quality index based on the quality prediction model; and based on the prediction result, the influence relationship and a preset control target, optimizing and adjusting the process parameters so as to carry out feedforward control on the manufacturing process. Through multi-modal sensing, feature fusion and model prediction, online feedforward closed-loop control in the manufacturing process is realized, and the product quality consistency and the production efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of precision manufacturing technology, and in particular to a method and equipment for manufacturing process sensing and quality control based on multimodal methods. Background Technology

[0002] In the traditional field of precision manufacturing, process sensing and quality control technologies currently generally adopt a technical model of "single sensing, offline detection, and open-loop control," which has obvious limitations in terms of process sensing, quality control, and system collaboration.

[0003] At the process perception level, existing technologies mainly rely on single-type sensors such as X-rays, beta rays, or laser thickness gauges, which can only detect local quality parameters such as areal density and thickness. They cannot comprehensively acquire multi-dimensional process information such as the state of the coating die slurry, substrate transmission vibration, and drying temperature field distribution. Due to the lack of simultaneous acquisition and fusion of multi-modal data such as visual, acoustic, and thermal data, there are "perception blind spots" for key features such as product surface texture defects, front and back homogeneity, abnormal equipment sound patterns, and electrode temperature gradients, making it difficult to achieve a comprehensive digital representation of the manufacturing process status.

[0004] At the quality control level, quality judgment heavily relies on offline sampling and inspection. This method results in a lag in process parameter adjustments, with response times often reaching several hours, making it unsuitable for high-speed coating operations exceeding 60 m / min. Furthermore, due to the limited dimension of the perceived data and its disconnect from the control system, it is difficult to establish a real-time, accurate correlation model between "equipment operating status - process parameter fluctuations - product quality deviations." This leads to frequent process quality problems such as poor areal density uniformity and large alignment errors, making it difficult to guarantee product quality consistency and production yield.

[0005] At the system integration level, existing sensing modules are typically simply spliced ​​onto manufacturing equipment in an "external" manner, with each subsystem (such as sensing, analysis, and control) operating independently. The data transmission and processing chain is long and has high latency, failing to achieve deep collaboration and real-time closed loop from perception, analysis, decision-making to execution. Essentially, it is still an open-loop or post-event adjustment system.

[0006] In summary, existing technologies have inherent defects such as limited sensing dimensions, lagging quality control, and poor system coordination, which cannot meet the real-time sensing and online collaborative control requirements of high-speed, high-precision intelligent manufacturing under complex working conditions. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a multimodal manufacturing process perception and quality control method and device. Through multimodal perception, feature fusion, and model prediction, it enables online feedforward closed-loop control of the manufacturing process, thereby improving product quality consistency and production efficiency.

[0008] In a first aspect, embodiments of the present invention provide a manufacturing process perception and quality control method based on multimodal sensing. The method includes: collecting multimodal sensing data during the manufacturing process, including image data, sound or vibration data, and thermal imaging data; extracting process features corresponding to each modality based on the multimodal sensing data, and adaptively fusing the process features of each modality based on an attention mechanism to generate a fused feature vector; inputting the fused feature vector and current process parameters into a quality prediction model to obtain prediction results for a preset quality index, and determining the influence relationship between process parameters and the preset quality index based on the quality prediction model; wherein the quality prediction model is trained based on multimodal fused features and historical process parameter data; and optimizing and adjusting process parameters based on the prediction results, influence relationships, and preset control objectives to perform feedforward control of the manufacturing process; wherein the preset control objective is to make the predicted quality approach a preset threshold.

[0009] In one possible embodiment, acquiring multimodal sensing data during the manufacturing process includes:

[0010] Data is acquired using visual sensors, acoustic or vibration sensors, and infrared thermal imaging sensors during the manufacturing process.

[0011] Based on a hardware synchronization mechanism, data from different sensors are aligned in time.

[0012] In one possible embodiment, based on multimodal sensing data, process features corresponding to each modality are extracted, including:

[0013] For image data, visual features related to surface defects and geometric dimensions are extracted using a convolutional neural network;

[0014] For sound or vibration data, frequency domain features related to the equipment's operating status are extracted through time-frequency analysis;

[0015] For thermal imaging data, thermal features related to temperature field distribution are extracted through statistical analysis.

[0016] In one possible embodiment, the process features of each modality are adaptively fused based on an attention mechanism to generate a fused feature vector, including:

[0017] Obtain the current manufacturing process's operating context information;

[0018] Based on the operating context information, dynamic weights are assigned to the process features of each mode;

[0019] Based on the assigned dynamic weights, the weighted features of each modal process are fused to generate a fused feature vector.

[0020] In one possible embodiment, the quality prediction model is a hybrid prediction model, including a data-driven sub-model and a physical mechanism sub-model; the data-driven sub-model is trained based on historical data to learn nonlinear mapping relationships; the physical mechanism sub-model is built based on the physical principles of the manufacturing process to provide prediction constraints.

[0021] In one possible embodiment, based on the prediction results, the influencing relationships, and the preset control objectives, process parameters are optimized and adjusted to perform feedforward control of the manufacturing process, including:

[0022] With the optimization objectives of minimizing the deviation between the prediction quality and the preset threshold, and the smoothness of the control action, a rolling time-domain optimization problem is constructed.

[0023] Based on the influence relationship, the model predictive control algorithm is used to solve the rolling time domain optimization problem and obtain the optimal process parameter adjustment sequence within a preset time period;

[0024] The real-time setpoints in the optimal process parameter adjustment sequence are sent to the manufacturing execution system.

[0025] In one possible embodiment, the method further includes:

[0026] Obtain actual quality measurement results of the products produced during the manufacturing process;

[0027] The actual quality measurement results are compared with the predicted results to generate the prediction deviation.

[0028] Based on prediction bias, the parameters of the quality prediction model are fine-tuned and adaptively calibrated online.

[0029] Secondly, embodiments of the present invention provide a multimodal manufacturing process sensing and quality control device, which includes: a data acquisition module, an extraction module, a prediction module, and a control module. Wherein:

[0030] The acquisition module is used to acquire multimodal sensing data during the manufacturing process. The multimodal sensing data includes image data, sound or vibration data, and thermal imaging data.

[0031] The extraction module is used to extract process features corresponding to each modality based on multimodal perception data, and to adaptively fuse the process features of each modality based on an attention mechanism to generate a fused feature vector.

[0032] The prediction module is used to input the fused feature vector and the current process parameters into the quality prediction model to obtain the prediction results of the preset quality indicators, and to determine the influence relationship of the process parameters on the preset quality indicators based on the quality prediction model; wherein, the quality prediction model is trained based on multimodal fused features and historical process parameter data;

[0033] The control module is used to optimize and adjust process parameters based on prediction results, influence relationships and preset control objectives to perform feedforward control of the manufacturing process; wherein, the preset control objective is to make the predicted quality approach a preset threshold.

[0034] Thirdly, embodiments of the present invention provide a computer storage medium storing multiple instructions adapted for loading by a processor and executing the steps of the above-described method.

[0035] Fourthly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being adapted to be loaded by the processor and to execute the steps of the above-described method.

[0036] The beneficial effects of the technical solutions provided by some embodiments of the present invention include at least the following: They enable real-time, high-precision, full-dimensional perception of the manufacturing process status and product quality, eliminating the "perception blind spot" in traditional single-sensor modes; by establishing dynamic quantitative correlation and online rolling prediction between process parameters and quality indicators, quality control is transformed from offline, lagging sampling inspection and manual adjustment to online, feedforward model prediction and automatic optimization, significantly improving system response speed and control accuracy, and solving the problem of control lag under high-speed production conditions; furthermore, by introducing online self-learning of models based on actual production feedback... The calibration mechanism enables the system to continuously track process drift and adapt to changes in operating conditions over a long period, thus ensuring the robustness of the control effect. Ultimately, this solution achieves deep collaboration and closed-loop optimization among perception, analysis, decision-making, execution, and learning, promoting the evolution of the manufacturing process towards autonomous and intelligent operation. While significantly improving product quality consistency, production yield, and process stability, it effectively reduces quality loss, material waste, and energy consumption, and reduces reliance on industry skills and human intervention. It provides an efficient, reliable, and self-optimizing solution for demanding large-scale precision manufacturing of new energy batteries and other applications. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A system architecture diagram of a multimodal manufacturing process sensing and quality control system provided for an embodiment of the present invention;

[0039] Figure 2A flowchart illustrating a multimodal manufacturing process sensing and quality control method provided in an embodiment of the present invention;

[0040] Figure 3 A structural block diagram of a multimodal manufacturing process sensing and quality control device provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the features and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0044] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances. Furthermore, in the description of this invention, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0045] As mentioned earlier, in the traditional precision manufacturing field, current process sensing and quality control generally adopt a "single sensor, offline detection, open-loop control" model. At the sensing level, it relies heavily on single sensors such as X / β-ray and laser thickness gauges, which can only acquire local parameters such as areal density and thickness. It cannot comprehensively capture multi-dimensional process information such as the state of the coating die slurry, substrate transmission vibration, and drying temperature field distribution, resulting in significant "sensing blind spots." Quality control heavily relies on offline sampling inspection, leading to process adjustments lagging by several hours, making it difficult to match high-speed production rhythms of over 60 m / min. Furthermore, it cannot establish a real-time dynamic correlation between "equipment status - process parameters - quality deviation," resulting in problems such as poor product uniformity and large alignment errors.

[0046] The aforementioned situation highlights three major shortcomings of existing technologies: limited sensing dimensions, lagging quality control, and poor system coordination. Each sensing, analysis, and control module often operates in isolation, forming "information silos" and lacking the collaborative capability to move from multi-dimensional real-time perception to online intelligent decision-making and precise closed-loop execution. This fundamentally hinders the development of high-speed, high-precision intelligent manufacturing under complex operating conditions.

[0047] In view of this, the present invention provides a method and device for manufacturing process perception and quality control based on multimodal sensing. The aim is to achieve full-dimensional, high-precision real-time perception of the manufacturing process status and product quality by constructing a technical solution of "multimodal synchronous perception - adaptive feature fusion - hybrid model quality prediction - feedforward closed-loop collaborative control," effectively eliminating the "perception blind spot" of traditional single-sensor mode. By establishing dynamic quantitative correlation and online prediction between process parameters and quality indicators, quality control is transformed from offline lagging adjustment to online feedforward regulation, improving system response speed and control accuracy, and solving the problem of regulation lag under high-speed production conditions. Finally, through deep collaboration and closed-loop optimization of perception, analysis, decision-making, and execution, adaptive and intelligent operation of the manufacturing process is achieved, significantly improving production efficiency and system stability while ensuring product consistency and yield. This is particularly suitable for large-scale industrial scenarios with stringent requirements for high-speed and high-precision manufacturing, such as new energy batteries.

[0048] Please see Figure 1 , Figure 1 This is an exemplary system architecture diagram of a multimodal manufacturing process perception and quality control method provided in an embodiment of the present invention.

[0049] like Figure 1As shown, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth communication links, Wireless-Fidelity (Wi-Fi) communication links, or microwave communication links, etc.

[0050] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to smartwatches, smartphones, tablets, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0051] In this embodiment of the invention, terminal 101 can collect multimodal sensing data during the manufacturing process, including image data, sound or vibration data, and thermal imaging data; based on the multimodal sensing data, process features corresponding to each mode are extracted, and based on an attention mechanism, the process features of each mode are adaptively fused to generate a fused feature vector; the fused feature vector and the current process parameters are input into a quality prediction model to obtain a prediction result for a preset quality index, and based on the quality prediction model, the influence relationship of process parameters on the preset quality index is determined; wherein, the quality prediction model is trained based on the multimodal fused features and historical process parameter data; based on the prediction result, the influence relationship, and the preset control objective, the process parameters are optimized and adjusted to perform feedforward control of the manufacturing process; wherein, the preset control objective is to make the predicted quality approach a preset threshold.

[0052] Server 103 can be a business server providing various services. It should be noted that server 103 can be either hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0053] Alternatively, the system architecture may not include server 103. In other words, server 103 may be an optional device in the embodiments of this specification. That is, the method provided in the embodiments of this specification can be applied to a system structure that only includes terminal 101. The embodiments of this invention do not limit this.

[0054] It should be understood that Figure 1 The number of terminals, networks, and servers shown is only illustrative; the number can be any number of terminals, networks, and servers depending on the implementation requirements.

[0055] Please see Figure 2 , Figure 2 This is a flowchart illustrating a multimodal manufacturing process perception and quality control method provided in an embodiment of the present invention. The executing entity in this embodiment can be a vehicle performing multimodal manufacturing process perception and quality control, a processor within the vehicle performing the multimodal manufacturing process perception and quality control method, or a multimodal manufacturing process perception and quality control service within the vehicle performing the multimodal manufacturing process perception and quality control method. For ease of description, the following uses a processor within a vehicle as an example to illustrate the specific execution process of the multimodal manufacturing process perception and quality control method.

[0056] like Figure 2 As shown, a multimodal manufacturing process awareness and quality control method can include at least the following:

[0057] S201. Collect multimodal sensing data during the manufacturing process. Multimodal sensing data includes image data, sound or vibration data, and thermal imaging data.

[0058] Specifically, during the manufacturing process, sensor groups with different sensing dimensions are deployed in parallel. These include high-speed visual sensors for capturing product appearance and geometry, high-fidelity acoustic or vibration sensor arrays for monitoring the mechanical state of equipment and process anomalies, and infrared thermal imagers for monitoring the temperature field distribution in the processing area. In one possible implementation, to achieve strict temporal comparability of these heterogeneous sensor data, a hardware synchronization mechanism is used to align data from different sensors in time. Specifically, a hardware-level synchronization scheme based on Precise Time Protocol (PTP) is adopted to provide a unified high-precision clock source for all sensors, using the main drive encoder pulse of the manufacturing equipment as a global trigger signal to ensure that visual frames, sound / vibration waveform slices, and thermal imaging matrices are acquired at the same physical moment. Furthermore, to achieve spatial correlation and fusion of multi-source information, a unified spatial coordinate system needs to be established. This step is accomplished through a specially designed multimodal calibration target, which simultaneously contains a high-contrast visual pattern, a sound source capable of emitting sound waves at a specific frequency, and a thermal imager with specific thermal radiation characteristics. The system precisely places the target on the path of the manufacturing workpiece and traverses the sensor's field of view. After synchronously collecting data from each sensor, a calibration algorithm is used to calculate the transformation relationship between the local coordinate system of each sensor and the world coordinate system based on the workpiece. For acoustic or vibration sensor arrays, it is necessary to additionally emit controllable sound pulses at known spatial locations on the coating machine (such as both sides of the coating head) and calculate the transformation relationship between the sound source positioning coordinate system and the aforementioned world coordinate system based on the Time Difference of Arrival (TDOA) principle. Finally, through the above-mentioned hardware and software coordinated spatiotemporal synchronization and calibration process, the raw data streams from vision, acoustic / vibration, and thermal imaging are transformed into a multimodal sensing data set that is strictly time-aligned and can map any detected abnormal signals (such as visual defects, abnormal vibrations, and temperature hotspots) to the same physical coordinate system.

[0059] S202. Based on multimodal perception data, extract the process features corresponding to each modality, and based on the attention mechanism, adaptively fuse the process features of each modality to generate a fused feature vector.

[0060] Specifically, for multimodal sensing data that has achieved spatiotemporal synchronization, process features corresponding to each modality are extracted based on the multimodal sensing data. Specifically, image data is first input into a pre-trained lightweight convolutional neural network, which automatically extracts depth visual features related to product appearance integrity, surface defects, and geometric dimensional accuracy, forming a visual feature vector. Simultaneously, sound or vibration data is preprocessed, including noise reduction and feature band filtering, and then converted to a time-frequency representation through short-time Fourier transform or wavelet packet decomposition. Features reflecting the mechanical state of the equipment and the stability of the process, such as frequency domain energy and spectral entropy, are extracted from this representation to form an acoustic / vibration feature vector. For infrared thermal imaging data, the statistical and dynamic characteristics of its temperature matrix are calculated. The statistical characteristics include the average temperature, standard deviation, and non-uniformity of the entire region of interest, while the dynamic characteristics are calculated based on the temperature rise rate and spatial temperature gradient using continuous frame data, thus forming a thermal imaging feature vector.

[0061] Furthermore, simple feature concatenation is insufficient to capture the complex interactions and dynamic importance between modalities. To address this, an attention-enhanced multimodal temporal fusion network is proposed. The core of this network is a cross-modal attention mechanism that dynamically evaluates which modal features are more critical to the current process state and quality assessment under specific process stages and conditions. For example, when vibration features show an abnormal surge in energy in a specific frequency band, the network automatically increases the weight of vibration features to focus on potential tool wear or spindle failure; when thermal imaging shows a sharp change in local temperature gradient, it pays more attention to thermal features to warn of potential thermal damage or deformation risks. Mathematically, for a visual feature sequence V, an acoustic / vibration feature sequence A, and a thermal feature sequence I, the fusion network calculates a set of context-dependent adaptive attention weights and performs deep fusion of the weighted multimodal features through a gated fusion unit, ultimately outputting a unified temporal fusion feature vector rich in multi-source complementary information. This vector provides a more comprehensive and robust digital representation of the "state-quality" relationship in the manufacturing process. For the visual feature sequence V… t Acoustic / vibration characteristic sequence A t and thermal characteristic sequence I t (The subscript t indicates the temporal dimension). The fusion network achieves deep fusion through context-aware attention and gating units. The model formula is as follows:

[0062]

[0063] in, The temporal fusion feature vector at time t is a unified digital representation of the "state-quality" correlation. , , For dependent working condition context vectors Dynamic attention weights (generated from processing parameters, process stages, etc.) are used to... calculate( These correspond to visual, acoustic / vibration, and infrared thermal imaging modes, respectively. (Representing the dimensions of the original features of each modality), enabling precise focusing of key modalities under specific operating conditions; The adaptation mapping function for each modality feature corresponds to the visual features extracted by the lightweight CNN, the acoustic / vibration features after time-frequency analysis, and the thermal features after temperature statistics calculation, respectively. It is used to unify the heterogeneous features of different modalities into the same dimensional space while retaining the key information of each modality. This represents element-wise multiplication. For the model's learnable parameters, The Sigmoid activation function is used to achieve effective information filtering and nonlinear fusion.

[0064] Additionally, the formula for the acoustic / vibration mode adaptation mapping function is as follows:

[0065]

[0066] in, The mapping process involves: first, performing time-frequency analysis by converting the one-dimensional time-domain signal into a two-dimensional time-frequency matrix using short-time Fourier transform or wavelet packet decomposition; then, extracting frequency domain features (such as peak values ​​in each frequency band, frequency band energy proportions, and spectral entropy) to form the original frequency domain feature vector. Then normalization is performed on the data. Standardization is performed to eliminate the influence of signal amplitude differences under different operating conditions; finally, linear mapping is performed using a learnable matrix. Mapping the normalized frequency domain features to a k-dimensional space, the bias term... Correct the offset. Output: The acoustic / vibration feature vector is a dimensionally unified vector that retains equipment status information such as spindle vibration and tool wear.

[0067] Formula for infrared thermal imaging mode adaptation mapping function:

[0068]

[0069] Among them, input This is the infrared thermal imaging temperature matrix of the processing area. The mapping process is as follows: First, statistical characteristic calculations are performed: global statistics of the temperature matrix are calculated, such as mean, standard deviation, maximum value, minimum value, and non-uniformity (standard deviation / mean), forming a statistical characteristic vector. Then, dynamic feature calculations are performed. Based on sequential continuous frame thermal imaging data, dynamic temperature change indicators are calculated, such as the rate of temperature rise (temperature difference between the current frame and the previous frame / time interval), temperature gradient (maximum rate of temperature change between adjacent pixels), and thermal diffusion rate, forming a dynamic feature vector. Perform feature concatenation again: and The vectors are concatenated to form the original hot feature vectors; finally, a linear mapping is performed using a learnable matrix. The concatenated features are mapped to a k-dimensional space, and the bias term is... Correct the offset. Output. The thermal feature vector is a dimensionally unified vector that retains information about the stability and uniformity of the thermal process.

[0070] In addition, the operating condition context vector It is a condensed digital representation of the current processing condition, used to guide the dynamic allocation of attention weights. Its calculation process is based on available process parameters and equipment status information. Specifically, it first acquires information sources about the industrial scenario, including process parameters: process parameter values ​​during the manufacturing process; process stage labels: labels obtained from the Manufacturing Execution System (MES) for the current process; and equipment historical status: equipment operating status indicators for the previous n time steps. Next, it calculates information encoding, directly standardizing numerical parameters (z-score normalization) to eliminate dimensional differences; categorical parameters (such as process stage labels): converting them into binary vectors using one-hot encoding; feature concatenation: concatenating the encoded numerical parameter vector, categorical parameter vector, and equipment historical status vector into the original processing condition information vector (the dimension is the sum of the dimensions of each input vector); dimensionality compression and feature fusion: compressing the original processing condition information vector through a lightweight fully connected network (2-layer perceptron, MLP) to obtain a low-dimensional context vector.

[0071]

[0072] in, , For learnable parameters of a fully connected network, The activation function introduces nonlinearity, which enhances the ability of the context vector to represent complex working conditions.

[0073] S203. Input the fused feature vector and the current process parameters into the quality prediction model to obtain the prediction results of the preset quality indicators, and determine the influence relationship of the process parameters on the preset quality indicators based on the quality prediction model.

[0074] Specifically, the time-series fusion feature vector Ft and the currently adjustable set of process parameters Pt are used as inputs to a pre-built quality prediction model. The set of process parameters Pt includes, but is not limited to, coating speed, slit gap, feeding pressure, and drying temperature setpoint. In one possible implementation, the quality prediction model is a hybrid prediction model consisting of two parallel sub-modules: a data-driven sub-module, employing a long short-term memory network or temporal Transformer architecture, responsible for learning the complex nonlinear temporal mapping relationship between massive historical fusion feature and process parameter sequence data and the final quality index; and a physical mechanism sub-module, based on a simplified mathematical model constructed from the first principles involved in the manufacturing process. This module uses the same process parameter Pt as input and calculates the prediction baseline of the quality index according to physical laws. The outputs of the two sub-modules, including the data-driven prediction value and the physical mechanism prediction value, are input to an adaptive weighted fusion layer. This layer dynamically calculates and allocates fusion weights based on the confidence level of the current operating condition. For example, when the process is in a stable production stage and historical data is sufficient, the weight of the data-driven sub-module is increased; when the process undergoes significant changes or new materials are used, the weight of the physical mechanism sub-module is increased. Finally, a unified and robust prediction result for key quality indicators over a future period is output through a weighted summation. Simultaneously, the system utilizes automatic differentiation technology embedded in the hybrid model to backpropagate the aforementioned prediction results and calculate the partial derivatives of each adjustable process parameter with respect to the predicted quality indicator. This generates a sensitivity matrix or Jacobian matrix J in real-time online, where each element J... ij It quantitatively reveals "the instantaneous impact of a tiny unit change in the j-th process parameter on the predicted value of the ith quality indicator in the future," which is the real-time dynamic impact relationship of process parameters on preset quality indicators, providing a clear quantitative gradient guide for subsequent precise process optimization.

[0075] S204. Based on the prediction results, the influence relationship and the preset control target, optimize and adjust the process parameters to carry out feedforward control of the manufacturing process.

[0076] Specifically, using the predicted results of key quality indicators for the future time period obtained from the aforementioned steps, the real-time sensitivity matrix J representing the dynamic correlation between the parameters and quality, and the preset control objectives as inputs, a finite-time domain rolling optimization control problem is constructed. In one possible implementation, a multi-objective cost function is constructed as the mathematical expression of the rolling time domain optimization problem, taking into account the deviation between the predicted results and the preset target thresholds (such as the target value of areal density or the upper limit of thickness uniformity) and the requirement to suppress the abrupt change in the control action. This cost function is typically composed of a weighted quadratic term of the prediction deviation and a weighted quadratic term of the control increment (i.e., the adjustment amount of the process parameters at adjacent time points). The constraints of the optimization problem include the upper and lower limits of the physical adjustable range of each process parameter (U_min≤U_t≤U_max), the maximum allowable rate of change within adjacent control cycles (|ΔU_t|≤ΔU_max), and the process model itself as an equality constraint. At each control time t, the system solves the constrained optimization problem online. This problem can usually be transformed into a quadratic programming problem, which yields a series of optimal process parameter adjustment sequences [U_t,U{t+1},...,U*{t+H-1}] from the current time to H future steps. Subsequently, only the first element of this sequence, i.e., the instantaneous control action U*_t, is used. The programmable logic controllers or actuators in the manufacturing process are sent to make real-time adjustments to key process parameters such as coating speed, feeding pressure, and drying temperature. This adjustment is proactively issued based on the prediction of future states, forming the core of feedforward control. In the next control cycle (time t+1), the system will acquire the latest actual process measurement data (including new multimodal sensing data and possible quality inspection feedback), and re-execute the complete process from feature fusion, quality prediction to rolling optimization, thereby forming a closed-loop control mechanism of "prediction-optimization-execution-feedback-update", enabling the system to continuously track and proactively compensate for disturbances and deviations in the process.

[0077] Furthermore, after completing real-time parameter adjustments based on model predictive control, the system initiates an online adaptive calibration process to maintain the long-term accuracy of the predictive model. This process first acquires the actual quality measurement results from the manufacturing process, sources including but not limited to: real-time data streams from high-precision online areal density gauges and thickness gauges deployed at the end of the production line, and results from the Manufacturing Execution System (MES) recording coating adhesion, residual solvent content, etc., from offline laboratory testing of periodically sampled electrode sheets. These measured data are strictly correlated with the corresponding product production batch and timestamp. Subsequently, the system automatically compares the actual quality measurement results with the predicted results, generating a prediction deviation. Specifically, the data synchronization module precisely matches and aligns the returned measured quality data with the predicted values ​​generated by the quality prediction model at the corresponding time point and process state in the historical database, based on the timestamp. The calculation module then calculates the absolute deviation and relative error for each set of matched data, storing and visualizing these deviation values ​​in a time series to form a real-time monitoring chart of the model prediction error. Based on the continuously accumulated prediction deviation sequence, the system periodically triggers the process of online parameter fine-tuning and adaptive calibration of the quality prediction model. The calibration process is completed within a separate online learning service, employing incremental learning or mini-batch gradient descent algorithms at its core. Specifically, the system uses recent (e.g., the past 24 hours) data pairs of "process parameters-fusion features-measured quality" as the fine-tuning training set. With the optimization objective of minimizing the mean squared error between the model's predicted output and the measured values, the trainable parameters of the data-driven sub-modules (such as LSTM networks) in the hybrid prediction model are updated via small-amplitude, low-learning-rate backpropagation. The parameters of the physical mechanism sub-module are typically kept fixed to maintain the model's physical interpretability. The entire fine-tuning process is performed silently in the production background, without interfering with the real-time control operations in the front end. After one round of fine-tuning, the system automatically hot-deploys the updated model parameters to the online prediction service, replacing the old model. Thus, the quality prediction model can continuously track process characteristic drift caused by slow time-varying factors such as batch fluctuations in slurry formulation, slow wear of the coating head slits, and seasonal changes in environmental temperature and humidity, achieving self-evolution of the model and ensuring the prediction accuracy and control robustness of the entire collaborative control system throughout its lifecycle.

[0078] This invention provides a multimodal manufacturing process perception and quality control method, enabling comprehensive, high-precision real-time perception of the manufacturing process status and product quality, eliminating the "perception blind spots" of traditional single-sensor modes. By establishing dynamic quantitative correlations and online rolling predictions between process parameters and quality indicators, quality control is transformed from offline, lagging sampling inspection and manual adjustment to online, feedforward model prediction and automatic optimization, significantly improving system response speed and control accuracy, and solving the problem of control lag under high-speed production conditions. Furthermore, by introducing online self-regulation based on actual production feedback... The learning and calibration mechanism enables the system to continuously track process drift and adapt to changes in operating conditions over a long period, thus ensuring the robustness of the control effect. Ultimately, this solution achieves deep collaboration and closed-loop optimization among perception, analysis, decision-making, execution, and learning, driving the manufacturing process towards autonomous and intelligent operation. While significantly improving product quality consistency, production yield, and process stability, it effectively reduces quality loss, material waste, and energy consumption, and lowers the reliance on industry skills and human intervention. It provides an efficient, reliable, and self-optimizing solution for demanding large-scale precision manufacturing such as new energy batteries.

[0079] Please see Figure 3 , Figure 3 This is a structural block diagram of a multimodal manufacturing process sensing and quality control device provided in an embodiment of the present invention. Figure 3 As shown: The device 300 for a multimodal manufacturing process sensing and quality control method includes: a data acquisition module 310, an extraction module 320, a prediction module 330, and a control module 340. Wherein:

[0080] The acquisition module 310 is used to acquire multimodal sensing data during the manufacturing process. The multimodal sensing data includes image data, sound or vibration data, and thermal imaging data.

[0081] The extraction module 320 is used to extract process features corresponding to each modality based on multimodal perception data, and to adaptively fuse the process features of each modality based on an attention mechanism to generate a fused feature vector.

[0082] The prediction module 330 is used to input the fused feature vector and the current process parameters into the quality prediction model to obtain the prediction results of the preset quality indicators, and to determine the influence relationship of the process parameters on the preset quality indicators based on the quality prediction model; wherein, the quality prediction model is trained based on multimodal fused features and historical process parameter data;

[0083] The control module 340 is used to optimize and adjust process parameters based on prediction results, influence relationships and preset control objectives to perform feedforward control on the manufacturing process; wherein, the preset control objective is to make the predicted quality approach a preset threshold.

[0084] In some possible embodiments, the acquisition module 310 includes:

[0085] The data acquisition unit is used to acquire data based on visual sensors, acoustic or vibration sensors, and infrared thermal imaging sensors used in the manufacturing process.

[0086] Alignment units are used to align data from different sensors in time based on hardware synchronization mechanisms.

[0087] In some possible embodiments, the extraction module 320 includes:

[0088] The first extraction unit is used to extract visual features related to surface defects and geometric dimensions from image data through a convolutional neural network.

[0089] The second extraction unit is used to extract frequency domain features related to the operating status of the equipment from sound or vibration data through time-frequency analysis.

[0090] The third extraction unit is used to extract thermal features related to the temperature field distribution from thermal imaging data through statistical analysis.

[0091] In some possible embodiments, the extraction module 320 includes:

[0092] The acquisition unit is used to acquire the operating context information of the current manufacturing process;

[0093] The allocation unit is used to assign dynamic weights to the process characteristics of each mode based on the operating condition context information.

[0094] The generation unit is used to fuse the weighted features of each modal process based on the assigned dynamic weights to generate a fused feature vector.

[0095] In some possible embodiments, the quality prediction model is a hybrid prediction model, including a data-driven sub-model and a physical mechanism sub-model; the data-driven sub-model is trained based on historical data to learn nonlinear mapping relationships; the physical mechanism sub-model is built based on the physical principles of the manufacturing process to provide prediction constraints.

[0096] In some possible embodiments, the control module 340 includes:

[0097] The building blocks are used to construct a rolling temporal optimization problem with the optimization objectives of minimizing the deviation between the prediction quality and the preset threshold, as well as the smoothness of the control action.

[0098] The unit is obtained and used to solve the rolling time-domain optimization problem based on the influence relationship using the model predictive control algorithm, so as to obtain the optimal process parameter adjustment sequence within a preset time period;

[0099] The distribution unit is used to distribute the real-time setpoints in the optimal process parameter adjustment sequence to the manufacturing execution system.

[0100] In some possible embodiments, the multimodal manufacturing process sensing and quality control method apparatus 300 further includes:

[0101] The acquisition module is used to acquire the actual quality measurement results of the products produced during the manufacturing process;

[0102] The generation module is used to compare the actual quality measurement results with the predicted results and generate the prediction deviation.

[0103] The calibration module is used to perform online fine-tuning and adaptive calibration of the parameters of the quality prediction model based on the prediction bias.

[0104] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 4 As shown, the electronic device 400 may include: at least one processor 401, at least one network interface 404, user interface 403, memory 405, and at least one communication bus 402.

[0105] The communication bus 402 is used to enable communication between these components.

[0106] The user interface 403 may include a display screen, and the optional user interface 403 may include a standard wired interface or a wireless interface.

[0107] The network interface 404 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0108] The processor 401 may include one or more processing cores. The processor 401 connects to various parts within the electronic device 400 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 405, and by calling data stored in the memory 405. Optionally, the processor 401 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 401 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 401.

[0109] The memory 405 may include random access memory (RAM) or read-only memory. Optionally, the memory 405 may include a non-transitory computer-readable storage medium. The memory 405 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 405 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 405 may also be at least one storage device located remotely from the aforementioned processor 401. Figure 4 As shown, the memory 405, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a multimodal manufacturing process awareness and quality control application.

[0110] exist Figure 4In the electronic device 400 shown, the user interface 403 is mainly used to provide an input interface for the user and acquire user input data; while the processor 401 can be used to call the railway catenary vibration monitoring application stored in the memory 405, and specifically perform the following operations: collect multimodal sensing data during the manufacturing process, including image data, sound or vibration data, and thermal imaging data; based on the multimodal sensing data, extract the process features corresponding to each mode, and based on the attention mechanism, adaptively fuse the process features of each mode to generate a fused feature vector; input the fused feature vector and the current process parameters into the quality prediction model to obtain the prediction results of the preset quality indicators, and based on the quality prediction model, determine the influence relationship of the process parameters on the preset quality indicators; wherein, the quality prediction model is trained based on the multimodal fused features and historical data of process parameters; based on the prediction results, influence relationships, and preset control objectives, optimize and adjust the process parameters to perform feedforward control of the manufacturing process; wherein, the preset control objective is to make the predicted quality approach a preset threshold.

[0111] In some possible embodiments, processor 401 performs the acquisition of multimodal sensing data during the manufacturing process, specifically for performing:

[0112] Data is acquired using visual sensors, acoustic or vibration sensors, and infrared thermal imaging sensors during the manufacturing process.

[0113] Based on a hardware synchronization mechanism, data from different sensors are aligned in time.

[0114] In some possible embodiments, the processor 401 performs process feature extraction based on multimodal sensing data, specifically for executing:

[0115] For image data, visual features related to surface defects and geometric dimensions are extracted using a convolutional neural network;

[0116] For sound or vibration data, frequency domain features related to the equipment's operating status are extracted through time-frequency analysis;

[0117] For thermal imaging data, thermal features related to temperature field distribution are extracted through statistical analysis.

[0118] In some possible embodiments, the processor 401 performs adaptive fusion of process features from each modality based on an attention mechanism to generate a fused feature vector, specifically used for:

[0119] Obtain the current manufacturing process's operating context information;

[0120] Based on the operating context information, dynamic weights are assigned to the process features of each mode;

[0121] Based on the assigned dynamic weights, the weighted features of each modal process are fused to generate a fused feature vector.

[0122] In some possible embodiments, the quality prediction model is a hybrid prediction model, including a data-driven sub-model and a physical mechanism sub-model; the data-driven sub-model is trained based on historical data to learn nonlinear mapping relationships; the physical mechanism sub-model is built based on the physical principles of the manufacturing process to provide prediction constraints.

[0123] In some possible embodiments, the processor 401 performs feedforward control of the manufacturing process by optimizing and adjusting process parameters based on prediction results, influencing relationships, and preset control objectives. Specifically, it performs the following:

[0124] With the optimization objectives of minimizing the deviation between the prediction quality and the preset threshold, and the smoothness of the control action, a rolling time-domain optimization problem is constructed.

[0125] Based on the influence relationship, the model predictive control algorithm is used to solve the rolling time domain optimization problem and obtain the optimal process parameter adjustment sequence within a preset time period;

[0126] The real-time setpoints in the optimal process parameter adjustment sequence are sent to the manufacturing execution system.

[0127] In some possible embodiments, processor 401 is also specifically configured to perform:

[0128] Obtain actual quality measurement results of the products produced during the manufacturing process;

[0129] The actual quality measurement results are compared with the predicted results to generate the prediction deviation.

[0130] Based on prediction bias, the parameters of the quality prediction model are fine-tuned and adaptively calibrated online.

[0131] This invention also provides a computer storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform the above-described actions. Figure 2 One or more steps in the illustrated embodiment. If the constituent modules of the above-described multimodal manufacturing process sensing and quality control device are implemented as software functional units and sold or used as independent products, they can be stored in the computer's storage medium.

[0132] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer storage medium or transmitted through the computer storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0133] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0134] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A manufacturing process sensing and quality control method based on multimodal methods, characterized in that, The method includes: Collect multimodal sensing data during the manufacturing process, including image data, sound or vibration data, and thermal imaging data; Based on the multimodal perception data, process features corresponding to each modality are extracted, and based on the attention mechanism, the process features of each modality are adaptively fused to generate a fused feature vector. The fused feature vector and the current process parameters are input into the quality prediction model to obtain the prediction results for the preset quality indicators. Based on the quality prediction model, the influence relationship between the process parameters and the preset quality indicators is determined. The quality prediction model is trained based on multimodal fused features and historical process parameter data. Based on the prediction results, the influence relationship, and the preset control target, the process parameters are optimized and adjusted to perform feedforward control of the manufacturing process; wherein, the preset control target is to make the predicted quality approach a preset threshold.

2. The method as described in claim 1, characterized in that, The multimodal sensing data collected during the manufacturing process includes: Data is acquired using visual sensors, acoustic or vibration sensors, and infrared thermal imaging sensors during the manufacturing process. Based on a hardware synchronization mechanism, data from different sensors are aligned in time.

3. The method as described in claim 1, characterized in that, The step of extracting process features corresponding to each modality based on the multimodal sensing data includes: For the image data, visual features related to surface defects and geometric dimensions are extracted using a convolutional neural network; For the sound or vibration data, frequency domain features related to the equipment's operating status are extracted through time-frequency analysis; The thermal imaging data is statistically analyzed to extract thermal features related to the temperature field distribution.

4. The method as described in claim 1, characterized in that, The adaptive fusion of process features from each modality based on an attention mechanism to generate a fused feature vector includes: Obtain the current manufacturing process's operating context information; Based on the operating context information, dynamic weights are assigned to the process features of each mode; Based on the assigned dynamic weights, the weighted modal process features are fused to generate the fused feature vector.

5. The method as described in claim 1, characterized in that, The quality prediction model is a hybrid prediction model, comprising a data-driven sub-model and a physical mechanism sub-model. The data-driven sub-model is trained based on historical data to learn nonlinear mapping relationships. The physical mechanism sub-model is constructed based on the physical principles of the manufacturing process and is used to provide prediction constraints.

6. The method as described in claim 5, characterized in that, The step of optimizing and adjusting the process parameters based on the prediction results, the influencing relationships, and the preset control objectives to perform feedforward control of the manufacturing process includes: With the optimization objectives of minimizing the deviation between the prediction quality and the preset threshold, and the smoothness of the control action, a rolling time-domain optimization problem is constructed. Based on the aforementioned influence relationship, the model predictive control algorithm is used to solve the rolling time-domain optimization problem to obtain the optimal process parameter adjustment sequence within a preset time period; The instantaneous set values ​​in the optimal process parameter adjustment sequence are sent to the manufacturing execution system.

7. The method as described in claim 1, characterized in that, The method further includes: Obtain actual quality measurement results of the products produced during the manufacturing process; The actual quality measurement results are compared with the predicted results to generate a prediction deviation. Based on the prediction deviation, the quality prediction model is subjected to online parameter fine-tuning and adaptive calibration.

8. A manufacturing process sensing and quality control device based on multimodal methods, characterized in that, The device includes: The acquisition module is used to acquire multimodal sensing data during the manufacturing process, including image data, sound or vibration data, and thermal imaging data. The extraction module is used to extract process features corresponding to each modality based on the multimodal perception data, and to adaptively fuse the process features of each modality based on an attention mechanism to generate a fused feature vector. The prediction module is used to input the fused feature vector and the current process parameters into the quality prediction model to obtain the prediction result of the preset quality index, and to determine the influence relationship of the process parameters on the preset quality index based on the quality prediction model; wherein, the quality prediction model is trained based on multimodal fused features and historical process parameter data; The control module is used to optimize and adjust the process parameters based on the prediction results, the influence relationship, and the preset control target, so as to perform feedforward control on the manufacturing process; wherein, the preset control target is to make the predicted quality approach a preset threshold.

9. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the steps of the method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described in any one of claims 1 to 7.

Images