A soft sensing method based on multimodal data of process industry digital twin platform

Abstract

The present application provides a soft measurement method based on multimodal data of the process industry digital twin platform, reads the corresponding process variable time series and production process image series and other multimodal data sequences from the data integration layer, unifies the time scale and aligns the time windows of the various multimodal data sequences read, and on this basis, extracts the image feature time series from the image sequence, uses the aligned various time series to perform soft measurement on the corresponding performance indicators, and feeds back the results of the soft measurement of the performance indicators to the data integration layer of the process industry digital twin platform, so as to realize the rapid call of the soft measurement results on other tasks and services, and form a closed loop of information flow. The above method can automatically, quickly and accurately perform soft measurement on some difficult-to-obtain performance indicators in the process industry, improve the effect of soft measurement of process industry performance indicators, and provide reference and guidance for downstream tasks such as monitoring, diagnosis, and optimization control.

Description

Technical Field

[0001] The present application relates to the fields of computer technology and artificial intelligence, and in particular to a soft measurement method based on multimodal data of a process industry digital twin platform. The method is mainly applicable to the real-time identification and estimation tasks of performance indicators that are difficult to obtain directly on the process industry digital twin platform. Background Art

[0002] The following is an introduction to some terms and background technologies involved in this application.

[0003] Process Industry: Process Industry, also known as process industry, refers to the industrial field where the production process involves a large number of physical and chemical changes. It includes but is not limited to metallurgy, energy, chemical industry and other industries. In process industry, the production of products usually needs to go through a series of complex physical and chemical processes, usually involving complex energy flow, material flow and information flow, etc. Therefore, real-time monitoring and control of process industry is very important.

[0004] Digital Twin Platform: A digital twin platform refers to a virtual model corresponding to an actual physical entity that is created and run through digital technology. Based on technologies such as real-time data acquisition, physical simulation, and data analysis, the digital twin platform can monitor, simulate, and optimize the operating status and performance of physical entities in real time. In the field of process industries, by building a digital twin platform for the corresponding production line, real-time monitoring, diagnosis, and optimization control of the process industry can be achieved, thereby improving the production efficiency of the process industry and reducing production costs, which is of great significance to the digital transformation and intelligent development of the process industry.

[0005] Multimodal data: Multimodal data refers to data from different sensors and different types of data sources. In the process industry, multimodal data can include but is not limited to numerical data collected by sensors and image data obtained by image acquisition devices. These data usually have different structures and carry diverse production process information.

[0006] Soft Sensing: Soft Sensing refers to the technology of indirectly inferring and estimating key process parameters that are difficult to measure directly or cannot be obtained in real time in related fields based on existing and easily available process variable data by using computers, mathematical modeling and statistical analysis.

[0007] Process variables: Process variables refer to key parameters or characteristics used to monitor, control and evaluate in industrial production processes or systems. They are variables closely related to the state of the production process or system and are used to describe and quantify the operation and performance of the process. Process variables can include various physical quantities, chemical indicators, process parameters, such as temperature, pressure, flow, etc. The specific types depend on the specific industrial process or system involved.

[0008] Time Series: A time series is a collection of data points or observations arranged in chronological order. It is a data sequence obtained by observing and recording a phenomenon, variable or event in a continuous time interval.

[0009] Performance Index: Performance Index is an indicator used to evaluate the performance of production products, production line equipment or the entire process industry system. In the process industry, performance indicators can include energy consumption intensity, product quality, production line reliability and other aspects. By setting and real-time monitoring of performance indicators, the production efficiency, sustainability and industrial competitiveness of the process industry production process can be evaluated and improved.

[0010] Convolutional Network: Convolutional Neural Network (CNN) is a deep learning neural network structure, which is mainly used to process grid-structured data, such as images and videos. Through specific network layer structures such as convolution operations and pooling operations, convolutional networks can effectively extract spatial and temporal features of input data. Compared with deep learning networks such as fully connected neural networks, convolutional networks can use convolution operations for local perception and parameter sharing when processing data, thereby reducing the number of parameters that need to be trained in the network. This parameter sharing feature makes convolutional networks have high computational efficiency when processing large-scale data. Therefore, convolutional networks have been widely used in image processing and computer vision tasks, such as image classification, object detection, image segmentation, etc.

[0011] Image Attention Mechanism: Image Attention Mechanism is a method that simulates human visual attention, which plays a role in selecting and focusing on important image areas and channels in image processing and computer vision. By introducing the image attention mechanism, the computer system can pay more attention to the parts of the image with significant information, improving the accuracy and efficiency of image processing and analysis.

[0012] Figure 1Figure 1 is a schematic diagram of the structure of an exemplary process industry digital twin platform. The process industry digital twin platform is established based on various physical entities such as production lines, systems, and equipment, and is mainly composed of multiple layers such as the data perception layer, the data integration layer, and the data service layer. The data perception layer is mainly used to perceive the operating conditions of various types of equipment in the process industry production line, that is, to obtain real-time data in various forms through industrial sensors, image acquisition devices, and other monitoring equipment installed and deployed in the process industry production line, and to test and upload some indicators through manual means to achieve data updates, and transmit them to the data integration layer through certain information transmission protocols. The function of the data integration layer is to integrate, preprocess, store, and integrate various types of data obtained by the data perception layer. At this level, data can be cleaned, calibrated, and format converted to ensure the consistency of data quality. The integrated and preprocessed data can be stored in the DCS database of the digital twin platform, or in the edge, cloud, and other places to achieve more extensive data interaction and remote management. Finally, data from different data sources are integrated to be applied to various downstream tasks. The data service layer provides access and management interfaces for digital twin models and data, and based on data input from the data integration layer, implements process monitoring, fault diagnosis, and optimization control of the corresponding production lines, and feeds back some service results to the data integration layer, such as the optimization results of some process variables.

[0013] Under the traditional process industry production model, some performance indicators of the production process, such as the quality of process industry products, often rely on manual testing and detection methods. These methods often take a lot of time and cost, and their accuracy is easily affected by the sampling method and the detection method. Therefore, there are problems such as high detection cost, poor real-time update, and large fluctuations in accuracy. As a result, these performance indicators cannot guide a series of tasks based on the digital twin platform in real time, such as monitoring, diagnosis and optimization control of process industry processes.

[0014] The data-driven soft measurement method of performance indicators is a method that uses historical data and machine learning technology to estimate the performance indicators of a system or process. It is based on statistical analysis and pattern recognition of data. By learning the patterns and trends in historical data, a soft measurement model is established, and the model is used to estimate the performance indicators. This method has been widely used in various fields. However, in the field of process industry, due to the harsh operating environment of the production line and the production process often involving a series of complex physical and chemical reaction changes, the correlation between the performance indicators of the process industry and various process variables is weak, and there are prominent nonlinear problems. This also leads to the fact that the use of process variable data alone may not be able to fully identify and estimate the performance indicators of the process industry. The processing and application of multimodal data is one of the important ways to solve the above problems.

[0015] In addition, in the process of deploying and applying the digital twin platform for process industries, modeling speed and resources are also important factors to consider. At present, some data-driven models rely on complex network structures and algorithms. The deployment of these models on the digital twin platform for process industries will take up a lot of computing resources. Therefore, they cannot adapt to the environment of digital twin platforms with multiple tasks in parallel and intensive information interaction, and are not conducive to the real-time establishment and update of digital twin models. The improvement of modeling speed and the saving of modeling resources can build and update digital twin models faster, and improve the real-time response capability of the digital twin platform for process industries. Summary of the invention

[0016] In view of the above background, the present application provides a soft measurement method based on multimodal data of the process industry digital twin platform. For a certain performance indicator that needs to be obtained, the relevant process variable time series and production process image series and other multimodal data sequences are obtained from the data integration layer of the digital twin platform. Considering the problem of multiple sampling intervals in the process industry data perception layer, the time scale of the multimodal sequence is unified, and the time windows of various sequences are aligned to solve the time lag problem between the process industry performance indicators and the process variables and image data. For the unified and aligned image sequence data, the image sequence feature extraction method is used to extract the image feature time series consistent with the process variable dimension from the image sequence. Subsequently, the process variable time series and the image feature time series are used to perform soft measurement on the corresponding performance indicators. Finally, the soft measurement results of the corresponding performance indicators are fed back to the data integration layer of the process industry digital twin platform, thereby providing reference and guidance for downstream tasks such as monitoring, diagnosis, and optimization control.

[0017] Multiple aspects of the present application provide a soft measurement method based on multimodal data of a process industry digital twin platform, which is used to automatically, quickly and accurately perform soft measurement on some difficult-to-obtain performance indicators in the process industry.

[0018] The present application provides a soft measurement method based on multimodal data of a process industry digital twin platform, comprising: reading corresponding multimodal data sequences such as process variable time series and production process image sequences from the data integration layer of the process industry digital twin platform, wherein process variables are parameters closely related to the operating conditions of process industry production lines, systems and equipment, and production process images refer to black-and-white, grayscale or color image data obtained by an image acquisition device deployed on the production line, containing important information of the production process; unifying the time scales of various types of multimodal data sequences read, wherein the time scale refers to the distance between two adjacent data points, observation values ​​or adjacent values ​​in a time series. The time span between two pictures; perform time window alignment on various multimodal sequences after the unified time scale, where the time scale refers to the time span between two adjacent data points, observations or two adjacent pictures in a time series; extract image feature time series from the image sequence, where the image feature sequence refers to the feature vector sequence containing various structural, texture, shape and other features of the image; use the aligned time series to perform soft measurement on the corresponding performance indicators; finally, feed back the results of the soft measurement of the performance indicators to the data integration layer of the process industry digital twin platform to realize the rapid call of the soft measurement results on other tasks and services, forming a closed loop of information flow.

[0019] The present application also provides an image sequence feature extraction network based on three-dimensional attention convolution, which is composed of modules such as a Gamma transformation module, a hole convolution module, an attention module, and various pooling layers to realize the function of converting an input image sequence of shape (L×C×H×W) to an output feature sequence of shape (L×O). Among them, the Gamma transformation method adopted by the Gamma transformation module is a nonlinear operation commonly used in image processing, which changes the brightness and contrast of the image as a whole by exponentially adjusting the intensity of each element in the image; the hole convolution module adopts a three-dimensional convolution kernel for feature extraction, and introduces holes in the length dimension of the three-dimensional convolution kernel to expand the time receptive field of the three-dimensional convolution kernel; the attention module mainly includes three algorithms: image sequence spatiotemporal attention calculation, image sequence channel attention calculation, and two types of attention fusion, which are used to give full-dimensional attention to the input image sequence; the pooling layer module is used to adjust the distribution of the generated feature map data, thereby reducing the noise in the feature map and retaining the overall characteristics of the feature map sequence.

[0020] The present application also provides a synchronous training method for an image sequence feature extraction network and a performance indicator soft measurement model, including: obtaining model training data, wherein the training data includes process variable data, multimodal data used for modeling such as production process image data, and process variable data as a modeling target; constructing a model training data set, including operations such as sequence time scale unification, sequence time window alignment, and data set division; designing an end-to-end network model structure, wherein the image feature extraction network and the performance indicator soft measurement network are connected to form an end-to-end network model, thereby realizing synchronous training of the two types of networks, and the design of the network structure includes, for example, the design of the number of layers, the amount of parameters, and the connection method of the network; finally, training the model and uploading the training results. Among them, the training results include, for example, the parameters of the network model, the input configuration information of the model, etc.

[0021] This application provides a soft measurement method based on multimodal data of the process industry digital twin platform. Based on the ability of the process industry digital twin platform to acquire, transmit and process data in real time, the multimodal data such as process variable time series and production process image series collected by the platform are used as model inputs. The multimodal data is processed and a performance indicator soft measurement model is established through methods such as time scale unification, time window alignment, and image sequence feature extraction. This method can automatically, quickly and accurately perform soft measurement on some difficult-to-obtain performance indicators in the process industry, improve the effect of soft measurement of process industry performance indicators, and provide reference and guidance for downstream tasks such as monitoring, diagnosis, and optimization control. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a structural diagram of an exemplary process industry digital twin platform.

[0023] Figure 2 This is a flow chart of a soft measurement method based on multimodal data of a process industry digital twin platform provided in this application.

[0024] Figure 3 This is a schematic diagram of an image sequence feature extraction network structure based on three-dimensional attention convolution provided in this embodiment.

[0025] Figure 4 It is a schematic diagram of the structure of the hole convolution module in the image sequence feature extraction network provided in this embodiment.

[0026] Figure 5 Schematic diagram of the structure of the attention module in the image sequence feature extraction network provided in this embodiment.

[0027] Figure 6 This is a flow chart of a synchronous training method for an image sequence feature extraction network and a performance indicator soft measurement model provided in this embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0029] In the embodiments of the present application, "and / or" describes the access relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In addition, in the present application, "first", "second", "third", etc. are only used to distinguish the contents of different objects and have no other special meanings.

[0030] The technical solution provided by the present application is described in detail below in conjunction with the accompanying drawings.

[0031] Figure 2 A flowchart of a soft sensing method based on multimodal data of a process industry digital twin platform provided in this application. Figure 2 , the method may include the following steps:

[0032] 101. Read corresponding multimodal data sequences such as process variable time series and production process image series from the data integration layer of the process industry digital twin platform.

[0033] In this embodiment, as mentioned above, process variables are parameters closely related to the operation status of process industrial production lines, systems and equipment, such as temperature, pressure, flow rate, etc. of key links of the production line. These parameters are usually obtained through various types of sensors deployed in the data perception layer and stored through the data integration layer. In addition, in the data integration layer, some parameters that cannot be directly measured by sensors and are obtained by processing sensor data can also be regarded as process variables, such as parameters such as energy consumption and efficiency.

[0034] In this embodiment, the production process image refers to black-and-white, grayscale or color image data containing important information of the production process, which is obtained by an image acquisition device deployed on the production line, such as microscopic inspection results of the product surface, flame imaging in a combustion device, liquid level images in a closed reactor, etc. The image acquisition device can be any device with an image acquisition function, for example, the image acquisition device can be divided into a CCD camera and a CMOS camera, etc. according to the type of its built-in sensor.

[0035] In practical applications, the multimodal sequences stored in the data integration layer of the process industry digital twin platform can be accessed and obtained in a variety of ways, such as API (application programming interface) access, database query, file system access, Web service access, etc., and parsed in the modeling platform of the data service layer. For example, in a modeling environment based on the Python language, tool libraries such as Numpy, Pandas, and OpenCV can be used to parse and convert the multimodal data read from the data integration layer into a data format that can be used for subsequent processing and modeling.

[0036] 102. Unify the time scales of various multimodal data sequences read.

[0037] In this embodiment, the time scale refers to the time span between two adjacent data points, observations, or two adjacent images in a time series. Considering that the sampling intervals of various process variables and images are different, for example, the sampling intervals of process variables such as temperature, pressure, and flow rate can usually reach the second level. However, considering the database storage capacity, the sampling interval of image data is generally at a relatively high level, such as 30 seconds or 1 minute. Therefore, it is necessary to unify the time scales of various data sequences to adapt to the input of the performance indicator soft measurement model.

[0038] In this embodiment, the unification of the time scale can be performed in a variety of ways, for example, the sequence with the largest sampling interval can be used as a benchmark to downsample other sequences, or the sequence with the smallest sampling interval can be used as a benchmark to upsample other sequences by interpolation, replication, etc. In addition, the unification of the time scale of multimodal sequences can also be achieved with the help of neural networks, such as convolutional networks and other tools.

[0039] 103. Perform time window alignment on various multimodal sequences after unifying the time scale.

[0040] In this embodiment, the time window refers to the selection range of the multimodal data sequence used for soft measurement of performance indicators. In the process industry, since the movement of raw materials, finished products and other materials used in industrial production in the process industry production lines, systems and equipment often takes a certain amount of time, and the data update of the data perception layer of the digital twin platform may also have a certain delay, for the same performance indicator, different data have different time ranges of influence on it. For example, for the product quality index of a certain production line, the data collected in the front process of the production line and the data collected in the terminal process have different time windows for the impact of this index, and the starting point of the time window of the impact of the front process data is earlier than the starting point of the time window of the terminal process data. Therefore, before modeling, it is necessary to select and align the time windows of the sequence to achieve the optimal soft measurement effect.

[0041] In this embodiment, first, according to actual needs, a suitable time window size is selected as the input length of the subsequent processing method. On this basis, the start and end time of each time window can be determined by a recursive method. For example, in a production line scenario containing multiple serial processes, for the soft measurement task of product quality performance indicators, based on the time when the product leaves the last process and the time required for the product from the second-to-last process to the last process, the time when the product leaves the second-to-last process, that is, the deadline of the time window of the data related to the second-to-last process, is determined; on this basis, based on the inferred time when the product leaves the second-to-last process, and the time required for the product from the third-to-last process to the second-to-last process, the time when the product leaves the third-to-last process, that is, the deadline of the time window of the data related to the third-to-last process, and so on. On this basis, according to the size of the time window, the start time of each type of data time window is determined, and the data in the corresponding time window is selected and intercepted to achieve time window alignment of various types of multimodal data.

[0042] Further optionally, the time distance relationship between each time window can be obtained by mathematical modeling or data-driven modeling. For example, in the above-mentioned soft measurement task of product quality performance indicators, the movement time of the product in each process can be obtained by establishing a movement model of the product in each process, and the movement time can be used to align the time windows, or these time distances can be adaptively learned through supervised or unsupervised neural networks to achieve alignment of data time windows.

[0043] 104. Extract image feature time series from image sequence.

[0044] In this embodiment, considering the structural characteristics of the image, the dimension of the image sequence is obviously higher than the dimension of the process variable time series, which brings difficulties to the modeling of the soft measurement of the performance index. Therefore, in order to facilitate the construction of the soft measurement model, it is necessary to unify the dimensions of the image sequence and the process variable time series. A common method is to extract features from the image sequence, thereby converting the high-dimensional image data sequence into a set of low-dimensional feature vector sequences. Through computer vision and image processing technology, various structural, texture and shape features of the image can be extracted. These features can be represented as feature vectors of fixed dimensions, where each dimension corresponds to a specific image feature.

[0045] 105. Use the aligned time series to perform soft measurement on the corresponding performance indicators.

[0046] In this embodiment, a performance indicator soft measurement model with performance indicator soft measurement function is pre-trained. In practical applications, according to the needs of specific scenarios, the performance indicator soft measurement model can take time series data as input and output a single performance indicator soft measurement result, or output a performance indicator soft measurement result sequence with multiple steps.

[0047] In this embodiment, the performance indicator soft measurement model is a model that can comprehensively input time series information and establish the relationship between the time series and the required predicted performance indicators. The form of the performance indicator soft measurement model includes, for example: recurrent neural networks (RNN), convolutional neural networks, or a combination of special network architectures such as autoencoders and self-attention mechanisms and the previous networks. This embodiment does not limit the form of the performance indicator soft measurement model.

[0048] 106. Feedback the results of soft measurement of performance indicators to the data integration layer of the process industry digital twin platform.

[0049] In this embodiment, the soft measurement results of the performance indicators obtained through steps 1-5 contain the key operating information of the corresponding process industry production line, which has important reference value for downstream tasks such as monitoring, diagnosis, and optimization control. By feeding back the soft measurement results to the data integration layer of the process industry digital twin platform, the soft measurement results can be quickly called on other tasks and services, forming a closed loop of information flow.

[0050] In practical applications, the soft measurement results of performance indicators generated in the data service layer can be fed back to the data integration layer in a variety of ways, such as through API interfaces, database operations, file system storage, Web service updates, etc., to achieve real-time updates of the data integration layer.

[0051] The technical solution provided in this application is based on the ability of the process industry digital twin platform to acquire, transmit and process data in real time. It uses multimodal data such as process variable time series and production process image series collected by the platform as model input, and processes multimodal data through time scale unification, time window alignment, image sequence feature extraction and other methods to establish a soft measurement model for performance indicators. Therefore, a soft measurement method based on multimodal data of the process industry digital twin platform is proposed. The above method can automatically, quickly and accurately perform soft measurement on some difficult-to-obtain performance indicators in the process industry, improve the effect of soft measurement of process industry performance indicators, and provide reference and guidance for downstream tasks such as monitoring, diagnosis, and optimization control.

[0052] The above is an introduction to a technical solution of a soft measurement method based on multimodal data of a process industry digital twin platform. Among them, the image sequence feature extraction link has problems such as large amount of calculation and high demand for computing resources. Therefore, it is very important to develop an image sequence feature extraction method adapted to the digital twin platform for the stable and efficient operation of the process industry digital twin platform. The following is an introduction to an efficient and fast image sequence feature extraction method provided in this embodiment.

[0053] Figure 3 is a schematic diagram of a network structure for extracting image sequence features based on three-dimensional attention convolution provided in this embodiment. Figure 3 , this network consists of modules such as Gamma transformation module, dilated convolution module, attention module and various pooling layers to realize the function of outputting feature sequences from input image sequences of shape (L×C×H×W) to output feature sequences of shape (L×O). Among them, L represents the length of the image sequence, C represents the number of channels of the image sequence, H represents the height of the image sequence, W represents the width of the image sequence, and O represents the dimension of the output feature sequence.

[0054] In this embodiment, Gamma transformation is a nonlinear operation commonly used in image processing, which changes the brightness and contrast of the image as a whole by exponentially adjusting the intensity of each element in the image. Further, the image can be linearly transformed based on the Gamma transformation to improve the brightness level of the image.

[0055] Figure 4 : is a schematic diagram of the structure of the dilated convolution module in the image sequence feature extraction network provided in this embodiment. The module uses a three-dimensional convolution network to realize feature extraction of image sequences. The three-dimensional convolution network uses a three-dimensional convolution kernel for feature extraction, and each convolution kernel can slide in the three dimensions of sequence length, width and height to complete feature extraction:

[0056]

[0057] Where I and M represent the input image and output feature map, respectively, K represents the trainable convolution kernel, (h, w, l) and (i, j, k) represent the pixel positions on the feature map and convolution kernel, respectively. in and c o are the number of channels of the input image and the output feature map, respectively. To save computing resources, holes are introduced in the length dimension of the 3D convolution kernel to expand the temporal receptive field of the 3D convolution kernel. By comprehensively adjusting the hole size and the appropriate temporal convolution stride, the 3D convolution kernel with a hole in the temporal dimension can effectively capture the main temporal dependencies in the input image sequence without traversing each image, thereby saving computing resources, speeding up the calculation, and improving the modeling effect.

[0058] Figure 5 : is a schematic diagram of the structure of the attention module in the image sequence feature extraction network provided in this embodiment. This module mainly includes three algorithms: image sequence spatiotemporal attention calculation, image sequence channel attention calculation, and the fusion of two types of attention. For the spatiotemporal attention of the image sequence, a three-dimensional convolution kernel with a dimension of (1×1×n) is used to implement this function:

[0059]

[0060] Where STA represents the spatiotemporal attention weights of the image sequence, whose length, width, and height shapes correspond to the input image sequence, but only have a channel dimension with shape 1. σ represents the Sigmoid transform. The spatiotemporal attention algorithm uses point-by-point convolution in the height and width dimensions to weight the spatial information, while in the time dimension, a specific receptive field size is used to consider the temporal relationship between adjacent images. For the channel attention of the image sequence, a method similar to Squeeze-and-Excitation is adopted, that is, a pair of feature vectors are first generated by the maximum pooling and average pooling methods of the image sequence, and then these feature vectors are converted into two channel weight vectors through a shared fully connected layer autoencoder. This pair of weight vectors is then averaged and Sigmoid transformed to finally form a set of channel attention weights consistent with the number of channels of the input image. When an image sequence of shape (L×C×H×W) is used as input, the channel weights output by the algorithm can be expressed as:

[0061]

[0062] Among them, CA represents the channel attention weight of the image sequence, δ represents the ReLU activation function used in the autoencoder, and W1 and W2 represent the weight matrices of the encoder and decoder, respectively. Finally, by combining the outputs of the two attention algorithms, we can get the full-dimensional scaled attention AA of the image sequence sc :

[0063] AA=STA⊙CA

[0064]

[0065] Where ⊙ represents the Hadamard product of two vectors. When the two vectors have different shapes, the broadcast mechanism is used for calculation. u and a l are the upper and lower bound factors for min-max scaling, which are used to scale the original full-dimensional attention AA. The min-max scaling process can help adjust the distribution of the absolute values ​​of the attention weights to better suit the image sequence feature extraction task.

[0066] In this embodiment, the role of various pooling layers is to adjust the distribution of the generated feature map data, thereby reducing the noise in the feature map and retaining the overall characteristics of the feature map sequence. The entire image sequence feature extraction network is composed of an attention module, a hole convolution module, and a maximum pooling module alternating to achieve the extraction of the image feature sequence. Finally, the feature map sequence processed by various modules will be reduced in dimension by the Squeeze method, and through a set of 1×1 two-dimensional convolutions, a shape of (L×O) is obtained to summarize and abstractly represent the image sequence information. Image feature time series. In practical applications, the number of convolutional neural network modules and comprehensive attention modules and their internal network structure parameters can be adjusted according to the complexity of the information contained in the image and the computing resources of the digital twin platform, so as to achieve the optimal image data feature extraction efficiency.

[0067] The present application provides an image sequence feature extraction network based on three-dimensional attention convolution, which can perform full-dimensional attention weighting on the input image sequence, and realize fast image sequence feature extraction while considering the temporal relationship between sequences through the application of three-dimensional void convolution. While reducing the computing resources of the digital twin platform, it improves the quality and effect of image sequence feature extraction.

[0068] Figure 6 This is a flow chart of a synchronous training method for an image sequence feature extraction network and a performance indicator soft measurement model provided by this embodiment. This training method is mainly used for synchronous training of an image sequence feature extraction network and a performance indicator soft measurement model constructed based on a network method. Figure 6 , the method may include the following steps:

[0069] 201. Obtain model training data.

[0070] In this embodiment, the training data consists of multiple parts, including process variable data, multimodal data used for modeling such as production process image data, and process variable data as the modeling target. As mentioned above, the acquisition of multimodal data can be obtained through API (application programming interface) access, database query, file system access, Web service access, etc., and parsed and cached in the data service layer of the digital twin platform.

[0071] In practical applications, it is necessary to collect training data of sufficient length to achieve accurate and reliable modeling. In addition, the collected multimodal training data needs to be cleaned, screened, and converted to other data types to improve the modeling effect.

[0072] 202. Build a model training data set.

[0073] In this embodiment, the multimodal data obtained for model training needs to be subjected to the above-mentioned operations such as sequence time scale unification and sequence time window alignment to form a set of multimodal data sequences that conform to the input format required by the image sequence feature extraction network and performance index soft measurement network model described above. On this basis, the multimodal data sequence is divided into a training set, a validation set, and a test set according to a certain ratio. During the training process, the training set is usually used for model training and parameter optimization, the validation set is used for model parameter adjustment and selection, and the test set is used to evaluate the generalization ability and performance of the model.

[0074] In some optional embodiments, the process variable data may be processed by normalization, standardization, etc. Standardization refers to ensuring that the mean of a single variable data is 0 and the variance is 1 by scaling and calculation. Normalization is to scale the feature vector of each sample to the unit norm so that the Euclidean norm of the feature vector is 1. These operations can eliminate the differences in the scale of the process variable data, enhance the convergence of the model during the training process, and avoid the occurrence of overfitting problems.

[0075] 203. Design an end-to-end network model structure.

[0076] In this embodiment, the image sequence feature extraction network and the performance indicator soft measurement network are connected to form an end-to-end network model, thereby realizing the synchronous training of the two types of networks, thereby saving the time, resources and labor costs consumed by the process industry digital twin platform in model training and improving the training effect of the model. The design of the end-to-end network model structure includes, for example, the design of the number of network layers, the number of parameters, and the connection method.

[0077] In practical applications, the appropriate end-to-end network model structure should be designed based on factors such as the richness of information contained in the image data and the complexity of the performance indicator prediction task.

[0078] 204. Train the model and upload the training results.

[0079] In this embodiment, the constructed end-to-end network model is trained by a certain training method. In multiple rounds of training, the network inputs batches of multimodal data sequences, and under the guidance of a certain loss function, the parameters are updated to minimize the loss, and finally a mapping relationship is established between the multimodal data sequence information and the performance indicators that need to be fitted.

[0080] In this embodiment, any loss function that meets the requirements can be used to guide the training direction of the model. For example, the loss function includes: absolute error loss, mean square error loss, L2 loss, L1 loss, etc. In practical applications, it is also possible to define the loss function by yourself in combination with the characteristics of the specific task to guide the parameter update of the complex model. The method of implementing parameter update through the loss function can be completed with the help of an optimizer. The algorithms used by the optimizer include, for example, gradient descent, Adam algorithm, Autograd algorithm, etc. The selection of a suitable optimizer depends on the specific task, data, and model structure. A common practice is to try multiple optimizers and select the optimizer with the best performance based on the experimental results.

[0081] In this embodiment, the training results of the model need to be uploaded and stored for real-time loading, calling and soft measurement of performance indicators of the digital twin platform. The training results are composed of multiple parts, such as the parameters of the network model, the input configuration information of the model, and some optional embodiments also include methods for multimodal data processing, such as normalization and standardization of production process variables. The training results can be uploaded and saved to the data service layer, or they can be temporarily saved by other methods and read through certain channels when they need to be called.

[0082] The present application provides a method for synchronously training an image sequence feature extraction network and a performance indicator soft measurement model, which can quickly and easily train the image sequence feature extraction network and the performance indicator soft measurement network model at the same time and upload the training results, saving the time, resources and labor costs consumed by the process industry digital twin platform in training the multimodal performance indicator soft measurement model, and improving the training effect of the model.

[0083] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0084] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A soft sensing method based on multimodal data of process industry digital twin platform, characterized in that: include: Read the corresponding process variable time series and production process image sequence multimodal data sequence from the data integration layer of the process industry digital twin platform. The process variables are parameters closely related to the operation status of the process industry production line, system and equipment, including the temperature, pressure and flow rate of the key links of the production line. The production process image refers to the black-and-white, grayscale or color image data containing important information of the production process obtained by the image acquisition device deployed on the production line, including the microscopic inspection results of the product surface, the flame imaging in the combustion equipment, and the liquid level image in the closed reactor; Unify the time scales of various multimodal data sequences read, where the time scale refers to the time span between two adjacent data points, observations, or two adjacent images in a time series; Performing time window alignment on various multimodal sequences after unifying the time scale, wherein the time window refers to the selection range of the multimodal data sequence used for soft measurement of performance indicators; Inputting the image sequence into an image sequence feature extraction network based on three-dimensional attention convolution to extract image feature time series, wherein the image feature time series refers to a feature vector sequence containing various structural, texture and shape features of the image; The performance indicator soft measurement is performed using the aligned time series. The performance indicator soft measurement is realized by a pre-trained performance indicator soft measurement model with performance indicator soft measurement function. The performance indicator soft measurement model is a model that can comprehensively input time series information and establish a relationship between the time series and the required predicted performance indicator. Feedback the results of soft measurement of performance indicators to the data integration layer of the process industry digital twin platform to enable the soft measurement results to be quickly called as needed, forming a closed loop of information flow; The image sequence feature extraction network based on three-dimensional attention convolution comprises: Gamma transformation module: the Gamma transformation method adopted by the Gamma transformation module is a nonlinear operation commonly used in image processing, which changes the brightness and contrast of the image as a whole by exponentially adjusting the intensity of each element in the image; A dilated convolution module, which uses a three-dimensional convolution kernel for feature extraction and introduces a hole in the length dimension of the three-dimensional convolution kernel to expand the temporal receptive field of the three-dimensional convolution kernel; An attention module, which includes three algorithms: image sequence spatiotemporal attention calculation, image sequence channel attention calculation, and two types of attention fusion, and is used to give full-dimensional attention to the input image sequence; A pooling layer module, wherein the pooling layer module is used to adjust the distribution of the generated feature map data, thereby reducing the noise in the feature map and retaining the overall characteristics of the feature map sequence; The dilated convolution module uses a three-dimensional convolutional network to extract features of image sequences. The three-dimensional convolutional network uses a three-dimensional convolutional kernel for feature extraction. Each convolutional kernel can slide in three dimensions of sequence length, width and height to complete feature extraction: Among them, I and M represent the input image and output feature map respectively, K represents the trainable convolution kernel, (h, w, l) and (i, j, k) represent the positions of pixels on the feature map and convolution kernel respectively; in addition, c in and c o are the number of channels of the input image and the output feature map respectively; a hole is introduced in the length dimension of the three-dimensional convolution kernel to expand the temporal receptive field of the three-dimensional convolution kernel; The attention module includes three algorithms: image sequence spatiotemporal attention calculation, image sequence channel attention calculation, and the fusion of the two types of attention; The spatiotemporal attention of the image sequence uses a 3D convolution kernel of dimension (1×1×n) to achieve this function: Among them, STA represents the spatiotemporal attention weight of the image sequence, whose length, width, and height shapes correspond to the input image sequence, but only the channel dimension with shape 1, σ represents the Sigmoid transform, and the spatiotemporal attention algorithm uses point-by-point convolution in the height and width dimensions to weight the spatial information, while in the time dimension, a specific receptive field size is used to consider the temporal relationship between adjacent images; The channel attention of the image sequence generates a pair of feature vectors by performing the maximum pooling and average pooling methods on the image sequence, and then converts these feature vectors into two channel weight vectors through a shared fully connected layer autoencoder. The weight vectors are then averaged and sigmoid transformed to finally form a set of channel attention weights consistent with the number of channels of the input image: Where CA represents the channel attention weight of the image sequence, δ represents the ReLU activation function used in the autoencoder, H, W, L represent the height, width and length of the image sequence respectively, W1 and W2 represent the weight matrices of the encoder and decoder respectively; The attention fusion method is: AA=STA⊙CA Among them, AA is the attention weight before scaling, AA sc is the scaled attention weight, ⊙ represents the Hadamard product of two vectors. When the two vectors have different shapes, the broadcast mechanism is used for operation; a u and a l It is the upper and lower bound factors for min-max scaling, which is used to scale the full-dimensional attention. The min-max scaling process helps adjust the distribution of attention weights to better adapt to the image sequence feature extraction task. The feature extraction network described, the entire image sequence feature extraction network is composed of an attention module, a hole convolution module, and a maximum pooling module alternating to achieve the extraction of image sequence feature sequences. Finally, the feature map sequence processed by various modules will be reduced in dimension through the Squeeze method, and through a set of 1×1 two-dimensional convolutions, an image feature time series that summarizes and abstractly represents the image sequence information is obtained.

2. The soft sensing method according to claim 1, characterized in that: A synchronous training method for an image feature extraction network and a performance indicator soft measurement model, comprising: Acquire model training data, wherein the training data consists of multiple parts, including process variable data, multimodal data used for modeling of production process image data, and process variable data as a modeling target; Constructing a model training data set, including performing sequence time scale unification and sequence time window alignment operations on the multimodal data obtained for model training to form a set of multimodal data sequences that meet the input format required by the image feature extraction network and the performance indicator soft measurement network model, and dividing the multimodal data sequence into a training set, a validation set, and a test set according to a certain ratio; Design an end-to-end network model structure, wherein the image feature extraction network and the performance indicator soft measurement network are connected to form an end-to-end network model, thereby realizing synchronous training of the two types of networks, and the network model structure includes the number of network layers, parameter quantity, and connection method; Train the model and upload the training results, which consist of multiple parts, including the parameters of the network model and the input configuration information of the model.

Images