A product multi-level assembly quality detection method and system based on multi-field spatiotemporal transfer learning

Through the combination of multi-field space-time transfer learning and OPC UA information model, the problem of multi-level assembly quality inspection in complex product assembly process is solved, the full process tracking and fault diagnosis of complex products is achieved, and the accuracy of assembly quality inspection is improved. and efficiency.

CN118917747BActive Publication Date: 2024-12-03CHENGDU WEICETONG INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411407971.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-10-10
Publication Date
2024-12-03
Estimated Expiration
2044-10-10

AI Technical Summary

Technical Problem

The existing assembly quality detection methods are difficult to apply to the multi-level assembly process of complex products, especially in the fields of aerospace, etc., and cannot effectively consider the coupling effect between product levels and timing decay effects, making it difficult to achieve long-term assembly quality diagnosis of complex products. .

Method used

Using a multi-field space-time transfer learning method, a multi-level assembly quality prediction model is built to realize the entire process of complex product assembly through a data acquisition and monitoring system oriented towards the full process of integrated assembly, combined with OPC UA information model and timing data processing. Tracking and troubleshooting.

Benefits of technology

It realizes dynamic tracking and virtual adaptation of complex product assembly processes, can comprehensively judge the assembly quality of multi-level complex products, and improves the accuracy and efficiency of assembly quality inspection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917747B_ABST
    Figure CN118917747B_ABST
Patent Text Reader

Abstract

The present invention provides a product multi-level assembly quality detection method and system based on multi-field spatiotemporal transfer learning, which belongs to the field of equipment manufacturing. The present invention can realize the full-process tracking of the assembly process of complex products with multi-level assembly characteristics, obtain the real-time status information of the assembly process of each component such as high-level components and low-level components, and realize dynamic tracking of the entire process of complex product assembly. A production line multi-category component virtual mapping method based on an improved OPC UA information model is proposed. This method realizes state-based real-time dynamic tracking of each component in a virtual space by assigning a virtual IP and a corresponding database to each component. Assembly quality prediction based on multi-field spatiotemporal transfer learning can track the assembly process of complex product process components and the final assembly quality of the product, thereby realizing a comprehensive judgment on the assembly quality of multi-level complex products.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of equipment manufacturing, and in particular relates to a product multi-level assembly quality detection method and system based on multi-field spatiotemporal transfer learning. Background Art

[0002] Assembly is a key link in the product development process, which largely determines the final quality and service life of the product. Assembly is an organic whole with rich connotations. It is not just a process of simply assembling parts together. More importantly, the assembled product can achieve the corresponding functions and reflect the quality of the product. With the continuous improvement of the precision of mechanical processing equipment, testing equipment, etc., the current mechanical processing and assembly links have become a key step that affects the quality of the final product and restricts the improvement of product performance. For this reason, product quality diagnosis and identification around the assembly process has become an important issue that needs to be studied urgently.

[0003] Assembly quality diagnosis can be mainly divided into single process quality diagnosis, T-based 2 Diagnostic methods based on statistical decomposition, fuzzy diagnostic methods, and neural network-based diagnostic methods. In terms of univariate process quality diagnostic methods, for a processing process that usually contains only one quality indicator (univariate) and only one factor affecting the quality indicator (single factor), the Shewhart control chart can be used to warn of abnormal process quality and point out the cause of abnormal process quality. However, for a univariate multi-factor process, the Shewhart control chart cannot diagnose the cause of abnormal process quality, resulting in it being rarely used in actual assembly tasks. Further, based on T 2 In the aspect of product assembly quality diagnosis based on statistical decomposition, due to the difference in the selection preference of statistical features, T 2 The selection of statistics has great differences. Therefore, when the number of mass components is large, the scale of the diagnosis system will increase dramatically, making it difficult to apply to the manufacturing process of aviation products containing many mass components. In terms of assembly quality based on fuzzy diagnosis, it is feasible to establish a fuzzy relationship equation group to diagnose the causes of process quality abnormalities under the condition that the membership function between abnormal factors and abnormal phenomena is known. However, for aviation products, the complexity of the manufacturing process leads to the fact that the membership function between abnormal factors and abnormal phenomena is often unknown. For the equation group that must rely on strict constraints to determine the fuzzy relationship, the complexity of the solution process largely limits the application of fuzzy diagnosis theory in the manufacturing process of aviation products.

[0004] At present, product assembly quality detection methods based on neural networks and machine learning have become the current mainstream. In terms of fault diagnosis based on neural networks, Hao Bo and others from Northeastern University proposed a digital detection system for parts assembly quality based on machine learning. The system realizes the quality detection of hole making and riveting in the process of parts assembly in a streamlined, modular, automated and efficient manner by constructing image acquisition modules and image analysis modules for the assembly process. Wang Zanzan from Hefei University of Technology proposed an assembly process quality control method based on LSTM (Long Short-Term Memory Network). This method predicts and identifies assembly quality by analyzing the coupling relationship between assemblies, thereby realizing assembly quality prediction within different time windows. Ma Zheng and others from Jiangsu University proposed a streamlined air curtain cleaning system assembly quality detection device and method in the US patent application. At present, many patents have been obtained around product assembly quality, and many significant results have been achieved.

[0005] However, the current lack of consideration of product-level assembly quality factors in product assembly quality inspection makes it difficult for existing methods to be applied to the assembly process of complex products. Existing complex products, such as aerospace products and military products, have obvious multi-level structures. Existing methods only consider the coupling effects between multiple processes and lack the calculation of coupling factors between product levels, making it difficult to directly use them in the assembly quality inspection process of complex products. In addition, existing methods lack relevant considerations for the timing decay effect of the assembly process and cannot solve the timing decay coupling effects between multiple levels and multiple processes, making it difficult to directly use them for long-term assembly quality diagnosis of complex products. Summary of the invention

[0006] Based on the above technical problems, the present invention proposes a product multi-level assembly quality detection method and system based on multi-field spatiotemporal transfer learning. First, a data collection, monitoring, and data cleaning system for the entire assembly process is proposed. On this basis, a dynamic tracking method for product component modules in the assembly process based on the Open Platform Communication Unified Architecture (OPC UA) information model is proposed to obtain the data of the entire assembly process. Furthermore, based on the acquired full-process data, the present invention integrates the proposed assembly process product fault diagnosis method based on spatiotemporal transfer learning, performs virtual assembly testing and fault diagnosis on product component modules, and realizes product fault diagnosis for the entire assembly process. Using this method, product fault diagnosis in the assembly process is realized.

[0007] In order to solve the above technical problems, the specific technical solutions of a product multi-level assembly quality detection method and system based on multi-field spatiotemporal transfer learning of the present invention are as follows:

[0008] Step 1: For assembly data tracking of multi-level assembly processes, the influencing factors of any two assembly links are compared through the fuzzy complementary judgment matrix, so as to identify the key error links of the assembly process and obtain the key factors of the assembly process;

[0009] Step 2: Build a cloud-edge-device collaborative architecture time series data model based on the time series base class and the high-dimensional time series base class, define the collected time series data based on key factors as a tree structure, and then divide it into dimensional vectors according to different dimensions;

[0010] Step 3: Based on the virtual dynamic mapping of assembly product components of the improved OPC UA information model, different types of time series data are obtained; first, the data based on the time series base class and the high-dimensional time series base class obtained in step 2 are collected in real time based on the OPC UA information model and sorted and organized, and the collected data is filtered based on the support vector machine-hidden Markov method for filtering abnormal data in time series sensor data, and the purpose and storage address of the filtered data are output; after completing the unification of the collection queue messages, the message queue telemetry transmission protocol is used to connect directly to the cloud, and the collected data is directly transmitted to the cloud queue to achieve dynamic and rapid tracking of messages;

[0011] Step 4: Multi-level assembly quality prediction based on multi-field spatiotemporal transfer learning;

[0012] Step 4.1: Clustering the data collected during the assembly process based on the similarity assembly process attribute clustering method of the regularized Gaussian mixture clustering model;

[0013] Step 4.2 Based on the multi-task random migration method of maximum mean difference-mean square error, transfer learning and training in multiple spatiotemporal fields are realized to build an assembly quality prediction model for multi-level long spatiotemporal queues;

[0014] Step 5: Based on the assembly quality prediction model of multi-level long-time and space queues, real-time quality inspection of the assembly process is performed.

[0015] The present invention also provides a product multi-level assembly quality detection system based on multi-field spatiotemporal transfer learning, including a front-end interactive interface, an assembly data acquisition module, a time series data processing module, an OPC UA client, and a multi-level assembly quality prediction module;

[0016] Among them, the assembly data acquisition module is responsible for collecting data on the assembly line, filtering the data according to the weight value, and transmitting it to the time series data processing module;

[0017] The time series data processing module is responsible for time series multi-dimensional processing of data. It defines the collected time series data based on key factors as a tree structure, and then divides it into dimension vectors according to different dimensions. Then, the processed time series multi-dimensional data is uploaded to the OPC UA client.

[0018] The OPC UA client performs different intermediate processing on the received data according to different types, filters the collected data based on the abnormal data screening method in time series sensor data based on the support vector machine-hidden Markov, and then directly transmits the filtered data to the cloud queue and the multi-level assembly quality prediction module; the OPC UA client interacts with the user through the user interaction interface;

[0019] The multi-level assembly quality prediction module is responsible for quality prediction of assembly data; specifically, the module includes a regularized Gaussian mixture clustering model and an assembly quality prediction model; first, based on the regularized Gaussian mixture clustering model, the different categories of attributes collected during the assembly process are clustered, and then based on the assembly quality prediction model, assembly quality prediction is achieved, and the final quality prediction results are displayed through the front-end interactive interface.

[0020] The beneficial effects of the present invention are as follows:

[0021] (1) Compared with the traditional assembly quality diagnosis method, the present invention can realize the full-process tracking of the assembly process of complex products with multi-level assembly characteristics, obtain the real-time status information of the assembly process of each component such as high-level components and low-level components, and realize dynamic tracking of the entire assembly process of complex products.

[0022] (2) Compared with traditional inventions and methods, which cannot realize dynamic data tracking and virtual adaptation of multi-level components, the present invention proposes a virtual mapping method for multi-category components of production lines based on an improved OPC UA information model. This method realizes state-based real-time dynamic tracking of each component in the virtual space by allocating a virtual IP and a corresponding database to each component.

[0023] (3) Compared with traditional inventions and methods, which can only analyze the coupling quality between multiple process sequences, the present invention proposes an assembly quality prediction method based on multi-field spatiotemporal transfer learning. This method can track the assembly process of complex product process components and the final assembly quality of the product, thereby achieving a comprehensive judgment on the assembly quality of multi-level complex products. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 The following is a schematic diagram of the entire assembly process of a complex product as an example; (a) is a multi-process flow chart, and (b) is a multi-component interaction flow chart;

[0025] Figure 2 The overall architecture diagram of the data model for the time series base class and the high-dimensional time series base class;

[0026] Figure 3 Modeling instance diagrams for time series base classes and high-dimensional time series base classes;

[0027] Figure 4 This is a diagram of the cloud-based direct data collection architecture based on OPC UA and MQTT;

[0028] Figure 5 It is a multi-level OPC UA information model interaction system for assembly process;

[0029] Figure 6 This is the architecture diagram for migration between models;

[0030] Figure 7 It is a schematic diagram of the dynamic CNN structure;

[0031] Figure 8 This is a schematic diagram of the structure of the array antenna in the embodiment;

[0032] Fig. 9 A schematic diagram of the front-end interactive interface of a product multi-level assembly quality inspection system provided in an embodiment. DETAILED DESCRIPTION

[0033] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a product multi-level assembly quality detection method and system based on multi-field spatiotemporal transfer learning in conjunction with the accompanying drawings.

[0034] Example 1

[0035] like Figure 1 The figure shows the multi-process flow and multi-component interaction flow involved in the present invention. Figure 1 As shown in (a) of the figure, the assembly firstly carries out multiple process transfers within the layers to realize the assembly of single accessories or parts. In this type of assembly process, multiple error sources will be directly introduced, thus affecting the quality of the final assembled product. In addition, Figure 1 As shown in (b), there is a hierarchical dominance relationship between products, and the hierarchical coupling relationship between the assembly quality of its product components will have a spatial coupling effect on the final assembly quality result of the product. Therefore, in order to realize the assembly quality prediction of complex products with multi-level characteristics, it is necessary to construct an assembly process analysis module for different hierarchical spaces and obtain assembly data at different levels of the product assembly process.

[0036] For example Figure 1 The multi-process flow and multi-component interactive assembly flow shown in the figure, this embodiment provides a product multi-level assembly quality detection method based on multi-field spatiotemporal transfer learning, the specific steps are as follows:

[0037] Step 1: Use the fuzzy complementary judgment matrix to identify the key error links of the assembly process and obtain the key factors of the assembly process;

[0038] Taking into account the problems of multiple assembly levels, complex assembly relationships, and the huge and high cost of building an assembly data management system for the entire process, the present invention first uses a fuzzy complementary judgment matrix to identify the key error links in the assembly process, thereby establishing corresponding data collection methods for such error links.

[0039] Specifically, we first analyze the data for all the influencing links of the assembly process. We arbitrarily define the different influencing factors in two assembly processes as and , construct the fuzzy complementary judgment matrix , its specific manifestations are as follows:

[0040]

[0041] The fuzzy matrix is ​​an n*n order matrix. The definitions are:

[0042]

[0043]

[0044] Can be used for elements and Quantitative impact relationship between: If , then the two are identified as equal. Approaching 1, we believe Compare important; otherwise, it is considered Compare Important: By constructing a decision matrix as shown above, the influencing factors can be compared pairwise.

[0045] Then construct an n-order non-negative regularized weight vector matrix W, which is expressed as follows:

[0046] W=[ w 1 , w 2 , ⋅⋅⋅ , w n ] T

[0047] In the formula, is the element of the non-negative regularized weight vector matrix W, that is, the weight value; W and The relationship between can be calculated by the following relationship formula:

[0048]

[0049] In the formula, represents a vector constant, are the i-th and j-th elements in the matrix W. On this basis, the following formula is used to transform the original result:

[0050]

[0051] Furthermore, the above formula can be simplified to:

[0052]

[0053] On this basis, the weight value of any element can be calculated

[0054]

[0055] The above formula can be simplified as follows:

[0056]

[0057] Based on the above-mentioned element theory, the present invention can obtain the weight values ​​of the impact of different links on the quality of the final product. In order to achieve simple calculation, the present invention only considers the top 30% of the weight influencing factors in the weight as key factors. After obtaining the key factors, further, the patent of the present invention proposes a data collection and processing architecture based on cloud-edge integration. For the key factors calculated in the preamble, for key points such as manual assembly stations, robot assembly stations, and transportation stations, diversified data collection methods such as cameras, torque sensors, and radio frequency identification (RFID) are constructed respectively. However, the amount of data collected is huge, involving multiple types of information such as timing sensors and cameras, and has the characteristics of large scale and high dimension.

[0058] Step 2: Build a cloud-edge-end collaborative architecture time series data model based on the time series base class and the high-dimensional time series base class; collect data based on key factors, and pre-process the data using the cloud-edge-end collaborative architecture time series data model;

[0059] The overall data model architecture of the proposed time series base class and high-dimensional time series base class is as follows: Figure 2 As shown, the collected multi-sensor time series data is first defined as a tree structure, and then this type of structure can be based on Figure 2As shown in the figure, the root-leaf continuous iterative distribution is performed. The distribution process starts with a high-dimensional array, that is, it starts from the first layer, and recursively loops the pointer to the second layer, finds its influencing factors for the second layer, and thus realizes the recursive iterative loop of all layers, that is, completes the evolution process of the high-dimensional time series tuple to the low-dimensional time series tuple. After defining and completing the above steps, the end-side data modeling based on the high-order time series base class data can be completed. The overall data modeling example is Figure 3 As shown, specifically, firstly, the time series information storage values ​​Value in different time periods are identified respectively, and Value is the data value corresponding to the specific moment, which can be defined as image, text, number and other formats, used to characterize video stream, text stream, sensor and other types of data, and then Value is divided into dimension vectors X, Y, Z according to different dimensions. Specifically, for a multi-process flow and multi-component interactive assembly process, there are many hierarchical relationships and diversified data formats. Firstly, the trunk of the tree structure can be formed according to the main process flow. A main process contains multiple sub-processes, and a sub-process has one or more components; and, since there are one or more types of data collection at the process point and each assembly process, the tree structure can be used to hierarchize the complex assembly process, and the collected high-order time series base class data can be classified according to different dimensions to form multiple dimensional vectors.

[0060] Step 3: Virtual dynamic mapping of assembly product components based on the improved OPC UA information model to achieve full-process tracking of collected data;

[0061] Considering the diversity of collected data, the present invention proposes a virtual dynamic mapping method for assembly product components based on an improved OPC UA information model to achieve full-process tracking of multi-level assembly processes. The core process of this method is as follows: Figure 4 As shown, firstly, data collection is carried out, and each data is automatically and efficiently collected by using the active subscription service of the OPC UA client; the data is sent to the OPC UA client in the form of message queues. For problems such as inconsistent content and categories of collected data, all message queues are sorted and organized based on the multi-level OPC UA information model in the client to achieve classification; finally, the message queue telemetry transmission protocol (MQTT) is used to connect directly to the cloud, and the classified message queue data is directly transmitted to the cloud queue, so as to achieve dynamic and rapid tracking of messages. Its core module is Figure 5 The OPC UA information model for multi-level assemblies.

[0062] In addition, considering the different sampling frequencies of sensors, resulting in different amounts of data in the message queue, in order to ensure the uniformity of the message queue time, the present invention adopts the upsampling method. The data collected by each sensor is defined as On this basis, the maximization function is used to formulate the function as follows:

[0063]

[0064] To maximize the sampling frequency After that, the upsampling method is used for the data with lower sampling frequency to increase the data samples. Upsampling is performed by the intermediate interpolation method for data interpolation:

[0065]

[0066] In the formula, represents the upsampled data, Indicates any data except the highest sampling frequency data, and Represents the data of the previous moment and the next moment respectively.

[0067] After completing the above upsampling, the present invention further elaborates on the proposed multi-level OPC UA information model. The assembly process information model established for the multi-level assembly process has a specific architecture as follows: Figure 5 As shown, the collected data is input into the input component module of the model, such as Figure 5 The gray module in the figure defines the input data to complete the collection of various types of data. Figure 8 Taking the assembly process of the array antenna shown in the figure as an example, data is collected for key factors such as testing, RF mainboard installation, antenna board insertion assignment, KK board insertion assignment, antenna base installation, antenna board branch line assembly, sub-component assembly, quality, etc. During the collection, the collected data is divided into different dimensions through the cloud-edge-end collaborative architecture time series data model and input into the input component module of the OPC UA information model; the model performs different intermediate processing for different types of data, such as Figure 5 The intermediate components in the model, such as the three-dimensional modal conversion for structured graphics, enable different types of data to be transmitted to the corresponding virtual space in a standard format; after data processing, the purpose of the output data and the storage address (the address symbol corresponding to the model data in the virtual space) can be used. By using this type of representation mode, the collected data can be effectively and accurately transferred to the corresponding virtual space. Figure 5 The visualization in the above diagram directly outputs the collected data to the corresponding display panel for visualization.

[0068] In the data processing part of the OPC UA information model, in order to realize multi-type data exception processing, the present invention proposes an abnormal data screening method in time series sensor data based on support vector machine (SVR)-hidden Markov (HMM).

[0069] The OPC UA information model assumes that the data distribution follows the HMM format and the random attribute rules of principal component analysis (PCA); then there is the collected sequence data obey Rules, formulas They represent the data transfer matrix, abnormal determination coefficient, and initial probability matrix respectively.

[0070] Compared with the traditional calculation method of abnormal determination coefficient B which is mainly determined by experience, the present invention proposes an abnormal determination coefficient matrix based on dynamic characteristics. The main calculation formula of the matrix is ​​performed by using a nonlinear inseparable support vector machine, and the overall solution process in the model is mainly performed by the following dual calculation formula:

[0071]

[0072]

[0073] in the formula , For the collected historical queue message data, is the data anomaly label, where 1 indicates normal and negative 1 indicates abnormal. is the Lagrange dual coefficient, and C is the penalty function coefficient value, which is usually given manually. The support vector machine regression model can be calculated through the dual formula. Based on the regression model, the abnormal determination coefficient matrix sequence at different stages can be obtained. Based on the abnormal determination coefficient matrix sequence, the accurate judgment of the results can be achieved. After data judgment, the model can automatically remove the abnormal data in the collection process, thereby achieving accurate judgment of the acquired data.

[0074] Step 4: Multi-level assembly quality prediction based on multi-field spatiotemporal transfer learning

[0075] After acquiring the above data, the present invention predicts the quality of the multi-level assembly process based on multi-field spatiotemporal transfer learning, and performs fault diagnosis on the assembly quality of products with multi-level characteristics. The method is mainly divided into two parts. First, a similarity assembly process attribute clustering method based on the regularized Gaussian mixture clustering model (RGMM) is constructed to cluster the different categories of attributes collected during the assembly process, reduce the type of modeling, and reduce the overall work complexity. On this basis, a multi-task random migration method based on the maximum mean difference-mean square error (MMD-MSE) is constructed to realize transfer learning and training in multiple spatiotemporal fields, and to construct an assembly quality prediction model for multi-level long spatiotemporal queues. The specific steps are as follows:

[0076] Step 4.1: Construct a regularized Gaussian mixture clustering model to cluster the different category attributes collected during the assembly process; clustering is performed by solving the maximum likelihood probability. The maximum likelihood probability is obtained by solving the likelihood function. After obtaining the likelihood function, the categories with uniform similarity can be completed.

[0077] First, the obtained complete data set is represented as:

[0078] in is the covariate matrix of the dataset, For each data Corresponding hidden variables (assignment) , N is the number of data points; data Assume a Gaussian distribution ,in Expressed as mean and covariance. Based on the above assumptions, the following likelihood function can be obtained:

[0079] log P(X , Z ∣ π , μ , Σ )= ∑ i=1 N ∑ j=1 K z ij [ log π j + log N( X i ∣ μ j , Σ j )]

[0080] in, Represents a preset constant value, K is the total number of rows and columns, here the number of rows is equal to the number of columns, is the specific value of the i-th row and j-th column in the covariate matrix X, is the population mean after the likelihood function is calculated. After the likelihood function is calculated, the overall covariance is obtained.

[0081] Based on the likelihood function, an optimization problem is used to obtain the maximum solution of the likelihood function. The optimization solution process is specifically as follows:

[0082]

[0083] in, is the conditional constant, q is the control constant, , is the time series number; represents the covariate matrix with the i-th diagonal element being 1, D represents the number of matrix elements, Represents the vector product of the i-th diagonal element in the covariate matrix I. Construct the constraints and objective function of the optimization problem shown above, and then use the following equation to obtain the expected logarithmic likelihood of the data, that is, the similarity index of the data:

[0084] Q( θ ; θ old )&= E z | X , θ old [log P ( X , Z )] &= ∑ i =1 N ∑ j =1 K E z | X , θ old [ z ij ][log π j +log p j ( X i | θ j )].

[0085] in the formula is the parameter of the likelihood function before iteration, is represented as the parameter of the likelihood function, is the Gaussian density in the assembly, where the key parameters are ; for Expectation of Z under event X, For events exist The conditional entropy under is the probability distribution of data X and latent variable Z; there are two iteration methods, one is E-step, which is performed in rotation order, and the other is M-step, which uses regularized covariance estimation as input in each iteration. The two iteration methods of the present invention are performed simultaneously. In E-step, the i-th row in the covariate matrix X Mixed component k Estimates as follows:

[0086] z ̂ ik &= E z|X , θ old [ z ij ] &=P( z ij =1|X , θ old )= π k p k ( X i | θ k ) ∑ j=1 K π j p j ( X i | θ j )

[0087] In the formula, For the parameter X, Down, The conditional probability of represents the preset constant value in the mixture component k; The key parameter for the mixture component k is Conditional data The probability distribution of data Specifically, the assembly event, is the key parameter in the mixed component k; The key parameter Conditional data The probability distribution of

[0088] Furthermore, the prior probability and average value of the kth component can be calculated by the following formula:

[0089]

[0090] In the formula, is the i-th row in the covariate matrix X Mixed component k The prior probability of is the i-th row in the covariate matrix X The kth component of the mixture component k of for The superscript t represents the time step. On this basis, combined with the aforementioned similarity index, further, the matrix method can be used for fitting to cluster the collected assembly data into several subsets.

[0091] Step 4.2: Construct an assembly quality prediction model for multi-level long-time and space queues, and realize transfer learning and training in multiple time and space fields based on the multi-task random migration method of maximum mean difference-mean square error;

[0092] In the next step, if Figure 6 As shown, deep learning will be used to establish quality prediction models M1, M2, M3…Mn-1, Mn for each subset D1, D2, D3, D…Dn. The quality prediction models established for the first time are collectively referred to as source models, and the models constructed by a single subset are referred to as sub-models. Free transfer learning can be performed between subsets. The above establishes independent scenarios for each category, thereby completing the training of the internal models of each source data set for transfer learning. On this basis, for each independent data model, the cascade method is used to achieve cooperative transition.

[0093] In terms of the basic model, the present invention proposes a training model system based on a dynamic convolutional neural network (Dynamic CNN)-conditional random field (CRF).

[0094] As mentioned above, after using the data set clustering division, multiple coupled training subsets D1, D2, D3, D...Dn with mutual influence relationships are formed, and free transfer learning can be performed between data sets. In terms of sub-model training, the present invention directly adopts a dynamic convolutional neural network, and its specific framework is as follows Figure 7 As shown in the figure, it is mainly divided into model data set deconstruction, feature extraction, feature fusion, modeling output and other parts. First, the data set is deconstructed. Taking the subset D1 as an example, the subset is repeatedly divided to construct a local data set set. , LD represents the local data set, and m represents the number of local data sets. Based on the construction of local data sets, feature extraction is performed for each local data set. Feature extraction uses the following method: Figure 7 The local cross-correlation feature and multivariate spatial feature extraction method shown in . For local cross-correlation feature extraction, the The continuous sliding window of express The specific data values ​​corresponding to the input and output at each moment, T represents transposition. Construct a matrix containing the previous and next correlation relationships under a continuous sliding window. The specific process is shown in the formula below:

[0095]

[0096]

[0097] Where z is the number of local data sets; is the first Data at the moment, subscript express The next moment of the moment; based on the matrix sequence constructed above, it can be considered that the assembly accuracy in the current assembly time period is dynamically characterized, and it is considered to be a dynamic matrix sequence, and then the convolution kernel is used to perform the following convolution operation on the dynamic matrix:

[0098] In the formula, the arrow represents the convolution operation, through which the cross-correlation features on the previous and next time series can be extracted and the cross-correlation feature map can be constructed. As the sliding window continues to move forward along the timestamp direction, the cross-correlation feature map on the continuous time variable will be constructed.

[0099] For the extraction of multivariate spatial features, it mainly performs the following convolution operations on each input using the convolution kernel:

[0100]

[0101] By using this convolution operation, the spatial variation characteristics of each type of input variable can be accurately extracted, thereby realizing the construction of a multivariate spatial feature map. After completing the above convolution operation, a pooling operation will be performed. In order to retain the original information as much as possible and reduce the amount of calculation, the maximum pooling operation is selected in this article. For the maximum pooling operation, its calculation formula is as follows:

[0102]

[0103] In this paper, a linear rectification function is used as the activation function of the inherent hidden layer in the dynamic convolutional neural network. After performing the maximum pooling operation on the cross-correlation feature map and the multivariate spatial feature map, the cross-correlation feature map and the multivariate spatial feature map are expanded in a fully connected manner, and the two are converted into a one-dimensional sequence matrix, and then directly connected on the fully connected layer to achieve feature fusion. At the same time, based on the fully connected layer, a regression layer is established to achieve assembly level accuracy prediction, and an independent sub-model M1 is obtained.

[0104] After the M1 model is established, in order to realize the establishment of models between other levels, the MMD-MSE method is used to migrate between sub-models. This method uses the maximum mean difference (MMD) to measure the probability distribution difference distance index between data, thereby realizing the migration of assembly accuracy prediction for multiple working conditions.

[0105] make , where D1 represents the maximum distance, is the posterior probability of the corresponding data; (hereinafter referred to as ) are used to represent the gap between the first established model and the subsequent models. Definition As a source model set relational dataset built on a dynamic convolutional neural network, and , the calculation formula can be based on To the original Calculate, where the superscript i represents the i-th data in data set A. Represented as input, is the corresponding output; in the formula and The data distributions of the first established model and subsequent models are shown below. Based on the above definition, the maximum mean difference MMDs can be obtained as follows:

[0106]

[0107]

[0108]

[0109] In the formula is the supremum of the aggregate, is a reproducible kernel Hilbert space (RKHS), n represents the amount of data inside, n1 represents the amount of data in data set A1, and n2 represents the amount of data in data set The amount of data in Represents the mean of the reproducible kernel Hilbert space. For the initial training model, Consider it as the optimization goal to adjust its weight. Then take a linear time fitting function of MMD as the optimized objective function, which is the following formula:

[0110]

[0111] M is the mean difference matrix, is the element in the mean difference matrix M, through Calculated, where , pass Calculated, pass Calculated; is the kernel operator defined on the quadruple, which can be calculated by the following formula:

[0112]

[0113] Here k (,) is the Hilbert kernel transform. At the same time, in the above-defined weight adjustment process, the prediction error should be minimized. Therefore, the prediction error is optimized as another optimization target. Then the total error can be further defined as The combination of MMD and mean absolute error (MSE) is MMD-MSE. The thresholds of MSE and Nadir points are different, so this paper performs regularization operations on both of them. In this embodiment, Nadir and Utopia points are used to perform regularization operations on MMD and MSE. , which is constructed by The minimization objective provides the minimum margins of MMD and MSE. i=1 represents the margin of MMD, and i=2 represents the margin of MSE, for the Nadir point. It is calculated by the formula.

[0114]

[0115] Based on the above definition, the regularized MMD and MSE can be calculated using the following formula:

[0116]

[0117]

[0118] In the formula For regularized ,and is the regularized MSE. Finally, the total loss can be obtained by adding the two regularized targets and the corresponding weight products. The specific calculation formula is as follows:

[0119]

[0120] in, and Assign and The weight of . This weight is used in transfer learning. is the temporary prediction value of the model, and Y is the actual value, which will be continuously iterated and optimized to determine the optimal weight value. Based on the above process, the training optimization algorithm will complete a retraining process. For the training optimization algorithm, this paper uses the adaptive moment algorithm (ADAM) to carry out.

[0121] Based on the above-mentioned MMD-MSE iteration, by running through the relationship between Dynamic CNN and conditional random fields, it is possible to achieve multi-level independent integrated modeling for assembly space, and then establish a multi-level prediction model for assembly quality, thereby achieving accurate monitoring of assembly.

[0122] Example 2

[0123] This embodiment provides a product multi-level assembly quality detection system based on multi-field spatiotemporal transfer learning, the system comprising a front-end interactive interface, an assembly data acquisition module, a time series data processing module, an OPC UA client, and a multi-level assembly quality prediction module.

[0124] Among them, the assembly data module is responsible for collecting data on the assembly line. It uses cameras, torque sensors, RFID and other diversified data collection methods to collect data at key points such as manual assembly stations, robotic assembly stations, and transportation stations. After filtering the data according to weight values, it is transmitted to the timing data processing module.

[0125] The time series data processing module is responsible for performing time series multi-dimensional processing on the collected data. Specifically, the module identifies the time series information storage values ​​according to different time periods. It can be defined as images, texts, numbers and other formats to represent video streams, text streams, sensors and other types of data. Then the Value is divided into dimension vectors X, Y, Z according to different dimensions, and the processed time series multi-dimensional data is uploaded to the OPC UA client. The OPC UA client performs different intermediate processing on the received data according to different types, and then outputs the purpose and storage address of each data (the address symbol corresponding to the model output data in the virtual space) to the multi-level assembly quality prediction module. The OPC UA client interacts with the user using the user interaction interface; among them, if Figure 5In the visualization module shown, the interaction includes: display of different types of input data, selection of paths for intermediate data processing, display of output data, etc.

[0126] The multi-level assembly quality prediction module is responsible for quality prediction of assembly data; specifically, the module includes a regularized Gaussian mixture clustering model and a multi-level prediction model for assembly quality; first, based on the regularized Gaussian mixture clustering model, the different categories of attributes collected during the assembly process are clustered, and then based on the multi-level prediction model for assembly quality, assembly quality prediction is achieved, and the final quality prediction results are displayed through the front-end interactive interface.

[0127] In this embodiment, Figure 8 The application is implemented in the array antenna assembly case shown in the figure. The front-end interactive interface of the system is as follows Fig. 9 As shown, the middle part of the page is the assembly progress tracking based on the whole process. The results show that the patent of this invention can well fit the multi-level assembly scene and perform visual dynamic display. The right part of the page is the result diagnosis part. The results show that the system constructed by the patent of this invention can accurately diagnose the algorithms at different assembly levels. The left part of the page provides the equipment type information for the patent of this invention, which further proves that the patent of this invention can achieve refined tracking.

[0128] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A product multi-level assembly quality detection method based on multi-field spatiotemporal transfer learning, characterized in that: The following steps are involved: Step 1: For assembly data tracking of multi-level assembly processes, the influencing factors of any two assembly links are compared through the fuzzy complementary judgment matrix, so as to identify the key error links of the assembly process and obtain the key factors of the assembly process; Step 2: Build a cloud-edge-device collaborative architecture time series data model based on the time series base class and the high-dimensional time series base class, define the collected time series data based on key factors as a tree structure, and then divide it into dimensional vectors according to different dimensions; Step 3: Based on the virtual dynamic mapping of assembly product components of the improved OPC UA information model, different types of time series data are obtained; first, the data based on the time series base class and the high-dimensional time series base class obtained in step 2 are collected in real time and sorted and organized based on the OPC UA information model, and the abnormal data screening method in the time series sensor data based on the support vector machine-hidden Markov is used to screen the collected data, and the purpose and storage address of the screened data are output; After completing the unification of the collected queue messages, the message queue telemetry transmission protocol is used to connect directly to the cloud, and the collected data is directly transmitted to the cloud queue to achieve dynamic and rapid tracking of messages; After automatic acquisition, upsampling and upsampling are used to expand the acquired data. Specifically, the data collected by each sensor is defined as S1, S2, ..., S n , the function is formulated using the maximization function as follows: S max =max(S1,S2,....,S n ); In order to obtain the maximum sampling frequency S max Finally, for data with a sampling frequency less than the maximum sampling frequency, the upsampling method is used to increase the data samples. The specific formula is as follows: Where V s Represents the upsampled data, V t+1 With V t Respectively represent the data of the previous moment and the next moment; Step 4: Multi-level assembly quality prediction based on multi-field spatiotemporal transfer learning; Step 4.1: Based on the similarity assembly process attribute clustering method of the regularized Gaussian mixture clustering model, the data collected during the assembly process are clustered into several subsets D1, D2, D3, ...Dn; Step 4.2 Based on the multi-task random migration method of maximum mean difference-mean square error, transfer learning and training in multiple spatiotemporal fields are realized to build an assembly quality prediction model for multi-level long spatiotemporal queues; The step 4.2 is as follows: Use deep learning to establish quality prediction models M1, M2, M3…Mn-1, Mn for each subset D1, D2, D3, D…Dn, which are collectively referred to as source models. The model constructed by a single subset is referred to as a sub-model, and free transfer learning can be performed between subsets; Step 4.2.1: First, taking subset D1 as an example, the sub-model M1 is established and trained based on the dynamic convolutional neural network-conditional random field. The specific steps are: model data set deconstruction, feature extraction, feature fusion, and modeling output; The data set deconstruction is specifically as follows: repeatedly split the subsets to construct a local data set set Z LD,m , LD represents the local data set, and m represents the number of local data sets; The feature extraction is specifically local cross-correlation feature and multivariate spatial feature extraction, and feature extraction is performed for each local data set; Local cross-correlation feature extraction and utilization The continuous sliding window of Indicates the specific data value corresponding to the input and output at time ki, and T represents transposition; construct a matrix containing the previous and next correlation relationships under a continuous sliding window. The specific process is shown in the formula below: Where z is the number of local data sets; is the kth local data set on the zth local data set i Data at time, subscript k i+1 Indicates k i The next moment of the moment; Based on the matrix sequence constructed above, as the dynamic matrix sequence for dynamically characterizing the assembly accuracy in the current assembly time period, the dynamic matrix is ​​subjected to the following convolution operation using the convolution kernel to complete the construction of the cross-correlation feature map; As the sliding window continues to move forward along the timestamp direction, the cross-correlation feature map on the continuous time variable is constructed; The extraction of multivariate spatial features is achieved by using convolution kernels to perform convolution operations on each input to construct a multivariate spatial feature map; The feature fusion is specifically as follows: after performing the maximum pooling operation on the cross-correlation feature map and the multivariate space feature map respectively, the cross-correlation feature map and the multivariate space feature map are respectively expanded in a fully connected manner, and the two are converted into a one-dimensional sequence matrix, and then directly connected on the fully connected layer to achieve feature fusion; at the same time, based on the fully connected layer, a regression layer is established to achieve assembly level accuracy prediction, and an independent sub-model M1 is established; Step 4.2.2: Then use the maximum mean difference-mean absolute error method to build sub-models for other subsets; Let D1 = {A1, P(A1)}, where D1 represents the maximum distance and P(A1) is the posterior probability of the corresponding data; D j ={A i , P(A i )},j=2,…,n, hereinafter referred to as A 2…n , respectively, are used to represent the gap between the first established model and the subsequent models; definition as well as As a source model set relational dataset based on a dynamic convolutional neural network, the superscript i represents the i-th data in the dataset A, x represents the input, and y represents the corresponding output; and The data distributions of the first established model and subsequent models are shown below. Based on the above definition, the maximum mean difference MMDs are obtained as follows: MMD s (A1A 2…n )=sup[Mean H (A1)-Mean H (A 2…n )]; In the formula, sup(·) is the supremum of the set, H(·) is the reproducible kernel Hilbert space, n1 represents the amount of data in the dataset A1, and n2 represents the amount of data in the dataset A 2…n The amount of data in MEan H represents the mean of the reproducible kernel Hilbert space; For the initial training model, MMD s (A1A 2…n ) is used as the optimization target to adjust its weight; a linear time fitting function of MMD is used as the optimized target function, which is the following formula: M is the mean difference matrix, z i is the element in the mean difference matrix M, through MMD s (A1A 2…n ) is calculated, By Mean H (A1) is calculated, By Mean H (A 2…n ) is calculated; h l is a kernel operator defined on a quadruple; Then the maximum mean difference and mean absolute error are regularized; for the Utopia point By constructing The minimization objective provides the minimum bounds for the maximum mean difference (MMD) and the mean absolute error (MSE). i = 1 represents the bounds for MMD and i = 2 represents the bounds for MSE. For Nadir points It is calculated by the following formula: Regularized MMD and MSE are calculated using the following formula: Where NMMD s is the regularized MMD s , and NMSE is the regularized MSE; finally, the specific calculation formula for the total loss is as follows: Among them, w1 and w2 are given to NMMD s and NMSE weight, and w1+w2=1; this weight is used in transfer learning. is the temporary prediction value of the model, and Y is the actual value. Based on the above MMD-MSE iteration, a multi-level prediction model for assembly quality is established to realize assembly monitoring. Step 5: Based on the assembly quality prediction model of multi-level long-time and space queues, real-time quality inspection of the assembly process is performed.

2. According to claim 1, a product multi-level assembly quality detection method based on multi-field spatiotemporal transfer learning is characterized in that: The step 1 is specifically as follows: First, for all the influencing links of the assembly process, we arbitrarily define the different influencing factors in two assembly processes as a i with a j , construct the fuzzy complementary judgment matrix R = (r ij ) n*n , its specific manifestations are as follows: The fuzzy matrix is ​​an n*n order matrix. For the element r in the fuzzy complementary judgment matrix, ij The definitions are: r ii =0.5,i=1,2,…,n; r ij +r ji =1,i,j=1,2,…,n; Using r ij For element a i with a j The quantitative influence relationship between: If r ij =0.5, then the two are marked as equal; if r ij If it is greater than 0.5, then a i Than a j important; otherwise, it is considered that a j Than a i important; Construct an n-order non-negative regularized weight vector matrix W, which is expressed as follows: W=[w1,w2,…,w n ] T ; In the formula, w n is the element of the non-negative regularized weight vector matrix W, that is, the weight value, which is obtained by the following formula: The weight influencing factors with the first set percentage in the weight value are taken as key factors.

3. The method for multi-level assembly quality inspection of products based on multi-field spatiotemporal transfer learning according to claim 1 is characterized in that: The process of screening the collected data by the method for screening abnormal data in time series sensor data based on support vector machine-hidden Markov is as follows: Assuming that the data distribution obeys the hidden Markov format and the random attribute rule of principal component analysis, there is a sequence data O = (o1, o2, ..., o T ) obeys the rule of λ=(A,B,π), where A, B, π represent the data transfer matrix, abnormal determination coefficient, and initial probability matrix respectively; The nonlinear inseparable support vector machine is used to solve the abnormal determination coefficient B and calculate the support vector machine regression model. Then, based on the regression model, the abnormal determination coefficient matrix sequence at different stages is obtained to achieve accurate judgment of the results and remove abnormal data.

4. The method for multi-level assembly quality inspection of products based on multi-field spatiotemporal transfer learning according to claim 1 is characterized in that: The step 4.1 is as follows: First, obtain the training data set, expressed as: {X,Z}={X1,z1,...,X i ,z i ,...,X N ,z N }; Where X is the covariate matrix of the data set, Z is the latent variable corresponding to the data, N is the number of data, and each data X i Corresponding to a hidden variable z i ; Data X i (i=1,…,N)Assume that it follows Gaussian distribution where μ j With Σ j Expressed as mean and covariance; based on the above assumptions, the following likelihood function is obtained: Among them, π is a preset constant, π j represents a preset constant value, K is the total number of rows and columns of the covariate matrix, z ij is the specific value of the i-th row and j-th column in the covariate matrix X, μ is the population mean after the likelihood function is calculated, and Σ is the population covariance after the likelihood function is calculated; Based on the likelihood function, an optimization problem is used to find the maximum solution of the likelihood function; then, the expectation of the logarithmic likelihood of the data, that is, the similarity index of the data, is obtained using the following equation: In the formula, θ old is the parameter of the likelihood function before iteration, θ represents the parameter of the likelihood function, p j is the Gaussian density in the assembly, where the key parameter is θ j ; is θ old The expectation of Z under event X, logp j (X i |θ j ) is event X i In θ j The conditional entropy under , P(X, Z) is the probability distribution of data X and hidden variable Z; Based on the obtained similarity index, a matrix method is used for fitting to achieve clustering of the collected assembly data into several subsets D1, D2, D3, ...Dn.

5. The method for multi-level assembly quality inspection of products based on multi-field spatiotemporal transfer learning according to claim 4 is characterized in that: There are two types of likelihood function iteration methods: one is E-step, which is performed in a rotation order; the other is M-step, which uses the regularized covariance estimate as input in each iteration, and the two iteration methods are performed simultaneously; in the E-step, the i-th row z in the covariate matrix X i Mixture component k in z ik Estimates as follows: In the formula, P(z ij =1|X,θ old ) is the probability distribution of θ with parameter X old Next, z ij =1 conditional probability; π k represents the preset constant value in the mixed component k; p k (X i |θ k ) is the key parameter θ in the mixture component k k Conditional data X i The probability distribution of data X i Specifically, for assembly events, θ k is the key parameter in the mixed component k; p j (X i |θ j ) is the key parameter θ j Conditional data X i The probability distribution of Furthermore, the prior probability and average value of the kth component can be calculated by the following formula: In the formula, is the i-th row z in the covariate matrix X i Mixture component k in z ik The prior probability, z ik is the i-th row z in the covariate matrix X i The kth component of the mixture component k of For z ik The superscript t represents the time step.

6. The method for multi-level assembly quality inspection of products based on multi-field spatiotemporal transfer learning according to claim 5 is characterized in that: The multi-level prediction model is trained using an adaptive moment algorithm ADAM during the training and optimization process.

7. A product multi-level assembly quality inspection system based on multi-field spatiotemporal transfer learning, characterized in that: The method according to any one of claims 1 to 6 can be implemented; specifically, the system includes a front-end interactive interface, an assembly data acquisition module, a time series data processing module, an OPC UA client, and a multi-level assembly quality prediction module; Among them, the assembly data acquisition module is responsible for collecting data on the assembly line, filtering the data according to the weight value, and transmitting it to the time series data processing module; The time series data processing module is responsible for time series multi-dimensional processing of data. It defines the collected time series data based on key factors as a tree structure, and then divides it into dimension vectors according to different dimensions. Then, the processed time series multi-dimensional data is uploaded to the OPC UA client. The OPC UA client performs different intermediate processing on the received data according to different types, filters the collected data based on the abnormal data screening method in time series sensor data based on the support vector machine-hidden Markov, and then directly transmits the filtered data to the cloud queue and the multi-level assembly quality prediction module; the OPC UA client interacts with the user through the user interaction interface; The multi-level assembly quality prediction module is responsible for quality prediction of assembly data; specifically, the module includes a regularized Gaussian mixture clustering model and an assembly quality prediction model; first, based on the regularized Gaussian mixture clustering model, the different categories of attribute data collected during the assembly process are clustered, and then based on the assembly quality prediction model, assembly quality prediction is realized, and the final quality prediction results are displayed through the front-end interactive interface.