Production knowledge automatic extraction and management system oriented to process standardization

Abstract

The invention relates to the technical field of electrical digital data processing, in particular to a process standardization-oriented production knowledge automatic extraction and management system, which comprises a data acquisition and processing module for acquiring multi-modal time sequence data, performing preprocessing and feature extraction and generating a multi-modal feature sequence; the skill identification module is used for segmenting a skill primitive sequence and acquiring a key parameter vector of each skill primitive based on the multi-modal feature sequence, and identifying an operation intention based on the skill primitive sequence and the key parameter vector; the skill modeling module is used for constructing a common skill model and a personalized skill model of the process operation process based on the skill primitive sequence; the model evaluation module is used for defining multi-dimensional evaluation indexes and evaluating operation effects corresponding to the generality and personalized skill models; and the knowledge management module is used for constructing a structured process knowledge base supporting dynamic updating and complex query. According to the method, a data-to-knowledge conversion system is established, and a scientific and efficient process standardization process is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and specifically to a production knowledge automatic extraction and management system for process standardization. Background Art

[0002] With the development of modern manufacturing towards intelligence and lean production, higher requirements are put forward for the process standardization of the production process. Process standardization is a key link to ensure product quality, improve production efficiency, reduce manufacturing costs, and achieve knowledge inheritance. In this context, using information technology to effectively manage and utilize the massive data generated during the production process has become an important means to promote process standardization. Basic data management technologies, such as database technologies for efficiently organizing structured data and file organization systems for managing unstructured information, have laid the foundation for storing and initially accessing massive production data.

[0003] Furthermore, in order to more effectively serve process standardization, simply storing raw data is far from enough. It is necessary to transform the data into understandable and reusable knowledge. In this regard, advanced information management methods, especially technologies related to knowledge representation, structured storage, and intelligent retrieval, play a core role. Some of the principles of these technologies can also be seen in large-scale data processing applications, such as the information retrieval and organization mechanisms relied on by Internet search services. By establishing a production knowledge base and using semantic data models, etc., explicit knowledge such as process rules, operation steps, and causal relationships extracted can be represented and stored in a structured manner. This enables process engineers to conveniently search for, compare, and revise standard operating procedures using mature query and retrieval mechanisms, and to a certain extent support process design and optimization decisions. Existing manufacturing execution systems, product lifecycle management systems, etc. exactly apply these principles of structured knowledge representation and retrieval to manage explicit process knowledge and support daily production operations and standardization processes.

[0004] However, there are still significant deficiencies in the existing technologies in terms of automatic extraction and management of knowledge for process standardization. First, it is difficult to effectively segment continuous multi-modal data streams and identify corresponding operation intentions. Second, it is difficult to distinguish and construct models to express common core operation paradigms and personalized skills.

[0005] Therefore, a production knowledge automatic extraction and management system for process standardization is proposed. Summary of the Invention

[0006] The object of the present invention is to provide a production knowledge automatic extraction and management system for process standardization, and to realize a scientific and efficient process standardization process by establishing a conversion system from data to knowledge. It includes: a data acquisition and processing module, which acquires multi-modal time-series data, performs preprocessing and feature extraction, and generates multi-modal feature sequences; a skill recognition module, which based on the multi-modal feature sequences, segments the skill primitive sequences and obtains the key parameter vectors of each skill primitive, and recognizes the operation intention based on the skill primitive sequences and the key parameter vectors; a skill modeling module, which constructs a common skill model and a personalized skill model for the process operation process based on the skill primitive sequences; a model evaluation module, which defines multi-dimensional evaluation indicators and evaluates the operation effects corresponding to the common and personalized skill models; a knowledge management module, which constructs a structured process knowledge base that supports dynamic update and complex query.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A production knowledge automatic extraction and management system for process standardization, including:

[0009] A data acquisition and processing module, which acquires multi-modal time-series data during the process operation executed by an operator, performs preprocessing and feature extraction on the multi-modal time-series data, and generates multi-modal feature sequences;

[0010] A skill recognition module, which based on the multi-modal feature sequences, segments the process operation process into skill primitive sequences and obtains the key parameter vectors of each skill primitive, and recognizes the operation intention based on the skill primitive sequences and the key parameter vectors;

[0011] A skill modeling module, which constructs a common skill model and a personalized skill model for the process operation process based on the skill primitive sequences;

[0012] A model evaluation module, which defines multi-dimensional evaluation indicators and evaluates the common skill model and the personalized skill model;

[0013] A knowledge management module, which based on the skill primitive sequences, the key parameter vectors, the operation intention, the common skill model, the personalized skill model and the operation effect evaluation results, constructs a process knowledge base that is dynamically updated, structured and can be complexly queried.

[0014] Further, the data acquisition and processing module includes: collecting attitude data, force sensor data, visual data, and acoustic data and performing time synchronization processing to form multi-modal time-series data; preprocessing the multi-modal time-series data, including denoising, filtering, calibration, and standardization; for attitude data and force sensor data, using a sliding window method to calculate statistical features, frequency domain features, and extract time-frequency features at different scales by combining wavelet transform within each time window, generating force feature vectors and attitude feature vectors; for visual data, using a pre-trained deep learning model to extract visual feature vectors; for acoustic data, also using a sliding window to extract Mel-frequency cepstral coefficients, spectral centroid, spectral entropy, and short-time energy, generating acoustic feature vectors; combining the features extracted from all modalities at each time step into a multi-modal feature vector, generating a multi-modal feature sequence.

[0015] Further, segmenting the process operation process into a sequence of skill primitives and obtaining the key parameter vectors of each skill primitive includes: inputting the multi-modal feature sequence into a temporal convolutional network to extract a high-level feature representation sequence; using the high-level feature representation sequence as the observation sequence of a hidden semi-Markov model, where the hidden semi-Markov model defines a set of hidden states, each of which corresponds to a skill primitive, introducing a duration distribution for each hidden state, and observing emission probabilities and state transition probabilities; based on the observation sequence, duration distribution, emission probabilities, and state transition probabilities, using the Viterbi algorithm to find the optimal state sequence and the corresponding start and end timestamps, obtaining the sequence of skill primitives, which includes the type, start time, and end time of each skill primitive; obtaining the multi-modal feature subsequence corresponding to each skill primitive based on the multi-modal feature sequence, and applying signal processing algorithms to obtain the key parameter vectors of each skill primitive.

[0016] Further, identifying the operation intention based on the sequence of skill primitives and key parameter features includes: using a first recurrent neural network to process the multi-modal feature subsequence corresponding to each skill primitive to obtain a first hidden feature sequence, applying a first attention mechanism within the first hidden feature sequence to generate a primitive representation vector; concatenating the primitive representation vector with the corresponding key parameter vector to obtain a comprehensive representation of the skill primitive; inputting all the comprehensive representations of the skill primitives into a second recurrent neural network to obtain a second hidden feature sequence; applying a second attention mechanism within the second hidden feature sequence to generate an operation representation vector; processing the operation representation vector through a fully connected layer and a Softmax activation function to obtain the operation intention.

[0017] Further, constructing a common skill model and a personalized skill model for the process operation process includes:

[0018] For each type of skill primitive, a common skill model based on conditional variational autoencoder is constructed, including: defining a conditional encoder to map the multi-modal feature subsequence of the skill primitive and the skill primitive type to a latent variable distribution; defining a conditional decoder to reconstruct the multi-modal feature subsequence according to the latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the evidence lower bound; using the trained conditional decoder as the common skill model;

[0019] Based on the common skill model, a personalized skill model is constructed, including: using the conditional encoder of the common skill model to obtain the sequence of latent vectors corresponding to each type of skill primitive executed by a specific operator; calculating the empirical mean and covariance matrix of the sequence of latent vectors as the personalized latent distribution of the specific operator for each type of skill primitive; the personalized skill model is jointly defined by the personalized latent distribution and the conditional decoder of the common skill model.

[0020] Furthermore, the model evaluation module includes: defining multi-dimensional evaluation indicators including efficiency, quality, consistency, resource consumption, and safety; generating a reconstructed multi-modal feature sequence based on the common skill model and the personalized skill model; performing the operations corresponding to the reconstructed multi-modal feature sequence through physical simulation, and calculating the values of the multi-dimensional evaluation indicators; normalizing the values of the multi-dimensional evaluation indicators, and calculating the comprehensive evaluation score of each model according to the preset weights; generating a model evaluation report including the values of each indicator and the comprehensive evaluation score.

[0021] Furthermore, the knowledge management module includes: designing a structured knowledge base schema for storing and associating information entities, where the information entities include processes, operators, operation records, skill primitive types, skill primitive instances, key parameters, operation intents, common skill models, personalized skill models, model evaluation results, and structured process knowledge; defining the associations between the information entities through foreign keys; constructing a knowledge base management sub-module to support operations of adding, deleting, modifying, and querying the information entities, and support complex queries, where the complex queries include standard operation retrieval, operator difference comparison, key parameter statistical analysis, and intent-based process mining; establishing a knowledge update mechanism to automatically update the content of the knowledge base after new data processing is completed, and implementing version control for models and knowledge entries; forming a structured process knowledge base.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] 1. Preprocess and extract features from multi-modal data to obtain multi-modal feature sequences. Then, through the combination of a temporal convolutional network and a hidden semi-Markov model, the system can accurately segment continuous operation processes into meaningful skill primitive sequences and extract key parameters for each primitive. Based on a recurrent neural network and a multi-level attention mechanism, the system can identify the operator's high-level operation intentions from the skill primitive sequences. The processing of multi-modal data significantly improves the accuracy and robustness of skill primitive recognition. The system can efficiently and accurately identify the operator's skill primitives and operation intentions, providing reliable data support for subsequent skill modeling and knowledge management, and significantly enhancing the automated extraction ability of production knowledge for process standardization.

[0024] 2. A skill modeling method that takes into account both commonality and individuality. Based on the conditional variational autoencoder architecture, by mapping the multi-modal feature subsequences and types of skill primitives to the latent variable space and then reconstructing the feature sequences from the latent variables, a deep understanding of the internal structure of skill primitives is achieved. In the construction of the common skill model, the system uses the data of all operators to train the CVAE to capture the general performance of skill primitives. In the stage of constructing the personalized skill model, the system retains the decoder structure of the common model but calculates personalized latent variable distribution parameters for each operator, thus retaining the unique skills of specific operators. This method not only overcomes the limitation of traditional skill models that can only express standardized operations and ignore individual differences but also provides a solution to organically integrate commonality and individuality, effectively integrating the common and personalized characteristics of skills, maintaining the standardization of process operations while retaining the effective personalized skills of operators.

[0025] 3. A dynamic update and structured process knowledge base is established to achieve systematic management of process knowledge. The knowledge base has powerful complex query capabilities, such as standard operation retrieval, operator difference comparison, key parameter statistical analysis, etc. The knowledge base has an automatic update mechanism. When the system processes new operation data, relevant knowledge entries will be automatically updated. At the same time, the version control function ensures the traceability of knowledge evolution. This knowledge base design solves problems such as information silos, lagging knowledge updates, and limited query capabilities in traditional process knowledge management, greatly improving the efficiency and accuracy of knowledge management. It realizes the efficient management and dynamic update of process knowledge, provides a knowledge foundation for process standardization, and supports the preservation and analysis of the personalized skills of different technicians. Brief Description of the Drawings

[0026] Figure 1 It is a schematic structural diagram of the production knowledge automated extraction and management system for process standardization of the present invention;

[0027] Figure 2 It is a schematic diagram of the operation intention recognition process of the present invention;

[0028] Figure 3 This is a flowchart showing the implementation steps of the production knowledge automatic extraction and management system for process standardization of the present invention. Specific implementation manners

[0029] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0030] Please refer to Figures 1 to 3 , the present invention provides a production knowledge automatic extraction and management system for process standardization, and the technical solutions are as follows:

[0031] Embodiment 1:

[0032] A production knowledge automatic extraction and management system for process standardization, see Figure 1 , including:

[0033] A data acquisition and processing module, which acquires multi-modal time-series data during the process operation performed by an operator, preprocesses and extracts features from the multi-modal time-series data, and generates a multi-modal feature sequence;

[0034] A skill recognition module, based on the multi-modal feature sequence, divides the process operation process into a skill primitive sequence and obtains the key parameter vectors of each skill primitive, and recognizes the operation intention based on the skill primitive sequence and the key parameter vectors;

[0035] A skill modeling module, which constructs a common skill model and a personalized skill model for the process operation process based on the skill primitive sequence;

[0036] A model evaluation module, which defines multi-dimensional evaluation indexes and evaluates the common skill model and the personalized skill model;

[0037] A knowledge management module, which constructs a dynamically updated, structured and complex queryable process knowledge base based on the skill primitive sequence, the key parameter vectors, the operation intention, the common skill model, the personalized skill model and the operation effect evaluation result.

[0038] Further, the data acquisition and processing module includes: collecting attitude data, force sensor data, visual data, and acoustic data, and performing time synchronization processing to form multi-modal time-series data; preprocessing the multi-modal time-series data, including denoising, filtering, calibration, and standardization; for attitude data and force sensor data, using a sliding window method to calculate statistical features, frequency-domain features, and extract time-frequency features at different scales by combining wavelet transform within each time window, generating force feature vectors and attitude feature vectors; for visual data, using a pre-trained deep learning model to extract visual feature vectors; for acoustic data, also using a sliding window to extract Mel-frequency cepstral coefficients, spectral centroid, spectral entropy, and short-time energy, generating acoustic feature vectors; combining the features extracted from all modalities at each time step into a multi-modal feature vector to generate a multi-modal feature sequence.

[0039] Through the multi-modal data synchronization and refined feature extraction method of the data acquisition and processing module, the system can effectively extract valuable information from multi-source data, ensuring data quality and the system's response speed. It improves the integrity and accuracy of the data, thus better supporting the subsequent skill recognition and modeling processes and providing a high-quality data foundation for process standardization.

[0040] Further, segmenting the process operation process into a skill primitive sequence and obtaining the key parameter vector of each skill primitive includes:

[0041] Inputting the multi-modal feature sequence into a temporal convolutional network to extract a high-level feature representation sequence; the temporal convolutional network learns the hierarchical temporal representation of the multi-modal feature sequence through multiple layers of causal convolution, dilated convolution, and residual connections to obtain a high-level feature representation sequence;

[0042] Taking the high-level feature representation sequence as the observation sequence of a hidden semi-Markov model, the hidden semi-Markov model defines a set of hidden states, where each hidden state corresponds to a skill primitive, introducing a duration distribution for each hidden state, for example, using a parameterized probability distribution to describe the possible duration length of each state, and observing the emission probability and state transition probability; the emission probability P(h t:t+d-1 |q i ,d) represents the probability of observing the high-level feature representation subsequence h i under the condition that the duration in the i-th state q t:t+d-1 is d, t represents the start time of the high-level feature representation subsequence; the state transition probability P(q j |q i ) represents the probability of transitioning from the end of the i-th state q i to the j-th state q j ;

[0043] Based on the observation sequence, duration distribution, emission probability, and state transition probability, use the Viterbi algorithm to find the optimal state sequence and the corresponding start and end timestamps, and obtain the skill primitive sequence, which includes the type, start time, and end time of each skill primitive; based on the multi-modal feature sequence, obtain the multi-modal feature subsequence corresponding to each skill primitive, and apply signal processing algorithms to obtain the key parameter vector of each skill primitive;

[0044] The key parameter vectors of different types of skill primitives contain different key parameters. It is necessary to pre-define a mapping relationship to associate each type of skill primitive with the corresponding key parameter list, which includes which key parameters need to be calculated for each type of skill primitive and the specific methods for calculating each key parameter; for example, for the "tightening" skill primitive, the key parameter vector includes: peak torque, time to reach peak torque, final tightening angle, average angular velocity, and tightening duration; it is calculated through peak detection and statistical analysis.

[0045] By introducing a temporal convolutional network and a hidden semi-Markov model, the system can accurately segment the process operation into a skill primitive sequence and extract the key parameter vector. It improves the segmentation accuracy of the process and effectively solves the problem of fuzzy boundaries of skill primitives in continuous operations, enabling the system to more accurately capture each core step in the operation process.

[0046] Furthermore, referring to Figure 2 , identifying the operation intention based on the skill primitive sequence and key parameter features includes: using a first recurrent neural network to process the multi-modal feature subsequence corresponding to each skill primitive to obtain a first hidden feature sequence; applying a first attention mechanism within the first hidden feature sequence to generate a primitive representation vector; concatenating the primitive representation vector with the corresponding key parameter vector to obtain a comprehensive representation of the skill primitive; inputting all the comprehensive representations of the skill primitives into a second recurrent neural network to capture the sequential dependence relationship between the skill primitives and obtain a second hidden feature sequence; applying a second attention mechanism within the second hidden feature sequence to generate an operation representation vector; processing the operation representation vector through a fully connected layer and a Softmax activation function to obtain the operation intention;

[0047] Furthermore, the first hidden feature sequence refers to the context representation within the multi-modal feature subsequence of a skill primitive. Each skill primitive corresponds to a first hidden feature sequence. Applying the first attention mechanism within the first hidden feature sequence means assigning different first importance weights to the features at different time steps within the skill primitive, and obtaining the primitive representation vector through weighted summation. Each second hidden feature in the second hidden feature sequence corresponds to a skill primitive. Applying the second attention mechanism within the second hidden feature sequence means calculating the second importance weight of each second hidden feature of the skill primitive for identifying the overall operation intention, and obtaining the operation representation vector of the entire operation sequence through weighted summation.

[0048] Using a recurrent neural network and a multi-level attention mechanism, the system can identify high-level operation intentions from the skill primitive sequence and its key parameters. This effectively improves the system's reasoning ability for operation intentions, enabling it to understand the purpose behind the operator's behavior and providing semantic-level support for the structured expression of process knowledge.

[0049] Furthermore, constructing the common skill model and the personalized skill model for the process operation process includes:

[0050] For each type of skill primitive, construct a common skill model based on a conditional variational autoencoder, including: defining a conditional encoder to map the multi-modal feature subsequence of the skill primitive and the skill primitive type to a latent variable distribution; defining a conditional decoder to reconstruct the multi-modal feature subsequence according to the latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the evidence lower bound; using the trained conditional decoder as the common skill model; the evidence lower bound includes a reconstruction loss term and a KL divergence term. The reconstruction loss measures the ability of the decoder to reconstruct the input, usually using mean squared error or cross-entropy. The KL divergence constrains the posterior distribution of the latent variables to be close to a preset prior distribution;

[0051] Based on the common skill model, construct a personalized skill model, including: using the conditional encoder of the common skill model to obtain the sequence of latent vectors corresponding to a specific operator's execution of each type of skill primitive; calculating the empirical mean and covariance matrix of the sequence of latent vectors as the personalized latent distribution of the specific operator for each type of skill primitive; the personalized skill model is jointly defined by the personalized latent distribution and the conditional decoder of the common skill model;

[0052] The construction method of the common skill model and the personalized skill model based on the conditional variational autoencoder realizes the balanced expression of skill commonality and individuality. This method can not only capture the standard patterns of process operations but also retain the personalized skills of operators, providing an effective solution to the contradiction between process standardization and the retention of personalized skills.

[0053] Furthermore, the model evaluation module includes: defining multi-dimensional evaluation indicators including efficiency, quality, consistency, resource consumption, and safety; generating a reconstructed multi-modal feature sequence based on the common skill model and the personalized skill model; performing operations corresponding to the reconstructed multi-modal feature sequence through physical simulation, and calculating the values of the multi-dimensional evaluation indicators; normalizing the values of the multi-dimensional evaluation indicators, and calculating the comprehensive evaluation scores of each model according to preset weights; generating a model evaluation report including the values of each indicator and the comprehensive evaluation scores.

[0054] The multi-dimensional evaluation indicators include: Efficiency: the total time to complete a task, the average duration of a single skill primitive, and the output per unit time; Quality: the pass rate of the final product, the critical dimension accuracy, and the surface roughness; Consistency: the standard deviation of the key parameters corresponding to the skill primitive when repeating the same task; Resource consumption: energy consumption and material loss rate; Safety: the maximum instantaneous force during the operation process.

[0055] Generating a reconstructed multi-modal feature sequence based on the common skill model and the personalized skill model means that, according to the process standard flow sequence, reconstructing the multi-modal feature subsequences of all skill primitives based on the common skill model, and obtaining the common reconstructed multi-modal feature sequence after merging; then, based on the common reconstructed multi-modal feature sequence, different personalized reconstructed multi-modal feature sequences are output through multiple personalized skill models.

[0056] Comprehensively evaluating the effects of skill models through multi-dimensional evaluation indicators provides a quantitative model evaluation standard, helps the system objectively evaluate the advantages and disadvantages of different skill models, and provides data support for the selection and optimization of skill models.

[0057] Furthermore, the knowledge management module includes: designing a structured knowledge base schema for storing and associating information entities, where the information entities include processes, operators, operation records, skill primitive types, skill primitive instances, key parameters, operation intentions, common skill models, personalized skill models, model evaluation results, and structured process knowledge, and the specific content is shown in Table 1; defining the associations between the information entities through foreign keys; constructing a knowledge base management sub-module to support operations of adding, deleting, modifying, and querying the information entities, and support complex queries, where the complex queries include standard operation retrieval, operator difference comparison, key parameter statistical analysis, and intention-based process mining; establishing a knowledge update mechanism to automatically update the content of the knowledge base after new data processing is completed, and implementing version control for models and knowledge entries; forming a structured process knowledge base.

[0058] Table 1 Information Entities and Contents

[0059]

[0060] The standard operation search queries the "best practice" knowledge entries for specific process steps (associated with primitive types), and obtains recommended models, standard parameter ranges, and operation descriptions; the operator difference comparison queries the personalized model parameters and evaluation results of different technicians for the same skill primitive, analyzes the differences in operation styles and their impacts on the effects; the key parameter statistical analysis calculates the distribution of the key parameters of a certain skill primitive in all records, or the parameter stability when a specific technician executes this primitive; the intention-based process mining queries the skill primitive sequence patterns associated with specific operation intentions.

[0061] When new operation data is entered, the system automatically triggers the entire processing flow, and updates the newly extracted primitives, parameters, models, and evaluation results to the knowledge base. Implement version control for model files and important knowledge entries, allowing tracing of historical versions and comparison of the effects of different versions.

[0062] The structured knowledge base design and management system realizes the systematic storage and efficient query of process knowledge. Through entity association and version control, the knowledge base can be dynamically updated and maintain data consistency, supporting complex knowledge retrieval and analysis functions, and providing a strong informatization support for process standardization and knowledge inheritance.

[0063] The production knowledge automatic extraction and management system for process standardization proposed by the present invention. First, collecting and processing multi-modal data ensures the quality of data from different sources, providing high-quality basic data for subsequent analysis; second, the skill primitive recognition method based on the temporal convolutional network and the hidden semi-Markov model greatly improves the accuracy of operation segmentation, and can accurately extract skill primitive segments with specific parameter characteristics from continuous operation flows; third, the intention recognition method based on the recurrent neural network and the multi-level attention mechanism realizes the inference from low-level skill primitives to high-level operation intentions, enhancing the system's understanding ability of operation purposes; the skill modeling method with the conditional variational autoencoder architecture realizes the balance between generality and individuality, ensuring both the requirements of process standardization and retaining valuable personalized skills of operators; at the same time, the multi-dimensional evaluation index provides an objective and comprehensive evaluation means for the skill model; finally, the design of the structured knowledge base realizes the systematic management of process knowledge, supports complex queries and automatic updates, and provides an informatization foundation for knowledge inheritance and application. The present invention not only solves the problems of skill recognition under multi-modal data and the handling of different technicians' operation styles, but also establishes a set of transformation systems from data to knowledge, making the process standardization process scientific and efficient, and providing technical support for enterprises to improve production quality, efficiency, and knowledge management level.

[0064] Embodiment 2:

[0065] This embodiment describes the specific implementation steps of the production knowledge automatic extraction and management system for process standardization. Refer to Figure 3 , including:

[0066] Step 1: Collect multi-modal time-series data during the process operation of the operator, preprocess and extract features from the multi-modal time-series data to generate a multi-modal feature sequence;

[0067] Furthermore, Step 1 specifically includes: collecting attitude data, force sensor data, visual data, and acoustic data and performing time synchronization processing to form multi-modal time-series data; preprocessing the multi-modal time-series data, including denoising, filtering, calibration, and standardization; for attitude data and force sensor data, using the sliding window method to calculate statistical features, frequency-domain features, and extract time-frequency features at different scales by combining wavelet transform within each time window to generate a force feature vector and an attitude feature vector; for visual data, using a pre-trained deep learning model to extract a visual feature vector; for acoustic data, also using the sliding window to extract Mel-frequency cepstral coefficients, spectral centroid, spectral entropy, and short-time energy to generate an acoustic feature vector; combining the features extracted from all modalities at each time step into a multi-modal feature vector to generate a multi-modal feature sequence.

[0068] In this embodiment, the following sensor devices are deployed to collect multi-modal data: force sensors are installed at the end of the operation tool; attitude sensors use wireless IMU (Inertial Measurement Unit) sensors, which are worn on the operator's wrists and the tool; visual sensors use two high-definition cameras, one fixed directly above the operation table and one fixed at a 45° angle on the side of the operation table; acoustic sensors use a four-channel microphone array; all sensor data is time-synchronized through a central processing unit, and the synchronization error is controlled within ±1ms.

[0069] Step 2: Based on the multi-modal feature sequence, segment the process operation process into a skill primitive sequence and obtain the key parameter vector of each skill primitive, and identify the operation intention based on the skill primitive sequence and the key parameter vector;

[0070] Further, splitting the process operation into a sequence of skill primitives and obtaining the key parameter vector of each skill primitive includes: inputting the multi-modal feature sequence into a temporal convolutional network to extract a high-level feature representation sequence; using the high-level feature representation sequence as the observation sequence of a hidden semi-Markov model, where the hidden semi-Markov model defines a set of hidden states, each of which corresponds to a skill primitive, introducing a duration distribution for each hidden state, and observing the emission probability and state transition probability; based on the observation sequence, duration distribution, emission probability, and state transition probability, using the Viterbi algorithm to find the optimal state sequence and the corresponding start and end timestamps, obtaining the skill primitive sequence, which includes the type, start time, and end time of each skill primitive; obtaining the multi-modal feature subsequence corresponding to each skill primitive based on the multi-modal feature sequence, and applying a signal processing algorithm to obtain the key parameter vector of each skill primitive.

[0071] This embodiment is mainly applied to the gyroscope precision assembly process of a certain instrument manufacturing enterprise. The gyroscope assembly involves many complex manual operation steps, including component positioning, precision debugging, fastening and tightening, etc., with extremely high requirements for operation skills and precision. There are many senior operators within the enterprise. The system collects multi-modal time-series data of multiple senior operators during the gyroscope assembly process, identifies skill primitives, constructs common and personalized skill models, and establishes a structured knowledge base, thereby realizing the extraction, preservation, optimization, and inheritance of process knowledge. Table 2 shows 8 typical skill primitives identified during the gyroscope assembly process, with an overall segmentation accuracy rate of 93% and an F1 score of 92.5%, indicating that the system has a high skill primitive segmentation ability.

[0072] Table 2 Skill Primitive Segmentation Results

[0073]

[0074] Further, identifying the operation intention based on the skill primitive sequence and key parameter features includes: using a first recurrent neural network to process the multi-modal feature subsequence corresponding to each skill primitive to obtain a first hidden feature sequence, applying a first attention mechanism within the first hidden feature sequence to generate a primitive representation vector; concatenating the primitive representation vector with the corresponding key parameter vector to obtain a comprehensive representation of the skill primitive; inputting all the comprehensive representations of the skill primitives into a second recurrent neural network to obtain a second hidden feature sequence; applying a second attention mechanism within the second hidden feature sequence to generate an operation representation vector; processing the operation representation vector through a fully connected layer and a Softmax activation function to obtain the operation intention.

[0075] Step 3: Construct a common skill model and a personalized skill model for the process operation based on the skill primitive sequence;

[0076] Furthermore, constructing the common skill model and personalized skill model for the process operation process includes:

[0077] For each type of skill primitive, construct a common skill model based on conditional variational autoencoder, including: defining a conditional encoder to map the multi-modal feature subsequence of the skill primitive and the skill primitive type to the latent variable distribution; defining a conditional decoder to reconstruct the multi-modal feature subsequence according to the latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the evidence lower bound; taking the trained conditional decoder as the common skill model;

[0078] Based on the common skill model, construct a personalized skill model, including: using the conditional encoder of the common skill model to obtain the sequence of latent vectors corresponding to a specific operator's execution of each type of skill primitive; calculating the empirical mean and covariance matrix of the sequence of latent vectors as the personalized latent distribution of the specific operator on each type of skill primitive; the personalized skill model is jointly defined by the personalized latent distribution and the conditional decoder of the common skill model.

[0079] Step Four: Define multi-dimensional evaluation indicators to evaluate the common skill model and the personalized skill model;

[0080] Furthermore, Step Four specifically includes: defining multi-dimensional evaluation indicators including efficiency, quality, consistency, resource consumption, and safety; generating reconstructed multi-modal feature sequences based on the common skill model and the personalized skill model; calculating the values of the multi-dimensional evaluation indicators by performing the operations corresponding to the reconstructed multi-modal feature sequences through physical simulation; normalizing the values of the multi-dimensional evaluation indicators and calculating the comprehensive evaluation scores of each model according to the preset weights; generating a model evaluation report including the values of each indicator and the comprehensive evaluation scores.

[0081] Step Five: Based on the skill primitive sequence, the key parameter vector, the operation intention, the common skill model, the personalized skill model, and the operation effect evaluation results, construct a dynamically updated, structured, and complex queryable process knowledge base;

[0082] Further, step five specifically includes: designing a structured knowledge base schema for storing and associating information entities, where the information entities include processes, operators, operation records, skill primitive types, skill primitive instances, key parameters, operation intentions, common skill models, personalized skill models, model evaluation results, and structured process knowledge; defining the associations between the information entities through foreign keys; supporting operations of adding, deleting, modifying, and querying the information entities, and supporting complex queries, where the complex queries include standard operation retrieval, operator difference comparison, key parameter statistical analysis, and intention-based process mining; establishing a knowledge update mechanism to automatically update the content of the knowledge base after new data processing is completed, and implementing version control for models and knowledge entries; and forming a structured process knowledge base.

[0083] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An automated extraction and management system for production knowledge oriented to process standardization, characterized in that Including: A data acquisition and processing module that acquires multi-modal time-series data during the process operation performed by an operator, preprocesses and extracts features from the multi-modal time-series data, and generates a multi-modal feature sequence; A skill recognition module that, based on the multi-modal feature sequence, segments the process operation into a skill primitive sequence and obtains the key parameter vectors of each skill primitive, and recognizes the operation intention based on the skill primitive sequence and the key parameter vectors; A skill modeling module that constructs a common skill model and a personalized skill model for the process operation based on the skill primitive sequence; A model evaluation module that defines multi-dimensional evaluation indicators and evaluates the common skill model and the personalized skill model; A knowledge management module that constructs a structured process knowledge base based on the skill primitive sequence, the key parameter vectors, the operation intention, the common skill model, the personalized skill model, and the model evaluation results, and the structured process knowledge base supports dynamic update and complex query.

2. The production knowledge automatic extraction and management system for process standardization according to claim 1, wherein The data acquisition and processing module includes: acquiring attitude data, force sensor data, visual data, and acoustic data and performing time synchronization processing to form multi-modal time-series data; preprocessing the multi-modal time-series data, including denoising, filtering, calibration, and standardization; for attitude data and force sensor data, using a sliding window method to calculate statistical features, frequency domain features, and time-frequency features at different scales by combining wavelet transform in each time window, and generating a force feature vector and an attitude feature vector; for visual data, using a pre-trained deep learning model to extract visual feature vectors; for acoustic data, also using a sliding window to extract Mel frequency cepstral coefficients, spectral centroid, spectral entropy, and short-time energy, and generating an acoustic feature vector; combining the features extracted from all modalities at each time step into a multi-modal feature vector to generate a multi-modal feature sequence.

3. The production knowledge automatic extraction and management system for process standardization according to claim 1, characterized in that Segmenting the process operation into a skill primitive sequence and obtaining the key parameter vectors of each skill primitive includes: inputting the multi-modal feature sequence into a temporal convolutional network to extract a high-level feature representation sequence; using the high-level feature representation sequence as the observation sequence of a hidden semi-Markov model, where the hidden semi-Markov model defines a set of hidden states, each of which corresponds to a skill primitive, introducing a duration distribution for each hidden state, and observing the emission probability and state transition probability; based on the observation sequence, duration distribution, emission probability, and state transition probability, using the Viterbi algorithm to find the optimal state sequence and the corresponding start and end timestamps, obtaining the skill primitive sequence, which includes the type, start time, and end time of each skill primitive; obtaining the multi-modal feature subsequence corresponding to each skill primitive based on the multi-modal feature sequence, and applying a signal processing algorithm to obtain the key parameter vector of each skill primitive.

4. The production knowledge automatic extraction and management system for process standardization according to claim 1, characterized in that Identifying the operation intention based on the skill primitive sequence and key parameter features includes: processing the multi-modal feature subsequence corresponding to each skill primitive using a first recurrent neural network to obtain a first hidden feature sequence, applying a first attention mechanism within the first hidden feature sequence to generate a primitive representation vector; concatenating the primitive representation vector with the corresponding key parameter vector to obtain a comprehensive representation of the skill primitive; inputting the comprehensive representations of all skill primitives into a second recurrent neural network to obtain a second hidden feature sequence; applying a second attention mechanism within the second hidden feature sequence to generate an operation representation vector; and processing the operation representation vector through a fully connected layer and a Softmax activation function to obtain the operation intention.

5. The production knowledge automated extraction and management system for process standardization according to claim 1, characterized in that Constructing the common skill model and personalized skill model for the process operation includes: For each type of skill primitive, constructing a common skill model based on a conditional variational autoencoder, including: defining a conditional encoder to map the multi-modal feature subsequence of the skill primitive and the skill primitive type to a latent variable distribution; defining a conditional decoder to reconstruct the multi-modal feature subsequence according to the latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the evidence lower bound; and using the trained conditional decoder as the common skill model; Based on the common skill model, constructing a personalized skill model, including: using the conditional encoder of the common skill model to obtain the sequence of latent vectors corresponding to each type of skill primitive executed by a specific operator; calculating the empirical mean and covariance matrix of the sequence of latent vectors as the personalized latent distribution of the specific operator for each type of skill primitive; and jointly defining the personalized skill model by the personalized latent distribution and the conditional decoder of the common skill model.

6. The production knowledge automatic extraction and management system for process standardization according to claim 1, characterized in that, The model evaluation module includes: defining multi-dimensional evaluation indicators including efficiency, quality, consistency, resource consumption, and safety; generating a reconstructed multi-modal feature sequence based on the common skill model and the personalized skill model; performing the operations corresponding to the reconstructed multi-modal feature sequence through physical simulation and calculating the values of the multi-dimensional evaluation indicators; normalizing the values of the multi-dimensional evaluation indicators and calculating the comprehensive evaluation score of each model according to the preset weights; and generating a model evaluation report including the values of each indicator and the comprehensive evaluation score.

7. The production knowledge automatic extraction and management system for process standardization according to claim 1, characterized in that The knowledge management module includes: designing a structured knowledge base schema for storing and associating information entities, where the information entities include processes, operators, operation records, skill primitive types, skill primitive instances, key parameters, operation intentions, common skill models, personalized skill models, model evaluation results, and structured process knowledge; defining the associations between the information entities through foreign keys; constructing a knowledge base management sub-module to support operations of adding, deleting, modifying, and querying the information entities, and support complex queries, where the complex queries include standard operation retrieval, operator difference comparison, key parameter statistical analysis, and intention-based process mining; establishing a knowledge update mechanism to automatically update the content of the knowledge base after new data processing is completed, and implementing version control for models and knowledge entries; and forming a structured process knowledge base.

Images