Production knowledge automatic extraction and management system oriented to process standardization

Abstract

The application relates to the technical field of electric digital data processing, in particular to a production knowledge automatic extraction and management system for process standardization, which comprises the following modules: a data acquisition and processing module, which acquires multi-modal time sequence data, carries out pretreatment and feature extraction, and generates multi-modal feature sequences; a skill identification module, which splits skill primitive sequences and acquires key parameter vectors of each skill primitive based on the multi-modal feature sequences, and identifies operation intentions based on the skill primitive sequences and the key parameter vectors; a skill modeling module, which constructs common skill models and individualized skill models of process operation processes based on the skill primitive sequences; a model evaluation module, which defines multi-dimensional evaluation indexes and evaluates operation effects corresponding to the common and individualized skill models; and a knowledge management module, which constructs a structured process knowledge base supporting dynamic updating and complex queries. The application establishes a conversion system from data to knowledge, and realizes a scientific and efficient process standardization process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, in particular to a production knowledge automatic extraction and management system for process standardization. BACKGROUND

[0002] With the development of modern manufacturing industry towards intelligence and lean, higher requirements are put forward for process standardization of production process. Process standardization is the key link to ensure product quality, improve production efficiency, reduce manufacturing cost and realize knowledge inheritance. Under this background, using information technology to effectively manage and utilize the massive data generated in the production process has become an important means to promote process standardization. Basic data management technology, such as database technology for efficient organization of structured data and file organization system for management of unstructured information, lays a foundation for storing and preliminary accessing massive production data.

[0003] Further, in order to more effectively serve the process standardization, it is far from enough to only store the original data, and it is necessary to transform the data into understandable and reusable knowledge. In this regard, advanced information management methods, especially the technologies related to knowledge representation, structured storage and intelligent retrieval, play a core role. Some principles of these technologies can also be found in large-scale data processing applications, such as the information retrieval and organization mechanism relied on by Internet search services. By establishing a production knowledge base and using a semantic data model, explicit knowledge such as refined process rules, operation steps and causal relationships can be represented and stored in a structured manner. This enables process engineers to conveniently find, compare and revise standard operating procedures by using mature query and retrieval mechanisms, and to a certain extent, to support process design and optimization decision-making. Existing manufacturing execution systems, product lifecycle management systems and other systems apply these principles of structured knowledge representation and retrieval to manage explicit process knowledge, support daily production operation and standardized processes.

[0004] However, the existing technology still has significant deficiencies in the automatic extraction and management of knowledge for process standardization. First, it is difficult to effectively segment continuous multi-modal data streams and identify the corresponding operation intent. Second, it is difficult to distinguish and construct models to express common core operation patterns and individualized skills.

[0005] Therefore, a production knowledge automatic extraction and management system for process standardization is proposed. SUMMARY

[0006] The application aims to provide a production knowledge automatic extraction and management system for process standardization, realize a scientific and efficient process standardization process by establishing a data-to-knowledge conversion system, comprising: a data acquisition and processing module, which acquires multi-modal time series data, performs preprocessing and feature extraction, and generates multi-modal feature sequences; a skill recognition module, which, based on the multi-modal feature sequences, segments skill primitive sequences and obtains key parameter vectors of each skill primitive, and identifies operation intention based on the skill primitive sequences and the key parameter vectors; a skill modeling module, which constructs common skill models and individualized skill models of process operation processes based on the skill primitive sequences; a model evaluation module, which defines multi-dimensional evaluation indexes and evaluates operation effects corresponding to the common and individualized skill models; and a knowledge management module, which constructs a structured process knowledge base supporting dynamic updating and complex queries.

[0007] To achieve the above-mentioned purpose, the application provides the following technical scheme:

[0008] The production knowledge automatic extraction and management system for process standardization comprises:

[0009] A data acquisition and processing module, which acquires multi-modal time series data in the process of operators performing process operation processes, performs preprocessing and feature extraction on the multi-modal time series data, and generates multi-modal feature sequences;

[0010] A skill recognition module, which, based on the multi-modal feature sequences, segments the process operation processes into skill primitive sequences and obtains key parameter vectors of each skill primitive, and identifies operation intention based on the skill primitive sequences and the key parameter vectors;

[0011] A skill modeling module, which constructs common skill models and individualized skill models of process operation processes based on the skill primitive sequences;

[0012] A model evaluation module, which defines multi-dimensional evaluation indexes and evaluates the common and individualized skill models;

[0013] A knowledge management module, which, based on the skill primitive sequences, the key parameter vectors, the operation intention, the common skill models, the individualized skill models and operation effect evaluation results, constructs a dynamic updating, structured and complex query process knowledge base.

[0014] Further, the data acquisition processing module comprises: acquiring posture data, force sensor data, visual data and acoustic data and performing time synchronization processing to form multi-modal time series data; pre-processing the multi-modal time series data, including denoising, filtering, calibration and standardization; for the posture data and the force sensor data, a sliding window method is adopted, and in each time window, statistical features, frequency domain features and time-frequency features at different scales extracted by combining wavelet transform are calculated to generate force feature vectors and posture feature vectors; for the visual data, a pre-trained deep learning model is used to extract visual feature vectors; for the acoustic data, a sliding window is also used to extract mel-frequency cepstrum coefficients, spectral centroid, spectral entropy and short-time energy to generate acoustic feature vectors; all modal features extracted at each time step are combined into multi-modal feature vectors to generate a multi-modal feature sequence.

[0015] Further, the process of dividing the process operation into a sequence of skill primitives and obtaining a key parameter vector of each skill primitive comprises: inputting the multi-modal feature sequence into a time convolution network to extract a high-level feature representation sequence; taking the high-level feature representation sequence as an observation sequence of a hidden semi-Markov model, the hidden semi-Markov model defining a set of hidden states, each of which corresponds to a skill primitive, introducing a duration distribution for each hidden state, and observing emission probability and state transition probability; based on the observation sequence, the duration distribution, the emission probability and the state transition probability, the Viterbi algorithm is used to find the optimal state sequence and the corresponding start and end time stamps to obtain the sequence of skill primitives, which contains the type, start time and end time of each skill primitive; based on the multi-modal feature sequence, a multi-modal feature sub-sequence corresponding to each skill primitive is obtained, and a signal processing algorithm is applied to obtain a key parameter vector of each skill primitive.

[0016] Further, identifying the operation intention based on the sequence of skill primitives and the key parameter features comprises: using a first recurrent neural network to process the multi-modal feature sub-sequence corresponding to each skill primitive to obtain a first hidden feature sequence, applying a first attention mechanism inside the first hidden feature sequence to generate a primitive representation vector; concatenating the primitive representation vector with the corresponding key parameter vector to obtain a skill primitive comprehensive representation; inputting all skill primitive comprehensive representations into a second recurrent neural network to obtain a second hidden feature sequence; applying a second attention mechanism inside the second hidden feature sequence to generate an operation representation vector; processing the operation representation vector through a fully connected layer and a Softmax activation function to obtain the operation intention.

[0017] Further, constructing the common skill model and the individualized skill model of the process operation process comprises:

[0018] For each skill primitive type, a common skill model based on a conditional variational autoencoder is constructed, including: defining a conditional encoder, mapping the multi-modal feature subsequence of the skill primitive and the skill primitive type to a latent variable distribution; defining a conditional decoder, reconstructing the multi-modal feature subsequence according to the latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the lower bound of the evidence; and taking the trained conditional decoder as the common skill model;

[0019] Based on the common skill model, a personalized skill model is constructed, including: obtaining the latent vector sequence corresponding to each skill primitive type performed by a specific operator by using the conditional encoder of the common skill model; calculating the empirical mean and covariance matrix of the latent vector sequence as the personalized latent distribution of the specific operator on each skill primitive type; and defining the personalized skill model by the personalized latent distribution and the conditional decoder of the common skill model.

[0020] Further, the model evaluation module includes: defining multi-dimensional evaluation indexes including efficiency, quality, consistency, resource consumption and safety; generating a reconstructed multi-modal feature sequence based on the common skill model and the personalized skill model; calculating the values of the multi-dimensional evaluation indexes by performing the operations corresponding to the reconstructed multi-modal feature sequence through physical simulation; normalizing the values of the multi-dimensional evaluation indexes and calculating the comprehensive evaluation score of each model according to a preset weight; and generating a model evaluation report containing the values of each index and the comprehensive evaluation score.

[0021] Further, the knowledge management module includes: designing a structured knowledge base mode for storing and associating information entities, the information entities including processes, operators, operation records, skill primitive types, skill primitive instances, key parameters, operation intentions, common skill models, personalized skill models, model evaluation results and structured process knowledge; defining the association between the information entities through foreign keys; constructing a knowledge base management submodule to support the add, delete, modify and query operations of the information entities, and support complex queries including standard operation retrieval, operator difference comparison, key parameter statistical analysis and intention-based process mining; establishing a knowledge updating mechanism to automatically update the knowledge base content after new data processing is completed, and implementing version control on the models and knowledge entries; and forming a structured process knowledge base.

[0022] Compared with the prior art, the present application has the following beneficial effects:

[0023] 1. Multimodal data is preprocessed and feature extracted to generate a multimodal feature sequence. Then, through a combination of a temporal convolutional network and a hidden semi-Markov model, the system accurately segments the continuous operation process into a meaningful sequence of skill primitives and extracts the key parameters of each primitive. Based on a recurrent neural network and a multi-level attention mechanism, the system can identify the operator's high-level operational intent from the skill primitive sequence. The processing of multimodal data significantly improves the accuracy and robustness of skill primitive identification. The system can efficiently and accurately identify the operator's skill primitives and operational intent, providing reliable data support for subsequent skill modeling and knowledge management, and significantly enhancing the ability to automatically extract production knowledge for process standardization.

[0024] 2. A skill modeling method that takes into account both commonality and individuality. Based on the conditional variational autoencoder architecture, it maps the multimodal feature subsequences and types of skill primitives to the latent variable space, and then reconstructs the feature sequence from the latent variables, thereby achieving a deep understanding of the inherent structure of skill primitives. In the construction of the common skill model, the system uses the data of all operators to train the CVAE to capture the general performance of the skill primitives; while in the personalized skill model construction stage, the system retains the decoder structure of the common model, but calculates personalized latent variable distribution parameters for each operator, thereby retaining the unique skills of the specific operator. This method not only overcomes the limitation of traditional skill models that can only express standardized operations while ignoring individual differences, but also provides a solution that organically integrates commonality and individuality, effectively integrating the commonality and individuality of skills, maintaining the standardization of process operations while retaining the effective personalized skills of operators.

[0025] 3. A dynamically updated, structured process knowledge base has been established, enabling systematic management of process knowledge. The knowledge base boasts powerful and complex query capabilities, such as standard operation retrieval, operator difference comparison, and statistical analysis of key parameters. The knowledge base also features an automatic update mechanism, automatically updating relevant knowledge entries when the system processes new operational data. Furthermore, version control ensures the traceability of knowledge evolution. This knowledge base design addresses issues inherent in traditional process knowledge management, such as information silos, delayed knowledge updates, and limited query capabilities, significantly improving the efficiency and accuracy of knowledge management. It enables efficient management and dynamic updating of process knowledge, provides a foundation for process standardization, and supports the preservation and analysis of the personalized skills of different technicians. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Fig. 1 This is a schematic diagram of the structure of the production knowledge automatic extraction and management system for process standardization of the present invention;

[0027] Fig. 2 This is a schematic diagram of the operation intention identification process of the present invention;

[0028] Fig. 3 A process standardization-oriented production knowledge automatic extraction and management system implementation step flowchart. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0030] Referring to Figs. 1 to 3 The present application provides a process standardization-oriented production knowledge automatic extraction and management system, and the technical solutions are as follows.

[0031] Embodiment one:

[0032] The process standardization-oriented production knowledge automatic extraction and management system, referring to Fig. 1 , comprises:

[0033] A data acquisition and processing module acquires multi-modal time series data in a process operation process performed by an operator, pre-processes and extracts features from the multi-modal time series data, and generates a multi-modal feature sequence.

[0034] A skill recognition module divides the process operation process into a skill primitive sequence and acquires a key parameter vector of each skill primitive based on the multi-modal feature sequence, and recognizes an operation intention based on the skill primitive sequence and the key parameter vector.

[0035] A skill modeling module constructs a common skill model and an individualized skill model of the process operation process based on the skill primitive sequence.

[0036] A model evaluation module defines multi-dimensional evaluation indexes and evaluates the common skill model and the individualized skill model.

[0037] A knowledge management module constructs a dynamic updating, structured, and complex queryable process knowledge base based on the skill primitive sequence, the key parameter vector, the operation intention, the common skill model, the individualized skill model, and an operation effect evaluation result.

[0038] Furthermore, the data acquisition and processing module includes: collecting posture data, force sensor data, visual data and acoustic data and performing time synchronization processing to form multimodal time series data; preprocessing the multimodal time series data, including denoising, filtering, calibration and standardization; for posture data and force sensor data, a sliding window method is used to calculate statistical features and frequency domain features in each time window and combine wavelet transform to extract time-frequency features at different scales to generate force feature vectors and posture feature vectors; for visual data, a pre-trained deep learning model is used to extract visual feature vectors; for acoustic data, a sliding window is also used to extract Mel-frequency cepstral coefficients, spectral centroid, spectral entropy and short-time energy to generate acoustic feature vectors; the features extracted from all modes at each time step are combined into a multimodal feature vector to generate a multimodal feature sequence.

[0039] Through the data acquisition and processing module's multimodal data synchronization and refined feature extraction methods, the system can effectively extract valuable information from multi-source data, ensuring data quality and system response speed. This improves data integrity and accuracy, thereby better supporting subsequent skill identification and modeling processes and providing a high-quality data foundation for process standardization.

[0040] Furthermore, dividing the process into skill primitive sequences and obtaining a key parameter vector of each skill primitive comprises:

[0041] The multimodal feature sequence is input into a temporal convolutional network to extract a high-level feature representation sequence; the temporal convolutional network learns a hierarchical temporal representation of the multimodal feature sequence through multiple layers of causal convolution, dilated convolution, and residual connection to obtain a high-level feature representation sequence;

[0042] The high-level feature representation sequence is used as the observation sequence of the hidden semi-Markov model. The hidden semi-Markov model defines a set of hidden states, where each hidden state corresponds to a skill primitive. A duration distribution is introduced for each hidden state, for example, a parameterized probability distribution is used to describe the possible duration of each state, and the emission probability and state transition probability are observed; the emission probability P(h t:t+d-1 |q i ,d) indicates that in the i-th state q i Under the condition of duration d, the high-level feature representation subsequence h is observed t:t+d-1 The probability of t represents the starting time of the high-level feature representation subsequence; the state transition probability P(q j |q i ) represents the transition from the i-th state q i After the end, transfer to the jth state q j probability;

[0043] Based on the observation sequence, duration distribution, emission probability and state transition probability, the optimal state sequence and corresponding start and end time stamps are found by using the Viterbi algorithm, and the skill primitive sequence is obtained, which contains the type, start time and end time of each skill primitive; Based on the multi-modal feature sequence, the multi-modal feature sub-sequence corresponding to each skill primitive is obtained, and the key parameter vector of each skill primitive is obtained by applying signal processing algorithm;

[0044] The key parameter vectors of different types of skill primitives contain different key parameters, and a mapping relationship needs to be defined in advance to associate each type of skill primitive to the corresponding key parameter list, which includes which key parameters need to be calculated for each type of skill primitive and the specific method for calculating each key parameter; For example, for the "tightening" skill primitive, the key parameter vector includes: peak torque, time to reach peak torque, final tightening angle, average angular velocity and tightening duration; calculated by peak detection and statistical analysis.

[0045] By introducing the time convolution network and the hidden semi-Markov model, the system can accurately segment the process operation into a sequence of skill primitives and extract the key parameter vector. Improve the segmentation accuracy of the process, effectively solve the problem of fuzzy skill primitive boundary in continuous operation, so that the system can more accurately capture each core step in the operation process.

[0046] Further, referring to Fig. 2 , based on the skill primitive sequence and key parameter feature, the operation intention is identified, including: using a first recurrent neural network to process the multi-modal feature sub-sequence corresponding to each skill primitive to obtain a first hidden feature sequence; Apply the first attention mechanism inside the first hidden feature sequence to generate a primitive representation vector; Concatenate the primitive representation vector with the corresponding key parameter vector to obtain a skill primitive comprehensive representation; Input all skill primitive comprehensive representations into a second recurrent neural network to capture the sequential dependency relationship between skill primitives to obtain a second hidden feature sequence; Apply the second attention mechanism inside the second hidden feature sequence to generate an operation representation vector; Process the operation representation vector through a fully connected layer and a Softmax activation function to obtain an operation intention;

[0047] Further, the first hidden feature sequence refers to the contextual representation within the multi-modal feature sub-sequence of the skill primitive, each skill primitive corresponds to a first hidden feature sequence, and the application of the first attention mechanism within the first hidden feature sequence refers to assigning different first importance weights to the features at different time steps within the skill primitive, and the primitive representation vector is obtained by weighted summation; each second hidden feature in the second hidden feature sequence corresponds to a skill primitive, and the application of the second attention mechanism within the second hidden feature sequence refers to calculating the second importance weight of each second hidden feature of the skill primitive for identifying the overall operation intention, and the operation representation vector of the entire operation sequence is obtained by weighted summation.

[0048] By using the recurrent neural network and the multi-level attention mechanism, the system can identify the high-level operation intention from the sequence of skill primitives and their key parameters. This effectively improves the reasoning ability of the system for operation intention, enabling it to understand the purpose behind the operator's behavior and providing semantic-level support for the structured expression of process knowledge.

[0049] Further, constructing the common skill model and the individualized skill model of the process operation process includes:

[0050] For each skill primitive type, a common skill model based on a conditional variational autoencoder is constructed, including: defining a conditional encoder to map the multi-modal feature sub-sequence of the skill primitive and the skill primitive type to a latent variable distribution; defining a conditional decoder to reconstruct the multi-modal feature sub-sequence according to the latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the lower bound of evidence; the trained conditional decoder is used as the common skill model; the lower bound of evidence includes a reconstruction loss term and a KL divergence term, the reconstruction loss measures the ability of the decoder to reconstruct the input, commonly using mean square error or cross entropy, and the KL divergence constrains the posterior distribution of the latent variable to be close to a preset prior distribution;

[0051] Based on the common skill model, an individualized skill model is constructed, including: using the conditional encoder of the common skill model to obtain a latent vector sequence corresponding to each skill primitive type performed by a specific operator; calculating the empirical mean and covariance matrix of the latent vector sequence as the individualized latent distribution of the specific operator for each skill primitive type; the individualized skill model is defined by the individualized latent distribution and the conditional decoder of the common skill model;

[0052] The common skill model and the individualized skill model based on the conditional variational autoencoder achieve a balanced expression of skill commonality and individuality. This method can capture both the standard mode of process operation and the individualized skills of operators, providing an effective solution to the contradiction between process standardization and individualized skills.

[0053] Further, the model evaluation module comprises: defining multi-dimensional evaluation indexes including efficiency, quality, consistency, resource consumption and safety; generating reconstructed multi-modal feature sequences based on the common skill model and the personalized skill model; performing operations corresponding to the reconstructed multi-modal feature sequences through physical simulation, calculating values of the multi-dimensional evaluation indexes; normalizing the values of the multi-dimensional evaluation indexes, and calculating a comprehensive evaluation score of each model according to a preset weight; generating a model evaluation report containing the values of each index and the comprehensive evaluation score;

[0054] The multi-dimensional evaluation indexes include: efficiency: total time for completing a task, average duration of a single skill primitive, output per unit time; quality: pass rate of final product, key dimension accuracy, surface roughness; consistency: standard deviation of key parameters corresponding to the skill primitive when the same task is repeatedly performed; resource consumption: energy consumption, material loss rate; safety: maximum instantaneous force during operation;

[0055] Generating the reconstructed multi-modal feature sequences based on the common skill model and the personalized skill model means that, according to the process standard flow sequence, the common skill model outputs reconstructed multi-modal feature subsequences of all skill primitives, and the common reconstructed multi-modal feature sequence is obtained after merging; then, based on the common reconstructed multi-modal feature sequence, different personalized reconstructed multi-modal feature sequences are output through multiple personalized skill models.

[0056] The multi-dimensional evaluation indexes are used to comprehensively evaluate the effect of the skill model, providing a quantitative model evaluation standard, helping the system to objectively evaluate the advantages and disadvantages of different skill models, and providing data support for the selection and optimization of the skill model.

[0057] Further, the knowledge management module comprises: designing a structured knowledge base mode for storing and associating information entities, the information entities including process, operator, operation record, skill primitive type, skill primitive instance, key parameter, operation intention, common skill model, personalized skill model, model evaluation result and structured process knowledge, and the specific content is shown in Table 1; defining the association between the information entities through foreign keys; constructing a knowledge base management submodule to support the add, delete, modify and query operations of the information entities, and supporting complex queries including standard operation retrieval, operator difference comparison, key parameter statistical analysis and intention-based flow mining; establishing a knowledge updating mechanism to automatically update the knowledge base content after new data processing is completed, and implementing version control on the model and knowledge entries; forming a structured process knowledge base.

[0058] Table 1 Information entities and contents

[0059]

[0060] Standard operation retrieval query the "best practice" knowledge items of specific process steps (associated to primitive types), get recommended models, standard parameter ranges and operation descriptions; Operator difference comparison query the personalized model parameters and evaluation results of the same skill primitive by different technicians, analyze the operation style differences and their impact on the results; Key parameter statistical analysis calculates the distribution of key parameters of a certain skill primitive in all records, or the parameter stability when a specific technician performs the primitive; Intention-based process mining queries the sequence patterns of skill primitives associated with a specific operation intention.

[0061] When new operation data is entered, the system automatically triggers all processing flows and updates the newly extracted primitives, parameters, models and evaluation results to the knowledge base. Version control is implemented for model files and important knowledge items, allowing to trace back to historical versions and compare the effects of different versions.

[0062] The structured knowledge base design and management system realizes the systematic storage and efficient query of process knowledge. Through entity association and version control, the knowledge base can dynamically update and maintain data consistency, support complex knowledge retrieval and analysis functions, and provide strong information support for process standardization and knowledge inheritance.

[0063] The production knowledge automatic extraction and management system for process standardization proposed in the application first collects and processes multi-modal data to ensure the quality of data from different sources, providing high-quality basic data for subsequent analysis. Second, the skill primitive recognition method based on time convolution network and hidden semi-Markov model greatly improves the accuracy of operation segmentation, and can accurately extract skill primitive fragments with specific parameter characteristics from continuous operation flow. Third, the intention recognition method based on recurrent neural network and multi-level attention mechanism realizes reasoning from low-level skill primitives to high-level operation intentions, enhancing the system's understanding ability of operation purpose. The skill modeling method based on conditional variational autoencoder architecture balances commonality and individuality, ensuring process standardization requirements while retaining valuable personalized skills of operators. At the same time, multi-dimensional evaluation indicators provide objective and comprehensive evaluation means for skill models. Finally, the design of structured knowledge base realizes the systematic management of process knowledge, supports complex queries and automatic updates, and provides an information-based foundation for knowledge inheritance and application. The application not only solves the problem of skill recognition under multi-modal data and the processing of different technician operation styles, but also establishes a data-to-knowledge conversion system, making the process of process standardization scientific and efficient, and providing technical support for enterprises to improve production quality, efficiency and knowledge management level.

[0064] Embodiment two:

[0065] The embodiment describes the specific implementation steps of the production knowledge automatic extraction and management system for process standardization, see Fig. 3 , including:

[0066] Step one, collect multi-modal time series data during the operator's execution of the process operation, pre-process and extract features from the multi-modal time series data, and generate multi-modal feature sequences;

[0067] Further, step one specifically includes: collecting posture data, force sensor data, visual data and acoustic data and performing time synchronization processing to form multi-modal time series data; pre-processing the multi-modal time series data, including denoising, filtering, calibration and standardization; for posture data and force sensor data, using a sliding window method, calculating statistical features, frequency domain features and time-frequency features at different scales combined with wavelet transform within each time window to generate force feature vectors and posture feature vectors; for visual data, use a pre-trained deep learning model to extract visual feature vectors; for acoustic data, also use a sliding window to extract mel-frequency cepstral coefficients, spectral centroid, spectral entropy and short-time energy to generate acoustic feature vectors; combine all modal features extracted at each time step into multi-modal feature vectors to generate multi-modal feature sequences.

[0068] In this embodiment, the following sensor devices are deployed to collect multi-modal data: a force sensor is installed at the end of the operating tool; a posture sensor is a wireless IMU (Inertial Measurement Unit) sensor, worn on the operator's wrists and the tool; a visual sensor uses two high-definition cameras, one fixed directly above the operating table and one fixed at a 45° angle to the side of the operating table; an acoustic sensor uses a four-channel microphone array; all sensor data is time-synchronized through a central processing unit with a synchronization error controlled within ±1ms.

[0069] Step two, based on the multi-modal feature sequence, divide the process operation into a sequence of skill primitives and obtain a key parameter vector for each skill primitive, and identify the operation intent based on the sequence of skill primitives and the key parameter vector;

[0070] Further, the process operation is segmented into a sequence of skill primitives and a key parameter vector of each skill primitive is obtained by inputting the multi-modal feature sequence into a time convolution network to extract a sequence of high-level feature representations, taking the sequence of high-level feature representations as an observation sequence of a hidden semi-Markov model, defining a set of hidden states for the hidden semi-Markov model, each of which corresponds to a skill primitive, introducing a duration distribution for each hidden state, and observing emission probability and state transition probability, and using a Viterbi algorithm to find an optimal state sequence and corresponding start and end time stamps based on the observation sequence, the duration distribution, the emission probability and the state transition probability, to obtain the sequence of skill primitives, which includes the type, start time and end time of each skill primitive, and obtaining a multi-modal feature sub-sequence corresponding to each skill primitive based on the multi-modal feature sequence and applying a signal processing algorithm to obtain the key parameter vector of each skill primitive.

[0071] The embodiment is mainly applied to a gyro precision assembly process in an instrument manufacturing enterprise. Gyro assembly involves many complex manual operation steps, including component positioning, precision debugging, fastening and tightening, and has very high requirements on operation skills and precision. The enterprise has many experienced operators, and the system identifies skill primitives by collecting multi-modal time series data of the experienced operators during the gyro assembly process, constructs common and individual skill models, and establishes a structured knowledge base, thereby realizing extraction, preservation, optimization and inheritance of process knowledge. Table 2 shows 8 typical skill primitives identified in the gyro assembly process, with an overall segmentation accuracy of 93% and an F1 score of 92.5%, indicating that the system has high skill primitive segmentation capability.

[0072] Table 2 Skill primitive segmentation results

[0073]

[0074] Further, the operation intention is identified based on the sequence of skill primitives and the key parameter features by using a first recurrent neural network to process the multi-modal feature sub-sequence corresponding to each skill primitive to obtain a first hidden feature sequence, applying a first attention mechanism inside the first hidden feature sequence to generate a primitive representation vector, concatenating the primitive representation vector with the corresponding key parameter vector to obtain a skill primitive comprehensive representation, inputting all skill primitive comprehensive representations into a second recurrent neural network to obtain a second hidden feature sequence, applying a second attention mechanism inside the second hidden feature sequence to generate an operation representation vector, and processing the operation representation vector through a fully connected layer and a Softmax activation function to obtain the operation intention.

[0075] Step three, constructing a common skill model and an individual skill model of the process operation based on the sequence of skill primitives;

[0076] Further, the common skill model and the personalized skill model of the process operation process are constructed, including:

[0077] For each skill primitive type, a common skill model based on a conditional variational autoencoder is constructed, including: defining a conditional encoder to map the multi-modal feature subsequence of the skill primitive and the skill primitive type to a latent variable distribution; defining a conditional decoder to reconstruct the multi-modal feature subsequence according to a latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the lower bound of the evidence; and taking the trained conditional decoder as the common skill model.

[0078] Based on the common skill model, a personalized skill model is constructed, including: obtaining a latent vector sequence corresponding to each skill primitive type performed by a specific operator using the conditional encoder of the common skill model; calculating the empirical mean and covariance matrix of the latent vector sequence as the personalized latent distribution of the specific operator on each skill primitive type; and defining the personalized skill model by the personalized latent distribution and the conditional decoder of the common skill model.

[0079] Step four, defining multi-dimensional evaluation indexes to evaluate the common skill model and the personalized skill model;

[0080] Further, step four specifically includes: defining multi-dimensional evaluation indexes including efficiency, quality, consistency, resource consumption, and safety; generating reconstructed multi-modal feature sequences based on the common skill model and the personalized skill model; calculating the values of the multi-dimensional evaluation indexes by performing the operations corresponding to the reconstructed multi-modal feature sequences through physical simulation; normalizing the values of the multi-dimensional evaluation indexes and calculating the comprehensive evaluation scores of each model according to the preset weights; and generating a model evaluation report containing the values of each index and the comprehensive evaluation scores.

[0081] Step five, based on the skill primitive sequence, the key parameter vector, the operation intention, the common skill model, the personalized skill model, and the operation effect evaluation result, a dynamic, structured, and complex queryable process knowledge base is constructed.

[0082] Further, the step five specifically comprises: designing a structured knowledge base mode for storing and associating information entities, the information entities including process, operator, operation record, skill primitive type, skill primitive instance, key parameter, operation intention, common skill model, individualized skill model, model evaluation result and structured process knowledge; defining the association between the information entities through foreign key; supporting adding, deleting, modifying and inquiring operations on the information entities, and supporting complex queries including standard operation retrieval, operator difference comparison, key parameter statistical analysis and intention-based flow mining; establishing a knowledge updating mechanism to automatically update the knowledge base content after new data processing is completed, and implementing version control on the models and knowledge entries; and forming a structured process knowledge base.

[0083] While embodiments of the present application have been shown and described, it is to be understood that the embodiments described are merely divergences, modifications, replacements and variations of the embodiments, which can be made by those skilled in the art without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A production knowledge automation extraction and management system oriented to process standardization, characterized in that, The method comprises the following steps: a data acquisition and processing module acquires multi-modal time series data of an operator performing a process operation, pre-processes and extracts features from the multi-modal time series data, and generates a multi-modal feature sequence; a skill recognition module divides the process operation into a skill primitive sequence based on the multi-modal feature sequence and obtains a key parameter vector of each skill primitive, and identifies an operation intention based on the skill primitive sequence and the key parameter vector; a skill modeling module constructs a common skill model and an individualized skill model of the process operation based on the skill primitive sequence; constructing a common skill model and an individualized skill model of the process operation comprises: for each skill primitive type, a common skill model based on a conditional variational autoencoder is constructed, including: defining a conditional encoder to map the multi-modal feature subsequence of the skill primitive and the skill primitive type to a latent variable distribution; defining a conditional decoder to reconstruct the multi-modal feature subsequence according to a latent vector sampled from the latent variable distribution and the skill primitive type; training the conditional variational autoencoder by maximizing the lower bound of the evidence; and taking the trained conditional decoder as the common skill model; based on the common skill model, an individualized skill model is constructed, including: obtaining a latent vector sequence corresponding to each skill primitive type performed by a specific operator by using the conditional encoder of the common skill model; calculating the empirical mean and covariance matrix of the latent vector sequence as the individualized latent distribution of the specific operator in each skill primitive type; and the individualized skill model is defined by the individualized latent distribution and the conditional decoder of the common skill model; a model evaluation module defines multi-dimensional evaluation indexes to evaluate the common skill model and the individualized skill model; a knowledge management module constructs a structured process knowledge base based on the skill primitive sequence, the key parameter vector, the operation intention, the common skill model, the individualized skill model, and the model evaluation result, and the structured process knowledge base supports dynamic updating and complex querying.

2. The production knowledge automation extraction and management system oriented to process standardization according to claim 1, characterized in that, The data acquisition and processing module comprises: acquiring posture data, force sensor data, visual data, and acoustic data and performing time synchronization processing to form multi-modal time series data; pre-processing the multi-modal time series data, including denoising, filtering, calibration, and standardization; for the posture data and the force sensor data, a sliding window method is adopted to calculate statistical features, frequency domain features, and time-frequency features at different scales combined with wavelet transform in each time window to generate force feature vectors and posture feature vectors; for the visual data, a pre-trained deep learning model is used to extract visual feature vectors; for the acoustic data, a sliding window is also used to extract mel-frequency cepstral coefficients, spectral centroid, spectral entropy, and short-time energy to generate acoustic feature vectors; and all modalities are combined into multi-modal feature vectors at each time step to generate a multi-modal feature sequence.

3. The production knowledge automation extraction and management system oriented to process standardization according to claim 1, characterized in that, The process operation is divided into a sequence of skill primitives and key parameter vectors of each skill primitive are obtained, including: inputting the multi-modal feature sequence into a time convolution network to extract a high-level feature representation sequence; taking the high-level feature representation sequence as an observation sequence of a hidden semi-Markov model, the hidden semi-Markov model defining a set of hidden states, each hidden state corresponding to a skill primitive, introducing a duration distribution for each hidden state, and observing emission probability and state transition probability; based on the observation sequence, the duration distribution, the emission probability and the state transition probability, using the Viterbi algorithm to find the optimal state sequence and the corresponding start and end time stamps to obtain the sequence of skill primitives, which includes the type, start time and end time of each skill primitive; based on the multi-modal feature sequence, obtaining the multi-modal feature sub-sequence corresponding to each skill primitive, and applying a signal processing algorithm to obtain the key parameter vector of each skill primitive.

4. The production knowledge automation extraction and management system oriented to process standardization according to claim 1, characterized in that, Identifying the operation intention based on the sequence of skill primitives and the key parameter feature includes: using a first recurrent neural network to process the multi-modal feature sub-sequence corresponding to each skill primitive to obtain a first hidden feature sequence, applying a first attention mechanism inside the first hidden feature sequence to generate a primitive representation vector; concatenating the primitive representation vector with the corresponding key parameter vector to obtain a skill primitive comprehensive representation; inputting all skill primitive comprehensive representations into a second recurrent neural network to obtain a second hidden feature sequence; applying a second attention mechanism inside the second hidden feature sequence to generate an operation representation vector; processing the operation representation vector through a fully connected layer and a Softmax activation function to obtain an operation intention.

5. The production knowledge automation extraction and management system oriented to process standardization according to claim 1, characterized in that, The model evaluation module includes: defining multi-dimensional evaluation indexes including efficiency, quality, consistency, resource consumption and safety; generating a reconstructed multi-modal feature sequence based on the common skill model and the individualized skill model; performing the operation corresponding to the reconstructed multi-modal feature sequence through physical simulation, calculating the values of the multi-dimensional evaluation indexes; normalizing the values of the multi-dimensional evaluation indexes and calculating the comprehensive evaluation score of each model according to the preset weight; generating a model evaluation report containing the values of each index and the comprehensive evaluation score.

6. The production knowledge automation extraction and management system oriented to process standardization according to claim 1, characterized in that, The knowledge management module includes: designing a structured knowledge base mode for storing and associating information entities, the information entities including processes, operators, operation records, skill primitive types, skill primitive instances, key parameters, operation intentions, common skill models, individualized skill models, model evaluation results and structured process knowledge; defining the association between the information entities through foreign keys; constructing a knowledge base management submodule to support add, delete, modify and query operations on the information entities, and supporting complex queries including standard operation retrieval, operator difference comparison, key parameter statistical analysis and intention-based process mining; establishing a knowledge updating mechanism to automatically update the knowledge base content after new data processing is completed, and implementing version control on models and knowledge entries; forming a structured process knowledge base.

Images