Human-computer interaction design method and device based on intention recognition and knowledge pushing
By using an LSTM-based sequence design intent prediction method and a design knowledge base, the problem of CAD software being unable to understand design intent is solved, enabling efficient human-computer collaborative design and improving design efficiency.
Patent Information
- Application Number
- CN202511051044.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-29
- Publication Date
- 2025-11-07
AI Technical Summary
Existing CAD software cannot understand the designer's design intent, causing the designer to spend a lot of time on trial and error, and making it difficult to effectively transfer design experience and knowledge.
We employ an LSTM-based sequence design intent prediction method. We reduce the dimensionality and encode the design behavior data using the FBS model, train the design intent using the LSTM model, and combine it with a design knowledge base for knowledge delivery.
It improves the application effect of human-computer interaction, reduces the trial and error time of designers, makes full use of designers' knowledge and experience, and achieves more efficient human-computer collaborative design.
Smart Images
Figure CN120910932A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of natural language processing and human-computer interaction, and particularly relates to a human-computer interaction design method and device based on intent recognition and knowledge pushing. BACKGROUND
[0002] With the popularization of computer-aided design (CAD) software, designers can more efficiently complete complex design tasks. However, existing CAD software still has some limitations, for example: CAD software can usually only operate according to the instructions of the designer, and cannot understand the design intent and goal of the designer. This leads to the designer needing to spend a lot of time trying and error to find a satisfactory design scheme; the designer accumulates rich experience and knowledge in long-term design practice, but this knowledge and experience is often difficult to effectively pass to the CAD software, resulting in the CAD software being unable to fully utilize the knowledge and experience of the designer. In order to solve the above problems, the present application explores a sequence design intent prediction method based on LSTM, aiming to let the computer better understand the design intent of the designer, to improve its application effect in human-computer interaction. SUMMARY
[0003] The purpose of the present application is to provide a human-computer interaction design method and device based on intent recognition and knowledge pushing, to overcome or alleviate the known technical defects.
[0004] The first aspect of the present application provides a human-computer interaction design method based on intent recognition and knowledge pushing, comprising the following steps:
[0005] S1 data set collection:
[0006] Set design requirements and design constraints, and obtain continuous design behavior data of the designer through CAD modeling software according to the set design requirements and design constraints;
[0007] S2 data set processing:
[0008] The design behavior data is reduced in dimension by the FBS model, the design behavior space is mapped to the design stage space, and the reduced data is encoded as model training data;
[0009] S3 model training:
[0010] The model training data is input into the LSTM model for training, and the performance of the model on the training set and the validation set needs to be monitored during training to prevent overfitting, and the model performance is evaluated using test data after training is completed;
[0011] S4 model hyperparameter adjustment:
[0012] The institute sets different hyperparameters of the LSTM model in advance, and studies the influence of the hyperparameters on the prediction accuracy of the algorithm. The sensitivity of the model is analyzed by constantly changing the values of the hyperparameters, and the corresponding prediction accuracy is studied to select the optimal hyperparameters.
[0013] S5 knowledge pushing:
[0014] After the designer's intention is recognized by the LSTM model, the design knowledge is matched from the design knowledge base constructed from the design image data based on the predicted design stage, and the knowledge is pushed to the designer.
[0015] In some embodiments of the application, the S1 data set processing further comprises design behavior data extraction;
[0016] The design requirements and the design constraints are specifically: using SolidWorks software to perform three-dimensional modeling on the crank slider mechanism, and the design goal is to meet the slider stroke requirement while ensuring the normal operation of the crank slider mechanism.
[0017] The design behavior data extraction is specifically: in the design task of the crank slider mechanism, the designer is required to complete the design of an organization and then take a break, which avoids the interruption of the designer's design ideas due to too long intervals, and video recording is performed during the design process to facilitate more detailed mining of the design actions after the design is completed.
[0018] In some embodiments of the application, in the S2 data set processing, the data dimension reduction is specifically: the FBS design process model (FBS model has three types of typical ontology variables: function (F), behavior (B), and structure (S)) is used, and a coding scheme is established combining the operation characteristics of SolidWorks software to transcribe different types of design actions to three design process stages (including Formulation (F), Analysis (A), and Reformulation (R)) to realize design space dimension reduction.
[0019] The data coding is specifically: after obtaining a set of continuous design stage data after dimension reduction, the design data is encoded using one-hot encoding. One-hot encoding converts n observation values and m different variables into m binary variables of n observation values. Each observation corresponds to a position, and if it exists, it is 1, and if it does not exist, it is 0. In order to reduce the model prediction variables while using all design stages, a column of data is added to record the design stage when each design action is observed.
[0020] In some embodiments of the application, the S3 model training specifically includes: data training, model monitoring, data evaluation,
[0021] The data training is specifically: inputting the single-hot encoded data into the LSTM model for training, and since there are three design process stages in the present research, the result of the final intention recognition is formed based on the learning of the operation rules of the designers in a certain process stage.
[0022] The model monitoring is specifically: when training the LSTM model, several preset hyperparameters are set, such as the number of LSTM layers, the size of LSTM, the number of fully connected layers, the Dropout value, the learning rate, the Batch size, etc. The Dropout value is set to prevent the model from overfitting;
[0023] The data evaluation is specifically: evaluating the prediction accuracy, precision, recall rate and F1 score of the model for the Formulation (F), Analysis (A) and Reformulation (R) design stages (i.e., the design stages based on the FBS model) and the average.
[0024] In some embodiments of the present application, in the S4 model hyperparameter adjustment, the hyperparameter adjustment adopts a one-layer LSTM arrangement and uses different node numbers, and the number and size of fully connected layers and the Dropout value are set to 1, 7 and 0.3, respectively. The learning rate adopts the callback function ReduceLROnPlateau carried by the Keras deep learning library, so as to quickly and accurately obtain the optimal model.
[0025] In some embodiments of the present application, the S5 knowledge pushing includes intention recognition and knowledge pushing. The intention recognition is specifically: based on the encoding scheme of the design stage, the prediction stage / current stage plus the previous three design stages are used.
[0026] The knowledge pushing is specifically: after the intention recognition, the picture in the knowledge base with the smallest distance will be fed back to the designer.
[0027] The second aspect of the present application provides a human-computer interaction design device based on intention recognition and knowledge pushing, comprising:
[0028] A data set collection unit is configured to set design requirements and design constraints, and obtain continuous design behavior data of a designer through CAD modeling software according to the set design requirements and design constraints;
[0029] A data set processing unit is configured to reduce the dimension of the design behavior data through the FBS model, map the design behavior space to the design stage space, and encode the reduced data into model training data;
[0030] A model training unit inputs the model training data into an LSTM model for training, monitors the performance of the model on the training set and the validation set during the training process to prevent overfitting, and evaluates the performance of the model using test data after the training is completed;
[0031] A model hyperparameter adjustment unit studies the influence of different hyperparameters preset in the LSTM model on the prediction accuracy of the algorithm, performs sensitivity analysis on the model by constantly changing the values of the hyperparameters, and studies the corresponding prediction accuracy to select the optimal hyperparameters.
[0032] A knowledge pushing unit matches design knowledge from a design knowledge base constructed from design image data based on the predicted design stage and pushes the knowledge to the designer after identifying the designer's intention through the LSTM model. BRIEF DESCRIPTION OF DRAWINGS
[0033] Figure 1 is an interactive design method flowchart based on intention recognition and knowledge pushing;
[0034] Figure 2 is an FBS model coding scheme diagram;
[0035] Figure 3 is a single-hot coding example diagram of a design sequence;
[0036] Figure 4 is a specific structure diagram of an LSTM model. DETAILED DESCRIPTION
[0037] The technical solutions in the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described examples are only a part of the examples of the present application, but not all the examples. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. DETAILED EMBODIMENT ONE
[0039] The technical solution of the present application is: the interactive design method based on intention recognition and knowledge pushing, the specific process is as shown in Figure 1 The method comprises the following five steps:
[0040] Step one:
[0041] Data set collection: The data set is the basis for training the deep learning model. The main design requirements and design constraints are set. The continuous design behavior data of the designer is obtained according to the set design requirements and design constraints through the CAD modeling software.
[0042] Step two:
[0043] Dataset processing: The design behavior data is reduced dimension by FBS model, the design behavior space is mapped to design stage space, and the reduced dimension data is encoded as model training data.
[0044] Step three:
[0045] Model training: The reduced dimension data is input into the LSTM model for training, and the performance of the model on the training set and validation set needs to be monitored during training to prevent overfitting. After training, the model performance is evaluated using test data.
[0046] Step four:
[0047] Model hyperparameter adjustment: Study the influence of different hyperparameters preset in the LSTM model on the prediction accuracy of the algorithm, and perform sensitivity analysis on the model by constantly changing the values of these hyperparameters, and study the corresponding prediction accuracy to select the optimal hyperparameters.
[0048] Step five:
[0049] Knowledge push: After identifying the designer's intention by LSTM, match the design knowledge from the design knowledge base constructed based on the predicted design stage to the designer for knowledge push.
[0050] Further technical solutions of the application are:
[0051] Design condition determination: Select SolidWorks software to perform three-dimensional modeling of the crank slider mechanism. The goal of the design is to meet the slider stroke requirement while ensuring the normal operation of the crank slider mechanism. The application sets the main design requirements and design constraints, so that the designer can start with configuration and parameterization design, and the design variables are limited to only the components that affect the design target. At the same time, the number of design constraints and design requirements is limited to ensure that the entire design process is within a controllable range.
[0052] Design behavior data extraction: In the design task of the crank slider mechanism, the designer is required to complete the design of the mechanism and then take a break, avoiding the interruption of the designer's design ideas due to too long intervals. At the same time, video recording is performed during the design process to facilitate more detailed mining of design actions after the design is completed. In the action extraction process, the application excludes some actions that have no effect on the design process, such as selecting the appropriate design perspective. After removing these actions, about 65 actions can be extracted for each crank slider mechanism.
[0053] The designer's design activities are carried out in the CAD environment of SolidWorks, which is commonly used to represent the morphology of structures or parts. SolidWorks has rich modeling functions such as lofting, scanning, interference checking, etc. These functions can help designers effectively explore and develop the design space. In addition, SolidWorks records the designer's design operations in sequence and displays them on the left side of the design interaction interface. This means that the data recording process is unknown to the designer, which helps to reduce the impact of environmental factors.
[0054] A further technical solution of the present application is that the step two of preprocessing data mainly includes data dimension reduction, data encoding, etc.
[0055] Data dimension reduction: Because the design action sequence has a very high design space dimension, in order to better understand the designer's design process intention and strategy, this paper adopts the FBS design process model (FBS model has three typical ontology variables: function (F), behavior (B), structure (S)), and establishes an encoding scheme combined with the operation characteristics of SolidWorks software, as shown in Figure 2 The different types of design actions are transcribed into three design process stages (including Formulation (F), Analysis (A), and Reformulation (R)) to realize design space dimension reduction, which helps better understand the designer's design thinking. In the design task, adding any design elements such as adding sketch lines, adding reference base surfaces, adding assembly parts, etc. are Formulation, and the designer adds design elements to achieve the design goal. According to the definition of FBS model, Synthesis occurs when setting component parameters to meet the expected behavior. Therefore, editing any part and adding cooperation when assembling are identified as Synthesis. Reformulation occurs when the design elements are added or the structure is generated, and then adjusted due to not meeting the design requirements.
[0056] Data encoding: After obtaining a set of continuous design stage data after dimension reduction, the design data is encoded using one-hot encoding. One-hot encoding converts n observations and m different variables into m binary variables for n observations. Each observation corresponds to a position, which is 1, and 0 is not present. In order to reduce the model prediction variables while using all design stages, a column of data is added to record the design stage when each design action observation is observed. Figure 3 A one-hot encoding example of a design sequence is shown. In order to reduce the model prediction variables while using all design stages, the present application adds a column of data to record the design stage when each design action observation is observed.
[0057] The further technical solution of the present application is that the step three model training mainly includes data training, model monitoring, data evaluation, etc.
[0058] Data training: input the single-hot encoded data into the LSTM model for training. LSTM is a variant of RNN, which contains a special unit of memory block in the loop hidden layer. Each memory block has three gate structures, which are input gate, forget gate and output gate, and the specific structure is as shown in Figure 4 To calculate the hidden state h t of LSTM, the following formula is used as the form of f RNN defined in Eq. (1):
[0059] i t =σ(W i x t +U i h t-1 +b i )
[0060] f t =σ(W f x t +U f h t-1 +b f )
[0061] g t =tanh(W g x t +U g h t-1 +b g )
[0062] o t =σ(W o x t +U o h t-1 +b o )
[0063] c t =f t ×c t-1 +i t ×g t
[0064] h t =o t ×tanh(c t ) (3)
[0065] where x t represents the design operation input vector at time t, h t represents the hidden design state at time t, ht-1 denotes the hidden state of the previous time step, c t-1 denotes the cell state at time t, i t , f t , g t , o t denote the input, forget, cell, and output gates at time t, respectively, and denotes the sigmoid function, and W, U, b denote the model parameters.
[0066] Model monitoring: When training the LSTM model, there are several pre-set hyperparameters, such as the number of LSTM layers, LSTM size, number of fully connected layers, Dropout value, learning rate, Batch size, etc. The Dropout value is set to monitor the model to prevent overfitting of the model.
[0067] Data evaluation: The prediction accuracy, precision, recall, and F1 score of the model for the Formulation (F), Analysis (A), and Reformulation (R) stages (i.e., the design stages based on the FBS model) and the average are evaluated, as shown in equations 5-7:
[0068]
[0069] The three indicators evaluate different aspects of the model, and are usually used simultaneously. Precision measures the ability of the classifier to correctly identify samples, representing the positive predictive rate. Recall represents the model's recall rate, indicating the proportion of all positive samples that can be correctly predicted. F1 score is a comprehensive reflection of the model's prediction, and the performance precision, recall, and F1 score of the LSTM prediction of the design sequence are 0.72, 0.75, and 0.73, respectively.
[0070] The further technical solution of the application is: the step four model hyperparameter adjustment adopts a one-layer LSTM arrangement and uses different node numbers. The number of fully connected layers and size, Dropout value are set to 1, 7, and 0.3, respectively, and the learning rate uses the callback function ReduceLROnPlateau carried by the Keras deep learning library to quickly and accurately obtain the optimal model.
[0071] LSTM size has a certain influence on the prediction accuracy of the model, and when it is set to 256, it has the optimal prediction performance (73.33%). When the Batch size is set to 5 (73.33%), it has better prediction performance than 4 (70.00%) or 8 (70.00%). In all settings, it is found that when LSTM size is 256 and Batch size is 5, the prediction accuracy is the best.
[0072] The further technical solution of the present application is that the step five knowledge pushing mainly includes intention recognition, knowledge pushing, etc.
[0073] Intention recognition: based on the coding scheme of the design phase, the prediction phase / current phase plus the previous three design phases are used.
[0074] Since there are three design process stages of Formulation (F), Analysis (A), and Reformulation (R) in this study, the final intention recognition result is based on the learning of the designer's operation rules to form a certain process stage. The present application uses the Keras deep learning library to run the LSTM model for designer intention recognition. Each round of training, forward and backward propagation is performed on the training set, so that various parameters of the model are updated. When the entire training set has completed this complete transmission process, it becomes an epoch of training. During testing, considering that the designer's next operation is often affected by the previous operation, the present application uses the previous three actions to predict the next operation. Therefore, if a design sequence has n actions, n-3 predictions will be made in the end. Then, by comparing with the true observed value, the total number of correct predictions (n ap ) is obtained, and then divided by the total number of predictions, i.e. n-3. In this way, the prediction accuracy of each design stage can be obtained, and if the average prediction accuracy is to be obtained, it is represented by the following formula:
[0075]
[0076] where n s represents the number of design stages, n s = 3 in this study, and n t represents the length of the design sequence in the test set. When training the model, a trial-and-error method is used to achieve the best prediction effect. The average prediction accuracy of the model for different design stages reaches 73.33%, among which the prediction accuracy of the synthesis (S) stage is the highest, reaching 81.82%, and the prediction accuracy of the Formulation (F) stage is the lowest, only 69.23%, which may be because the order of actions in the S stage is more regular.
[0077] Knowledge pushing: after intention recognition, the picture in the knowledge base that is most similar, i.e. the smallest distance, will be fed back to the designer.
[0078] After the designer's intention is recognized by the LSTM, the next design action is obtained, which only completes the process from human to machine, and does not further form the feedback from machine to human. In order to realize the true sense of man-machine collaborative design, it is necessary to form effective design knowledge recommendation combined with the design intention recognition result. The coding in this paper refers to the coding scheme based on the design stage, which adopts the mode of prediction stage / current stage plus the previous three design stages, such as 1123, which represents that the prediction design stage is Formulation, and the three stages from the back to the front are Formulation, Synthesis and Reformulation. The basis of knowledge retrieval is the Euclidean distance, and the to-be-retrieved code is expressed as a vector form x=(x1,x2,x3,x4), x1,x2,x3,x4 represent the prediction design stage number and the previous three design stage numbers respectively, and the knowledge base code extraction is transformed into y=(y1,y2,y3,y4) in the same way. The similarity judgment expression is as follows:
[0079]
[0080] Wherein, n represents the number of stages participating in coding, and in this study, n=4. After the judgment, the picture in the knowledge base that is most similar, that is, the smallest distance, will be fed back to the designer.
[0081] The application also provides a man-machine interactive design device based on intention recognition and knowledge pushing, which is applied to the above method and comprises: a data set collection unit for setting design requirements and design constraints, acquiring continuous design behavior data of a designer according to the set design requirements and design constraints through CAD modeling software; a data set processing unit for reducing the dimension of the design behavior data through an FBS model, mapping the design behavior space to a design stage space, and coding the reduced data into model training data; a model training unit for inputting the model training data into an LSTM model for training, monitoring the performance of the model on the training set and the validation set during the training process to prevent overfitting, and evaluating the model performance using test data after the training is completed; a model hyperparameter adjustment unit for studying the influence of different hyperparameters preset in the LSTM model on the prediction accuracy of the algorithm, performing sensitivity analysis on the model by constantly changing the values of these hyperparameters, and studying the corresponding prediction accuracy to select the optimal hyperparameters; and a knowledge pushing unit for pushing design knowledge to the designer by matching the design knowledge from a design knowledge base constructed based on the predicted design stage after the designer's intention is recognized by the LSTM model.
[0082] Invention effect: The average accuracy of the LSTM model in the three stages is at a high level, and the test accuracy of the Synthesis stage is higher than that of other design stages. Although all design stages of sequence design are affected by short-term and long-term memory, the Synthesis stage is more affected. Combined with the prediction results of the sequence design operation of the designer and the built design knowledge base, the design knowledge related to the prediction stage is successfully pushed. It is concluded that after identifying the design intention of the designer, it is feasible to push the design knowledge to the designer combined with the past design experience.
[0083] In summary, combining a deep learning model similar to an LSTM model to identify the designer's intention and then forming a knowledge push combined with design knowledge can become a new method of human-computer collaborative design. Moreover, the method introduced in the present application has universality, and as long as the sequence design actions of the designer can be collected and the relevant field design knowledge base can be built, it can be applied to other design environments, thereby realizing multi-field human-computer collaborative design.
Claims
1. A human-machine interaction design method based on intent recognition and knowledge pushing, characterized in that, The method comprises the following steps: S1 data set collection: Set design requirements and design constraints, and obtain continuous design behavior data of designers according to the set design requirements and design constraints through CAD modeling software; S2 data set processing: The design behavior data is dimensionally reduced through an FBS model, the design behavior space is mapped to a design stage space, and the dimensionally reduced data is encoded as model training data; S3 model training: The model training data is input into an LSTM model for training, the performance of the model on the training set and the validation set needs to be monitored during the training process to prevent overfitting, and the model performance is evaluated using test data after the training is completed; S4 model hyperparameter adjustment: The influence of different hyperparameters of the LSTM model on the prediction accuracy of the algorithm is studied, the sensitivity of the model is analyzed by constantly changing the values of these hyperparameters, and the corresponding prediction accuracy is studied to select the optimal hyperparameters; S5 knowledge pushing: After the designer's intention is identified through the LSTM model, design knowledge is matched from a design knowledge base constructed based on the predicted design stage to push knowledge to the designer.
2. The method of claim 1, wherein, The S1 data set processing further comprises design behavior data extraction; The design requirements and the design constraints are specifically that SolidWorks software is selected to perform three-dimensional modeling on the crank slider mechanism, and the goal of the design is to meet the slider stroke requirement while ensuring the normal operation of the crank slider mechanism; The design behavior data extraction is specifically that in the design task of the crank slider mechanism, the designer is required to complete the design of an organization and then take a break, which avoids the interruption of the designer's design ideas due to too long intervals, and video recording is performed during the design process to facilitate more detailed mining of the design actions after the design is completed.
3. The method of claim 1, wherein, In the S2 data set processing, the data dimension reduction is specifically that an FBS design process model (the FBS model has three types of typical ontology variables: function (F), behavior (B), and structure (S)) is adopted, and an encoding scheme is established in combination with the operation characteristics of the SolidWorks software to transcribe different types of design actions to three design process stages (including Formulation (F), Analysis (A), and Reformulation (R)) to realize design space dimension reduction; The data encoding is specifically that after a set of dimensionally reduced continuous design stage data is obtained, the set of design data is encoded using one-hot encoding, which converts n observation values and m different variables into m binary variables of n observation values, each observation result corresponds to a position, and 1 exists, and 0 does not exist. In order to reduce the model prediction variables while using all design stages, a column of data records the design stage in which each design action observation is located.
4. The method of claim 1, wherein, The S3 model training specifically comprises data training, model monitoring, and data evaluation. The data training is specifically: inputting the single-hot encoded data into the LSTM model for training. Since there are three design process stages in this study, the final intent recognition result is based on the learning of the designer's operation rules to form a certain process stage. The model monitoring is specifically: when training the LSTM model, several pre-set hyperparameters are used, such as the number of LSTM layers, LSTM size, number of fully connected layers, Dropout value, learning rate, Batch size, etc. The Dropout value is set to prevent model overfitting; The data evaluation is specifically: evaluating the prediction accuracy, precision, recall rate, and F1 score of the model for the Formulation (F), Analysis (A), and Reformulation (R) design stages (i.e., based on the FBS model division) and the average.
5. The method of claim 1, wherein, In the S4 model hyperparameter adjustment, the hyperparameter adjustment uses a one-layer LSTM arrangement with different node numbers, and the number of fully connected layers and size, Dropout value are set to 1, 7, and 0.3, respectively. The learning rate uses the callback function ReduceLROnPlateau carried by the Keras deep learning library to quickly and accurately obtain the optimal model.
6. The method of claim 1, wherein, S5 knowledge pushing includes intent recognition and knowledge pushing, The intent recognition is specifically: based on the coding scheme of the design stage, using the prediction stage / current stage plus the previous three design stages; The knowledge pushing is specifically: after intent recognition, the picture in the knowledge base with the smallest distance will be fed back to the designer.
7. An interactive design device based on intent recognition and knowledge pushing, characterized by, including: a data set collection unit for setting design requirements and design constraints, and obtaining continuous design behavior data of designers through CAD modeling software based on the set design requirements and design constraints; a data set processing unit for dimension reduction of design behavior data through the FBS model, mapping design behavior space to design stage space, and encoding the reduced data as model training data; a model training unit for inputting the model training data into the LSTM model for training, monitoring the model performance on the training set and validation set during training to prevent overfitting, and evaluating the model performance using test data after training; a model hyperparameter adjustment unit for studying the influence of different pre-set hyperparameters of the LSTM model on algorithm prediction accuracy, performing sensitivity analysis on the model by constantly changing the values of these hyperparameters, and studying the corresponding prediction accuracy to select the optimal hyperparameters; a knowledge pushing unit for identifying the designer's intent through the LSTM model, matching design knowledge from the design image data based on the predicted design stage to construct a design knowledge base, and pushing the design knowledge to the designer.