A method for automatic operation optimization of simulation software based on twin neural network
Through the automatic operation optimization method based on twin neural networks, the problem of inefficient automation operation of simulation software is solved, the automation operation of simulation software and the control recognition capability are improved, and the problem that the existing technology cannot cope with the automated operation of simulation software is solved.
Patent Information
- Application Number
- CN202311229782.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-09-22
AI Technical Summary
The existing computer automation operation methods cannot effectively deal with the automatic operation of simulation software, especially in the case of unbalanced control frequency and different data set sizes, resulting in inefficient automation operation of simulation software.
Adopt automatic operation optimization method based on twin neural networks, and obtain automated operation processes by recording operation processes, use twin neural network training data sets, train twin neural network models, realize the marking and adjustment of control screenshots, and improve the recognition ability of neural networks. Implement automated operations of simulation software.
It realizes automatic switching of the simulation software's examples and automation of the computing process, improves the simulation work efficiency, enhances the neural network's adaptability to changes, and improves the control recognition ability.
Smart Images

Figure BDA0004463500830000031 
Figure BDA0004463500830000051 
Figure FDA0004986149960000021
Abstract
Description
Technical Field
[0001] The present invention relates to the field of industrial simulation and computer technology, and in particular to an automatic operation optimization method for simulation software based on twin neural networks. Background Art
[0002] Simulation software is the cornerstone of digitalization, networking and intelligence in the manufacturing industry, and the core element of a new round of industrial revolution. In the process of simulation software development, in order to verify the performance of simulation software, a large number of examples need to be repeatedly calculated. However, the calculation of examples by industrial simulation software is limited by the influence of computing resources, and the simulation software is usually operated manually, which is time-consuming and extremely inefficient. Simulation software has its particularity. For example, during operation, some controls will change, and the same example will run on different operating systems. However, the existing computer automation operation method cannot deal with the above problems well, and thus cannot perform automatic operation of simulation software. In simulation software, different controls are used at different frequencies, and the data sets accumulated by various controls are of different sizes, which are typical class imbalance tasks. Compared with other neural networks, twin neural networks are more robust to class imbalance tasks and have strong advantages. Therefore, it is very necessary for this application to propose an automatic operation method for simulation software based on twin neural networks. Summary of the invention
[0003] In order to overcome the above-mentioned defects of the prior art, the present invention proposes an automatic operation optimization method for simulation software based on twin neural networks. The method realizes automatic switching of calculation examples and automation of calculation process, intelligent image assertion, and improves simulation work efficiency.
[0004] In order to solve the above technical problems, the present invention is implemented in the following ways:
[0005] A method for automatic operation optimization of simulation software based on twin neural network, comprising the following steps:
[0006] S1, record the simulation software operation process, obtain the simulation software case automation operation process, execute each operation step of the process in sequence, and enter S2 when executing the image assertion operation step;
[0007] S2, if the assertion result is successful, return to S1 to continue to execute the remaining operations. If the assertion result is failed, end the operation process of this example and determine whether there is an example that has not been executed. If so, execute the next example. After all examples are executed, the user evaluates the image assertion result and the corresponding O s <V a The screenshot of the control is sent to the database;
[0008] S3, the management program marks the newly sent screenshots according to user evaluation, adjusts them according to needs, and sends them to the twin neural network training data set;
[0009] S4, train the twin neural network using the adjusted training dataset;
[0010] S5. After the training is completed, if the image contrast ability is stronger than that of the existing neural network through the test set test, a request is sent to the model user to obtain permission to update, and then the neural network model parameters are updated.
[0011] Furthermore, the specific method of step S2 is as follows:
[0012] When the execution step requires image assertion, block the X2 main path, send the template image to the twin neural network X1 branch network, and send the control screenshot to the X3 branch network. The high-dimensional features of the two images are respectively proposed, and the absolute difference between the features d4 = |h2(x3)-h2(x1)| is used to calculate the difference between the two images under the high-dimensional features, and then map them through the fully connected layer and the Sigmoid() function to output the image similarity O s A value between 0 and 1;
[0013] The similarity O s With threshold V a For comparison, if O s ≥V a The result is asserted to be successful, and the remaining steps are continued; if s <V a The assertion result is failure, the operation process ends, and it is determined whether there are any unexecuted examples. If so, the next example is executed.
[0014] Furthermore, the specific method of step S3 is as follows:
[0015] After user evaluation, the control enters the database, which includes control screenshots of the control image marking management software. The management program marks the control screenshots with the user evaluation image assertion result of "satisfactory" as the negative sample of the control, and marks the control screenshots with the user evaluation image assertion result of "unsatisfactory" as the positive sample of the control. When the demand is to improve the control recognition ability, the marked screenshots of the control are increased and sent to the data set. When the demand is to reduce the control recognition ability, the marked screenshots of the control are reduced and sent to the data set.
[0016] Furthermore, the specific process of the twin neural network training in step S4 is as follows:
[0017] The twin neural network includes 1 main network, 2 branch networks and a backend processing part. The branch networks are identical and share weight parameters. The backend processing part includes an absolute difference calculation module between features, 3 fully connected layers and a Sigmoid() function. A picture is selected as a template picture and sent to the X1 branch network. A picture whose label is consistent with the template picture is selected as a positive sample and sent to the X3 branch network. A picture whose label is inconsistent with the template picture is selected as a negative sample and sent to the X2 main network.
[0018] The input image passes through the first layer of 3×3-64 convolution, increasing the number of image channels to 64, and the Maxpool2×2 pooling layer reduces the image to 1 / 2 of the original image. The second layer increases the number of image channels to 128 through 3×3-128 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 4 of the original image. The third layer increases the number of channels to 256 through 3×3-256 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 8 of the original image. The fourth layer increases the number of channels to 512 through 3×3-512 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 16 of the original image.
[0019] The X1 branch network and the X3 branch network are directly input to the fifth layer. The X2 main network calculates the absolute difference between its fourth layer image and the image of the X1 branch network and its fourth layer image and the image output of the X3 branch network as the input of the fifth layer. After the fifth layer convolution processing but the number of image channels remains unchanged, it is output after spatial pyramid pooling SPP processing. X2 outputs the perturbation term b. The X1 branch network and the X3 branch network calculate the absolute difference a between the image features of this layer and output it, and calculate the total absolute difference T d ; The calculated absolute difference between the image features is mapped to a single-value output through the linear layers FC-4028, FC-2014, and FC-1, and the value is input into the activation function Sigmoid() to fix the value at the similarity of (0,1).
[0020] Preferably, in the twin neural network of the present invention, the expected output of similar pictures is 1, and the expected output of dissimilar pictures is 0, and the loss function adopts the binary cross entropy loss BCEWithLogitsLoss(), and its loss formula is as follows:
[0021]
[0022] Where N represents the number of samples, y i represents the true label of sample i, Represents the probability of sample prediction, log represents the natural logarithm, and σ() represents the Sigmoid function.
[0023] Compared with the prior art, the present invention has the following beneficial effects:
[0024] In response to the software automation operation requirements of simulation software, the present invention provides a simulation software automatic operation optimization method based on a twin neural network. The method can collect screenshots of controls where automation operation fails, and make corrections through a human-in-the-loop method, thereby enabling the simulation software automation operation to have the ability to continuously learn and enhance the ability to cope with changes in the simulation software. The actual deployment of the present invention provides intelligent image assertions of controls including recording and playback of simulation software automation operation behaviors to verify changes, adjusts the neural network model according to the automation operation results, and improves the neural network recognition capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a schematic diagram of the step operation flow of the present invention;
[0026] Figure 2 It is a schematic diagram of the execution flow of the present invention;
[0027] Figure 3 It is a schematic diagram of the structure of the twin neural network training model in the present invention;
[0028] Figure 4 It is a schematic diagram of a calculation example operation process record format of the present invention;
[0029] Figure 5 It is a schematic diagram of another example operation process recording format of the present invention. DETAILED DESCRIPTION
[0030] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solution protected by the present invention will be clearly and completely described below using specific embodiments and drawings. It is obvious that the embodiments described below are only part of the embodiments of the present invention, rather than all of the embodiments.
[0031] like Figures 1-2 As shown, a method for automatic operation optimization of simulation software based on twin neural network includes the following steps:
[0032] S1, record the simulation software operation process, obtain the simulation software case automation operation process, execute each operation step of the process in sequence, and enter S2 when executing the image assertion operation step;
[0033] S2, if the assertion result is successful, return to S1 to continue to execute the remaining operations; if the assertion result is failed, end the operation process of this example, and determine whether there is an example that has not been executed. If so, execute the next example. After all examples are executed, the user evaluates the image assertion results and sends the corresponding control screenshot to the database; after all examples are run, the user evaluates the image assertion results and sends the screenshot to the database. s <Va The screenshot of the control is automatically sent to the database;
[0034] S3, the management program marks the newly sent screenshots according to user evaluation, adjusts them according to needs, and sends them to the twin neural network training data set;
[0035] S4, train the twin neural network using the adjusted training dataset;
[0036] S5. After the training is completed, when the image contrast ability is stronger than the existing neural network through the test set test, a request is sent to all users of the model in the local area network. After obtaining permission to update, the neural network model parameters are updated, and the twin neural network in the automated operation software has enhanced adaptability and effect to changes.
[0037] Furthermore, the specific method of step S2 is as follows:
[0038] When the execution step requires image assertion, the X2 branch is blocked, the template image is sent to the X1 branch network of the twin neural network, and the control screenshot is sent to the X3 branch network. The high-dimensional features of the two images are respectively proposed, and the absolute difference between the features d4 = |h2(x3)-h2(x1)| is used to calculate the difference between the two images under the high-dimensional features, and mapped to a value between 0 and 1 through the fully connected layer and the Sigmoid() function, which is the similarity O of the output image. s ; Set the similarity O s With threshold V a For comparison, if O s ≥V a The result is asserted to be successful, and the remaining steps are continued; if s <V a The assertion result is failure, the operation process ends, and it is determined whether there are any unexecuted examples. If so, the next example is executed.
[0039] The specific method of step S3 is as follows:
[0040] After user evaluation, the control enters the database, which includes control screenshots of the control image marking management software. The management program marks the control screenshots with the user evaluation image assertion result of "satisfactory" as the negative sample of the control, and marks the control screenshots with the user evaluation image assertion result of "unsatisfactory" as the positive sample of the control. When the demand is to improve the control recognition ability, the marked screenshots of the control are increased and sent to the data set. When the demand is to reduce the control recognition ability, the marked screenshots of the control are reduced and sent to the data set.
[0041] like Figure 3As shown, the twin neural network in the present invention includes a main network, two branch networks and a back-end processing part, the branch networks are identical and share weight parameters, and the back-end processing part includes an absolute difference calculation module between features, three fully connected layers and a Sigmoid() function; select one picture as a template picture and send it to the branch network X1, select one picture whose label is consistent with the template picture as a positive sample and send it to the branch network X3, and select one picture whose label is inconsistent with the template picture as a negative sample and send it to the main network X2;
[0042] The input image passes through the first layer of 3×3-64 convolution, increasing the number of image channels to 64, and the Maxpool2×2 pooling layer reduces the image to 1 / 2 of the original image. The second layer increases the number of image channels to 128 through 3×3-128 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 4 of the original image. The third layer increases the number of channels to 256 through 3×3-256 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 8 of the original image. The fourth layer increases the number of channels to 512 through 3×3-512 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 16 of the original image.
[0043] Branch network X1 and branch network X3 are directly input to the fifth layer. The main network X2 calculates the absolute difference between its fourth layer image and the image of branch network X1 and its fourth layer image and the image of branch network X3 as the input of the fifth layer. After the fifth layer convolution processing but the number of image channels remains unchanged, it is output after spatial pyramid pooling SPP processing. X2 outputs the perturbation term b. Branch network X1 and branch network X3 calculate the absolute difference a between the image features of this layer and output them. The total absolute difference T is calculated. d ; The calculated absolute difference between the image features is mapped to a single-value output through the linear layers FC-4028, FC-2014, and FC-1, and the value is input into the activation function Sigmoid() to fix the value at the similarity of (0,1).
[0044] Preferably, in the twin neural network of the present invention, the expected output of similar pictures is 1, and the expected output of non-similar pictures is 0, so it can be equivalent to a binary classification network; the loss function adopts the binary cross entropy loss BCEWithLogitsLoss(), and its loss formula is as follows:
[0045]
[0046] Where N represents the number of samples, y i represents the true label of sample i, Represents the probability of sample prediction, log represents the natural logarithm, and σ() represents the Sigmoid function.
[0047] In the present invention, the automated operation process of the simulation software includes: a series of sequential operations performed by the mouse and keyboard on each control of the simulation software interface, which are recorded in advance, and each operation step includes mouse single-click, double-click, drag, wheel scrolling and keyboard input operations; the control operation records are divided into two categories: controls that do not need to assert operations, recording the control-specific properties and the operations performed by the mouse and keyboard on the controls; operations that need to be asserted, recording the control-specific properties, the operations performed by the mouse and keyboard on the controls and the template image of the control.
[0048] In this embodiment, the original training data set is preliminarily collected for 50 types of controls commonly used in simulation software, that is, 30 positive sample images and 20 negative sample images for each control, a total of 2500 images, and the twin neural network is preliminarily trained.
[0049] 1) Prepare the recorded simulation software automation operation process and obtain the pre-set threshold V a 0.9
[0050] The automation operation process file includes multiple automation operations. Each automation operation field includes control-specific attributes (Attribute), control operations (Operator), image assertions (Assert), and control template images (Template). In the automation operation record file, each line represents an automation operation.
[0051] like Figures 4-5 As shown, in this embodiment, the value of the control-specific attribute is unique to the entire simulation software control and has unique attributes. The control-specific attribute recording method is {'attribute type' = 'property value'}, such as {'name' = 'solid'}; the control operations include mouse single-click, double-click, drag, wheel scrolling and keyboard input, and the control operation recording method is {'Operator' = 'operation name'}, such as {'Operator' = 'Mouse_click'}; the picture assertion records whether the picture enters the assertion operation, and the assertion recording method is {'Assert' = 'Yes / No'}, such as {'Assert' = 'No'}; the control template picture records the standard comparison picture, and the picture in this method is stored in .png format.
[0052] 2) Execute the automated operation process operations one by one
[0053] Take the first row of control-specific properties {'name' = 'solid'}, and locate the control whose property name value is soild according to the specific properties and their values; take the image assertion {'Assert' = 'No'}, and determine that the operation does not require an image assertion operation. Take the control operation {'Operator' = 'Mouse_click'}, and perform a mouse click operation according to the value of Operator.
[0054] Take the second row of control-specific properties {'name'='status_View'}, and locate the control whose property name value is status_View based on the specific properties and their values; take the image assertion {'Assert'='Yes'}, and if it is determined that the operation requires an image assertion operation, then enter the image assertion operation.
[0055] The real-time screenshot of the control whose current attribute name value is status_View is represented as a matrix of H×W×C, where the image size is H×W and the number of feature channels is C. Input the twin neural network branch network to extract high-dimensional feature 2, and take the control template image input into the twin neural network main network to extract high-dimensional feature 1. In this embodiment, the twin neural network image assertion network is as follows Figure 4 As shown in the figure, the input image is subjected to a combination of 3×3 convolution and pooling layers to achieve feature extraction. After 4 operations, the size of the feature layer becomes (H×W) / 16 of the original image, and the number of channels increases to C*512; after another 3×3 convolution operation, the output is unified through spatial pyramid pooling SPP, and then input into 3 fully connected layers and ReLU function to obtain the similarity O of the input image. s is 0.95. s Compared with the threshold value of 0.95, O s >V a , output assertion result is successful. Take the current control operation: {'Operator' = 'Mouse_doubleclick'}, according to the value of Operator Mouse_doubleclick, perform the mouse double-click operation on the control, check that the current operation is not the last line, and continue to execute.
[0056] Take the third row of control-specific properties: {'name' = 'status_View'}, locate the control with the property name value of status_View according to the specific properties and their values, take the image assertion {'Assert' = 'Yes'}, determine that the operation requires an image assertion operation, and then enter the image assertion operation.
[0057] Take a real-time screenshot of the control whose current attribute name value is status_View, input the twin neural network branch network to extract high-dimensional feature 1, and take the control template image and input the twin neural network main network to extract high-dimensional feature 2. After the high-dimensional feature extraction is completed, use the Euclidean distance to calculate the difference of the high-dimensional features, and get a one-dimensional vector. After passing through three layers of fully connected layers and one layer of activation function, the similarity O is obtained. s =0.4.
[0058] Compare the current similarity value with the threshold, s<V a , the output assertion result fails, the program sends the screenshot of the control to the database, and the current example operation process ends. Check that Example 2 has not been executed yet. The simulation software reads Example 2 and its operation process, takes the first line of control proprietary attributes {'name'='solid'}, locates the control with the attribute name value of soild according to the proprietary attributes and their values, takes the image assertion {'Assert'='No'}, and determines that the operation does not require an image assertion operation; takes the control operation {'Operator'='Mouse_click'}, and performs a mouse click operation on the control according to the value of Operator. If there is no subsequent operation after the execution is completed, Example 2 is executed. No other examples to be executed are detected, and the entire automation operation ends.
[0059] 3) Modify the control training data set based on the annotation of the assertion results to improve the recognition ability of the neural network model
[0060] The management program annotates the image. According to the user's evaluation, the assertion result is "unsatisfactory". The management program is used to mark the control screenshot as a positive sample and send it to the training data set. At this time, the number of positive samples of the control with the attribute name value of status_View becomes 31, and the log automatically records the changes in the training data set.
[0061] The twin neural network adopts the neural network training framework of the present invention, and regularly uses the modified data set for training, and obtains the optimal network parameter network by minimizing the Loss through the optimizer. During training, the initialization learning rate is set to 0.01, the weight decay coefficient is set to 0.0005, the minimum learning rate is 0.0001, the stochastic gradient descent method (SGD) is used, and the batch_size is set to 16. After the training is completed, the twin neural network is tested with a test set to have a stronger ability to recognize control features, and the twin neural network model used for assertion by the specified user is automatically updated. In this embodiment, the user only needs to evaluate the assertion results to improve the recognition ability of the twin neural network according to user needs.
[0062] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.
Claims
1. A method for automatic operation optimization of simulation software based on twin neural network, characterized in that: The following steps are involved: S1, record the simulation software operation process, obtain the simulation software case automation operation process, execute each operation step of the process in sequence, and enter S2 when the image assertion operation is required; S2, if the assertion result is successful, return to S1 to continue to execute the remaining operations. If the assertion result is failed, end the operation process of this example and determine whether there is an example that has not been executed. If so, execute the next example. After all examples are executed, the user evaluates the image assertion result and the corresponding O s <V a The screenshot of the control is sent to the database; S3, the management program marks the newly sent screenshots according to user evaluation, adjusts them according to needs, and sends them to the twin neural network training data set; S4, train the twin neural network using the adjusted training dataset; S5. After the training is completed, if the image contrast capability is stronger than that of the existing neural network through the test set test, a request is sent to the model user to obtain permission to update, and then the neural network model parameters are updated; The specific method of step S2 is as follows: When the execution step requires image assertion, block the X2 main path, send the template image to the twin neural network X1 branch network, and send the control screenshot to the X3 branch network. The high-dimensional features of the two images are respectively proposed, and the absolute difference between the features d4 = |h2(x3)-h2(x1)| is used to calculate the difference between the two images under the high-dimensional features, and then map them through the fully connected layer and the Sigmoid() function to output the image similarity O s A value between 0 and 1; The similarity O s With threshold V a For comparison, if O s ≥V a The result is asserted to be successful, and the remaining steps are continued; if s <V a If the assertion result is failure, the operation process ends and a check is made to see if there is any unexecuted example. If so, the next example is executed. The specific method of step S3 is as follows: After being evaluated by the user, the control is entered into the database, which includes control screenshots of the control image tagging management software. The management program marks the control screenshots with the user's evaluation image assertion result of "satisfactory" as the negative sample of the control, and marks the control screenshots with the user's evaluation image assertion result of "unsatisfactory" as the positive sample of the control; when the demand is to improve the control recognition ability, the control's marked screenshot data is sent to the data set, and when the demand is to reduce the control recognition ability, the control's marked screenshot data is reduced. The specific process of the twin neural network training in step S4 is as follows: The twin neural network includes 1 main network, 2 branch networks and a backend processing part. The branch networks are identical and share weight parameters. The backend processing part includes an absolute difference calculation module between features, 3 fully connected layers and a Sigmoid() function. Select a picture as a template picture and send it to the X1 branch network, select a picture whose label is consistent with the template picture as a positive sample and send it to the X3 branch network, select a picture whose label is inconsistent with the template picture as a negative sample and send it to the X2 main network; The input image passes through the first layer of 3×3-64 convolution, increasing the number of image channels to 64, and the Maxpool2×2 pooling layer reduces the image to 1 / 2 of the original image. The second layer increases the number of image channels to 128 through 3×3-128 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 4 of the original image. The third layer increases the number of channels to 256 through 3×3-256 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 8 of the original image. The fourth layer increases the number of channels to 512 through 3×3-512 convolution and Maxpool2×2 pooling, and the size is reduced to 1 / 16 of the original image. The X1 branch network and the X3 branch network are directly input to the fifth layer. The X2 main network calculates the absolute difference between its fourth layer image and the image of the X1 branch network and its fourth layer image and the image output of the X3 branch network as the input of the fifth layer. After the fifth layer convolution processing but the number of image channels remains unchanged, it is output after spatial pyramid pooling SPP processing. X2 outputs the perturbation term b. The X1 branch network and the X3 branch network calculate the absolute difference a between the image features of this layer and output it, and calculate the total absolute difference T d ; The calculated absolute difference between the image features is mapped to a single-value output through the linear layers FC-4028, FC-2014, and FC-1, and the value is input into the activation function Sigmoid() to fix the value at the similarity of (0,1).
2. The method for automatic operation optimization of simulation software based on twin neural network according to claim 1, characterized in that: The expected output of similar images in the twin neural network is 1, and the expected output of dissimilar images is 0. The loss function uses the binary cross entropy loss BCEWithLogitsLoss(), and its loss formula is as follows: Where N represents the number of samples, y i represents the true label of sample i, Represents the probability of sample prediction, log represents the natural logarithm, and σ() represents the Sigmoid function.
