Robot control software testing method fusing dqn and pso

Abstract

This invention discloses a robot control software testing method integrating DQN and PSO. First, based on the path association characteristics in the robot control software, paths are divided into a high-association path set and a low-association path set. Then, for the low-association path set, a suitable sample set is updated, and the parameters of the DQN network are fine-tuned to achieve reuse of the DQN model for high-association paths by low-association paths. Finally, based on the PSO algorithm, starting with an initial population with high contribution, a test case set covering the target path is generated. This method, through the differentiation of path association characteristics and the efficient reuse of the DQN model, significantly improves testing efficiency while ensuring the sufficiency of robot control software testing, such as path coverage integrity and control accuracy compliance, providing technical support for the reliability verification of robot control strategy software.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer software testing and intelligent optimization algorithms, specifically a hybrid testing method for robot control software. This method utilizes Deep Q-Network (DQN) and Particle Swarm Optimization (PSO) algorithms to accurately generate test data, thereby improving the path coverage and defect detection efficiency of robot control software. Background Technology

[0002] As the "brain" of a robot, the reliability of robot control software directly determines the safety and accuracy of the robot in performing tasks in complex, unstructured environments. With the widespread application of robotics technology in safety-critical fields such as industrial automation, medical services, and space exploration, thorough and efficient testing of robot control software has become a crucial step in ensuring its quality.

[0003] Software testing is a crucial method for detecting software quality. By discovering and fixing defects in programs, it can effectively improve software reliability. Mutation testing in software testing is often used to evaluate the quality of sample datasets and assist in generating test cases, thereby improving software quality. There are two main types of mutation testing: strong mutation testing and weak mutation testing. Strong mutation testing requires test cases to comprehensively cover all potential changes in the variants, which typically requires significant computational resources and time, especially for complex, large-scale software systems, resulting in very high costs. Weak mutation testing, on the other hand, focuses only on whether the variants affect the intermediate states of the program or certain important output results, without requiring comprehensive testing of the variants, significantly reducing the cost of software testing.

[0004] Deep Q-Network (DQN) is a reinforcement learning algorithm that innovatively combines deep neural networks with Q-learning algorithms, using deep neural networks to approximate the Q-value function. The core mechanism of DQN includes two key technologies: experience replay and the target network. Experience replay stores the agent's experience in the form of states, actions, rewards, and next states in an experience pool. During training, random sampling breaks the temporal correlation between data, improving sample utilization efficiency. The target network uses an independent network structure to calculate the target value of the Q-value and updates it periodically, thereby reducing oscillations during training and enhancing algorithm stability.

[0005] In robot control, Directed QN (DQN) is widely used in tasks such as robot path planning, grasping operations, and multi-robot collaboration. It can autonomously learn optimal control strategies and achieve intelligent decision-making, fully demonstrating its effectiveness in complex decision-making tasks. However, reinforcement learning algorithms typically require a large number of exploration samples, and the training process suffers from low sampling efficiency and slow convergence speed. This invention introduces DQN into the field of software testing. DQN uses the program's input data as states and adjustments to input parameters as actions. Through continuous interaction with the program under test, it receives rewards based on path coverage, learning to generate effective strategies with high coverage test cases.

[0006] Pseudo-Swarm Optimization (PSO) is a swarm intelligence-based optimization algorithm inspired by the foraging behavior of birds. In PSO, each particle moves in the search space and adjusts its position based on historical best and global best positions. The core mechanism of PSO is to quickly converge to the optimal solution through information sharing and cooperation among particles.

[0007] In software testing, PSO (Physical Strain Analysis) can be used to automatically generate test cases. By using a custom fitness function, the test case generation problem is transformed into an optimization problem, allowing PSO to improve test case coverage by guiding the search direction. Especially for paths that are difficult to cover, PSO can find a higher quality number of tests. However, when facing test targets common in robot control software, characterized by long temporal dependencies and complex state transitions, traditional PSO algorithms are prone to getting trapped in local optima. This invention improves the PSO algorithm by proposing a comprehensive coefficient of variation (COP) to enhance local convergence detection capabilities. The core value of the COP lies in providing a dual evaluation of the subpopulation state: the global standard deviation term measures the overall dispersion of fitness values ​​across all particles, while the neighboring difference term captures the jump changes in fitness values ​​between neighboring particles. This design enables it to sensitively identify local optimum traps. Summary of the Invention

[0008] This invention proposes a robot control software testing method that integrates DQN and PSO. The key difference between this method and existing methods lies in its approach to addressing a large number of difficult-to-cover variant paths. It groups highly correlated and low-correlation paths based on path similarity, constructs and trains a Deep Q-Network (DQN), and combines it with a Particle Swarm Optimization (PSO) algorithm to generate test cases covering the target paths, thereby improving the efficiency of software testing.

[0009] The technical solution adopted in this invention is a robot control software testing method integrating DQN and PSO, comprising the following steps:

[0010] S1: Grouping highly correlated and low-correlation paths based on path similarity;

[0011] S2: Co-construction based on highly correlated paths Model;

[0012] S2.1: Filter high-quality sample states based on highly correlated path features;

[0013] S2.2: Determining the state based on high-quality samples The initial experience pool of the model;

[0014] S2.3: Training based on the initial experience pool Model;

[0015] S2.3.1: Select high-priority experiences from the initial experience pool;

[0016] S2.3.2: Iterative training based on high-priority experience Model;

[0017] S3: Reuse highly associated paths Low-association paths in model training Model;

[0018] S3.1: Filter high-quality sample states based on low-association path features;

[0019] S3.2: High-quality sample state training based on low-association paths Model;

[0020] S3.3: Based on Model and The model selects sample data with high reward values.

[0021] S4: Test cases for initial population generation coverage paths based on high reward value sample data for PSO;

[0022] S4.1: Define the fitness values ​​for test cases covering the path;

[0023] S4.2: Dynamic detection of particle swarms trapped in local optima based on fitness value calculation of comprehensive coefficient of variation;

[0024] S4.3: For the local optimum problem, an inverse particle swarm optimization algorithm is proposed to improve search performance;

[0025] S4.4: Inverse particle swarm optimization algorithm generates test cases for covered paths.

[0026] Furthermore, step S2.1, which involves filtering high-quality sample states based on highly correlated path features, specifically includes:

[0027] Set highly associated paths The set of input sample states for the model is the set of input data for the program. , After executing the program under test, a set of traversal paths is obtained, denoted as... ;

[0028] Determined status Corresponding traversal path and target path Similarity Normalized value of the difference in length between the traversing path and the target path and state perturbation stability Three evaluation indicators:

[0029] For the traversal path With the target path Similarity;

[0030] yes and The normalized value of the path length difference is: ;in It is a path Length is also the number of nodes contained in the path; The target path Length;

[0031] It is the sample state The perturbation stability; assuming the sample state Include Dimensional variables ;

[0032] In order to calculate ,first Each dimension of variable Nearby positive and negative perturbations were obtained respectively. and ,in For the first The perturbation value of the dimension; then, that is... The original data and the data after positive and negative perturbations are used to obtain the following through mathematical permutations and combinations: New status;

[0033] set up The path traversed is the target path. ,So, The traversal path corresponding to the new state and The average similarity is ;

[0034] Define the state based on three indicators. Comprehensive indicators ;in The following determines reasonable weight values ​​for the three indicators:

[0035] First, random selection Individual sample states; assumption ,calculate The average of the three indicators for each sample state is denoted as . ;

[0036] Then, from the three indicators, one evaluation indicator is deleted in turn, and the weights of the remaining two indicators are each set to 0.50. The results are then recalculated. The average value of the comprehensive index of each sample state is denoted as: , , ;

[0037] The formula for estimating weights is as follows:

[0038] ;

[0039] Calculate the comprehensive index for each sample based on weights, then sort them from largest to smallest, and select... Individual sample status Used for training highly correlated paths Model.

[0040] Furthermore, step S2.2.2 calculates the priority of each experience based on the current state and action, specifically as follows:

[0041] In experience middle, Measured by path similarity:

[0042] set up The optimized task objective is to have the target path as ;state The corresponding traversal path is , and similarity ,So When the traversal path can completely cover the target path, take... ;otherwise ; The value range is greater than 1;

[0043] In the model, predictions based on policy networks The value is denoted as ; Target network prediction The value is denoted as ;in For state The acquired action;

[0044] Defined as the priority of each experience tuple in the experience pool. for:

[0045] .

[0046] Furthermore, step S2.3.1, which involves selecting high-priority experiences from the experience pool, specifically involves:

[0047] In experience In China, experience-based priority Calculate the sampling probability as for:

[0048]

[0049] in This is the sum of all experience points in the experience pool. Control priority intensity, It degenerates into uniform sampling at times. Sampling is performed entirely according to priority.

[0050] After determining the sampling probability of each experience, samples are drawn from the experience pool. One experience point for training Model and calculate the loss function; The size is determined by the capacity of the experience pool.

[0051] Furthermore, S3.1, which filters high-quality sample states based on low-association path features, specifically involves calculating a comprehensive index of the sample state.

[0052] Low-association paths The initial sample state set of the model follows the steps in S2.1, but with the original selection of highly correlated training samples... Based on the comprehensive index ranking formula of the initial sample state set of the model, a fourth index is added to generate a new comprehensive index ranking update formula:

[0053]

[0054] in Indicates the use of highly related paths The model calculates the sample state. After normalizing the value, take the complement value;

[0055] Value normalization methods: High-relevance paths The model will predict the state of each sample. Find the maximum value among all actions. The value is denoted as and minimum The value is denoted as ,calculate The normalization formula is:

[0056] .

[0057] Furthermore, step S4.2, which calculates the comprehensive coefficient of variation based on the fitness value to dynamically detect when the particle swarm is trapped in a local optimum, is specifically executed as follows:

[0058] First, based on highly associated paths Model or low-association path Model, targeting the target path ,get The initial particle swarm constructed from 100 particles is denoted as _p_s_i ... ;

[0059] Then, in the PSO evolution process, the comprehensive coefficient of variation is used to determine whether the particle swarm has a local optimum, as follows:

[0060]

[0061] in Indicates the number of particles in the subpopulation; This represents the fitness value of a particle. This represents the average fitness value of all particles in the subpopulation.

[0062] The numerator: Unlike the ordinary coefficient of variation, it consists of two components that form the combined square root;

[0063] First item: It is the variance of the fitness values ​​of particles in the subpopulation;

[0064] Second item: It is the mean of the fitness differences between adjacent particles in the subpopulation; denominator: This is the average fitness value of particles in the subpopulation.

[0065] Furthermore, step S4.3, the inverse particle swarm optimization algorithm, solves for local optima, specifically as follows:

[0066] Let the range of values ​​for each dimension of the program input variables be defined. Iterate to the ; Next time, the Each particle is denoted as: ,in Indicates the first The first particle Dimensional variable values;

[0067] When the subpopulation gets stuck in a local optimum, from Particles are randomly selected from each subpopulation. Each particle is reversed;

[0068] particle The Dimensional variable values Calculate the reversed new variable value using the following formula:

[0069]

[0070] in This refers to the boundary values ​​of the particles; This represents the value of the dimension corresponding to the particle; Indicates the disturbance value;

[0071] For each dimension of the particle, the inverse variable value is calculated, and the result is obtained. We take the inverse variable value and then replace the corresponding dimension variable value in the original particle to obtain... A new particle;

[0072] From the above One new particle is randomly selected to replace the original particle. This is to achieve particle position updates.

[0073] The beneficial effects of this invention are:

[0074] (1) Precise path segmentation to improve the comprehensiveness and relevance of defect path coverage: Based on the path association characteristics of robot control software, this invention divides the path into high association and low association sets. For high association paths (high incidence of defects and concentrated mutation branches), a DQN model is constructed in a targeted manner. Through state design, action space definition and priority experience playback mechanism, training is strengthened to achieve accurate coverage of complex defect paths. For low association paths, a differentiated sample update strategy is adopted to avoid coverage blind spots or resource waste caused by "one-size-fits-all" testing, and fully meet the safety requirements of robot control software for path coverage integrity.

[0075] (2) Model reuse is highly efficient, taking into account both test sufficiency and test efficiency: The low-association path model is reused by fine-tuning the parameters of the DQN model with high correlation path, which greatly reduces the time cost of repeated modeling and training; at the same time, high reward value samples in the DQN experience pool are used as the initial population of PSO, avoiding the iteration inefficiency problem caused by the randomization of the initial population of traditional PSO. Under the premise of ensuring control accuracy compliance and path coverage sufficiency, the efficiency of test case generation is significantly improved, perfectly adapting to the actual needs of robot control software for test efficiency.

[0076] (3) Intelligent algorithm optimization to ensure the reliability and stability of test case generation: This invention introduces dynamic detection of local optima by comprehensive coefficient of variation in the PSO algorithm and combines it with the reverse particle swarm optimization algorithm to make breakthroughs, effectively solving the technical pain point of traditional PSO being prone to getting trapped in local optima, and ensuring that the generated test cases can cover more potential defect paths; it integrates the deep reinforcement learning capability of DQN with the intelligent optimization advantage of PSO, giving full play to the learning and adaptation capability of DQN to complex paths, and using the optimized PSO to achieve efficient search of test cases, providing stable and reliable technical support for the reliability verification of robot control strategy software and improving the credibility of test results. Attached Figure Description

[0077] Figure 1 The overall flowchart of a robot control software testing method integrating DQN and PSO of the present invention;

[0078] Figure 2 Example program for implementing this invention

[0079] Figure 3 Deep Q-network framework diagram of the present invention Detailed Implementation

[0080] The technical solution adopted in this invention is a robot control software testing method integrating DQN and PSO, which includes the following steps:

[0081] S1: Grouping highly correlated and low-correlation paths based on path similarity.

[0082] S1.1: Based on Calculate path similarity

[0083] Based on existing methods, a set of mutation paths can be obtained. A mutated path is a path in which nodes contain only mutated branches.

[0084] path and of Similarity is:

[0085] (1)

[0086] In formula (1) Representing a path and The number of identical variant branches in the intersection of two sets; Representing a path and The total number of mutated branches of the union of two sets, minus the number of duplicate nodes.

[0087] Based on equation (1), it can be seen that and The more similar, The larger the value. The value ranges from 0 to 1. When When, it means the two paths are exactly the same; when When, it means that the two paths are completely different.

[0088] S1.2: Grouping paths based on path similarity

[0089] Based on equation (1), all paths can be constructed. Similarity matrix .

[0090] For the target mutation path This allows us to obtain its similarity to other paths, and from this, we can calculate the path. Other path correlation :

[0091] (2)

[0092] Where N is the total number of mutation paths in the tested program. From equation (2), it can be seen that... The correlation score is calculated by removing the similarity of paths within the same path, which allows for a more accurate assessment. Relevance to other paths.

[0093] threshold Defined as the average of the correlation of all paths.

[0094] Select the highest correlation. The corresponding path is used as the baseline path; then, paths with a similarity greater than a threshold to the baseline path are selected. Paths that are highly correlated are defined as high-association paths, and these are grouped into a high-association path group. Other paths are grouped into a low-association path group.

[0095] Example: such as Figure 2 The tested program contains 24 variant branches, which generate 4 target paths, namely:

[0096]

[0097]

[0098]

[0099] The similarity between paths is calculated based on equation (1). A path similarity matrix is ​​generated for all paths.

[0100]

[0101] According to formula (2), the specific data can be obtained as follows:

[0102] , , , .

[0103] Therefore, the association with the largest mean is Its corresponding path As the baseline path; then, select the similarity. Greater than the threshold The path, as Highly associated paths ,at this time, They are divided into a highly correlated group. and Low correlation Each person forms a group.

[0104] threshold The average of all correlations .

[0105] because It is the highest degree of relevance, therefore this invention selects As the baseline path; with Similarity greater than the threshold The path is and Therefore, , , They are divided into a group of highly correlated paths, and This is a group of low-association paths.

[0106] S2: Co-construction based on highly correlated paths Model

[0107] When building Then, set the input layer dimensions; design the hidden layer structure, including selecting a CNN, setting the number of layers and neurons; and then set the output layer dimensions, which is to output the dimensions corresponding to different actions. value.

[0108] Select any path from the set of highly correlated paths. As a build The target path.

[0109] First, build The model's experience pool is a storage... By leveraging historical interaction experiences of the model, including buffers of states, actions, and rewards, and employing certain strategies to utilize sample correlations and reuse data, the stability and efficiency of deep reinforcement learning can be significantly improved.

[0110] Let the sample be Experience suggests setting it up as a 5-tuple data structure:

[0111] Experience = <State, Action, Reward, Next State, Experience Priority>

[0112] Recorded as: Each empirical tuple is denoted as .

[0113] Sample status This refers to the input data for the program; the program's input dataset is... The sample state set of the model is denoted as ,in Included 3D variable, represented as .

[0114] action Is In the range of values, A grammatically correct change. The number of actions in each dimension can be the same or different. Let the state be... The total number of corresponding actions is Therefore, the set of actions is obtained as follows: .

[0115] Regarding the setting of actions, this invention determines the magnitude of the action based on the value range of the sample state, and determines the action according to perturbations of ±10%, ±5%, ±3%, ±2%, and ±1% in each dimension. For example, in the sample program, the sample state... According to its value range ,exist By going up, you can get the first action; By analogy, we can obtain... There are a total of 10 actions:

[0116] , , , , , , , , , .

[0117] for Execute action The reward received later.

[0118] for Execute action The next state after that.

[0119] The colon (:) indicates the priority of experience. This is the innovation of this invention compared to traditional experience tuples, adding a priority of experience to the experience tuple. The priority mechanism, by "focusing" on high-value experience, can significantly improve performance with the same amount of data. It improves learning speed and final performance, especially in environments with sparse rewards or complex state spaces.

[0120] Each target path in the set of highly correlated paths Train the common ones in turn Model. For Implement screening of high-quality sample status and determine based on high-quality status. The model's initial experience pool and training based on the experience pool The model and other steps continue until the loss function value falls below a preset threshold, or the maximum number of iterations is reached, at which point the current path stops. of Model training. Then, the next path is trained following the same steps until all highly correlated paths are involved in the training. The model ultimately yields a set of highly correlated paths. Model.

[0121] S2.1: Filtering high-quality sample states based on highly correlated path features

[0122] train When developing a model, the quality and characteristics of the sample states directly affect the model's learning efficiency, stability, and final performance. For highly correlated target paths, it is necessary to select high-quality initial sample states. This is because high-quality sample states are crucial for training. The model can quickly find the features corresponding to the target path, improving... Model training efficiency.

[0123] The sample state set is also the input of the program. , After executing the program under test, a set of traversal paths is obtained, denoted as... .

[0124] To filter for high-quality states, select states. Corresponding traversal path and target path Similarity Normalized value of the difference in length between the traversing path and the target path and state perturbation stability Three evaluation indicators.

[0125] (1) Calculation

[0126] For the traversal path With the target path The similarity can be obtained based on equation (1).

[0127] For example, status The crossing path is The corresponding target path is .

[0128] Based on equation (1), the state Corresponding traversal path With the target path Similarity .

[0129] (2) Calculation

[0130] yes and The normalized value of the path length difference can be expressed as:

[0131] (3)

[0132] in It is a path Length, which is the number of nodes contained in the path. The target path The length of . From equation (3), we can see that, The larger the value, the smaller the difference in path length from the target path.

[0133] Given the length of the crossing path Target path length Based on equation (3) .

[0134] (3) Calculation

[0135] It is the sample state The perturbation stability.

[0136] Let the sample state Include Dimensional variables .

[0137] In order to calculate ,first Each dimension of variable Nearby positive and negative perturbations were obtained respectively. and ,in For the first The perturbation value of the dimension; then, that is... The original data and the data after positive and negative perturbations can be obtained through mathematical permutations and combinations. New status.

[0138] set up The path traversed is the target path. ,So, The traversal path corresponding to the new state and The average similarity is defined as . The bigger the better. The larger the value, the more stable it is. Corresponding state More suitable high-quality samples.

[0139] For example, state It can be known that .set up Corresponding traversal path This is the target path. When , After the perturbation, a new state is obtained. . The corresponding traversal path is Based on equation (1), we can obtain and The path similarity is 1.0. Similarly, the path similarity is calculated... The state of the newly generated sample and The similarity values ​​are calculated by averaging these similarities.

[0140] .

[0141] (4) Calculate the comprehensive index

[0142] Define the state based on three indicators. Comprehensive indicators :

[0143] (4)

[0144] in The weights of the three indicators.

[0145] set up Sample status The comprehensive index value can be calculated based on equation (4):

[0146] (5) Determine reasonable weights

[0147] Considering The three indicators have different degrees of importance in selecting high-quality states. In order to obtain reasonable weights, this invention estimates the importance of the evaluation indicators.

[0148] First, random selection Individual sample states. Assume Calculated based on equation (4) The average of the three indicators for each sample state is denoted as . .

[0149] Then, from the three indicators, one evaluation indicator is deleted in turn, and the weights of the remaining two indicators are each set to 0.50. The results are then recalculated. The average value of the comprehensive index of each sample state is denoted as: , , .

[0150] The formula for estimating weights is as follows:

[0151] (5)

[0152] continue Figure 1 This example program illustrates a method for achieving a high-quality state. (Based on the target path...) Taking this example, we will explain the calculation method for determining the reasonable weight values ​​of the three evaluation indicators.

[0153] Through the above steps, this invention targets the target path. The initial sample state is ranked based on the comprehensive index calculated by equation (4), and the following is selected. Individual sample status Used for training Model.

[0154] Example: Suppose we select 5 sample states to calculate three evaluation indicators. as well as As shown in Table 1:

[0155] Table 1 When weight Calculate the comprehensive evaluation index value of the three indicators.

[0156]

[0157] when The average value can be seen from Table 1. ,

[0158] When one review is deleted, the weight of the other two is set to 0.5. For example, when one review is deleted... : The comprehensive index values ​​for these five sample states are as follows: Therefore, the average value of their comprehensive evaluation index can be calculated. Similarly, when deleted... At that time, the average value of the comprehensive evaluation index When deleted : .

[0159] The sum of the differences between the average of the three indicators and the average of the two indicators is... .

[0160] Based on equation (5), weight Similarly, , .

[0161] Finally, based on the new weights, the comprehensive index values ​​of all samples are calculated according to equation (4), as shown in Table 2.

[0162] Table 2 shows the status of high-quality samples selected according to the new weights.

[0163]

[0164] Based on Table 2, assuming that the top two samples with the highest comprehensive index values ​​are selected... and As training The input sample state of the model.

[0165] S2.2: Determining the state based on high-quality samples The initial experience pool of the model

[0166] S2.2.1: Based on Greedy strategy selects the action corresponding to the state.

[0167] The model's goal is to select an action in the current state that maximizes the similarity between the new state's path and the target path. Therefore, the following is based on... A greedy strategy selects the action corresponding to a state.

[0168] (1) Calculation Method

[0169] In the model, Set it to a value that decays exponentially with the number of iterations. Decay refers to the gradual reduction of [something] as training progresses. The value of . Attenuation is a key means of optimizing this strategy.

[0170] calculate The formula is:

[0171] (6)

[0172] In formula (6): This is the maximum value originally designed; : This is the final minimum value of the design; : is the current Number of steps interacting with the environment; This refers to the attenuation coefficient, which controls the attenuation rate. The calculation method is as follows:

[0173] (7)

[0174] In formula (7): This refers to the maximum number of iterations set by the system.

[0175] (2) Based on - Greedy strategy determines the action corresponding to the state.

[0176] based on Select status The method is as follows:

[0177] (8)

[0178] Equation (8) is explained in detail as follows: using probability To conduct random "exploration," that is, to randomly select an action; with probability Based on the current policy network, perform the optimal action that "utilizes" the current estimate, i.e. Execute the action with the highest value to maximize immediate rewards.

[0179] The state can be determined by equation (8). corresponding actions and perform actions , to obtain a new state .

[0180] This invention employs a roulette wheel method to achieve probabilistic selection: when a random number is drawn... When the random number is within the range, a random action strategy is selected; when the random number is within the range... When within the range, select the previously corresponding option. The action strategy with the highest value.

[0181] Example: Settings and Given the current number of iterations Total number of iterations Based on equation (7) in the example Therefore, based on equation (6), we can calculate... .

[0182] state ,when At that time, randomly select one action, for example, in the variable Select action -10 on the top, in If the value changes from 8 to -2, then... Transformation into a new state At this point, the action causes the variable to... The value changed from 8 to -2, causing an "action out of bounds" error, therefore it was discarded. The -10 action above. Re-according to probability. exist Select action -5 above, and the state changes. The action did not result in "action exceeding the boundary", so the action is retained.

[0183] If in Based on the probability, choose the current state. The following correspondence The method for taking the action with the highest value is as follows: 30 movements The value is represented by the following matrix:

[0184]

[0185] In the matrix above, rows represent three-dimensional variables, and columns represent the actions performed by each variable. The matrix displays... Value. At this time. The value is predicted through the policy objective, because the initial experience pool is built at this time.

[0186] As can be seen from the matrix, the choice The +3 action above, sample state Become "Action crossed the line", select the next one. Actions with large values, i.e. +10 on top If the requirements are met, the action will be retained.

[0187] S2.2.2: Calculate the priority of each experience based on the current state and action.

[0188] (1) Based on the current state Calculate reward value

[0189] In experience In the middle, reward value Used to measure the current state Execute action Post-new state for The effect of the model on the task objectives.

[0190] Measured by path similarity. Let... The optimized task objective is to have the target path as ;state The corresponding traversal path is Then, based on equation (1), we get and similarity ,So It can be represented as:

[0191] (9)

[0192] When the traversal path can completely cover the target path, take ;otherwise . The value can be greater than 1. The larger the value, the better. Execute action The more positively the new state contributes to achieving the mission objectives.

[0193] For example, state The corresponding target path is Sample status The crossing path is .

[0194] Based on equation (1), we get Take at this time Its reward value is calculated based on (9). .

[0195] (2) Calculation based on policy network and target network

[0196] In the DQN algorithm, the policy network and the target network are two important networks.

[0197] This invention In the model, predictions based on policy networks The value is denoted as ; Target network prediction The value is denoted as .in For state The action is obtained based on equation (8).

[0198] Defined as the priority of each experience tuple in the experience pool. It can be represented as:

[0199] (10)

[0200] As can be seen from equation (10), The larger the value, the higher the corresponding empirical tuple. The greater the value.

[0201] Example: Sample status The action to choose is The +3 action above, so the sample state Become Based on policy networks as state Predicted Meanwhile, based on the target network as state Predicted Therefore, based on equation (9), the following can be calculated: Therefore, this lesson is:

[0202] .

[0203] S2.3: Training based on the initial experience pool Model

[0204] S2.3.1: Select high-priority experience from the initial experience pool.

[0205] Given the large amount of experience in the experience pool, this invention selects appropriate experience from the pool based on sampling probability. A well-designed sampling probability ensures that individuals with different characteristics in the population are selected fairly, reducing sampling bias.

[0206] Experience-based priorities The sampling probability can be calculated as follows: for:

[0207] (11)

[0208] in: It is the sum of all experience in the experience pool. Control priority intensity: It degenerates into uniform sampling at times. Sampling is performed entirely according to priority. Based on equation (11), the sampling probability for all experiences can be calculated.

[0209] After determining the sampling probability of each experience, samples are drawn from the experience pool. One experience point for training Model and calculate the loss function. The size is determined by the capacity of the experience pool. (This invention) Take the experience pool capacity arrive The numbers in between are reasonable.

[0210] The following example, using five experiences from Table 3, illustrates how to select experiences from the experience pool.

[0211] Table 3 Experience extraction probability table

[0212]

[0213] set up Based on equation (11). .

[0214] Therefore, based on equation (11), the probability of each experience being extracted is:

[0215] ; ; ; ; ; Assumption Therefore, according to the roulette method, experience 5 has the highest probability of being drawn.

[0216] S2.3.2: Iterative training based on high-priority experience Model

[0217] (1) Calculate the policy network using Bellman's formula value: :

[0218] (12)

[0219] (2) Calculate the target network using the Bellman formula. value: :

[0220] (13)

[0221] in: For sample state based and The calculated instant reward value. This is a discount factor between 0 and 1, representing the value of future rewards, and is typically set to 0.99. and Representing states respectively Using a policy network and a target network, predict the corresponding actions for all actions. Values, and take the largest one. value.

[0222] (3) Calculate the loss function :

[0223] (14)

[0224] (4) Calculate network parameters using backpropagation strategy gradient:

[0225] (15)

[0226] in: Represents the loss function Regarding policy network parameters The gradients of the weights and biases are applied through the loss function. Policy network parameters The partial derivative is calculated to obtain the result.

[0227] (5) Update policy network parameters :

[0228] (16)

[0229] in: The parameters of the policy network are used to control the process of selecting appropriate actions by the policy, and are continuously adjusted. To optimize The model's behavioral strategy aims to achieve higher returns. This indicates the current replay training parameters. Representative at the The parameters of the policy network model before the next parameter update. Indicates the first The parameters of the policy network model after the next parameter update. The network parameters of the strategy are calculated using backpropagation for all experiences. The mean of the gradient.

[0230] The learning rate is 0.01 in this invention.

[0231] The current path stops when the loss function value falls below a preset threshold or the maximum number of iterations is reached. of The model is trained, and then training begins on the next path. The preset threshold is an empirical value obtained from multiple experiments, generally between 0.01 and 0.05.

[0232] Let's illustrate this with an example. First, let's assume that the experience points were drawn from the experience pool. There are 6 experiences, each corresponding to a sample state, as shown in the table below.

[0233] Table 4 Extracted experience

[0234]

[0235] Let the target path be In terms of state For example, in variables Go up, select action -1, and get the new state. Traversing Path Based on equation (9), .

[0236] Set discount factor The value is 0.99, the preset threshold for stopping is 0.05, and the maximum value predicted by the policy network is... value The target network predicts the maximum value Then, based on equation (12), the calculation... Based on equation (13), .Depend on and Based on the loss function (14), the loss function is calculated. Since 0.144 is greater than 0.05, the early termination condition is not met. Therefore, continuing, based on equation (16), we can obtain... This is one of them. There are a total of indivual The values ​​are shown in column 4 of Table 4. Therefore, the network parameters using the strategy are calculated. The average gradient .

[0237] Assuming the previous It is 0.850, the learning weight. If the value is 0.01, then based on equation (16), the policy network parameters are updated to obtain... .

[0238] S3: Reuse highly relevant paths Model training low-association paths

[0239] Both low-association paths and high-association paths share the characteristic that the DQN model aims to solve the coverage path optimization problem, and the DQN model structures for both types of path sets are identical. Therefore, by adjusting the parameters, the high-association paths can be optimized. The model is used to train low-association paths Model.

[0240] This invention has low correlation The model is still constructed strictly according to steps S2.2.1 to S2.2.2.

[0241] S3.1: Filtering high-quality sample states based on low-association path features

[0242] (1) Calculate the comprehensive index of sample status

[0243] Training low-association paths The initial sample state set method for the model follows the steps in S2.1, but the original method of selecting and training highly correlated samples is used. Based on the comprehensive index ranking formula of the initial sample state set of the model, a fourth index is added to generate a new comprehensive index ranking update formula:

[0244] (17)

[0245] in: Indicates the use of highly related paths The model calculates the sample state. The value is normalized and then the complement is taken.

[0246] (2) value normalization

[0247] Highly related paths The model will predict the state of each sample. Find the maximum value among all actions. The value is denoted as and minimum The value is denoted as ,calculate The normalization formula is:

[0248] (18)

[0249] For example: Sample status The action to be chosen is -5, in middle , The value is 22.57, which is the highest among all sample states. , Then, the sample state is calculated according to equation (18). Thus we obtain .

[0250] (3) Recalculation of weights

[0251] To improve the overall ranking, different weights were assigned to each of the four evaluation indicators. The importance of selecting high-quality samples varies, and in order to obtain reasonable weights, this invention selects N samples and calculates their comprehensive scores.

[0252] First, randomly select N samples, assuming... The average value of the comprehensive index of N states is calculated based on equation (17), and denoted as . Then, delete one evaluation indicator at a time, keeping the remaining three indicators. Set the weights of these three indicators to 0.33 each, and calculate the average of the comprehensive indicators for the N sample states, denoted as:

[0253] , , , .

[0254] The formula for estimating weights based on sample states is as follows:

[0255] (19)

[0256] Just like S2.1 before, here it is... For example, when all four indicators are present, this invention first sets the weight to 0.25, and calculates the average value of the first 100 sample state indicators to 0.651. When one indicator is removed and the weight is set to 0.33, the comprehensive indicator values ​​of the first 100 sample states are as follows:

[0257] Delete : ; Delete :

[0258] Delete : ; Delete :

[0259] The following calculation sums the differences between the mean scores when each indicator is missing and when all four indicators are present. The weights are then calculated by dividing each individual difference by the sum of the differences. Therefore, the weights are set as follows:

[0260]

[0261] S3.2: High-quality sample state training based on low-association paths Model

[0262] Sample status This refers to a specific input data set for the program; the input dataset is the sample state set, used to construct low-association paths. The model sample state set is a new sample state selected using a new comprehensive index ranking formula.

[0263] The value is a low-association path. The estimated value of the cumulative future rewards that the model expects to obtain.

[0264] The experience pool is a buffer that stores historical interaction experiences (states, actions, rewards, etc.) of the DQN model. It uses a certain strategy to adopt sample correlation and reuse data.

[0265] Fine-tuning strategy network parameters The process and training of highly correlated paths The model is the same, extraction Each experience is calculated. and Then calculate the loss function for each experience. Finally, the updated policy network parameters are calculated. .

[0266] Finally, based on The model selects sample data as the initial population for PSO.

[0267] Continue with Figure 2 The example program will be used to illustrate this. Let's assume the original... Model parameters are Still sampling 6 experiences from Table 4, the first experience sample state , The action selected by the model for this sample state is The +5 (action 2) update the state, and the sample state becomes... Its traversal path is: Sample status The target path is: According to calculation Equation (9) shows their reward value is 0.46. Assuming the discount factor is 0.99, the predicted reward for all actions of the policy network is... The maximum value is 25.89, representing the predicted values ​​for all actions of the target network. The maximum value is 27.20. Therefore, according to equation (12)... And according to equation (13), the target network is used for calculation. The value is 27.39, which can be calculated using the loss function (14): Because of the previous The model parameter is 0.850, and the learning weights are assumed to be... The value is 0.01, according to equation (15) Similarly, the gradient of the remaining experiences is , obtained Then the updated parameters according to equation (16) are:

[0268] S3.3: Based on Model and The model selects sample data with high reward values.

[0269] For S2.3 trained The model and S3.2 trained The model, in this invention, will use the corresponding experience from each experience in the experience pool. Reward value is used as an evaluation metric. (Through...) The reward values ​​are used to prioritize the experience data, selecting high-reward data from the pre-trained experience pool. The sample data of the values ​​are used as the initial population of PSO.

[0270] S4: Test cases for initial population generation coverage paths of PSO based on high reward value sample data.

[0271] S4.1: Define the fitness values ​​for test cases covering the path;

[0272] set up For the target path, use the test cases cover Let the objective function be denoted as . .

[0273] When a test case Executing the program under test yields the following traversal path: .

[0274] Based on the target path (1) and The similarity is used as the objective function , can be represented as:

[0275] (20)

[0276] As shown in the above formula, if a test case can cover the target path, then the traversal path is the same as the target path. That is Take the maximum value.

[0277] Therefore, the problem of generating test cases for path coverage can be transformed into the problem of finding the maximum value of the objective function.

[0278] Based on equation (20), the objective function is defined as the fitness function, expressed as:

[0279] (twenty one)

[0280] S4.2 Dynamic detection of particle swarms trapped in local optima based on fitness value calculation of comprehensive coefficient of variation;

[0281] First, based on step S2.4 or S3.3, for the target path ,get The initial particle swarm constructed from 100 particles is denoted as _p_s_i ... .

[0282] Then, during the PSO evolution process, it is necessary to determine whether the particle swarm has identified a local optimum.

[0283] The comprehensive coefficient of variation in this invention is calculated using the following formula:

[0284] (twenty two)

[0285] in:

[0286] Indicates the number of particles in the subpopulation; The fitness value of a particle is represented by equation (21); This represents the average fitness value of all particles in the subpopulation.

[0287] The numerator: Unlike the ordinary coefficient of variation, it consists of two components that form the square root.

[0288] First item: It is the variance of the fitness values ​​of particles in the subpopulation;

[0289] Second item: It is the mean of the fitness differences between adjacent particles in the subpopulation; denominator: That is, the average fitness value of particles in the subpopulation (the overall average of the response fitness value), and The purpose is to balance the threshold expansion problem caused by the fusion of the two molecular components.

[0290] In detecting whether particles are trapped in local optima using the particle swarm optimization algorithm, this invention uses a comprehensive coefficient of variation. The ordinary fitness coefficient of variation is the ratio of the standard deviation of the fitness values ​​of all particles to the mean, and it only reflects the overall dispersion of the fitness values ​​of particles within a subpopulation; while the comprehensive coefficient of variation is a combination of the fitness coefficient of variation and adjacent fluctuation terms. By introducing adjacent difference terms, it can simultaneously assess the local clustering and continuity of particle distribution.

[0291] The core value of the overall coefficient of variation lies in providing a dual assessment of the subpopulation state: the global standard deviation measures the overall dispersion of fitness values ​​across all particles, while the neighboring difference term captures jumps in fitness values ​​between neighboring particles. This design allows it to sensitively identify local optimum traps. When particles cluster near a local optimum, the overall standard deviation may be small, but if there are local gradient changes in fitness values, the neighboring difference term effectively captures this pattern.

[0292] When the overall variation coefficient of the subpopulation Less than the preset threshold When this happens, the subpopulation is determined to be trapped in a local optimum.

[0293] S4.3 proposes an inverse particle swarm optimization algorithm to improve search performance for the local optimum problem;

[0294] To address the problem that the PSO algorithm is prone to getting trapped in local optima, this invention employs the inverse particle swarm optimization algorithm.

[0295] Let the range of values ​​for each dimension of the program input variable be an interval. Inside. Let the iteration reach the... Next time, the Each particle is denoted as: ,in Indicates the first In the iteration, the first The first particle Dimensional variables.

[0296] When this invention determines that the subpopulation is trapped in a local optimum, from Particles are randomly selected from each subpopulation. Perform the reverse operation on each particle. Calculate The formula for the value is:

[0297] (twenty three)

[0298] particle The Dimensional variable values Calculate the reversed new variable value using the following formula:

[0299] (twenty four)

[0300] in: It refers to the boundary of particle values. This represents the value of the dimension corresponding to the particle. This represents the disturbance value.

[0301] The inverse variable value for each dimension is calculated using equation (24), resulting in... We take the inverse variable value and then replace the corresponding dimension variable value in the original particle to obtain... A new particle;

[0302] From the above One new particle is randomly selected to replace the original particle. This is to achieve particle position updates.

[0303] continue Figure 2 The example program will be used to illustrate this.

[0304] Let the target path corresponding to the current particle swarm be... The particles within this subpopulation have respectively According to equation (21), their fitness values ​​are respectively , , .

[0305] The comprehensive coefficient of variation of this subpopulation is calculated according to equation (22). .

[0306] After obtaining empirical values ​​through multiple experiments, let... .because If the subpopulation of the present invention is found to be trapped in a local optimum, then the inverse particle swarm optimization algorithm needs to be used.

[0307] According to equation (23). Randomly select from subpopulation one Perform the reverse operation on each particle, that is, select and ;

[0308] Depend on Figure 2 As shown in the example program, the range of each dimension of the program's input variables is [1, 128]. Empirical values ​​were obtained through multiple trials. The range of values ​​is .at this time The range of values ​​is Randomly generated

[0309] With the current particle For example, the reverse coordinates of each dimension of the particle are calculated using equation (24): , , Corresponding The new particles with opposite coordinates are: , , , , , , Randomly select a reverse coordinate Replace the original .

[0310] S4.4 Inverse Particle Swarm Optimization Algorithm generates test cases for covered paths.

[0311] Input: Path set; DQN model;

[0312] Output: Test dataset for the covered paths;

[0313] S4.4.1: Set the various parameter values ​​for the particle swarm optimization algorithm;

[0314] S4.4.2: Randomly generated initial candidate test data, which are then input into the DQN model for evaluation;

[0315] Based on the results selected from the experience pool by the DQN model, test data with high fitness values ​​are selected to form the initial particle swarm;

[0316] S4.4.3: Particle Swarm Each particle in the test executes the program being tested;

[0317] S4.4.4: If a certain particle Covered path Save test data and marking Overwritten and stopped The evolution of the corresponding subpopulation;

[0318] For uncovered paths, when the maximum number of iterations has not been reached, the corresponding particle swarm implements the PSO evolutionary strategy: calculating fitness values, updating individual values ​​and the global optimum;

[0319] S4.4.5: Calculate the overall coefficient of variation When the overall coefficient of variation Less than the preset threshold If the current particle swarm is found to be trapped in a local optimum, the following operations are performed;

[0320] Select from the current particle swarm Each particle is the target that needs to be replaced in this iteration;

[0321] Then, the reverse operation is performed to generate new particles to replace the selected particles; a new particle swarm is then formed.

[0322] S4.4.6: If the path set If all paths are marked as covered, proceed to step 4.4.8;

[0323] S4.4.7: Iterate continuously. If the maximum number of iterations has not been reached, execute step S4.4.3 after completing the particle reverse operation and updating the particle swarm.

[0324] S4.4.8: Output the complete test dataset when all paths have found covered use cases or the maximum number of iterations has been reached.

Claims

1. A robot control software testing method integrating DQN and PSO, characterized in that, Includes the following steps: S1: Grouping highly correlated and low-correlation paths based on path similarity; S2: Co-construction based on highly correlated paths Model; S2.1: Filter high-quality sample states based on highly correlated path features; S2.2: Determining the state based on high-quality samples The initial experience pool of the model; S2.3: Training based on the initial experience pool Model; S2.3.1: Select high-priority experiences from the initial experience pool; S2.3.2: Iterative training based on high-priority experience Model; S3: Reuse highly associated paths Low-association paths in model training Model; S3.1: Filter high-quality sample states based on low-association path features; S3.2: High-quality sample state training based on low-association paths Model; S3.3: Based on Model and The model selects sample data with high reward values. S4: Test cases for initial population generation coverage paths based on high reward value sample data for PSO; S4.1: Define the fitness values ​​for test cases covering the path; S4.2: Dynamic detection of particle swarms trapped in local optima based on fitness value calculation of comprehensive coefficient of variation; S4.3: For the local optimum problem, an inverse particle swarm optimization algorithm is proposed to improve search performance; S4.4: Inverse particle swarm optimization algorithm generates test cases for covered paths.

2. The method according to claim 1, characterized in that, Step S2.1, which involves selecting high-quality sample states based on highly correlated path features, specifically includes: Set highly associated paths The set of input sample states for the model is the set of input data for the program. , After executing the program under test, a set of traversal paths is obtained, denoted as... ; Determined status Corresponding traversal path and target path Similarity Normalized value of the difference in length between the traversing path and the target path and state perturbation stability Three evaluation indicators: For the traversal path With the target path Similarity; yes and The normalized value of the path length difference is: ;in It is a path Length is also the number of nodes contained in the path; The target path Length; It is the sample state The perturbation stability; assuming the sample state Include Dimensional variables ; In order to calculate ,first Each dimension of variable Nearby positive and negative perturbations were obtained respectively. and ,in For the first The perturbation value of the dimension; then, that is... The original data and the data after positive and negative perturbations are used to obtain the following through mathematical permutations and combinations: New status; set up The path traversed is the target path. ,So, The traversal path corresponding to the new state and The average similarity is ; Define the state based on three indicators. Comprehensive indicators ;in The following determines reasonable weight values ​​for the three indicators: First, random selection Individual sample states; assumption ,calculate The average of the three indicators for each sample state is denoted as . ; Then, from the three indicators, one evaluation indicator is deleted in turn, and the weights of the remaining two indicators are each set to 0.

50. The results are then recalculated. The average value of the comprehensive index of each sample state is denoted as: , , ; The formula for estimating weights is as follows: ； Calculate the comprehensive index for each sample based on weights, then sort them from largest to smallest, and select... Individual sample status Used for training highly correlated paths Model.

3. The method according to claim 1, characterized in that, Step S2.2.2 calculates the priority of each experience based on the current state and action, specifically as follows: In experience middle, Measured by path similarity: set up The optimized task objective is to have the target path as ;state The corresponding traversal path is , and similarity ,So When the traversal path can completely cover the target path, take... ; otherwise ; The value range is greater than 1; In the model, predictions based on policy networks The value is denoted as ; Target network prediction The value is denoted as ;in For state The acquired action; Defined as the priority of each experience tuple in the experience pool. for: 。 4. The method according to claim 1, characterized in that, Step S2.3.1, which involves selecting high-priority experiences from the experience pool, specifically involves: In experience In China, experience-based priority Calculate the sampling probability as for: in This is the sum of all experience points in the experience pool. Control priority intensity, It degenerates into uniform sampling at times. Sampling is performed entirely according to priority. After determining the sampling probability of each experience, samples are drawn from the experience pool. One experience point for training Model and calculate the loss function; The size is determined by the capacity of the experience pool.

5. The method according to claim 1, characterized in that, Step S3.1, which involves screening high-quality sample states based on low-association path features, specifically includes: calculating a comprehensive index of sample states. Low-association paths The initial sample state set of the model follows the steps in S2.1, but with the original selection of highly correlated training samples... Based on the comprehensive index ranking formula of the initial sample state set of the model, a fourth index is added to generate a new comprehensive index ranking update formula: in Indicates the use of highly related paths The model calculates the sample state. After normalizing the value, take the complement value; Value normalization methods: High-relevance paths The model will predict the state of each sample. Find the maximum value among all actions. The value is denoted as and minimum The value is denoted as ,calculate The normalization formula is: 。 6. The method according to claim 1, characterized in that, Step S4.2, which calculates the comprehensive coefficient of variation based on fitness values ​​to dynamically detect when the particle swarm is trapped in a local optimum, is specifically executed as follows: First, based on highly associated paths Model or low-association path Model, targeting the target path ,get The initial particle swarm constructed from 100 particles is denoted as _p_s_i ... ; Then, in the PSO evolution process, the comprehensive coefficient of variation is used to determine whether the particle swarm has a local optimum, as follows: in Indicates the number of particles in the subpopulation; This represents the fitness value of a particle. This represents the average fitness value of all particles in the subpopulation. The numerator: Unlike the ordinary coefficient of variation, it consists of two components that form the combined square root; First item: It is the variance of the fitness values ​​of particles in the subpopulation; Second item: It is the mean of the fitness differences between adjacent particles in the subpopulation; denominator: This is the average fitness value of particles in the subpopulation.

7. The method according to claim 1, characterized in that, Step S4.3, the inverse particle swarm optimization algorithm, solves local optima, specifically as follows: Let the range of values ​​for each dimension of the program input variables be defined. Iterate to the ; Next time, the Each particle is denoted as: ,in Indicates the first The first particle Dimensional variable values; When the subpopulation gets stuck in a local optimum, from Particles are randomly selected from each subpopulation. Each particle is reversed; particle The Dimensional variable values Calculate the reversed new variable value using the following formula: in This refers to the boundary values ​​of the particles; This represents the value of the dimension corresponding to the particle; Indicates the disturbance value; For each dimension of the particle, the inverse variable value is calculated, and the result is obtained. We take the inverse variable value and then replace the corresponding dimension variable value in the original particle to obtain... A new particle; From the above One new particle is randomly selected to replace the original particle. This is to achieve particle position updates.

Images