Path information generation model training method, electronic equipment and storage medium

Abstract

The invention provides a path information generation model training method. The method comprises the following steps: acquiring a sample path sequence information set as a sample data set; inputting the training sample data of the current batch into the current path information generation model to obtain a corresponding current exchange weight set, and obtaining a current action corresponding to the training sample data to obtain a corresponding current state and a current reward; setting c1 to be equal to c1 + 1, if c1 is smaller than or equal to n, entering the next step, otherwise, setting c2 to be equal to c2 + 1, if c2 is smaller than or equal to C0, entering the previous step, and otherwise, taking the current path information generation model as a target path information generation model; on the basis of the current exchange weight corresponding to the current action corresponding to each piece of training sample data and the previous exchange weight, obtaining an importance coefficient corresponding to the sample data, updating the importance coefficient, taking the next batch of sample data as the current training sample data, and entering the previous step. The method can improve the calculation efficiency.

Description

Technical Field

[0001] The present invention relates to the field of information processing, and in particular to a path information generation model training method, electronic equipment and storage medium. Background Art

[0002] In some application scenarios, it is necessary to sort the sequences containing multiple given coordinate information to obtain the optimal path sequence with the optimal distance. The current method to obtain the optimal path sequence is generally to put the input sequence into a scoring model. The model will assign a score to each object according to the characteristics of the sequence object, and then obtain a permutation by mapping the score to the indicator function. The scoring model aims to score each object according to the characteristics of the object. The indicator function aims to convert the object set into a permutation sequence through the object score. The permutation learning process has the problem that the sorting process itself is discontinuous due to the discrete characteristics of the indicator function, and the sorting result is biased due to the limitations of the approximation function itself, because a slight change in the object score may lead to a drastic change in the permutation result, which indicates that there is a problem that the score is closely related to the permutation position. The existing sorting generation model generates one permutation at a time, and the search space of large-scale problems grows exponentially, which leads to high computational complexity. Therefore, when facing large-scale permutation problems, it is very difficult to generate an approximate optimal solution at a time. Summary of the invention

[0003] In view of the above technical problems, the technical solution adopted by the present invention is:

[0004] According to a first aspect of the present invention, a path information generation model training method is provided, the method comprising the following steps:

[0005] S100, obtaining a sample path sequence information set I as a sample data set, where I=(I 1 , I 2 , ..., I i , ..., I m ), I i is the i-th sample path sequence information set in I, i ranges from 1 to m, m is the number of sample path sequence information in I, I i =(I i1 , I i2 , ..., I it , ..., I in ), I it For I i The t-th path sequence information in I, where t ranges from 1 to n, and n is the number of time steps; it =(S it , A it , r it , P it ), Sit For I it The state at time t, S it ={(x i1 , π i1 t ), (x i2 , π i2 t ),……,(x ij , π ij t ),……,(x if(i) , π if(i) t )},x ij For I i The coordinate information of the jth path node among the corresponding f(i) path nodes, π ij t is the access position corresponding to the jth path node at time t, and the value of j ranges from 1 to f(i); A it For I it The action at time t, r it For I it The reward at time t, P it For I it The initial exchange weight set of all actions corresponding to time t, P it = {P it hr} h，r∈(1,2，……，f(i)) , P it hr For I it The initial exchange weights corresponding to the actions corresponding to any exchange of the hth path node and the rth path node at time t, the values ​​of h and r are 1 to f(i); where P it hr = z(Δf it hr ), △f it hr is the feature difference between the hth path node and the rth path node, and z() is a preset function expression.

[0006] S200, input the current batch of training sample data into the current path information generation model to obtain the corresponding current exchange weight set, and obtain the current action corresponding to the training sample data based on the current exchange weight set corresponding to each training sample data, and obtain the corresponding current state and current reward; set c1=c1+1, if c1≤n, execute S300, otherwise, set c2=c2+1, if c2≤C0, execute S200, otherwise, execute S400; c1 is the first counter, the initial value of c1 is 0, c2 is the second counter, the initial value of c2 is 0, and C0 is the preset iteration number threshold.

[0007] S300, based on the current exchange weight and the previous exchange weight corresponding to the current action corresponding to each training sample data, obtain the importance coefficient corresponding to the sample data, and update the parameters of the current path information generation model based on the currently obtained importance coefficient, and use the next batch of sample data as the current training sample data to execute S200.

[0008] S400: Using the current path information generation model as the target path information generation model.

[0009] According to a second aspect of the present invention, there is provided an electronic device comprising a processor and a memory; the processor is used to execute the steps of the method described in the first aspect of the present invention by calling a program or instruction stored in the memory.

[0010] According to a second aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program or instructions, wherein the program or instructions enable a computer to execute the steps of the method described in the first aspect of the present invention.

[0011] The present invention has at least the following beneficial effects:

[0012] The path information generation model training method provided by the embodiment of the present invention can improve the speed of obtaining the optimal solution in the path information generation process and improve the generation efficiency of the path information because the exchange weight corresponding to each node is considered when selecting an action.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 A flow chart of a path information generation model training method provided by an embodiment of the present invention;

[0016] Figure 2 A structural block diagram of a path information generation model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0019] It should be noted that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0020] The embodiment of the present invention provides a path information generation model training method, which can be applied to traveling salesman problem, vehicle routing problem, etc.

[0021] like Figure 1 As shown, the method may include the following steps:

[0022] S100, obtaining a sample path sequence information set I as a sample data set, where I=(I 1 , I 2 , ..., I i , ..., I m ), I i is the i-th sample path sequence information set in I, i ranges from 1 to m, m is the number of sample path sequence information in I, I i =(I i1 , I i2 , ..., I it , ..., I in ), I it For I i The t-th path sequence information in I, where t ranges from 1 to n, and n is the number of time steps; it =(S it , A it , r it , P it ), Sit For I it The state at time t, S it ={(x i1 , π i1 t ), (x i2 , π i2 t ),……,(x ij , π ij t ),……,(x if(i) , π if(i) t )},x ij For I i The coordinate information of the jth path node among the corresponding f(i) path nodes, π ij t is the access position corresponding to the jth path node at time t, and the value of j ranges from 1 to f(i); A it For I it The action at time t, r it For I it The reward at time t, P it For I it The initial exchange weight set of all actions corresponding to time t, P it = {P it hr} h，r∈(1,2，……，f(i)) , P it hr For I it At time t, the initial exchange weights corresponding to the actions corresponding to any exchange of the hth path node and the rth path node are taken from 1 to f(i), and f(i) = n!.

[0023] In an embodiment of the present invention, the coordinate information of a node may be the latitude and longitude information of the node on a map. The access position indicates the order of access. For example, for a path sequence 54321 having 5 nodes, it indicates that the first node visited is the 5th node, the second node visited is the 4th node, the third node visited is the 3rd node, the fourth node visited is the 2nd node, and the fifth node visited is the 1st node. The path sequence corresponding to the state in the path sequence information is a loop, which means that a certain node is used as the starting node, each node is visited once, and finally returns to the starting node. For example, for the path sequence 54321, it indicates that the 5th node is used as the starting node, and the 4th node, the 3rd node, the 2nd node and the 1st node are visited in sequence, and finally returns to the 5th node.

[0024] In this embodiment of the present invention, the reward r corresponding to the current time t t The following conditions can be met: t=D(s t * )-min(D(s t+1 * ), D(s t * )), where D(s t * ) is the best solution that can be found before the current moment, that is, the shortest path distance that can be found, D(s t+1 * ) is the optimal solution that can be found at the next moment.

[0025] The reward calculation method in the embodiment of the present invention can ensure that the cumulative reward is equal to the total reduced cost of the initial solution. Therefore, the reward r is increased if and only if a better solution is found. t >0.

[0026] In an exemplary embodiment of the present invention, n coordinates may be randomly generated in the interval [0, 1] as nodes corresponding to each sample path sequence information, and an initial solution, ie, an action, may be randomly generated.

[0027] In another illustrative embodiment of the present invention, a randomly generated initial path sequence information can be input into the initial path information generation model to obtain the next path sequence information. Then, the next path sequence information is further input into the initial path information generation model to obtain another path sequence information. By analogy, n path sequence information can be obtained.

[0028] In an embodiment of the present invention, the path information generation model is a reinforcement learning model. Figure 2 As shown, the path information generation model includes a permutation-equivariant node embedding representation network, a permutation-invariant interactive feature strategy network, and an evaluation network. The input of the node embedding representation network is the state, and the output of the node embedding representation network is the input of the interactive feature strategy network and the evaluation network, respectively.

[0029] Furthermore, if Figure 2 As shown, the node embedding representation network 1 includes a first embedding representation module and a second embedding representation module. The first embedding representation module and the second embedding representation module have the same structure, both including a first linear layer, a self-attention mechanism layer, a second linear layer, a feedforward neural network layer, a third linear layer and a normalization layer connected in sequence. The second embedding representation module includes a first linear layer, a self-attention mechanism layer, a second linear layer and a normalization layer connected in sequence.

[0030] Further, in an embodiment of the present invention, the output of the node embedding representation network includes a coordinate information embedding feature and an access location embedding feature, wherein the coordinate information embedding feature is obtained by the following steps:

[0031] S10, inputting the coordinate information input into the first embedding representation module into the first linear layer of the first embedding representation module for feature extraction to obtain a corresponding first output feature Fh1.

[0032] S11, input Fh1 into the self-attention mechanism layer of the first embedding representation module to obtain the corresponding self-attention coefficient Kh.

[0033] S12, concatenate Fh1 and Kh, and input the concatenated features into the second linear layer of the first embedding representation module for feature extraction to obtain the corresponding second output feature Fh2.

[0034] S13, input Fh2 into the feedforward neural network layer of the first embedding representation module for feature extraction to obtain the corresponding feedforward feature FFh2.

[0035] S14, input FFh2 into the normalization layer of the first embedding representation module to obtain the corresponding normalized features as the coordinate information embedding features.

[0036] The access location embedding feature is obtained by the following steps:

[0037] S20, inputting the coordinate information input into the first embedding representation module into the first linear layer of the first embedding representation module for feature extraction to obtain a corresponding first output feature Fh1.

[0038] S21, input Fg1 into the self-attention mechanism layer of the second embedding representation module to obtain the corresponding self-attention coefficient Kg.

[0039] S22, concatenate Fg1 and Kg, and input the concatenated features into the second linear layer of the second embedding representation module for feature extraction to obtain the corresponding second output feature Fg2.

[0040] S23, input Fg2 into the feedforward neural network layer of the second embedding representation module for feature extraction to obtain the corresponding feedforward feature FFg2.

[0041] S24, input FFg2 into the normalization layer of the second embedding representation module to obtain the corresponding normalized features as the access position embedding features.

[0042] Those skilled in the art know that the structures of the linear layer, the self-attention mechanism layer, the feedforward neural network layer and the normalization layer may be existing structures, and their specific working principles may also refer to existing working principles.

[0043] In the embodiment of the present invention, the technical effect of using permutation equivariant nodes to embed the network is that it can maintain the ability to perceive the order of input nodes, that is, when the permutation of the input nodes changes, the characteristics of the network output also change accordingly. Using self-attention for node embedding makes the network naturally have permutation equivariance. During the node embedding process, each node only relies on its own attributes to update its representation, rather than through interactions with other nodes. This method emphasizes the independence of nodes, reduces external influences, and makes the model more robust when dealing with noisy data or outliers.

[0044] Furthermore, if Figure 2 As shown, the interactive feature strategy network is used to output the corresponding action in the current state based on the received input information, and may include a first interactive feature strategy module, a second interactive feature strategy module, a feedforward neural network layer, an exchange weight calculation module and an action acquisition module. The first interactive feature strategy module and the second interactive feature strategy module have the same structure, and both include a self-attention mechanism layer, a normalization layer, a cross-attention mechanism layer, a maximum pooling layer, and a multi-layer perceptron.

[0045] Furthermore, the corresponding action in the current state can be obtained through the following steps:

[0046] S30, input the coordinate information embedding feature output by the first embedding representation module into the self-attention mechanism layer of the first exchange feature strategy module to obtain the corresponding self-attention coefficient Ks1, and input the access position embedding feature output by the first embedding representation module into the self-attention mechanism layer of the second exchange feature strategy module to obtain the corresponding self-attention coefficient Ks2.

[0047] S31, input Ks1 to the normalization layer of the first exchange feature strategy module to obtain the corresponding normalized feature Ns1, and input Ks2 to the normalization layer of the second exchange feature strategy module to obtain the corresponding normalized feature Ns2.

[0048] S32, input Ns1 and Ks2 into the cross-attention mechanism layer of the first exchange feature strategy module to obtain the corresponding cross-attention coefficient KC1, and input Ns2 and Ks1 into the cross-attention mechanism layer of the second exchange feature strategy module to obtain the corresponding cross-attention coefficient KC2.

[0049] S33, concatenate Ks1 and KC1 to obtain the corresponding concatenation feature as the coding feature FC1 of the coordinate information, and concatenate Ks2 and KC2 to obtain the corresponding concatenation feature as the coding feature FC2 of the access position.

[0050] S34, input FC1 into the maximum pooling layer of the first exchange feature strategy module to obtain the corresponding pooling feature FP1, and input FC2 into the maximum pooling layer of the second exchange feature strategy module to obtain the corresponding pooling feature FP2.

[0051] S35, the added feature obtained by adding FC1 and FP2 is input into the multilayer perceptron of the first exchange feature strategy module to obtain the corresponding multi-head attention coefficient MAK1, and the added feature obtained by adding FC2 and FP1 is input into the multilayer perceptron of the second exchange feature strategy module to obtain the corresponding multi-head attention coefficient MAK2.

[0052] S36, concatenating MAK1 and MAK2 and inputting the concatenated result into the feedforward neural network layer of the interactive feature strategy network to obtain corresponding feedforward features.

[0053] S37, normalizing the obtained feedforward features to obtain corresponding feature difference information and inputting it into the exchange weight calculation module, wherein the feature difference information includes the feature difference between any two nodes in the corresponding nodes.

[0054] That is, in the embodiment of the present invention, Δf it hr =FFA(MHA(Maxpooling(out it ha )+out it hg ), MHA(Maxpooling(out it ra )+out it rg )),

[0055] Among them, out it ha is the encoding feature of the coordinate information corresponding to the hth node, out it hg is the encoding feature of the access position corresponding to the hth node, out it ra is the encoding feature of the coordinate information corresponding to the rth node, out it rg is the encoding feature of the access position corresponding to the rth node, is the attention coefficient obtained after normalizing the attention coefficient between the embedded features of the coordinate information corresponding to the p-th node and the s-th node in the f(i) nodes, p is h or r, is the attention coefficient obtained by normalizing the attention coefficient between the embedded features of the access position corresponding to the p-th node and the s-th node in the f(i) nodes, a it s is the embedding feature of the coordinate information corresponding to the sth node, g it s is the embedded feature of the access position corresponding to the sth node, MAxPooling() represents the maximum pooling, and contact() represents the concatenation operation.

[0056] S38, the exchange weight calculation module obtains the corresponding exchange weight set based on the feature difference information, and inputs it to the action acquisition module.

[0057] In the embodiment of the present invention, P it hr = z(Δf it hr ), △f it hr is the feature difference between the hth path node and the rth path node, and z() is a preset function expression.

[0058] In an exemplary embodiment, P it hr =1 / (1+exp(-△f it hr )(π ih t -π ir t )) / β, exp() represents an exponential function with the natural number e as the base, β is a hyperparameter, which can be an empirical value. The larger β is, the more likely different positions are to be selected, and the larger the exploration range is. The smaller β is, the more different probabilities of different positions being selected are, and the smaller the exploration range is.

[0059] S39, the action acquisition module acquires an exchange weight as a target exchange weight based on the currently received exchange weight set, and exchanges the access positions corresponding to the two path nodes corresponding to the target exchange weight to obtain the current action.

[0060] In the embodiment of the present invention, the action acquisition module acquires an exchange weight as the target exchange weight from the current exchange weight set based on a polynomial sampling method.

[0061] Those skilled in the art are aware that any method based on polynomial sampling to obtain an exchange weight from the current exchange weight set as the target exchange weight, and to exchange the access positions corresponding to the two path nodes corresponding to the target exchange weight to obtain the current action falls within the scope of protection of the present invention.

[0062] The permutation-invariant interactive feature strategy network in the embodiment of the present invention is used to ensure that the network output does not depend on the order of input nodes. When dealing with existing loop problems such as the traveling salesman problem, the permutation-invariant network can ensure the consistency and fairness of the solution. Regardless of the order of input cities, the same result needs to be obtained because in the traveling salesman problem, the final total length of the journey should be independent of the input order of the cities.

[0063] Furthermore, in an embodiment of the present invention, the evaluation network is used to output the value, and may include a multi-layer perceptron, an average pooling layer, a first linear layer, a second linear layer, and a fully connected layer. The output of the multi-head attention mechanism layer of the evaluation network is respectively used as the input of the average pooling layer and the first linear layer of the average network, the sum of the output of the first linear layer of the evaluation network and the output of the evaluation pooling layer is used as the input of the second linear layer of the evaluation network, the output of the second linear layer of the evaluation network is used as the input of the fully connected layer of the evaluation network, and the output of the fully connected layer is the value.

[0064] In an embodiment of the present invention, the input of the evaluation network includes ((h1, g1), (h2, g3), ..., (hu, gu), ..., (hx, gx)), hu and gu are respectively the coordinate information embedding features and access position embedding features corresponding to the u-th node in any path information, u ranges from 1 to x, and x is the number of nodes corresponding to the path information. In an embodiment of the present invention, directly combining the coordinate information corresponding to a node with its corresponding access position and comparing it with the combination corresponding to other nodes can help the model better understand the spatial relationship between nodes. Compared with processing coordinate information or access position separately, this combination can provide richer contextual information.

[0065] S200, input the current batch of training sample data into the current path information generation model to obtain the corresponding current exchange weight set, and obtain the current action corresponding to the training sample data based on the current exchange weight set corresponding to each training sample data, and obtain the corresponding current state and current reward; set c1=c1+1, if c1≤n, execute S300, otherwise, set c2=c2+1, if c2≤C0, execute S200, otherwise, execute S400; c1 is the first counter, the initial value of c1 is 0, c2 is the second counter, the initial value of c2 is 0, C0 is the preset iteration number threshold, which can be an empirical value.

[0066] In the embodiment of the present invention, the training sample data of each batch may be set based on actual needs, for example, the path sequence information corresponding to each time step.

[0067] In the embodiment of the present invention, the acquiring of the current action corresponding to each training sample data based on the current exchange weight set corresponding to the training sample data specifically includes:

[0068] Based on the polynomial sampling method, an exchange weight is obtained from the current exchange weight set as the target exchange weight, and the access positions corresponding to the two path nodes corresponding to the target exchange weight are exchanged to obtain the current action.

[0069] Those skilled in the art are aware that any method based on polynomial sampling to obtain an exchange weight from the current exchange weight set as the target exchange weight, and to exchange the access positions corresponding to the two path nodes corresponding to the target exchange weight to obtain the current action falls within the scope of protection of the present invention.

[0070] S300, based on the current exchange weight and the previous exchange weight corresponding to the current action corresponding to each training sample data, obtain the importance coefficient corresponding to the sample data, and update the parameters of the current path information generation model based on the currently obtained importance coefficient, and use the next batch of sample data as the current training sample data to execute S200.

[0071] In the embodiment of the present invention, the importance coefficient ratios corresponding to each training sample data satisfies the following condition: ratios = P c / P f , where P c is the current exchange weight corresponding to the current action corresponding to the training sample data, P f It is the previous exchange weight corresponding to the training sample data, that is, the exchange weight corresponding to the previous batch of sample data.

[0072] Furthermore, in an embodiment of the present invention, the parameter update of the policy network satisfies the following conditions:

[0073]

[0074] Among them, θ is the parameter of the policy network, η θ is the learning rate of the policy network, is the gradient of the policy network, Represents the gradient.

[0075] In the embodiment of the present invention, the PP02 algorithm can be used to obtain the gradient of the policy network. Specifically, in the embodiment of the present invention, J ppo (θ)=(∑ m j＝1 min(ratios j t A j t-1,clip(ratios j , 1-ε, 1+ε))A j t ) / m, where ratios j t A is the importance coefficient corresponding to the jth training sample data in the current batch of training sample data. The value of j ranges from 1 to m, where m is the number of training sample data in each batch. j t is the current advantage function corresponding to the jth training sample data, A j t-1 is the previous advantage function corresponding to the j-th training sample data, that is, the advantage function corresponding to the previous training sample data corresponding to the j-th training sample data, A j t =R t j -V(S j t ), R t j is the cumulative reward corresponding to the jth training sample data, V(S j t ) is the value of the jth training sample data obtained by the evaluation network at the current moment, R t j =r j t +γV(S j t+1 ), V(S j t+1 ) evaluates the value of the j-th training sample data obtained by the network at the next moment, γ is the discount factor, and clip() is the clip function.

[0076] Furthermore, in an embodiment of the present invention, the parameter update of the evaluation network satisfies the following conditions:

[0077]

[0078] Among them, φ is the parameter of the evaluation network, η φ To evaluate the learning rate of the network, To evaluate the gradient of the network.

[0079] In the embodiment of the present invention, the PP02 algorithm can be used to obtain the gradient of the policy network. Specifically, L Critic (φ) satisfies the following conditions:

[0080] L Critic (φ)=(∑ m j＝1 ∣V(S jt )-r j t ∣ 2 ) / m

[0081] S400: Using the current path information generation model as the target path information generation model.

[0082] In a specific application process, the designated path node information including the coordinate information of the path node and the access position of the path node can be input into the target path generation model to obtain the corresponding path information.

[0083] The path information generation model training method provided by the embodiment of the present invention can efficiently optimize permutation problems of various sizes, and realizes effective solution space exploration through posterior probability calculation and multiple iterative updates. By adjusting the temperature using the hyperparameter β, the exploration range of the solution space can be expanded, and the algorithm optimization gradient can be used to ensure the stability and efficiency of the strategy update. In addition, through the effective use of historical data and the consideration of comprehensive optimization goals, significant optimization can be achieved in reducing costs, improving efficiency, and accelerating convergence time.

[0084] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the embodiment of the present invention.

[0085] The embodiment of the present invention further provides a non-transitory computer-readable storage medium storing computer-executable instructions, wherein the computer instructions are used to execute the method described in the embodiment of the present invention.

[0086] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0087] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A path information generation model training method, It is characterized in that The method comprises the following steps: S100, obtaining a sample path sequence information set I as a sample data set, where I=(I 1 , I 2 , ..., I i , ..., I m ), I i is the i-th sample path sequence information set in I, i ranges from 1 to m, m is the number of sample path sequence information in I, I i =(I i1 , I i2 , ..., I it , ..., I in ), I it For I i The t-th path sequence information in I, where t ranges from 1 to n, and n is the number of time steps; it =(S it , A it , r it , P it ), S it For I it The state at time t, S it ={(x i1 , π i1 t ), (x i2 , π i2 t ),……,(x ij , π ij t ),……,(x if(i) , π if(i) t )},x ij For I i The coordinate information of the jth path node among the corresponding f(i) path nodes, π ij t is the access position corresponding to the jth path node at time t, and the value of j ranges from 1 to f(i); A it For I it The action at time t, r it For I it The reward at time t, P it For I it The initial exchange weight set of all actions corresponding to time t, P it = {P it hr } h，r∈(1,2，……，f(i)) , P it hr For I it The initial exchange weights corresponding to the actions corresponding to any exchange of the hth path node and the rth path node at time t, the values ​​of h and r are 1 to f(i); where P it hr = z(Δf it hr ), △f it hr is the characteristic difference between the hth path node and the rth path node, z() is a preset function expression; S200, input the current batch of training sample data into the current path information generation model to obtain the corresponding current exchange weight set, and obtain the current action corresponding to the training sample data based on the current exchange weight set corresponding to each training sample data, and obtain the corresponding current state and current reward; set c1 = c1 + 1, if c1 ≤ n, execute S300, otherwise, set c2 = c2 + 1, if c2 ≤ C0, execute S200, otherwise, execute S400; c1 is the first counter, the initial value of c1 is 0, c2 is the second counter, the initial value of c2 is 0, and C0 is the preset iteration number threshold; S300, based on the current exchange weight and the previous exchange weight corresponding to the current action corresponding to each training sample data, obtain the importance coefficient corresponding to the sample data, and update the parameters of the current path information generation model based on the currently obtained importance coefficient, and use the next batch of sample data as the current training sample data to execute S200; S400: Using the current path information generation model as the target path information generation model.

2. The method according to claim 1, It is characterized in that P it hr =1 / (1+exp(-△f it hr )(π ih t -π ir t )) / β, exp() represents the exponential function with the natural number e as the base, and β is a hyperparameter.

3. The method according to claim 1, It is characterized in that The path information generation model is a reinforcement learning model.

4. The method according to claim 3, It is characterized in that The path information generation model includes a permutation-equivariant node embedding representation network, a permutation-invariant interactive feature strategy network, and an evaluation network; The input of the node embedding representation network is the state, and the output of the node embedding representation network is the input of the interactive feature strategy network and the evaluation network respectively; The node embedding representation network includes a first embedding representation module and a second embedding representation module. The first embedding representation module and the second embedding representation module have the same structure and both include a first linear layer, a self-attention mechanism layer, a second linear layer, a feedforward neural network layer, a third linear layer and a normalization layer connected in sequence. The second embedding representation module includes a first linear layer, a self-attention mechanism layer, a second linear layer and a normalization layer connected in sequence. The interactive feature strategy network includes a first interactive feature strategy module, a second interactive feature strategy module, a feedforward neural network layer, an exchange weight calculation module and an action acquisition module. The first interactive feature strategy module and the second interactive feature strategy module have the same structure and both include a self-attention mechanism layer, a normalization layer, a cross attention mechanism layer, a maximum pooling layer, and a multilayer perceptron. The evaluation network includes a multilayer perceptron, an average pooling layer, a first linear layer, a second linear layer and a fully connected layer. The outputs of the multilayer perceptron of the evaluation network are respectively used as inputs of the average pooling layer and the first linear layer of the average network. The sum of the output of the first linear layer of the evaluation network and the output of the evaluation pooling layer is used as the input of the second linear layer of the evaluation network. The output of the second linear layer of the evaluation network is used as the input of the fully connected layer of the evaluation network.

5. The method according to claim 4, It is characterized in that The output of the node embedding representation network includes a coordinate information embedding feature and an access location embedding feature, wherein the coordinate information embedding feature is obtained by the following steps: S10, inputting the coordinate information input into the first embedding representation module into the first linear layer of the first embedding representation module for feature extraction, and obtaining the corresponding first output feature Fh1; S11, input Fh1 into the self-attention mechanism layer of the first embedding representation module to obtain the corresponding self-attention coefficient Kh; S12, concatenating Fh1 and Kh, and inputting the concatenated features into the second linear layer of the first embedding representation module for feature extraction, to obtain the corresponding second output feature Fh2; S13, inputting Fh2 into the feedforward neural network layer of the first embedding representation module for feature extraction to obtain the corresponding feedforward feature FFh2; S14, inputting FFh2 into the normalization layer of the first embedding representation module to obtain the corresponding normalized features as the coordinate information embedding features; The access location embedding feature is obtained by the following steps: S20, inputting the coordinate information input into the first embedding representation module into the first linear layer of the first embedding representation module for feature extraction, and obtaining a corresponding first output feature Fh1; S21, input Fg1 into the self-attention mechanism layer of the second embedding representation module to obtain the corresponding self-attention coefficient Kg; S22, concatenating Fg1 and Kg, and inputting the concatenated features into the second linear layer of the second embedding representation module for feature extraction, to obtain the corresponding second output feature Fg2; S23, inputting Fg2 into the feedforward neural network layer of the second embedding representation module for feature extraction to obtain the corresponding feedforward feature FFg2; S24, input FFg2 into the normalization layer of the second embedding representation module to obtain the corresponding normalized features as the access position embedding features.

6. The method according to claim 5, It is characterized in that The output of the interactive feature strategy network is an action, which is obtained through the following steps: S30, inputting the coordinate information embedding feature output by the first embedding representation module into the self-attention mechanism layer of the first exchange feature strategy module to obtain the corresponding self-attention coefficient Ks1, and inputting the access position embedding feature output by the first embedding representation module into the self-attention mechanism layer of the second exchange feature strategy module to obtain the corresponding self-attention coefficient Ks2; S31, inputting Ks1 into the normalization layer of the first exchange feature strategy module to obtain the corresponding normalized feature Ns1, and inputting Ks2 into the normalization layer of the second exchange feature strategy module to obtain the corresponding normalized feature Ns2; S32, inputting Ns1 and Ks2 into the cross attention mechanism layer of the first exchange feature strategy module to obtain the corresponding cross attention coefficient KC1, and inputting Ns2 and Ks1 into the cross attention mechanism layer of the second exchange feature strategy module to obtain the corresponding cross attention coefficient KC2; S33, concatenating Ks1 and KC1 to obtain a corresponding concatenated feature as a coding feature FC1 of the coordinate information, and concatenating Ks2 and KC2 to obtain a corresponding concatenated feature as a coding feature FC2 of the access position; S34, inputting FC1 into the maximum pooling layer of the first exchange feature strategy module to obtain the corresponding pooling feature FP1, and inputting FC2 into the maximum pooling layer of the second exchange feature strategy module to obtain the corresponding pooling feature FP2; S35, input the added feature obtained by adding FC1 and FP2 into qi of the first exchange feature strategy module to obtain the corresponding multi-head attention coefficient MAK1, and input the added feature obtained by adding FC2 and FP1 into the multilayer perceptron of the second exchange feature strategy module to obtain the corresponding multi-head attention coefficient MAK2; S36, concatenating MAK1 and MAK2 and inputting the concatenated result into the feedforward neural network layer of the interactive feature strategy network to obtain corresponding feedforward features; S37, normalizing the obtained feedforward features to obtain corresponding feature difference information and inputting it into the exchange weight calculation module, wherein the feature difference information includes the feature difference between any two nodes in the corresponding nodes; S38, the exchange weight calculation module obtains a corresponding exchange weight set based on the feature difference information, and inputs the set to the action acquisition module; S39, the action acquisition module acquires an exchange weight as a target exchange weight based on the currently received exchange weight set, and exchanges the access positions corresponding to the two path nodes corresponding to the target exchange weight to obtain the current action.

7. The method according to claim 6, It is characterized in that The action acquisition module acquires an exchange weight as a target exchange weight from the current exchange weight set based on a polynomial sampling method.

8. The method according to claim 5, It is characterized in that The input of the evaluation network includes ((h1, g1), (h2, g3), ..., (hu, gu), ..., (hx, gx)), where hu and gu are respectively the coordinate information embedding features and access position embedding features corresponding to the u-th node in any path information, and the value of u ranges from 1 to x, where x is the number of nodes corresponding to the path information.

9. An electronic device, It is characterized in that including a processor and a memory; The processor is used to execute the steps of the method according to any one of claims 1 to 8 by calling the program or instruction stored in the memory.

10. A non-transitory computer-readable storage medium, It is characterized in that The non-transitory computer-readable storage medium is used to store a program or instruction, and the program or instruction enables a computer to execute the steps of the method according to any one of claims 1 to 8.

Images