Intelligent decision optimization method for cross-border trade based on reinforcement learning

Abstract

The invention relates to the technical field of artificial intelligence and reinforcement learning, and discloses a cross-border trade intelligent decision optimization method based on reinforcement learning. The method comprises the following steps: constructing a composite state representation fusing tax, logistics, exchange rate, demand and competitive behavior; high-dimensional state compression is realized through an auto-encoder; designing a hierarchical action architecture to decouple a macroscopic strategy and a microscopic operation; a Nash equilibrium guided reward shaping function is introduced, and an equilibrium income deviation is estimated in combination with anti-factual reasoning; carrying out stable training by adopting a double-delay depth deterministic strategy gradient algorithm with state transition consistency constraint; an online fine tuning mechanism is deployed to adapt to a real business environment. According to the method, the strategy convergence speed, the annual profit rate and the responsiveness to policy mutation are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and reinforcement learning, specifically relating to a method for optimizing intelligent decision-making in cross-border trade based on reinforcement learning. Background Technology

[0002] As global supply chain networks become increasingly complex and digitalized, cross-border trade decision-making is evolving from traditional rule-driven models to data-driven and intelligent optimization. Artificial intelligence technologies, particularly reinforcement learning methods, are widely being explored for their ability to autonomously explore optimal strategies in dynamic environments, particularly in core scenarios such as trade route selection, tariff optimization, risk hedging, and multi-party negotiation. However, cross-border trade is inherently a complex system characterized by high-dimensionality, incomplete information, and multi-party game dynamics. Its state space encompasses multiple heterogeneous factors, including exchange rate fluctuations, policy changes, logistical delays, credit risk, and cultural differences, posing significant challenges to traditional single-agent or shallow multi-agent reinforcement learning models in practical applications.

[0003] Reinforcement learning frameworks based on classical Markov decision processes struggle to effectively model the policy dependencies and dynamic evolutionary relationships among participants in cross-border trade multilateral games. Existing methods typically treat each trading entity as an independent decision-making unit, neglecting the collaborative or competitive structures formed in their policy spaces over long-term interactions. This leads to slow policy convergence or even local equilibria. Furthermore, the state representations commonly employ discretization or low-dimensional embedding, failing to capture the fine-grained relational features in high-dimensional trade environments. This further exacerbates the curse of dimensionality, causing a sharp decline in algorithm training efficiency in real-world simulations involving dozens or more participants, making it difficult to meet real-time decision-making requirements.

[0004] While existing technologies have made initial explorations in integrating game theory and reinforcement learning, they are mostly limited to static Nash equilibrium solutions or two-player zero-sum game scenarios, lacking effective integration of dynamic adjustment mechanisms for population strategies in evolutionary games. Furthermore, mainstream methods have not introduced new computational paradigms to reconstruct the expressive power of state and action spaces, resulting in a large and redundant policy search space that cannot achieve efficient compression and rapid convergence. In highly uncertain scenarios involving multiple countries' policy changes, sudden geopolitical risks, or access to emerging markets, traditional models struggle to discover robust non-dominant trade strategies and cannot reach stable equilibrium within a reasonable timeframe, severely restricting the deployment and effectiveness of intelligent decision-making systems in real-world cross-border trade ecosystems. Therefore, a novel intelligent decision optimization method that deeply integrates quantum heuristic representations, multi-agent collaborative learning, and evolutionary game dynamics is urgently needed. Summary of the Invention

[0005] This invention provides a cross-border trade intelligent decision optimization method based on reinforcement learning. By constructing a multi-granularity state space compression mechanism and a hierarchical action abstraction strategy, combined with a reward shaping function guided by game equilibrium, it effectively alleviates the curse of dimensionality problem of traditional reinforcement learning in high-dimensional continuous state-action space and significantly improves the policy convergence speed.

[0006] The proposed method first performs structured modeling on multi-source heterogeneous data in the cross-border trade environment, forming a composite state representation that includes tariff policies, logistics costs, exchange rate fluctuations, market demand, and competitor behavior. Then, it introduces a state embedding network based on an autoencoder to nonlinearly reduce the dimensionality of the original high-dimensional state vector, generating a low-dimensional, dense implicit state representation. Based on this, a hierarchical action space architecture is designed to decouple macro-level trade strategies (such as market entry, product pricing, and supply chain layout) from micro-level operational instructions (such as customs declaration completion, transportation route selection, and payment method configuration) into a high-level strategy network and a low-level execution network. The high-level strategy network outputs discretized strategic intentions, and the low-level execution network generates continuous control signals within a restricted local action subspace based on these intentions.

[0007] Simultaneously, a reward function integrating Nash equilibrium prior knowledge is constructed. This function not only includes direct business indicators such as corporate profits, compliance costs, and delivery timeliness, but also embeds dynamic game feedback terms on competitor strategy responses. The equilibrium payoff deviation under different strategy combinations is estimated through a counterfactual reasoning module, thereby shaping the immediate reward. Finally, a dual-delay deep deterministic policy gradient algorithm framework is adopted, introducing state transition consistency constraints in the target network update to ensure the stability and sample efficiency of the policy learning process.

[0008] Furthermore, the construction of the composite state representation specifically includes: obtaining real-time tariff rates, rules of origin, and lists of trade barriers for the target country or region from the General Administration of Customs database; collecting current freight rates, space availability, customs clearance times, and historical delay records for sea, air, and land transportation from international logistics platform interfaces; extracting spot exchange rates, forward spreads, and volatility surfaces of major settlement currency pairs from the central bank's foreign exchange market data stream; subscribing to consumer price indices, product category demand elasticity coefficients, and seasonal sales trends for the target market from third-party market research institutions; analyzing the capacity utilization rate, export destination distribution, and pricing range of major competitors from publicly available bidding information and corporate annual reports; and then standardizing the above five types of data and concatenating them into an original state vector with a dimension of no less than 120.

[0009] Preferably, the autoencoder-based state embedding network includes an encoder subnetwork and a decoder subnetwork. The encoder subnetwork consists of three fully connected layers, with the number of neurons in the input layer equal to the dimension of the original state vector, the number of neurons in the first hidden layer being 60% of that in the input layer, the number of neurons in the second hidden layer being 50% of that in the first hidden layer, and the number of neurons in the output layer being 16. The activation function of each layer is a modified linear unit. The decoder subnetwork has a structure symmetrical to the encoder subnetwork and is used to reconstruct the original state vector. During training, the mean squared reconstruction error is used as the loss function, and training is stopped when the reconstruction error on the validation set decreases by less than 1 / 1000 for five consecutive training epochs. After training, only the encoder subnetwork is retained, and its output is the low-dimensional dense implicit representation of the state.

[0010] Preferably, in the hierarchical action space architecture, the output of the high-level strategy network is a discrete action index, whose value range corresponds to 10 predefined macro-trade strategy templates, including full market entry, trial sales, localized production, establishment of regional distribution centers, parallel export to multiple countries, single product focus, full category coverage, high premium positioning, cost leadership strategy, and hybrid channel model; the bottom execution network receives the action index output by the high-level strategy network as conditional input, and combines it with the implicit representation of the current state to generate a continuous action vector through two fully connected layers; the continuous action vector includes five dimensions: product export unit price, transportation mode weight allocation coefficient, insurance coverage ratio, letter of credit payment period, and local marketing expense ratio. The values ​​of each dimension are mapped to the range of -1 to +1 by the hyperbolic tangent function, and then converted into actual operation parameters according to preset business rules.

[0011] Preferably, the reward function that incorporates prior knowledge of Nash equilibrium is defined as: ,in , , , The values ​​are 0.6, -0.2, -0.1, and 0.15, respectively. The company's net profit at time t is equal to sales revenue minus production costs, logistics costs, tariff costs, and marketing expenses. This is a compliance cost penalty item, which takes a negative value when key fields are missing from customs documents or when export control regulations are violated. Its absolute value is equal to the average amount of fines caused by the violation. The penalty for delivery timeliness deviation is equal to the square of the difference between the actual delivery period and the contractually agreed period; The equilibrium return deviation term is calculated using the counterfactual reasoning module.

[0012] As one embodiment of the present invention, the workflow of the counterfactual reasoning module is as follows: In each decision cycle, based on the currently observed competitor behavior sequence, a long short-term memory network is used to predict its strategy distribution over the next three cycles; combined with the current strategy of the player, a two-player non-cooperative game matrix is ​​constructed; a fictional game algorithm is used to iteratively solve for the approximate Nash equilibrium point of the matrix; the difference between the player's expected payoff at the equilibrium point and the actual payoff under the current strategy is calculated, and this difference is the equilibrium payoff deviation term. If the difference is positive, it means that the current strategy is better than the equilibrium strategy. Positive values ​​are used to enhance exploration; negative differences indicate that the current strategy is inferior to the equilibrium strategy. Negative values ​​are used to accelerate strategy adjustments.

[0013] As one embodiment of the present invention, the dual-delay deep deterministic policy gradient algorithm framework comprises four neural networks: a current policy network, a target policy network, a current value network, and a target value network. The policy network outputs deterministic actions, and the value network evaluates the long-term reward of state-action pairs. During each parameter update, the target network is only updated synchronously if the gradient norm of the loss function of the policy network is greater than a preset threshold of 0.01 and the Bellman error of the value network is less than 0.5. The parameters of the target network are updated using a soft update method, and the update formula is as follows: ,in The update rate is set to 0.005. In addition, in the experience replay buffer, samples with a change in the value function before and after the state transition exceeding 1.5 times the standard deviation are sampled first to improve the reuse efficiency of key experiences.

[0014] Preferably, the state transition consistency constraint is implemented by adding a regularization term to the loss function of the value network; this regularization term is defined as the current state. Actions generated via policy network Then, transition to the next state. Then the policy network generates actions. },calculate( , )and( , The first-order difference on the value network output requires that the absolute value of the difference does not exceed a preset tolerance threshold of 0.03; this constraint forces the value function to maintain a smooth change on adjacent state-action pairs, avoiding value estimation oscillations caused by sparse sampling in high-dimensional space.

[0015] Preferably, the method further includes an online strategy fine-tuning mechanism; when the system is deployed in actual cross-border trade business, after each complete trade cycle, the actual transaction data, customer feedback and regulatory review results within that cycle are collected to construct a fine-tuning dataset; the trained strategy network is updated with gradients for no more than 10 training steps using this dataset, with the update step size fixed at 0.001; during the fine-tuning process, the parameters of the state embedding network are frozen, and only the output layer weights of the strategy network and the value network are adjusted to prevent catastrophic forgetting and maintain the stability of the core state representation.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] 1. This invention effectively maps the original high-dimensional cross-border trade state vector to a low-dimensional implicit representation space through a multi-granularity state space compression mechanism, fundamentally alleviating the curse of dimensionality problem in reinforcement learning under high-dimensional input. The hierarchical action space architecture decouples macro-strategy from micro-operation, significantly reducing the effective action search space and improving the sample efficiency of policy learning. The reward shaping function, which integrates Nash equilibrium priors, introduces rational decision-making criteria guided by game theory, enabling the agent to quickly approximate the equilibrium strategy in complex multi-agent environments and significantly accelerating the convergence speed. The state transition consistency constraint and the priority experience replay mechanism jointly ensure the stability and accuracy of the value function estimation, avoiding the training collapse of traditional deep reinforcement learning in sparse reward scenarios. The online policy fine-tuning mechanism enables the model to continuously adapt and evolve in real business environments, ensuring the timeliness and robustness of the decision-making strategy.

[0018] 2. Experiments show that in a simulated environment containing 15 trade variables and 8 competitors, the method of this invention reduces the number of interaction steps required for policy convergence by 72% compared to the standard deep Q-network algorithm, increases the annualized profit rate of the final strategy by 18.5%, and shortens the strategy adjustment response time to 1 / 3 of the original method when facing sudden changes in trade policy. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention;

[0020] Figure 2 This is a schematic diagram of the core principle framework of the multi-granularity state space compression and hierarchical action abstraction strategy in this invention;

[0021] Figure 3 This is a logical flowchart of the construction of composite state representation and the embedding of autoencoders into the network in this invention;

[0022] Figure 4 This is a hierarchical action generation framework diagram for collaborative decision-making between the high-level strategy network and the low-level execution network in this invention;

[0023] Figure 5 This is a data flow diagram of the reward shaping and counterfactual reasoning module that integrates Nash equilibrium prior knowledge in this invention;

[0024] Figure 6 This is a training optimization framework diagram of the dual-delay deep deterministic policy gradient algorithm and state transition consistency constraint in this invention. Detailed Implementation

[0025] Please refer to the attached document. Figure 1 To be continued Figure 6 This invention provides a reinforcement learning-based intelligent decision-making optimization method for cross-border trade, aiming to solve the problems of dimensionality curse and slow policy convergence faced by traditional reinforcement learning algorithms in high-dimensional continuous state-action spaces. This method achieves efficient modeling and intelligent decision-making in complex cross-border trade environments by constructing a multi-granularity state space compression mechanism, a hierarchical action abstract policy architecture, and a reward shaping function that integrates prior knowledge of game equilibrium. The specific implementation process of this invention will be described in detail below, combining each technical step.

[0026] The method first performs step S1: constructing a composite state representation. This step synchronously collects key indicators related to cross-border trade from five independent data sources and integrates them in a structured manner. Specifically, it obtains real-time tariff rates, rules of origin, and lists of trade barriers for the target country or region from the General Administration of Customs database; it collects current freight rates, space availability, customs clearance times, and historical delay records for sea, air, and land transportation from international logistics platform interfaces; it extracts spot exchange rates, forward spreads, and volatility surfaces of major settlement currency pairs from the central bank's foreign exchange market data stream; it subscribes to the target market's consumer price index, category demand elasticity coefficients, and seasonal sales trends from third-party market research institutions; and it analyzes the capacity utilization rate, export destination distribution, and pricing range of major competitors from publicly available bidding information and corporate annual reports. These five types of data are standardized and then concatenated into an original state vector with a dimension of no less than 120. The standardization process uses the Z-score method, which involves subtracting the historical mean from each feature dimension and then dividing by its standard deviation to ensure that all input variables are on a uniform scale with zero mean and unit variance. This original state vector fully characterizes the five core dimensions upon which cross-border trade decisions depend: external policies, logistical conditions, financial environment, market demand, and competitive landscape.

[0027] Then, step S2 is executed: the original state vector is nonlinearly dimensionality-reduced using an autoencoder-based state embedding network to generate a low-dimensional, dense implicit state representation. The state embedding network consists of an encoder subnetwork and a decoder subnetwork. The encoder subnetwork contains three fully connected layers: the number of neurons in the input layer equals the dimension of the original state vector; the number of neurons in the first hidden layer is 60% of that in the input layer; the number of neurons in the second hidden layer is 50% of that in the first hidden layer; and the number of neurons in the output layer is fixed at 16. All activation functions are modified linear units. The decoder subnetwork structure is strictly symmetrical to the encoder subnetwork, i.e., the input layer has 16 neurons, the first hidden layer has 32 neurons, the second hidden layer has 48 neurons, and the number of neurons in the output layer equals the dimension of the original state vector. During training, the mean squared reconstruction error is used as the loss function, defined as the squared Euclidean distance between the original state vector and the vector reconstructed through the encoder-decoder path. Training uses batch gradient descent with a batch size of 256 and an initial learning rate of 0.001. When the reconstruction error on the validation set decreases by less than 1 / 1000 for five consecutive training epochs, the model is considered converged, and training is stopped. After training, only the encoder subnetwork is retained, and its output is a 16-dimensional low-dimensional dense implicit representation of the state. This representation effectively preserves the key semantic information in the original high-dimensional state, while significantly reducing the input dimensionality of the subsequent policy network, thus mitigating the curse of dimensionality.

[0028] Next, proceed to step S3: Design a hierarchical action space architecture to decouple macro-level trade strategies from micro-level operational instructions. This architecture comprises a high-level strategy network and a low-level execution network. The high-level strategy network receives implicit state representations as input and outputs discrete action indices, ranging from 0 to 9, corresponding to 10 predefined macro-level trade strategy templates: full market entry, exploratory sales, localized production, establishment of regional distribution centers, parallel export to multiple countries, single product focus, full category coverage, high-premium positioning, cost leadership strategy, and hybrid channel model. Each strategy template corresponds to a set of business logic rules to guide the action generation scope of the low-level execution network.

[0029] The bottom-level execution network receives two inputs: first, the action index output from the higher-level policy network, expanded into a ten-dimensional vector after one-hot encoding; second, the implicit representation of the current state. These two inputs are concatenated and fed into two fully connected layers, each containing 64 neurons and using rectified linear units as the activation function. The final output is a five-dimensional continuous action vector. This vector contains five dimensions: product export unit price, transportation mode weighting coefficient, insurance coverage ratio, letter of credit payment term, and local marketing expense ratio. The values ​​of each dimension are mapped to the -1 to +1 interval using a hyperbolic tangent function, and then converted into actual operational parameters according to preset business rules. For example, the mapping formula for the product export unit price is: ,in Basic pricing, The first dimension is the action vector; the weighting coefficients for transportation modes are normalized using Softmax and then distributed to the three modes of transportation: sea, air, and land; the payment period for letters of credit is linearly mapped to the range of 30 to 180 days using a_4. This hierarchical architecture significantly reduces the effective action search space, allowing policy learning to focus on the synergistic optimization of high-level strategic choices and fine-tuning of local parameters.

[0030] Then, step S4 is performed: A reward function incorporating prior knowledge of Nash equilibrium is constructed. This reward function is defined as follows: .in, For a moment The overall reward value; For businesses at all times Net profit is calculated by subtracting production costs, logistics costs, tariff costs, and marketing expenses from sales revenue. As a compliance cost penalty, when customs documents are missing key fields or violate export control regulations, Take a negative value; its absolute value equals the average fine amount caused by the violation. The penalty for delivery timeliness deviation is equal to the square of the difference between the actual delivery period and the contractually agreed period; The equilibrium return deviation term is calculated using the counterfactual reasoning module; , , , The preset weighting coefficients have values ​​of 0.6, -0.2, -0.1, and 0.15. This reward function not only reflects direct business performance but also introduces a game equilibrium guidance mechanism to encourage agents to adopt rational strategies in multi-agent environments.

[0031] For calculation Step S5: Run the counterfactual reasoning module. In each decision cycle, this module predicts the strategy distribution of competitors over the next three cycles based on the currently observed competitor behavior sequence using a Long Short-Term Memory (LSTM) network. The LSM input consists of observable behaviors of competitors over the past 10 cycles, such as export volume, pricing, and transportation methods, with 128 hidden units. The predicted output is the probability distribution of competitors over the next three cycles across 10 macro-strategy templates. Combining this with the current strategy, a two-player non-cooperative game matrix is ​​constructed, where each element represents the expected profit for both players under different strategy combinations. A hypothetical game algorithm is used to iteratively solve for the approximate Nash equilibrium point of this matrix. The maximum number of iterations for the hypothetical game is set to 50, and the convergence threshold is set to 0.001. The difference between the player's expected payoff at the equilibrium point and the actual payoff under the current strategy is calculated; this difference is the equilibrium payoff deviation term. If the difference is positive, it means that the current strategy is better than the equilibrium strategy. Positive values ​​are used to enhance exploration; negative differences indicate that the current strategy is inferior to the equilibrium strategy. Negative values ​​are used to accelerate policy correction. This mechanism allows the agent to dynamically evaluate the merits of its own policy relative to the game equilibrium, thereby adjusting its learning direction.

[0032] The following step, S6, is executed: policy training is performed using a dual-delay deep deterministic policy gradient algorithm framework. This framework comprises four neural networks: the current policy network, the target policy network, the current value network, and the target value network. The policy network outputs deterministic actions, while the value network evaluates the long-term reward of the state-action pair. Both the policy and value networks employ a 3-layer fully connected structure, with 256 neurons per layer and modified linear units (MRUs) as activation functions. During each parameter update, the target network is only updated synchronously if the gradient norm of the policy network's loss function is greater than a preset threshold of 0.01 and the Bellman error of the value network is less than 0.5. The target network's parameters are updated using a soft update method, with the update formula being... ,in The value is 0.005. Furthermore, in the experience replay buffer, samples with a change in the value function exceeding 1.5 times the standard deviation before and after the state transition are preferentially sampled. The preferential sampling probability is defined as...

[0033]

[0034] in The standard deviation of the value function. The value is 0.01. This mechanism improves the efficiency of reusing key experiences and accelerates policy convergence.

[0035] To ensure training stability, step S7 is performed: a state transition consistency constraint regularization term is added to the loss function of the value network. This regularization term is defined as the current state... Actions generated via policy network Then, transition to the next state. Then the policy network generates actions. ,calculate( , )and( , The first-order difference at the value network output must have an absolute value not exceeding a preset tolerance threshold of 0.03. Specifically, the regularization term... This constraint forces the value function to change smoothly over adjacent state-action pairs, avoiding value estimation oscillations caused by sparse sampling in high-dimensional space, thereby improving the robustness of policy learning.

[0036] Finally, step S8 is executed: Deploying the online policy fine-tuning mechanism. When the system is deployed in actual cross-border trade operations, after each complete trade cycle, actual transaction data, customer feedback, and regulatory review results within that cycle are collected to construct a fine-tuning dataset. The fine-tuning dataset contains a four-tuple of state, action, actual reward, and next state. This dataset is used to update the gradients of the trained policy network for no more than 10 training steps, with a fixed update step size of 0.001. During fine-tuning, the parameters of the state embedding network are frozen, and only the output layer weights of the policy network and value network are adjusted. This design prevents catastrophic forgetting while maintaining the stability of the core state representation, enabling the model to continuously adapt to dynamic changes in the real business environment.

[0037] Throughout the methodology, the data flow and processing logic between each step are highly coordinated. The original high-dimensional state is compressed by an autoencoder and input into the hierarchical action architecture. The high-level strategy determines the strategic direction, while the low-level execution generates specific parameters. The reward function integrates business indicators and game equilibrium deviations to guide the strategy towards rational equilibrium. A dual-delay deep deterministic policy gradient algorithm, combined with state transition consistency constraints and priority experience replay, ensures efficient and stable training. An online fine-tuning mechanism enables the model to continuously evolve in real-world scenarios. In a simulated environment with 15 trade variables and 8 competitors, this method reduces the number of interaction steps required for strategy convergence by 72% compared to the standard deep Q-network algorithm, ultimately increasing the annualized profit margin of the strategy by 18.5%. Furthermore, when facing sudden changes in trade policy, the strategy adjustment response time is reduced to one-third of the original method.

[0038] At the system implementation level, the cross-border trade intelligent decision optimization system of this invention includes a state perception module, a state embedding module, a hierarchical decision module, a reward calculation module, a strategy training module, and an online fine-tuning module. The state perception module is responsible for collecting and standardizing raw data in real time from five major data sources: customs, logistics, finance, market, and competitive intelligence, forming a high-dimensional state vector. The state embedding module deploys a trained autoencoder subnetwork to map the high-dimensional state into a 16-dimensional implicit representation. The hierarchical decision module includes a high-level strategy network and a low-level execution network; the former outputs discrete strategy indices, and the latter generates continuous operational parameters. The reward calculation module integrates a net profit calculator, a compliance checker, a timeliness assessor, and a counterfactual reasoning engine, outputting a comprehensive reward signal.

[0039] The policy training module implements a dual-latency deep deterministic policy gradient algorithm, comprising four neural networks and an experience replay buffer, supporting priority sampling and state transition consistency constraints. The online fine-tuning module performs lightweight updates to the policy network using actual business data after each trade cycle. All modules are interconnected via an internal data bus, ensuring efficient transmission and synchronization of state, action, reward, and gradient information. The system runs on a distributed computing cluster; the state awareness and embedding module is deployed on edge nodes to reduce latency, the policy training module runs on a GPU server to accelerate computation, and the fine-tuning module is integrated into the business execution engine to achieve closed-loop feedback. This system architecture ensures the deployability and scalability of the method in real-world cross-border trade scenarios.

Claims

1. A method for intelligent decision-making optimization in cross-border trade based on reinforcement learning, characterized in that, include: A composite state representation is constructed, which is formed by splicing together tariff policy data obtained from the General Administration of Customs database, logistics cost data collected from the international logistics platform interface, exchange rate fluctuation data extracted from the central bank's foreign exchange market data stream, market demand data subscribed from third-party market research institutions, and competitor behavior data analyzed from public bidding information and corporate annual reports. The composite state representation is nonlinearly dimensionality-reduced by an autoencoder-based state embedding network to generate a low-dimensional, dense implicit state representation. Based on a hierarchical action space architecture, macro-trade strategies and micro-operation instructions are decoupled. The high-level strategy network outputs a discretized strategic intent index based on the implicit state representation, and the low-level execution network generates a continuous action vector based on the strategic intent index and the implicit state representation. Construct a reward function that integrates prior knowledge of Nash equilibrium, wherein the reward function includes a net profit term, a compliance cost penalty term, a delivery timeliness deviation penalty term, and an equilibrium revenue deviation term; The policy network is trained using a dual-delay deep deterministic policy gradient algorithm framework, and a state transition consistency constraint is introduced into the value network loss function to stabilize the training process. The execution of cross-border trade operations is controlled by the continuous action vectors output by the trained policy network.

2. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 1, characterized in that, Constructing composite state representations includes: The tariff policy data, logistics cost data, exchange rate fluctuation data, market demand data, and competitor behavior data are all Z-score standardized. The standardized five types of data are concatenated to form an original state vector, the original state vector having a dimension of no less than 120.

3. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 2, characterized in that, The composite state representation is nonlinearly dimensionality-reduced using an autoencoder-based state embedding network to generate a low-dimensional, dense implicit state representation, including: The original state vector is compressed layer by layer using an encoder subnetwork. The encoder subnetwork contains three fully connected layers. The number of neurons in the input layer is equal to the dimension of the original state vector. The number of neurons in the first hidden layer is 60% of that in the input layer. The number of neurons in the second hidden layer is 50% of that in the first hidden layer. The number of neurons in the output layer is 16. The activation function of each layer is a modified linear unit. The encoder subnetwork after training is retained, and its output is a 16-dimensional low-dimensional dense state implicit representation.

4. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 3, characterized in that, Based on a hierarchical action space architecture, macro-level trade strategies and micro-level operational instructions are decoupled, including: The discrete strategic intent index output by the high-level strategy network ranges from 0 to 9, corresponding to ten predefined macro trade strategy templates, including full market entry, exploratory marketing, localized production, establishment of regional distribution centers, parallel export to multiple countries, single product focus, full category coverage, high premium positioning, cost leadership strategy, and hybrid channel model. The underlying execution network receives the strategic intent index after one-hot encoding and the implicit state representation as input, and generates a 5-dimensional continuous action vector through two fully connected layers. The 5-dimensional continuous action vector includes the product export unit price, transportation mode weight allocation coefficient, insurance coverage ratio, letter of credit payment period, and local marketing expense ratio.

5. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 4, characterized in that, The values ​​of each dimension in the 5-dimensional continuous action vector are mapped to the interval from -1 to +1 using the hyperbolic tangent function, and then converted into actual operation parameters according to preset business rules, including: The export unit price of the product is calculated using the formula. Calculation, where Basic pricing, The first dimension is the action vector; The weighting coefficients for different modes of transport are assigned to sea, air, and land transport after being normalized by Softmax. The payment period for letters of credit is linearly mapped to the range of 30 to 180 days using the fourth dimension of the action vector.

6. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 5, characterized in that, Construct a reward function that incorporates prior knowledge of Nash equilibrium, including: Define the overall reward value ,in , , , The values ​​are 0.6, -0.2, -0.1, and 0.15, respectively. For businesses at all times Net profit equals sales revenue minus production costs, logistics costs, tariff costs, and marketing expenses; This is a compliance cost penalty item, which takes a negative value when key fields are missing from customs declarations or when export control regulations are violated. Its absolute value is equal to the average amount of fines caused by the violation. The penalty for delivery timeliness deviation is equal to the square of the difference between the actual delivery period and the contractually agreed period; The equilibrium return deviation term is calculated using the counterfactual reasoning module.

7. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 6, characterized in that, The equilibrium return deviation term is calculated using the counterfactual reasoning module. ,include: Based on the currently observed competitor behavior sequence, a long short-term memory network is used to predict their strategy distribution over the next three periods. We construct a two-player non-cooperative game matrix based on our current strategy, and use a hypothetical game algorithm to iteratively solve for the approximate Nash equilibrium point. Calculate the difference between the expected return of this player at the equilibrium point and the actual return under the current strategy; this difference is the equilibrium return deviation term. .

8. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 7, characterized in that, The policy network is trained using a dual-delay deep deterministic policy gradient algorithm framework, including: Maintain four neural networks: the current policy network, the target policy network, the current value network, and the target value network. The target network is updated synchronously only when the gradient norm of the loss function of the policy network is greater than 0.01 and the Bellman error of the value network is less than 0.

5. The target network parameters are updated using a soft update method, and the update formula is as follows: ,in It is 0.

005.

9. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 8, characterized in that, In the experience replay buffer, samples whose value function changes by more than 1.5 times the standard deviation before and after the state transition are preferentially sampled. The preferential sampling probability is defined as follows: in The standard deviation of the value function. It is 0.

01.

10. The intelligent decision-making optimization method for cross-border trade based on reinforcement learning according to claim 9, characterized in that, Introduce state transition consistency constraints into the loss function of the value network, including: defining a regularization term. in Current state For the current action, For the next state, For the next action; The regularization term is added to the loss function of the value network to force the value output of adjacent state-action pairs to change smoothly.

Images