DQN-based enterprise AI introduction opportunity optimization method and device

Abstract

The invention relates to a DQN-based enterprise AI introduction opportunity optimization method. The method comprises the steps of S1, obtaining state information of a real-time current price, a current yield, a warehouse product stock, a worker number, an AI level and a subsidy condition; s2, constructing an MDP model of an enterprise AI introduction opportunity control system, wherein the MDP model comprises a state space, an action space and a reward function; s3, according to the constructed MDP model of the enterprise AI introduction opportunity system, obtaining an optimal control strategy of the enterprise AI introduction opportunity system based on a DQN algorithm neural network; the DQN algorithm neural network comprises a main network and a target network, and the blindness of a traditional method in technical transformation opportunity selection is solved by constructing a dynamic decision model. The method is based on the U-shaped curve theory, long-term benefits of different stages are quantified in combination with the DQN algorithm, the stage 3 is finally determined as the optimal introduction opportunity, the enterprise profit is remarkably improved, and the transformation risk is reduced.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, and in particular to a method and apparatus for optimizing the timing of enterprise AI introduction based on DQN. Background Technology

[0002] In recent years, with the rapid development of technologies such as biometrics, the Internet of Things, and artificial intelligence, competition among enterprises has become fierce, product homogenization has become serious, and the industry has been mired in homogenized competition and price wars for a long time. Enterprises are facing pressure and challenges in technological innovation and industrial upgrading.

[0003] Traditional enterprises lack a scientific basis for decision-making regarding the timing of AI technology introduction, often relying on experience-based judgments, resulting in high investment costs, long transformation cycles, and large profit fluctuations. The benefits of introducing AI for enterprises exhibit a U-shaped curve: high initial technology investment costs, potential market adaptation risks in the mid-term, and profit recovery after the technology matures in the later stages. Existing methods struggle to dynamically balance costs and benefits at each stage. Static models (such as cost-benefit analysis) cannot adapt to dynamic market environments and technological iterations. Traditional machine learning methods (such as regression analysis) are ill-suited to handling complex time-series decision-making problems. Given the numerous uncertainties in existing enterprise transformation and upgrading strategies, a stable and effective control strategy is needed, providing a new approach to optimizing enterprise transformation and upgrading. Summary of the Invention

[0004] The purpose of this invention is to at least address one of the shortcomings of the prior art by providing a method for optimizing the timing of enterprise AI introduction based on DQN.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: Specifically, a method for optimizing the timing of enterprise AI introduction based on DQN is proposed, including the following steps: S1. Obtain relevant data of the target enterprise, including current pricing, current output, warehouse product inventory, number of workers, AI level, and subsidy status information; S2. Based on the relevant data, construct an MDP model for the enterprise AI introduction timing control system. The MDP model includes: state space, action space, and reward function. The state space includes: current pricing, current output, warehouse product inventory, number of workers, AI level, and subsidy status. The action space selects whether to upgrade AI as the control action. The reward function is used for quantitative evaluation of the performance of the enterprise AI introduction timing system strategy. S3. Based on the constructed MDP model of the enterprise AI introduction timing control system, the optimal control strategy of the enterprise AI introduction timing control system is obtained based on the DQN algorithm neural network. The DQN algorithm neural network includes two neural networks: a main network Q(S,A;θ) and a target network Q(S,A;θ'). The main network Q(S,A;θ) is used to predict the Q value of the current state-action pair of the enterprise AI introduction timing control system, and the target network Q(S,A;θ') is used to calculate the target Q value. Where S represents the current environment state, A represents the action taken, and θ and θ' are network parameters, respectively.

[0006] Furthermore, specifically, the state space, dynamic space, and reward function in step S2 are expressed as follows: The state space is represented as follows: Product pricing represents the selling price of goods; output represents the quantity of products produced per round; warehouse inventory represents the current quantity of products stored in the warehouse; worker count represents the number of workers employed by the company; AI level represents the current AI stage of the company; and subsidy status represents the government subsidies received by the company and consumers, as detailed below: Formula (1); in, The state of the enterprise at time t constitutes the enterprise's state space S. , , , , and Let represent the product price, output, warehouse product inventory, number of workers, AI stage, and subsidy at time t, respectively, where t is the time series label, and t = 0, 1, 2, ..., T; The action space is expressed as follows: Formula (2); The reward function is expressed as follows: Formula (3); in, It is the initial capital for the current round. α is the number of product chambers produced in the current round, and α is the raw material cost per product. β is the number of workers, and β is the labor cost per unit of product. γ represents the labor cost coefficient at the current stage of AI development, and γ represents the energy cost per unit of product. The energy cost coefficient at the current stage of AI. It is the transportation and logistics cost of a single product. The logistics and transportation cost coefficient at the current stage of AI. It is the storage cost in the current round. The cost of introducing AI in the current round. It is the subsidy given by the government in the current round. It is the selling price of the product in the current round. This represents the number of products sold in the current round.

[0007] Furthermore, specifically, the training steps of the DQN algorithm neural network in step S3 are as follows: Step 3-1: Initialize the DQN network parameters and parameters during training, including initializing the network parameters θ of the main network Q(S,A;θ), the network parameters θ' of the target network Q(S,A;θ'), and the capacity N of the experience buffer pool; Step 3-2: The agent selects an action to interact with the environment according to the policy π, and introduces a probability parameter ε through the ε-greedy decay mechanism to control the balance between exploration and exploitation. Step 3-3: Store the five-element array (S, A, R, S', done) generated during the interaction between the agent and the environment in the experience buffer pool, where S represents the current environment state, A represents the action taken, R represents the reward obtained, S' represents the new environment state entered, and done marks whether S is a termination state. Steps 3-4: Obtain the next environment state S' and calculate the reward R. Store the five-element array (S, A, R, S', done) generated by the next environment state in the experience buffer pool. Randomly obtain batch samples from the experience buffer pool to train the main network. Q(S,A;θ) is used to determine whether the step size requirement is met. If it is met, the parameters of the main network Q(S,A;θ) are copied to the target network Q(S,A;θ').

[0008] Furthermore, specifically, step 3-1 initializes the DQN network parameters and the parameters used during training, including: During initialization, the network parameters θ of the main network and θ' of the target network are the same, i.e., θ = θ'. During training, θ is updated by minimizing the error between the output values ​​of network parameters θ and θ', and the difference between the predicted value and the target Q value is reduced by the loss function through formula (4), as follows: Formula (4); in, Let Q be the target Q value at time j. Let A and B represent the state at time j and the action taken in the current state, respectively. This indicates that the main network is in a given state. The predicted Q value for action A is E[], which represents the expectation. In practice, this expectation value is approximated by randomly selecting samples from the experience replay pool. Then, the parameters are adjusted using formula (5). Calculate the partial derivative to obtain the error loss gradient, and then use the optimization algorithm to update the parameters. ; Formula (5); in, and Represent the loss function respectively And the predicted Q-value with respect to the main network parameters The gradient.

[0009] Furthermore, specifically, in step 3-2, the agent selects an action to interact with the environment based on policy π, including: As shown in the following formula, the training objective of the agent is to find the optimal policy. ,make The highest expected return can be obtained in all states. Formula (6); The process of introducing a probability parameter ε through an ε-greedy decay mechanism to control the balance between exploration and exploitation is as follows: Formula (7); A large probability parameter ε is set at the beginning of training, and after each training round, the probability parameter ε is slowly reduced using formula (8). It is the ε value set during the initial training phase. and The preset iteration parameters are t, where t = 0, 1, 2, ..., T; Formula (8).

[0010] Furthermore, specifically, Set to 1, Set to 0.1, Set it to 10000.

[0011] This invention also proposes a DQN-based enterprise AI introduction timing optimization device, comprising the following: The data acquisition module is used to acquire relevant data of the target enterprise in real time. The relevant data includes current pricing, current output, warehouse product inventory, number of workers, AI level and subsidy status information. The model building module is used to construct an MDP model for the enterprise AI introduction timing control system based on the relevant data. The MDP model includes: state space, action space, and reward function; wherein, the state space includes: current pricing, current output, warehouse product inventory, number of workers, AI level, and subsidy status; the action space selects whether to upgrade AI as the control action; the reward function is used for quantitative evaluation of the performance of the enterprise AI introduction timing system strategy. The allocation module is used to obtain the optimal control strategy of the enterprise AI introduction timing control system based on the constructed MDP model of the enterprise AI introduction timing control system and the DQN algorithm neural network. The DQN algorithm neural network includes two neural networks: a main network Q(S,A;θ) and a target network Q(S,A;θ'). The main network Q(S,A;θ) is used to predict the Q value of the current state-action pair of the enterprise AI introduction timing control system, and the target network Q(S,A;θ') is used to calculate the target Q value. Where S represents the current environment state, A represents the action taken, and θ and θ' are network parameters, respectively.

[0012] The present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the DQN-based enterprise AI introduction timing method.

[0013] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the DQN-based enterprise AI introduction timing method.

[0014] The beneficial effects of this invention are as follows: This invention proposes a timing optimization method for enterprise AI introduction based on DQN. During training, the main network trains and updates normally according to the steps, while the target network periodically copies the parameters of the main network for updates, thereby stabilizing the calculation of the target Q-value and thus stabilizing the training process. The DQN algorithm can significantly improve its learning stability and efficiency. Since the target network remains unchanged for a period of time, the agent can learn in a relatively static environment, making it easier to find the correct optimization direction. This effectively solves the problem of training instability and low efficiency caused by the inclusion of the Q-network to be learned in the target value in DQN, significantly improving the performance of the DQN algorithm's neural network. The Replay Buffer mechanism stores historical data of the agent's interaction with the environment and randomly extracts small batches of data for learning during training, effectively breaking the temporal correlation between data and improving learning efficiency. As training progresses, the agent's decision-making ability is enhanced, enabling it to make optimal choices based on the current state, achieving high reward returns and maximizing enterprise profits. Attached Figure Description

[0015] The above and other features of this disclosure will become more apparent from the detailed description of the embodiments illustrated in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals denote the same or similar elements. Obviously, the drawings described below are merely some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort. In the drawings: Figure 1 This is a flowchart of the control system in the DQN-based enterprise AI introduction timing optimization method provided by the present invention; Figure 2 This is a state-action process diagram in the DQN-based enterprise AI introduction timing optimization method provided by the present invention; Figure 3 This is a diagram of the Q-Network update process in the DQN-based enterprise AI introduction timing optimization method provided by the present invention; Figure 4 This is a flowchart of the DQN strategy optimization control process in the DQN-based enterprise AI introduction timing optimization method provided by the present invention. Figure 5 This is a schematic diagram of the experience buffer pool in the DQN-based enterprise AI introduction timing optimization method provided by the present invention; Figure 6 This invention provides a pre-training process in the experience buffer pool of the enterprise AI introduction timing optimization method based on DQN. Figure 7 This is a diagram of the target Q network update process in the DQN-based enterprise AI introduction timing optimization method provided by this invention; Figure 8 This is a diagram illustrating the training process for stabilizing the target Q value in the DQN-based enterprise AI introduction timing optimization method provided by this invention. Figure 9 This is a schematic diagram of the neural network for approximating the action value function Q in the enterprise AI introduction timing optimization method based on DQN provided by the present invention; Figure 10 This is a cost diagram for each stage of the enterprise AI introduction timing optimization method based on DQN provided by the present invention. Detailed Implementation

[0016] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The same reference numerals used throughout the accompanying drawings indicate the same or similar parts.

[0017] like Figure 1 As shown, the present invention provides a method for optimizing the timing of enterprise AI introduction based on DQN, including the following embodiments: S1, real-time monitoring of status information such as current pricing, current output, warehouse product inventory, number of workers, AI level and subsidy status through sensors; S2. Construct an MDP model for the enterprise AI introduction timing control system. The MDP model includes: state space, action space, and reward function. The state space includes: current pricing, current output, warehouse product inventory, number of workers, AI level, and subsidy status. The action space selects whether to upgrade AI as the control action. The reward function is used for quantitative evaluation of the performance of the enterprise AI introduction timing system strategy. S3. Based on the constructed MDP model of the enterprise AI introduction timing control system, the optimal control strategy of the enterprise AI introduction timing control system is obtained based on the DQN algorithm neural network. The DQN algorithm neural network includes two neural networks: a main network Q(S,A;θ) and a target network Q(S,A;θ'). The main network Q(S,A;θ) is used to predict the Q value of the current state-action pair of the enterprise production system, and the target network Q(S,A;θ') is used to calculate the target Q value.

[0018] Sensors monitor real-time status information including current pricing, current production volume, warehouse inventory, number of workers, AI level, and subsidy status. The controller, as the system's central hub, receives sensor data and adjusts its control strategy to determine whether to introduce AI for upgrades. Subsequently, the enterprise system operates production according to the strategy, resulting in the next state.

[0019] In the implementation of AI-driven timing control systems, intelligent agents play a crucial role. The agent intelligently formulates control decisions based on the current environmental state and translates these decisions into specific control commands, which are then transmitted to the control system. The control system adjusts its operating state according to the received commands, thereby influencing the indoor environment and creating new environmental states. Finally, the agent calculates reward values ​​by comparing feedback-based corporate profits, evaluating the effectiveness of the control actions. Through continuous learning and optimization, the agent can gradually find the optimal control strategy that maximizes rewards under various environmental states, ensuring the efficient and stable operation of the enterprise's production system and ultimately improving corporate profits.

[0020] The control process of an enterprise production system can be viewed as an MDP, which defines the interaction between the production control system and the environment in the form of states, actions, and rewards.

[0021] To establish an MDP, this section defines the relevant MDP elements, which are described in detail below:

[0022] ①State Space

[0023] Specifically, it is expressed as follows: ; Where St represents the enterprise state at time t, forming the enterprise's state space S. , , , , and Let t represent the product price at time t, the number of products produced per round, the inventory of products in the warehouse, the number of workers, the AI ​​stage, and the subsidy, respectively. t is the time sequence label, where: t = 0, 1, 2, ..., T;

[0024] ② Action Space Enterprise production control systems primarily drive enterprise transformation and upgrading through the introduction of AI. Therefore, this paper selects the introduction of AI as a control action, encompassing two possible actions: ; Here, At represents the action selected at time t, forming the action space A. The number 0 indicates that AI is not introduced, and the number 1 indicates that AI is introduced, such as... Figure 2 The diagram shows the state and action process of the system.

[0025] ③ Reward and punishment functions ; in, It is the initial capital for the current round. The number of products produced in the current round, α (preferred value 550) represents the raw material cost per product, is the number of workers, and β (preferred value 300) represents the labor cost per product. The labor cost coefficient at the current AI stage, the optimal γ value of 78 represents the energy cost per unit of product. The energy cost coefficient at the current stage of AI. The preferred value of 50 represents the transportation and logistics cost of a single product. The logistics and transportation cost coefficient at the current stage of AI. It is the storage cost in the current round. The cost of introducing AI in the current round. It is the subsidy given by the government in the current round. It is the selling price of the product in the current round. This represents the number of products sold in the current round. It helps agents clarify their goals, enabling them to learn effectively within the enterprise's production system and find the optimal strategy.

[0026] DQN Algorithm Flow DQN is an important algorithm in deep reinforcement learning. It combines the advantages of Q-learning and deep learning, replacing the traditional Q-table with a neural network, enabling the agent to handle high-dimensional state spaces. The core of the algorithm is to use a deep neural network to approximate the value function Q(S,A), which represents the expected cumulative reward obtained by performing action A in state S. The specific update formula is as follows: ; in, Let be the target Q value at time t. Let Q be the predicted Q value at time t. This represents the error at time, which can be used to update the DQN.

[0027] In DQN, two neural networks are typically used: a main network and a target network, where Q(Sj, A;θj) and Q(Sj, A;θj') represent the outputs of the main network and the target network, respectively. The main network predicts the Q-value of the current state-action pair, while the target network calculates the target Q-value. Therefore, in DQN, instead of updating the Q-table, the Q-network is updated. In this way, DQN can learn the optimal policy in a continuous environment without needing to know the specific model of the environment in advance. The specific update process of the Q-network is as follows: Figure 3 As shown; During initialization, the parameters of the main network and the target network are the same, i.e., θ = θ'. During training, θ is updated by minimizing the error between their output values. The loss function aims to reduce the difference between the prediction and the target Q-value, and is calculated as follows: L(θj )=E[Yj -Q(Sj ,A;θj )] 2 ; Then, with respect to the parameter θj, the partial derivative is taken to obtain the error loss gradient (as shown in the following equation), and then the parameter θj is updated using the optimization algorithm: ; This invention applies the DQN algorithm to enterprise production systems. An agent can learn whether to introduce AI to ensure enterprise profits by interacting with the production control system environment. By observing the current state of the environment, the agent uses a neural network to predict the possible Q-values ​​under different actions, thereby selecting the action that maximizes the expected cumulative reward. In this way, the agent can gradually learn how to make efficient decisions and adapt to different environments and operating conditions.

[0028] DQN strategy optimization

[0029] ①ε-greedy decay mechanism like Figure 4 As shown, in DQN, the agent selects an action to interact with the environment based on policy π, and the training objective of the agent is to find the optimal policy that maximizes the expected reward in all states. ; To achieve a balance between environmental exploration and exploitation, ε- The greedy selection strategy. ε-greedy controls the balance between exploration and exploitation by introducing a probability parameter ε, as follows: ; A large ε value is set at the beginning of each round, and the ε value is linearly decreased after each round of training. The specific process is as follows: ; The ε decay mechanism ensures that the agent fully explores the environment, collects information, and discovers potential better strategies in the early stages of training, while in the later stages of training, it gradually shifts to using the learned knowledge to optimize behavior and improve agent performance.

[0030] ②Replay Buffer Experience Replay Mechanism like Figure 5 As shown, in order to effectively utilize the data generated by the interaction between the agent and the environment during agent training, this invention introduces the concept of an experience buffer pool. The experience buffer pool is a specially designed container used to store a five-element array (S, A, R, S', done) generated during the interaction between the agent and the environment, where S represents the current state, A represents the action taken, R represents the reward obtained, S' represents the new state entered, and done indicates whether S is a termination state.

[0031] like Figure 6 As shown, each interaction between the agent and the environment packages and stores the resulting experience data in a buffer pool. When the Q-network parameters need to be updated and the amount of data in the buffer pool reaches a preset threshold, the agent randomly selects a fixed-size subset of data to update the Q-network. This mini-batch gradient learning method allows the agent to optimize its parameters using historical data while avoiding the problem of high correlation between consecutive data samples.

[0032] By utilizing historical data in the buffer pool, the agent can more effectively use each experience during training. This not only improves the efficiency and stability of learning but also reduces the risk of overfitting. At the same time, the diversity of historical data ensures that the agent can learn more robust strategies.

[0033] ③Fixed Q-targets network mechanism like Figure 7 As shown, when DQN is trained using the target value Yt, since Yt contains a Q network to be learned, and the update of the Q network affects the calculation of the target value, the core idea of ​​this method is to separate the calculation of the target value from the Q network used to estimate the action value, and use it to calculate Yt.

[0034] like Figure 8 As shown, during the training process, the main network trains and updates normally according to the steps, while the target network periodically copies the parameters of the main network for updates, thereby stabilizing the calculation of the target Q value and thus stabilizing the training process.

[0035] By introducing the Fixed Q-targets mechanism, the DQN algorithm can significantly improve its learning stability and efficiency. Since the target network remains unchanged over a period of time, the agent can learn in a relatively static environment, making it easier to find the correct optimization direction. This effectively solves the problems of training instability and inefficiency caused by the inclusion of the Q-network to be learned in the target value in DQN, significantly improving the performance of the DQN algorithm.

[0036] DQN Algorithm Design This invention optimizes the control strategy of an enterprise production system based on the DQN algorithm. The enterprise production system acts as an intelligent agent, continuously interacting and training with the environment to select and execute the optimal strategy, thereby achieving optimized control of the enterprise production system. The algorithm code flow is shown below.

[0037] DQN approximates the action-value function using a deep neural network and optimizes network parameters using backpropagation and gradient descent. The key to this algorithm lies in the introduction of the Replay Buffer and Fixed Q-targets mechanisms. The Replay Buffer mechanism effectively breaks the temporal correlation between data by storing historical data of the agent's interactions with the environment and randomly sampling small batches of data for learning during training, thus improving learning efficiency. Fixed Q-targets, on the other hand, improve training stability and convergence speed by decoupling the calculation of target values ​​from the updates of the main network.

[0038] Experimental Design of Enterprise Production Control System This invention focuses on a smart lock from a smart lock manufacturer that integrates a peephole, large screen, and facial recognition. Preprocessed data from December 28, 2023 to December 28, 2024 was selected as the research sample. To train and optimize the heating control model, the data was divided into training and testing sets, with approximately 80% used for model training and the remaining 20% ​​used for testing model performance. In the DQN algorithm, a deep neural network is used to approximate the Q-value function. This neural network structure includes an input layer, a hidden layer, and an output layer. Information is transferred between layers using a fully connected approach. The input layer receives the state information of the environment, the hidden layer uses ReLU as the activation function, and the output layer outputs the Q-values ​​corresponding to each action under different states. The hyperparameter settings of the production control algorithm are shown in Table 1 below.

[0039]

[0040]

[0041] Table 1 The specific process of approximating the action value function Q using a neural network is as follows: Figure 9 As shown, the neural network takes the state St observed by the agent at time t as input and extracts features through two hidden layers, each containing 88 neurons. Finally, the number of neurons in the output layer matches the number of available actions in the action space A to predict the expected reward for each action in a given state. This network structure effectively approximates a complex Q-function, thus guiding the agent's decision-making process.

[0042] This experiment on the introduction of AI for enterprise transformation and upgrading includes six stages. The first stage is the initial stage, where labor costs, energy costs, warehousing and logistics costs, R&D costs, and marketing costs are each set to an initial value of 100, before the introduction of AI. After introducing a personnel information management system in the first stage, labor costs and energy costs decreased by 10% and 5% respectively compared to the initial stage. In the second stage, based on the first stage, intelligent production lines were introduced to automate quality inspection and optimize employee task allocation, resulting in a 30% and 15% reduction in labor costs and energy costs respectively compared to the initial stage. In the third stage, based on the second stage, intelligent warehousing and logistics, unmanned vehicles, and optimized logistics scheduling were introduced, resulting in a 50% reduction in labor costs, energy costs, and warehousing and logistics costs respectively compared to the initial stage. In the fourth stage, based on the third stage, enterprise R&D investment was significantly increased, with R&D costs increasing by 20% compared to the initial stage. In the fifth stage, based on the fourth stage, marketing costs were increased by 20% compared to the initial stage, as shown in Table 2 below.

[0043]

[0044] Table 2 like Figure 10 The diagram shows the reward variation during the training phase of the DQN-based control algorithm. Figure 10 It is evident that in the early stages of training, due to the agent's lack of experience, it mainly adopts control actions through exploration, resulting in large fluctuations in rewards.

[0045] As the training rounds increase, the agent learns and updates its strategy by accumulating experience and utilizing historical data, and the reward gradually stabilizes. Experimental results show that after 200 training rounds, the total reward of the control algorithm stabilizes within the range of [180, 200], indicating that the agent is now able to effectively cope with the environment and achieve the goal of increasing corporate profits.

[0046] The results show that, assuming the company does not go bankrupt, the company can achieve the greatest profit by transforming and upgrading itself as soon as possible and reaching the third stage, which minimizes the various costs required for production.

[0047] Although the description of the invention has been quite detailed and particularly of several described embodiments, it is not intended to limit it to any of these details or embodiments or any particular embodiment, but should be considered as providing a broad possible interpretation of the claims by referring to the appended claims and taking into account the prior art, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

[0048] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any embodiment that achieves the technical effects of the present invention using the same means should fall within the protection scope of the present invention. Within the protection scope of the present invention, various modifications and variations can be made to the technical solutions and / or implementation methods.

Claims

1. A method for optimizing the timing of enterprise AI introduction based on DQN, characterized in that, Includes the following steps: S1. Obtain relevant data of the target enterprise, including current pricing, current output, warehouse product inventory, number of workers, AI level, and subsidy status information; S2. Based on the relevant data, construct an MDP model for the enterprise AI introduction timing control system. The MDP model includes: state space, action space, and reward function. The state space includes: current pricing, current output, warehouse product inventory, number of workers, AI level, and subsidy status. The action space selects whether to upgrade AI as the control action. The reward function is used for quantitative evaluation of the performance of the enterprise AI introduction timing system strategy. S3. Based on the constructed MDP model of the enterprise AI introduction timing control system, the optimal control strategy of the enterprise AI introduction timing control system is obtained based on the DQN algorithm neural network. The DQN algorithm neural network includes two neural networks: a main network Q(S,A;θ) and a target network Q(S,A;θ'). The main network Q(S,A;θ) is used to predict the Q value of the current state-action pair of the enterprise AI introduction timing control system, and the target network Q(S,A;θ') is used to calculate the target Q value. Where S represents the current environment state, A represents the action taken, and θ and θ' are network parameters, respectively.

2. The method for optimizing the timing of enterprise AI introduction based on DQN according to claim 1, characterized in that, Specifically, the state space, dynamic space, and reward function in step S2 are expressed as follows: The state space is represented as follows: Product pricing represents the selling price of goods; output represents the quantity of products produced per round; warehouse inventory represents the current quantity of products stored in the warehouse; worker count represents the number of workers employed by the company; AI level represents the current AI stage of the company; and subsidy status represents the government subsidies received by the company and consumers, as detailed below: Formula (1): in, The state of the enterprise at time t constitutes the enterprise's state space S. , , , , and Let represent the product price, output, warehouse product inventory, number of workers, AI stage, and subsidy at time t, respectively, where t is the time series label, and t = 0, 1, 2, ..., T; The action space is expressed as follows: Formula (2): The reward function is expressed as follows: Formula (3): in, It is the initial capital for the current round. α is the number of product chambers produced in the current round, and α is the raw material cost per product. β is the number of workers, and β is the labor cost per unit of product. γ represents the labor cost coefficient at the current stage of AI development, and γ represents the energy cost per unit of product. The energy cost coefficient at the current stage of AI. It is the transportation and logistics cost of a single product. The logistics and transportation cost coefficient at the current stage of AI. It is the storage cost in the current round. The cost of introducing AI in the current round. It is the subsidy given by the government in the current round. It is the selling price of the product in the current round. This represents the number of products sold in the current round.

3. The method for optimizing the timing of enterprise AI introduction based on DQN according to claim 2, characterized in that, Specifically, the training steps of the DQN algorithm neural network in step S3 are as follows: Step 3-1: Initialize the DQN network parameters and parameters during training, including initializing the network parameters θ of the main network Q(S,A;θ), the network parameters θ' of the target network Q(S,A;θ'), and the capacity N of the experience buffer pool; Step 3-2: The agent selects an action to interact with the environment according to the policy π, and introduces a probability parameter ε through the ε-greedy decay mechanism to control the balance between exploration and exploitation. Step 3-3: Store the five-element array (S, A, R, S', done) generated during the interaction between the agent and the environment in the experience buffer pool, where S represents the current environment state, A represents the action taken, R represents the reward obtained, S' represents the new environment state entered, and done marks whether S is a termination state. Steps 3-4: Obtain the next environment state S' and calculate the reward R. Store the five-element array (S, A, R, S', done) generated by the next environment state in the experience buffer pool. Randomly obtain batch samples from the experience buffer pool to train the main network. Q(S,A;θ) is used to determine whether the step size requirement is met. If it is met, the parameters of the main network Q(S,A;θ) are copied to the target network Q(S,A;θ').

4. The method for optimizing the timing of enterprise AI introduction based on DQN according to claim 3, characterized in that, Specifically, step 3-1 initializes the DQN network parameters and the parameters used during training, including: During initialization, the network parameters θ of the main network and θ' of the target network are the same, i.e., θ = θ'. During training, θ is updated by minimizing the error between the output values ​​of network parameters θ and θ', and the difference between the predicted value and the target Q value is reduced by the loss function through formula (4), as follows: Formula (4): in, Let Q be the target Q value at time j. Let A and B represent the state at time j and the action taken in the current state, respectively. This indicates that the main network is in a given state. The predicted Q value for action A is E[], which represents the expectation. In practice, this expectation value is approximated by randomly selecting samples from the experience replay pool. Then, the parameters are adjusted using formula (5). Calculate the partial derivative to obtain the error loss gradient, and then use the optimization algorithm to update the parameters. ; Formula (5): in, and Represent the loss function respectively And the predicted Q-value with respect to the main network parameters The gradient.

5. The method for optimizing the timing of enterprise AI introduction based on DQN according to claim 3, characterized in that, Specifically, in step 3-2, the agent selects an action to interact with the environment based on policy π, including: As shown in the following formula, the training objective of the agent is to find the optimal policy. ,make The highest expected return can be obtained in all states. Formula (6): The process of introducing a probability parameter ε through an ε-greedy decay mechanism to control the balance between exploration and exploitation is as follows: Formula (7): A large probability parameter ε is set at the beginning of training, and after each training round, the probability parameter ε is slowly reduced using formula (8). It is the ε value set during the initial training phase. and The preset iteration parameters are t, where t = 0, 1, 2, ..., T; Formula (8).

6. The method for optimizing the timing of enterprise AI introduction based on DQN according to claim 5, characterized in that, Specifically, Set to 1, Set to 0.1, Set it to 10000.

7. A device for optimizing the timing of enterprise AI introduction based on DQN, characterized in that, Including the following: The data acquisition module is used to acquire relevant data of the target enterprise in real time. The relevant data includes current pricing, current output, warehouse product inventory, number of workers, AI level and subsidy status information. The model building module is used to construct an MDP model for the enterprise AI introduction timing control system based on the relevant data. The MDP model includes: state space, action space, and reward function; wherein, the state space includes: current pricing, current output, warehouse product inventory, number of workers, AI level, and subsidy status; the action space selects whether to upgrade AI as the control action; the reward function is used for quantitative evaluation of the performance of the enterprise AI introduction timing system strategy. The allocation module is used to obtain the optimal control strategy of the enterprise AI introduction timing control system based on the constructed MDP model of the enterprise AI introduction timing control system and the DQN algorithm neural network. The DQN algorithm neural network includes two neural networks: a main network Q(S,A;θ) and a target network Q(S,A;θ'). The main network Q(S,A;θ) is used to predict the Q value of the current state-action pair of the enterprise AI introduction timing control system, and the target network Q(S,A;θ') is used to calculate the target Q value. Where S represents the current environment state, A represents the action taken, and θ and θ' are network parameters, respectively.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the DQN-based enterprise AI introduction timing method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the DQN-based enterprise AI introduction timing method according to any one of claims 1-6.

Images