Fishery management game problem optimal strategy determination method and device based on reinforcement learning, medium and product

Abstract

The invention discloses a fishery management game problem optimal strategy determination method and device based on reinforcement learning, a medium and a product, and relates to the technical field of cooperative games, and the method comprises the steps: obtaining historical data, a state vector of a to-be-treated water area at a current moment, and a correlation matrix of all enterprises; initializing an environmental governance strategy parameter matrix of each enterprise; performing multi-round iterative optimization on the environmental governance strategy parameter matrix of each enterprise based on historical data and the correlation matrix of each enterprise by using a strategy iterative algorithm to obtain an optimal environmental governance strategy parameter matrix of each enterprise; and determining the optimal environment governance strategy of each enterprise based on the optimal environment governance strategy parameter matrix of each enterprise and the state vector at the current moment. According to the method, the problem of making the optimal decision of the fishery management game enterprise under the condition that the game environment and the resource configuration parameters are unknown is solved.

Description

Technical Field

[0001] The present application relates to the field of cooperative game technology, and in particular to a method, device, medium and product for determining the optimal strategy for a fishery management game problem based on reinforcement learning. Background Art

[0002] Fishery management technology originates from the needs of sustainable development and resource management, especially in dealing with the problems of overexploitation of global fishery resources and environmental damage. In recent years, cross-regional fishery resource management problems have frequently occurred, showing that the level of fishery management lags far behind the actual needs of fishery resource protection. Problems such as water pollution and overfishing caused by competition in the actual fishing process seriously endanger the sustainable development of fishery resources. New technical theories are urgently needed to solve such problems.

[0003] Cooperative game theory is often applied to fishery management to help analyze the behavior of different fishery managers in resource utilization and develop optimal strategies to achieve sustainable resource utilization and long-term development. By studying the consequences of the interactions between the behaviors of different fishery managers, game theory aims to determine the optimal strategy for each fishery manager.

[0004] Reinforcement learning, as a cutting-edge machine learning method, originated from the study of animal behavior and plays an important role in solving optimal control problems. Unlike other machine learning methods, it focuses on the reward and punishment mechanism of behavior. This requires the intelligent agent (fishery manager) to interact with the environment and modify its behavior based on the feedback from the environment to obtain the maximum reward. Since the intelligent agent improves its strategy based on environmental information, it only needs a small amount of prior knowledge of the system. Based on these advantages, reinforcement learning technology is introduced into control systems to obtain the optimal feedback controller.

[0005] At present, fishery management game problems are usually modeled and solved when the game system is known, and the optimal strategy of the enterprise is obtained. However, in practical applications, the system matrix parameters of fishery management games are usually unknown or partially unknown, so traditional fishery management game solving methods are difficult to solve such problems. Summary of the invention

[0006] The purpose of this application is to provide a method, device, medium and product for determining the optimal strategy for fishery management game problems based on reinforcement learning, so as to solve the problem of optimal decision-making for fishery management game enterprises when the game environment and resource allocation parameters are unknown.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In the first aspect, the present application provides a method for determining the optimal strategy for a fishery management game problem based on reinforcement learning, comprising:

[0009] Obtain historical data, the current state vector of the water area to be managed and the correlation matrix of all enterprises; the historical data includes: the historical environmental management strategies of all enterprises at multiple historical moments and the state vector of the water area to be managed at each historical moment; the state vector is the difference between the actual value and the equilibrium value of the fish resources in the water area to be managed; the correlation matrix includes: a weight matrix, a cross-environmental management cost parameter matrix and an instantaneous benefit parameter matrix;

[0010] Initialize the environmental governance strategy parameter matrix of each enterprise;

[0011] Using a strategy iteration algorithm, based on the historical data and the correlation matrix of each enterprise, the environmental governance strategy parameter matrix of each enterprise is optimized through multiple rounds of iterations to obtain the optimal environmental governance strategy parameter matrix of each enterprise;

[0012] The optimal environmental governance strategy for each enterprise is determined based on the optimal environmental governance strategy parameter matrix of each enterprise and the state vector at the current moment.

[0013] Optionally, a policy iteration algorithm is used to perform multiple rounds of iterative optimization on the environmental governance strategy parameter matrix of each enterprise based on the historical data and the correlation matrix of each enterprise to obtain the optimal environmental governance strategy parameter matrix of each enterprise, including:

[0014] Calculate a parameter dimension expansion matrix based on the historical data;

[0015] Determine any round as the current round;

[0016] Based on the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round are determined; when the current round is the initial round, the environmental governance strategy parameter matrix of each enterprise in the previous round is the initialized environmental governance strategy parameter matrix of each enterprise;

[0017] Based on the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round, determine the environmental governance strategy parameter matrix of each enterprise after matrix reconstruction in the current round;

[0018] Determine the environmental governance strategy parameter matrix of each enterprise in the current round based on the environmental governance strategy parameter matrix after matrix reconstruction of each enterprise in the current round;

[0019] Determine whether an iteration stop condition is met; the iteration stop condition is that the absolute value of the difference between the environmental governance strategy parameter matrix of each enterprise in the current round and the environmental governance strategy parameter matrix of each enterprise in the previous round is less than a preset threshold or reaches a preset number of iterations;

[0020] If so, the environmental governance strategy parameter matrix of each enterprise in the current round is determined as the optimal environmental governance strategy parameter matrix of each enterprise;

[0021] If not, the environmental governance strategy parameter matrix of each enterprise in the previous round is updated to the environmental governance strategy parameter matrix of each enterprise in the current round, the current round is updated to the next round, and the method returns to "based on the parameter expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, determine the state parameters and control parameter coupling matrix and the instantaneous benefit parameters and control parameter coupling matrix of the previous round" until the optimal environmental governance strategy parameter matrix of each enterprise is obtained.

[0022] Optionally, the parameter dimension expansion matrix includes: a first dimension expansion matrix, a second dimension expansion matrix, a third dimension expansion matrix and a fourth dimension expansion matrix;

[0023] Calculating a parameter dimension expansion matrix based on the historical data includes:

[0024] Calculating the first expanded-dimension matrix, the second expanded-dimension matrix, and the fourth expanded-dimension matrix according to the state vector of the water area to be governed at each historical moment;

[0025] The third dimension expansion matrix is ​​calculated according to the historical environmental governance strategies of all enterprises at each historical moment and the state vectors of the water area to be governed at each historical moment.

[0026] Optionally, based on the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round are determined, including:

[0027] Calculate the state parameters and control parameter coupling matrix of the previous round according to the first dimension expansion matrix, the second dimension expansion matrix, the third dimension expansion matrix, the weight matrix and the cross environmental governance cost parameter matrix of each enterprise, and the environmental governance strategy parameter matrix of each enterprise in the previous round;

[0028] According to the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, the instantaneous benefit parameters and the control parameter coupling matrix of the previous round are calculated.

[0029] Optionally, the calculation formula of the state parameter and control parameter coupling matrix includes:

[0030]

[0031] Among them, k-1 is the coupling matrix of state parameters and control parameters in the k-1th round; δ xxis the first dimension-expanding matrix; I XX is the second dimension-expanding matrix; I n is the unit matrix; R α is the cross-environment governance cost parameter matrix after weighted summation, N is the number of enterprises, α i is the weight matrix of enterprise i, R i is the cross-environmental governance cost parameter matrix of enterprise i; When α is the weight matrix, the environmental governance strategy parameter matrix of all enterprises in the k-1th round; I XU is the third expanded dimension matrix.

[0032] Optionally, the calculation formula of the instantaneous benefit parameter and the control parameter coupling matrix includes:

[0033]

[0034] Among them, Ω k-1 is the coupling matrix of instantaneous profit parameter and control parameter in the k-1th round; I xx is the fourth dimension-expanded matrix; vec(·) is the matrix column reconstruction; Q α is the instantaneous return parameter matrix after weighted summation, Q i is the instantaneous profit parameter matrix of enterprise i; T is the transpose.

[0035] Optionally, the environmental governance strategy parameter matrix after the enterprise's matrix reconstruction includes:

[0036]

[0037]

[0038] Among them, svec(·) is the upper triangular reconstruction of the matrix; is the intermediate parameter matrix in the k-1th round; The matrix is ​​composed of the matrix of environmental governance strategy parameter matrices of all enterprises after matrix reconstruction; Reconstruct the matrix; When is the weight matrix α, the environmental governance strategy parameter matrix of all enterprises in the kth round; vec(·) is the matrix column reconstruction; When is the weight matrix α, the environmental governance strategy parameter matrix of enterprise 1 in the kth round; When is the weight matrix α, the environmental governance strategy parameter matrix of enterprise 2 in the kth round; When α is the weight matrix, it is the environmental governance strategy parameter matrix of enterprise N in the kth round.

[0039] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for determining the optimal strategy for a fishery management game problem based on reinforcement learning.

[0040] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned methods for determining the optimal strategy for the fishery management game problem based on reinforcement learning.

[0041] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for determining the optimal strategy for the fishery management game problem based on reinforcement learning.

[0042] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0043] The present application discloses a method, device, medium and product for determining the optimal strategy of a fishery management game problem based on reinforcement learning. First, historical data, the state vector of the water area to be governed at the current moment and the correlation matrix of all enterprises are obtained; the historical data include: the historical environmental governance strategies of all enterprises at multiple historical moments and the state vector of the water area to be governed at each historical moment; the state vector is the difference between the actual value and the equilibrium value of the fish resources in the water area to be governed; the correlation matrix includes: the weight matrix, the cross-environmental governance cost parameter matrix and the instantaneous benefit parameter matrix; secondly, the environmental governance strategy parameter matrix of each enterprise is initialized; then, based on the historical data and the correlation matrix of each enterprise, the environmental governance strategy parameter matrix of each enterprise is iteratively optimized for multiple rounds using a policy iteration algorithm to obtain the optimal environmental governance strategy parameter matrix of each enterprise; finally, based on the optimal environmental governance strategy parameter matrix of each enterprise and the state vector at the current moment, the optimal environmental governance strategy of each enterprise is determined. The present application uses a policy iteration method to determine the optimal environmental governance strategy of each enterprise, and can obtain the optimal environmental governance strategy between multiple enterprises by alternating between strategy evaluation and strategy improvement when the game environment information is completely unknown. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1A schematic flow chart of a method for determining an optimal strategy for a fishery management game problem based on reinforcement learning provided in an embodiment of the present application;

[0046] Figure 2 It is a schematic diagram of the convergence of the environmental governance strategy parameter matrix of the policy iteration algorithm;

[0047] Figure 3 This is a schematic diagram of the convergence of the intermediate parameter matrix of the strategy iteration algorithm;

[0048] Figure 4 It is a schematic diagram of the state vector trajectory of the water area to be managed;

[0049] Figure 5 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] The purpose of this application is to provide a method, device, medium and product for determining the optimal strategy for fishery management game problems based on reinforcement learning, aiming to solve the problem of optimal decision-making for fishery management game enterprises when the game environment and resource allocation parameters are unknown.

[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0053] In an exemplary embodiment, Figure 1 As shown, the method for determining the optimal strategy for the fishery management game problem based on reinforcement learning in this embodiment includes:

[0054] Step 1: Obtain historical data, the current state vector of the water area to be managed, and the correlation matrix of all enterprises.

[0055] Among them, the historical data include: the historical environmental governance strategies of all enterprises at multiple historical moments and the state vectors of the waters to be governed at each historical moment; the state vector is the difference between the actual value and the equilibrium value of the fish resources in the waters to be governed; the relevant matrix includes: the weight matrix, the cross-environmental governance cost parameter matrix and the instantaneous benefit parameter matrix.

[0056] Step 2: Initialize the environmental governance strategy parameter matrix of each enterprise.

[0057] Step 3: Using the strategy iteration algorithm, based on historical data and the correlation matrix of each enterprise, perform multiple rounds of iterative optimization on the environmental governance strategy parameter matrix of each enterprise to obtain the optimal environmental governance strategy parameter matrix of each enterprise.

[0058] As an optional implementation, step 3 includes:

[0059] Step 31: Calculate the parameter expansion matrix based on historical data.

[0060] As an optional implementation, the parameter dimension expansion matrix includes: a first dimension expansion matrix, a second dimension expansion matrix, a third dimension expansion matrix and a fourth dimension expansion matrix.

[0061] Step 31 includes:

[0062] Step 311: Calculate the first expanded dimension matrix, the second expanded dimension matrix and the fourth expanded dimension matrix according to the state vector of the water area to be governed at each historical moment.

[0063] Specifically, the calculation formula of the first dimension expansion matrix includes:

[0064]

[0065]

[0066] Among them, δ xx is the first expanded dimension matrix; is the intermediate state quantity of the water area to be managed at historical time t, t = t0, t1, t2, ..., t l-1 ,t l ; T is the transpose; svec(·) is the upper triangular reconstruction of the matrix.

[0067] The expression for the upper triangular reconstruction of any matrix or vector A is:

[0068]

[0069] Among them, svec(A) is the matrix A after upper triangular reconstruction; a 11 is the element in the first row and first column of A; a 12 is the element in the first row and second column of A; a 1m is the element in the 1st row and the mth column of A; a 22 is the element in the 2nd row and 2nd column of A; a m-1,m is the element in the m-1th row and mth column of A; a m,m is the element in the mth row and mth column of A.

[0070] The calculation formula of the second dimension expansion matrix includes:

[0071]

[0072]

[0073] Among them, I XX is the second expanded dimension matrix; XX is the parameter of the second expanded dimension matrix; x(τ) is the state vector of the water area to be governed at the historical moment τ. is the Kronecker product.

[0074] The calculation formula of the fourth dimension expansion matrix includes:

[0075]

[0076] Among them, I xx is the fourth extended-dimensional matrix.

[0077] Step 312: Calculate the third dimension expansion matrix based on the historical environmental governance strategies of all enterprises at each historical moment and the state vectors of the water area to be governed at each historical moment.

[0078] Specifically, the calculation formula of the third dimension expansion matrix includes:

[0079]

[0080]

[0081] Among them, I XU is the third dimension expansion matrix; XU is the parameter of the third dimension expansion matrix; u1(τ) is the historical environmental governance strategy of enterprise 1; u2(τ) is the historical environmental governance strategy of enterprise 2; u N (τ) is the historical environmental governance strategy of enterprise N.

[0082] Step 32: Determine any round as the current round.

[0083] Step 33: Based on the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, determine the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round; when the current round is the initial round, the environmental governance strategy parameter matrix of each enterprise in the previous round is the initialized environmental governance strategy parameter matrix of each enterprise.

[0084] As an optional implementation, step 33 includes:

[0085] Step 331: Calculate the state parameters and control parameter coupling matrix of the previous round based on the first expanded dimension matrix, the second expanded dimension matrix, the third expanded dimension matrix, the weight matrix and cross-environmental governance cost parameter matrix of each enterprise, and the environmental governance strategy parameter matrix of each enterprise in the previous round.

[0086] As an optional implementation, the calculation formula of the state parameter and control parameter coupling matrix includes:

[0087]

[0088] Among them, k-1 is the coupling matrix of state parameters and control parameters in the k-1th round; δ xx is the first dimension-expanding matrix; I XX is the second dimension-expanding matrix; I n is the unit matrix; R α is the cross-environment governance cost parameter matrix after weighted summation, N is the number of enterprises, α i is the weight matrix of enterprise i, R i is the cross-environmental governance cost parameter matrix of enterprise i; When α is the weight matrix, the environmental governance strategy parameter matrix of all enterprises in the k-1th round; I XU is the third expanded-dimensional matrix.

[0089] Step 332: Calculate the instantaneous benefit parameters and control parameter coupling matrix of the previous round based on the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round.

[0090] As an optional implementation, the calculation formula of the instantaneous benefit parameter and the control parameter coupling matrix includes:

[0091]

[0092] Among them, Ω k-1 is the coupling matrix of instantaneous profit parameter and control parameter in the k-1th round; I xx is the fourth dimension-expanded matrix; vec(·) is the matrix column reconstruction; Q α is the instantaneous return parameter matrix after weighted summation, Q i is the instantaneous profit parameter matrix of enterprise i; T is the transpose.

[0093] Step 34: Based on the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round, determine the environmental governance strategy parameter matrix of each enterprise after matrix reconstruction in the current round.

[0094] As an optional implementation method, the environmental governance strategy parameter matrix after the enterprise's matrix reconstruction includes:

[0095]

[0096]

[0097] Among them, svec(·) is the upper triangular reconstruction of the matrix; is the intermediate parameter matrix in the k-1th round; The matrix is ​​composed of the matrix of environmental governance strategy parameter matrices of all enterprises after matrix reconstruction; Reconstruct the matrix; When is the weight matrix α, the environmental governance strategy parameter matrix of all enterprises in the kth round; vec(·) is the matrix column reconstruction; When is the weight matrix α, the environmental governance strategy parameter matrix of enterprise 1 in the kth round; When is the weight matrix α, the environmental governance strategy parameter matrix of enterprise 2 in the kth round; When α is the weight matrix, it is the environmental governance strategy parameter matrix of enterprise N in the kth round.

[0098] Step 35: Determine the environmental governance strategy parameter matrix of each enterprise in the current round based on the environmental governance strategy parameter matrix after matrix reconstruction of each enterprise in the current round.

[0099] Specifically, the calculation formula of the enterprise's environmental governance strategy parameter matrix includes:

[0100]

[0101]

[0102] in, When is the weight matrix α, the environmental governance strategy parameter matrix of enterprise i in the kth round; res(·) is the inverse matrix reconstruction; for The element in the e-th row and f-th column, e = 1, 2, …, n, f = 1, 2, …, n.

[0103] Step 36: Determine whether the iteration stop condition is met; the iteration stop condition is that the absolute value of the difference between the environmental governance strategy parameter matrix of each enterprise in the current round and the environmental governance strategy parameter matrix of each enterprise in the previous round is less than a preset threshold or reaches a preset number of iterations.

[0104] Specifically, the expression of the iteration stop condition is:

[0105]

[0106] Among them, ε is the preset threshold.

[0107] Step 37: If yes, then the environmental governance strategy parameter matrix of each enterprise in the current round is determined as the optimal environmental governance strategy parameter matrix of each enterprise.

[0108] Step 38: If not, update the environmental governance strategy parameter matrix of each enterprise in the previous round to the environmental governance strategy parameter matrix of each enterprise in the current round, update the current round to the next round, and return to step 33 until the optimal environmental governance strategy parameter matrix of each enterprise is obtained.

[0109] Step 4: Determine the optimal environmental governance strategy for each enterprise based on the optimal environmental governance strategy parameter matrix of each enterprise and the state vector at the current moment.

[0110] Furthermore, the method of this application was verified by taking the example of two enterprises jointly developing fish resources in a body of water.

[0111] Figure 2 and Figure 3 It shows that after five iterations, the enterprise environmental governance strategy obtained by the policy iteration algorithm converges to the optimal environmental governance strategy. Figure 2 and Figure 3 The horizontal axis represents the number of iterations. Figure 2 The ordinate represents the absolute value of the difference between the environmental governance strategy parameter matrix in any round and the optimal environmental governance strategy parameter matrix. Figure 2 The ordinate represents the absolute value of the difference between the intermediate parameter matrix in any round and the intermediate parameter matrix corresponding to the optimal environmental governance strategy parameter matrix. Execute the optimal environmental governance strategy at this time, and get the difference between the balance value of fishery resources in the waters and the natural environment as follows: Figure 4 As shown, the aquatic ecosystem returned to stability in a very short time. Figure 4 The horizontal axis represents time, and the vertical axis represents the difference between the balance value of fishery resources and the natural environment. Figure 4 The environment was not governed before the black point in the figure. The black point began to be controlled by the policy iteration algorithm. It can be observed that the difference is in a random and chaotic state when no control is applied, and it quickly returns to zero after the optimal control is applied.

[0112] In an exemplary embodiment, a computer device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method for determining an optimal strategy for a fishery management game problem based on reinforcement learning.

[0113] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a method for determining an optimal strategy for a fishery management game problem based on reinforcement learning is implemented.

[0114] In an exemplary embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements a method for determining an optimal strategy for a fishery management game problem based on reinforcement learning.

[0115] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for determining the optimal strategy for a fishery management game problem based on reinforcement learning is implemented.

[0116] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0118] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0119] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for determining the optimal strategy for fishery management game problems based on reinforcement learning, characterized in that: The method for determining the optimal strategy for the fishery management game problem based on reinforcement learning includes: Obtain historical data, the current state vector of the water area to be managed and the correlation matrix of all enterprises; the historical data includes: the historical environmental management strategies of all enterprises at multiple historical moments and the state vector of the water area to be managed at each historical moment; the state vector is the difference between the actual value and the equilibrium value of the fish resources in the water area to be managed; the correlation matrix includes: a weight matrix, a cross-environmental management cost parameter matrix and an instantaneous benefit parameter matrix; Initialize the environmental governance strategy parameter matrix of each enterprise; Using a strategy iteration algorithm, based on the historical data and the correlation matrix of each enterprise, the environmental governance strategy parameter matrix of each enterprise is optimized through multiple rounds of iterations to obtain the optimal environmental governance strategy parameter matrix of each enterprise; The optimal environmental governance strategy for each enterprise is determined based on the optimal environmental governance strategy parameter matrix of each enterprise and the state vector at the current moment.

2. The method for determining the optimal strategy for fishery management game problems based on reinforcement learning according to claim 1 is characterized in that: Using the strategy iteration algorithm, based on the historical data and the correlation matrix of each enterprise, the environmental governance strategy parameter matrix of each enterprise is optimized through multiple rounds of iterations to obtain the optimal environmental governance strategy parameter matrix of each enterprise, including: Calculate a parameter dimension expansion matrix based on the historical data; Determine any round as the current round; Based on the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round are determined; when the current round is the initial round, the environmental governance strategy parameter matrix of each enterprise in the previous round is the initialized environmental governance strategy parameter matrix of each enterprise; Based on the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round, determine the environmental governance strategy parameter matrix of each enterprise after matrix reconstruction in the current round; Determine the environmental governance strategy parameter matrix of each enterprise in the current round based on the environmental governance strategy parameter matrix after matrix reconstruction of each enterprise in the current round; Determine whether an iteration stop condition is met; the iteration stop condition is that the absolute value of the difference between the environmental governance strategy parameter matrix of each enterprise in the current round and the environmental governance strategy parameter matrix of each enterprise in the previous round is less than a preset threshold or reaches a preset number of iterations; If so, the environmental governance strategy parameter matrix of each enterprise in the current round is determined as the optimal environmental governance strategy parameter matrix of each enterprise; If not, the environmental governance strategy parameter matrix of each enterprise in the previous round is updated to the environmental governance strategy parameter matrix of each enterprise in the current round, the current round is updated to the next round, and the "based on the parameter expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, determine the state parameters and control parameter coupling matrix and the instantaneous benefit parameters and control parameter coupling matrix of the previous round" is returned until the optimal environmental governance strategy parameter matrix of each enterprise is obtained.

3. The method for determining the optimal strategy for fishery management game problems based on reinforcement learning according to claim 2 is characterized in that: The parameter dimension expansion matrix includes: a first dimension expansion matrix, a second dimension expansion matrix, a third dimension expansion matrix and a fourth dimension expansion matrix; Calculating a parameter dimension expansion matrix based on the historical data includes: Calculating the first expanded-dimension matrix, the second expanded-dimension matrix, and the fourth expanded-dimension matrix according to the state vector of the water area to be governed at each historical moment; The third dimension expansion matrix is ​​calculated according to the historical environmental governance strategies of all enterprises at each historical moment and the state vectors of the water area to be governed at each historical moment.

4. The method for determining the optimal strategy for fishery management game problems based on reinforcement learning according to claim 3 is characterized in that: Based on the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, the state parameter and control parameter coupling matrix and the instantaneous benefit parameter and control parameter coupling matrix of the previous round are determined, including: Calculate the state parameters and control parameter coupling matrix of the previous round according to the first dimension expansion matrix, the second dimension expansion matrix, the third dimension expansion matrix, the weight matrix and the cross environmental governance cost parameter matrix of each enterprise, and the environmental governance strategy parameter matrix of each enterprise in the previous round; According to the parameter dimension expansion matrix, the correlation matrix of each enterprise and the environmental governance strategy parameter matrix of each enterprise in the previous round, the instantaneous benefit parameters and the control parameter coupling matrix of the previous round are calculated.

5. The method for determining the optimal strategy for fishery management game problems based on reinforcement learning according to claim 4 is characterized in that: The calculation formula of the state parameter and control parameter coupling matrix includes: Among them, k-1 is the coupling matrix of state parameters and control parameters in the k-1th round; δ xx is the first dimension-expanding matrix; I XX is the second dimension-expanding matrix; I n is the unit matrix; R α is the cross-environment governance cost parameter matrix after weighted summation, N is the number of enterprises, α i is the weight matrix of enterprise i, R i is the cross-environmental governance cost parameter matrix of enterprise i; When α is the weight matrix, the environmental governance strategy parameter matrix of all enterprises in the k-1th round; I XU is the third expanded-dimensional matrix.

6. The method for determining the optimal strategy for fishery management game problems based on reinforcement learning according to claim 5 is characterized in that: The calculation formula of the instantaneous benefit parameter and the control parameter coupling matrix includes: Among them, Ω k-1 is the coupling matrix of instantaneous profit parameter and control parameter in the k-1th round; I xx is the fourth dimension-expanded matrix; vec(·) is the matrix column reconstruction; Q α is the instantaneous return parameter matrix after weighted summation, Q i is the instantaneous profit parameter matrix of enterprise i; T is the transpose.

7. The method for determining the optimal strategy for fishery management game problems based on reinforcement learning according to claim 6 is characterized in that: The environmental governance strategy parameter matrix of the enterprise after matrix reconstruction includes: Among them, svec(·) is the upper triangular reconstruction of the matrix; is the intermediate parameter matrix in the k-1th round; The matrix is ​​composed of the matrix of environmental governance strategy parameter matrices of all enterprises after matrix reconstruction; Reconstruct the matrix; When is the weight matrix α, the environmental governance strategy parameter matrix of all enterprises in the kth round; vec(·) is the matrix column reconstruction; When is the weight matrix α, the environmental governance strategy parameter matrix of enterprise 1 in the kth round; When is the weight matrix α, the environmental governance strategy parameter matrix of enterprise 2 in the kth round; When α is the weight matrix, it is the environmental governance strategy parameter matrix of enterprise N in the kth round.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for determining the optimal strategy for the fishery management game problem based on reinforcement learning as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for determining the optimal strategy for the fishery management game problem based on reinforcement learning as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for determining the optimal strategy for the fishery management game problem based on reinforcement learning as described in any one of claims 1 to 7.

Images