Construction site monitoring arrangement method based on deep reinforcement learning

Abstract

The invention discloses a construction site monitoring arrangement method based on deep reinforcement learning, and relates to the technical field of model application. Monitoring arrangement modeling is design of a multi-agent reinforcement learning model, through deep reinforcement learning of multiple agents and guidance of a common target, all the agents are promoted to act in a cooperative and self-adaptive mode, and finally intelligent arrangement of construction site monitoring is achieved. According to the construction site monitoring intelligent arrangement method based on deep reinforcement learning, monitoring arrangement design can be efficiently and automatically carried out, compared with a traditional artificial experience and fixed rule method, the arrangement efficiency and precision can be remarkably improved, the monitoring equipment is regarded as an intelligent agent for joint optimization, and the construction site monitoring intelligent arrangement method based on deep reinforcement learning has the advantages that the arrangement efficiency and precision are greatly improved. Global coordination and self-adaptive arrangement can be realized, the problems of inflexibility and low efficiency caused by a manual method are avoided, and the monitoring arrangement efficiency and precision are improved.

Description

Technical Field

[0001] The present invention relates to the field of model application technology, and in particular to a construction site monitoring layout method based on deep reinforcement learning. Background Art

[0002] As the scale of construction sites continues to expand, the demand for site monitoring arrangements is increasing; Traditional monitoring deployment methods often rely on manual experience and fixed rules, which are inefficient and inflexible. Construction site monitoring layout methods based on intelligent algorithms often require long iterative calculations, making it difficult to quickly provide layout plans.

[0003] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0004] The purpose of the present invention is to provide a construction site monitoring layout method based on deep reinforcement learning to solve the technical problems raised in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a construction site monitoring arrangement method based on deep reinforcement learning, comprising at least the following steps: S1: The monitoring layout problem is modeled as a multi-agent reinforcement learning model. Each monitor is regarded as an agent, that is, a monitoring-agent. Each monitoring-agent is free and automatic in the construction site. The monitoring-agents cooperate with each other to achieve comprehensive monitoring coverage and minimize costs, and finally determine their own positions; S2: Build a monitoring-agent based on deep reinforcement learning. Through deep reinforcement learning and the guidance of common goals, each agent is prompted to act in a coordinated and adaptive manner, and finally realize the intelligent arrangement of construction site monitoring; S3: Build an attention-based intelligent network architecture.

[0006] Furthermore, the calculation process of performing monitoring arrangement modeling includes at least the following steps: Transform the monitoring layout problem from a plane design problem to a path planning problem with time and environment changes; At this point, the mathematical model conforms to the mathematical modeling principles of multi-agent reinforcement learning; As each monitoring-agent i is based on the current state Perform an action with the current time step t , the environment is based on the current state and actions Feedback a new status and rewards ; For each agent, its goal is to maximize the expected cumulative reward, as shown in formula (1):

[0007] in, is the discount factor, which is 0.99 and is used to represent the current value of future rewards; N represents the number of monitoring agents.

[0008] Furthermore, the deep reinforcement learning of the monitoring-agent includes at least the actions of the monitoring-agent, the observations of the monitoring-agent and the rewards of the monitoring-agent.

[0009] Furthermore, the action format output by each agent in the monitoring-agent action is the same, that is, ; in, Indicates the type of monitoring i, Indicates the position of monitoring i in the plane, Indicates the height of monitoring i, Indicates the direction of monitoring i. It should be noted that is a discrete action, and its action list is ; in, Indicates the type of the mth monitoring device, and 0 means that this agent is invalid, that is, this monitoring device is not deployed in the venue; By taking 0 as an alternative, the algorithm can not only decide the position of the agent but also the number of agents, making the design results more flexible and diverse; In addition, in order to monitor and improve the generalization ability of the agent neural network and accelerate the convergence speed, The action space of these four parameters is designed as a continuous action and is scaled to .

[0010] Furthermore, the monitoring-agent's observation content includes at least the following: The monitoring layout cannot exceed the boundary of the construction area, so each agent needs to observe the boundary of the construction area, that is, ,in, Indicates the kth construction area boundary coordinate point; Similarly, each monitoring-agent needs to observe the area of ​​the construction structure, i.e.

[0011] in, represents the x-coordinate of the qth boundary point of the mth engineering structure; Each monitoring-agent i needs to observe its own model, observation range, plane position, height and orientation angle, that is, .

[0012] It represents the angle that monitor i can observe, which is an inherent attribute of monitor i; Indicates the position of monitoring i in the plane; represents the height of monitoring i; represents the direction of monitoring i; Each monitoring agent i needs to observe the positions and parameters of other agents, that is, .

[0013] Further, the reward of the monitoring-agent is the feedback of the environment to the monitoring arrangement; The rewards obtained by the monitoring-agent guide the agent to move to a better position and encourage the agent to choose a more suitable device type. In addition, since the goals of the monitoring-agents are common goals and there is no competition between them, the global reward is used as the reward for each monitoring-agent. Of course, the reward setting of the intelligent agent needs to take into account many aspects, as shown in formula (2), where: Indicates the weights of different factors, determined by the designer or through experimental adjustment:

[0014] in, is the monitoring coverage reward, which is used to guide the agent to cover more areas, as shown in formula (3):

[0015] is the area covered by the monitoring, is the area of ​​the site; is the monitored equipment purchase reward, guiding the agent to use less equipment purchase cost, as shown in formula (4):

[0016] represents the purchase cost of the i-th device, and N represents the number of all devices; is the installation and construction reward for monitoring, guiding the agent to use less installation and construction costs, as shown in formula (5):

[0017] Where N represents the number of all equipment, that is, the fewer the number of equipment, the lower the installation and construction costs; It indicates the number of types of equipment. That is, the fewer the types of equipment, the less complicated the installation, which means the lower the cost.

[0018] Furthermore, in the agent network architecture based on the attention mechanism, each monitoring-agent has its own set of action (Actor) network and evaluation (Critic) network. In addition, it can be seen from the observation content of the monitoring-agent that the monitoring-agent observes more content, so the monitoring-agent needs to pay more attention to important content. The network based on the attention mechanism is used as the action (Actor) network of the agent, and it is also used as the evaluation (Critic) network of the agent. In the proposed network, Transformer is the final output network.

[0019] Furthermore, the Transformer network queries Q (Query), key K (Key) and value V (Value) are the three core components in the attention mechanism, and the output of the LSTM network is used as the input of the three core components respectively; The inputs of the three LSTMs are observation content A, observation content A, and observation content B; The entire network architecture can be expressed as formula (6):

[0020] in: LSTM Q Indicates the formation Q LSTM network of values, d k yes LSTM Q The vector dimensions of the output of Obv A It represents the observed content A, and softmax is the activation function, which represents the ratio of the index of the element to the sum of the indexes of all elements. Similarly, LSTM K Indicates the formation K LSTM network of values, LSTM V Indicates the formation V LSTM network of values, Obv B It indicates the observation content B; In the action network, the parameters of the As Obv B , other parameters as Obv A ； In the evaluation network, the action to be taken a As Obv B , other parameters as Obv A ； The two networks obtain actions and rewards respectively through the attention mechanism expressed in formula (6).

[0021] Compared with the prior art, the present invention has the following beneficial effects: 1. The monitoring layout modeling in the present invention is designed as a multi-agent reinforcement learning model. Through the deep reinforcement learning of multi-agents and the guidance of common goals, each agent is prompted to act in a coordinated and adaptive manner, and finally the intelligent layout of construction site monitoring is realized; 2. The intelligent layout method for construction site monitoring based on deep reinforcement learning proposed in the present invention can efficiently and automatically design the monitoring layout. Compared with the traditional manual experience and fixed rule methods, it can significantly improve the layout efficiency and accuracy. By treating the monitoring equipment as an intelligent body for joint optimization, it can achieve global coordination and adaptive layout, avoiding the inflexibility and inefficiency caused by manual methods, and improving the efficiency and accuracy of monitoring layout; 3. The method proposed in this invention can balance the number and layout of monitoring equipment, thereby maximizing the monitoring coverage and minimizing the equipment purchase, installation and construction costs. Through deep reinforcement learning, the intelligent agent can automatically adjust the location and number of equipment according to the dynamic conditions of the site, avoiding the waste of resources caused by excessive layout, while ensuring that the monitoring needs of the site are met; 4. The present invention introduces a combination of discrete action space (including equipment type and location decision) and continuous action space. This method enables each agent to not only select its own optimal position, but also dynamically decide whether to deploy monitoring equipment, thereby increasing the flexibility and diversity of the algorithm. This method solves the limitation brought by the fixed number of agents and gives the deployment plan more adaptability. 5. The Transformer network architecture in the present invention is applied to the action and evaluation (Critic) network of the intelligent agent, which can process a large amount of observation data and accurately focus on important information through the attention mechanism. This design enhances the information processing ability of the intelligent agent in a complex environment, improves the decision-making quality and the generalization ability of the network, and ensures that the deep reinforcement learning algorithm can quickly adapt to the changeable and complex construction environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a schematic diagram of monitoring-multi-agent of the present invention; Figure 2 Schematic diagram of parameters for monitoring i in the present invention; Figure 3 Schematic diagram of the network architecture of the present invention. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0025] In order to achieve comprehensive monitoring of the construction site, the monitoring needs to cooperate with each other to cover as much as possible. However, too much monitoring will increase the cost. Therefore, it is necessary to arrange monitoring reasonably and efficiently. Each monitoring is regarded as an intelligent agent, and each intelligent agent is free and automatic in the construction site. Figure 1 The optimal strategy of an agent depends on the strategies of other agents, which leads to significant strategy interactions and coordination problems in the learning process.

[0026] A construction site monitoring layout method based on deep reinforcement learning is proposed, which includes at least the following steps: S1: The monitoring layout problem is modeled as a multi-agent reinforcement learning model. Each monitor is regarded as an agent, that is, a monitoring-agent. Each monitoring-agent is free and automatic in the construction site. The monitoring-agents cooperate with each other to achieve comprehensive monitoring coverage and minimize costs, and finally determine their own positions; S2: Build a monitoring-agent based on deep reinforcement learning. Through deep reinforcement learning and the guidance of common goals, each agent is prompted to act in a coordinated and adaptive manner, and finally realize the intelligent arrangement of construction site monitoring; S3: Build an attention-based intelligent network architecture.

[0027] The calculation process for monitoring arrangement modeling includes at least the following steps: Transform the monitoring layout problem from a plane design problem to a path planning problem with time and environment changes; At this point, the mathematical model conforms to the mathematical modeling principles of multi-agent reinforcement learning; As each monitoring-agent i is based on the current state Perform an action with the current time step t , the environment is based on the current state and actions Feedback a new status and rewards ; For each agent, its goal is to maximize the expected cumulative reward, as shown in formula (1):

[0028] in, is the discount factor, which is 0.99 and is used to represent the current value of future rewards; N represents the number of monitoring agents.

[0029] The deep reinforcement learning of the supervisory-agent includes at least the actions of the supervisory-agent, the observations of the supervisory-agent and the rewards of the supervisory-agent.

[0030] The action format of each agent output in the monitoring-agent action is the same, that is, ,like Figure 2 As shown; in, Indicates the type of monitoring i, Indicates the position of monitoring i in the plane, Indicates the height of monitoring i, Indicates the direction of monitoring i. It should be noted that is a discrete action, and its action list is ; in, Indicates the type of the mth monitoring device, and 0 means that this agent is invalid, that is, this monitoring device is not deployed in the venue; By taking 0 as an alternative, the problem of the number of agents being fixed when designing deep reinforcement learning for multi-agents is solved. This allows the algorithm to decide not only the position of the agents but also the number of agents, making the design results more flexible and diverse. In addition, in order to monitor and improve the generalization ability of the agent neural network and accelerate the convergence speed, The action space of these four parameters is designed as a continuous action and is scaled to .

[0031] The monitoring agent's observations include at least the following: The monitoring layout cannot exceed the boundary of the construction area, so each agent needs to observe the boundary of the construction area, that is, ,in, Indicates the kth construction area boundary coordinate point; Similarly, each monitoring-agent needs to observe the area of ​​the construction structure, i.e.

[0032] in, represents the x-coordinate of the qth boundary point of the mth engineering structure; Each monitoring-agent i needs to observe its own model, observation range, plane position, height and orientation angle, that is, , It represents the angle that monitor i can observe, which is an inherent attribute of monitor i; Indicates the position of monitoring i in the plane; represents the height of monitoring i; represents the direction of monitoring i; Each monitoring agent i needs to observe the positions and parameters of other agents, that is, .

[0033] The reward of the monitoring-agent is the feedback from the environment on the monitoring arrangement; The rewards obtained by the monitoring-agent guide the agent to move to a better position and encourage the agent to choose a more suitable device type. In addition, since the goals of the monitoring-agents are common goals and there is no competition between them, the global reward is used as the reward for each monitoring-agent. Of course, the reward setting of the intelligent agent needs to take into account many aspects, as shown in formula (2), where: Indicates the weights of different factors, determined by the designer or through experimental adjustment:

[0034] in, is the monitoring coverage reward, which is used to guide the agent to cover more areas, as shown in formula (3): , is the area covered by the monitoring, is the area of ​​the site; is the monitored equipment purchase reward, guiding the agent to use less equipment purchase cost, as shown in formula (4): , represents the purchase cost of the i-th device, and N represents the number of all devices (i.e., the number of agents); is the installation and construction reward for monitoring, guiding the agent to use less installation and construction costs, as shown in formula (5):

[0035] Where N represents the number of all devices (i.e., the number of monitoring-agents), that is, the fewer the number of devices, the lower the installation and construction costs; It indicates the number of types of equipment. That is, the fewer the types of equipment, the less complicated the installation, which means the lower the cost.

[0036] In the agent network architecture based on the attention mechanism, each monitoring-agent has its own set of action (Actor) network and evaluation (Critic) network. In addition, it can be seen from the observation content of the monitoring-agent that the monitoring-agent observes more content, so the monitoring-agent needs to pay more attention to important content. The network based on the attention mechanism is used as the action (Actor) network of the agent, and it is also used as the evaluation (Critic) network of the agent. In the proposed network, Transformer is the final output network.

[0037] Query in Transformer Network Q (Query), key K (Key) and value V (Value) are the three core components in the attention mechanism, and the output of the LSTM network is used as the input of the three core components respectively; The inputs of the three LSTMs are observation content A, observation content A, and observation content B; The entire network architecture can be expressed as formula (6):

[0038] in: LSTM Q Indicates the formation Q LSTM network of values, d k yes LSTM Q The vector dimensions of the output of Obv A It represents the observed content A, and softmax is the activation function, which represents the ratio of the index of the element to the sum of the indexes of all elements. Similarly, LSTM K Indicates the formation K LSTM network of values, LSTM V Indicates the formation V LSTM network of values, ObvB It indicates the observation content B; In the action network, the parameters of the As Obv B , other parameters as Obv A ； In the evaluation network, the action to be taken a As Obv B , other parameters as Obv A ； The two networks obtain actions and rewards respectively through the attention mechanism expressed in formula (6); In addition, the multi-agent reinforcement learning of the present invention adopts a centralized training and distributed execution training method, and adopts an experience replay mechanism for learning. This algorithm framework can effectively handle learning tasks in large-scale, highly interactive environments.

[0039] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A construction site monitoring arrangement method based on deep reinforcement learning, characterized in that: At least the following steps are included: S1: The monitoring layout problem is modeled as a multi-agent reinforcement learning model. Each monitor is regarded as an agent, that is, a monitoring-agent. Each monitoring-agent is free and automatic in the construction site. The monitoring-agents cooperate with each other to achieve comprehensive monitoring coverage and minimize costs, and finally determine their own positions; The calculation process for monitoring arrangement modeling includes at least the following steps: Transform the monitoring layout problem from a plane design problem to a path planning problem with time and environment changes; At this point, the mathematical model conforms to the mathematical modeling principles of multi-agent reinforcement learning; As each monitoring-agent i is based on the current state Perform an action with the current time step t , the environment is based on the current state and actions Feedback a new status and rewards ; For each agent, its goal is to maximize the expected cumulative reward, as shown in formula (1): ， in, is the discount factor, which is 0.99 and is used to represent the current value of future rewards; N represents the number of monitoring-agents; S2: Build a monitoring-agent based on deep reinforcement learning. Through deep reinforcement learning and the guidance of common goals, each agent is prompted to act in a coordinated and adaptive manner, and finally realize the intelligent arrangement of construction site monitoring; The deep reinforcement learning of the monitoring-agent includes at least the actions of the monitoring-agent, the observations of the monitoring-agent and the rewards of the monitoring-agent; The action format output by each agent in the monitoring-agent action is the same, that is, ; in, Indicates the type of monitoring i, Indicates the position of monitoring i in the plane, Indicates the height of monitoring i, Indicates the direction of monitoring i. It should be noted that is a discrete action, and its action list is ; in, Indicates the type of the mth monitoring device, and 0 means that this agent is invalid, that is, this monitoring device is not deployed in the venue; By taking 0 as an alternative, the algorithm can not only decide the position of the agent but also the number of agents, making the design results more flexible and diverse; In addition, in order to monitor and improve the generalization ability of the agent neural network and accelerate the convergence speed, The action space of these four parameters is designed as a continuous action and is scaled to ; The monitoring-agent's observation content includes at least the following: The monitoring layout cannot exceed the boundary of the construction area, so each agent needs to observe the boundary of the construction area, that is, ,in, Indicates the kth construction area boundary coordinate point; Similarly, each monitoring-agent needs to observe the area of ​​the construction structure, i.e. ， in, represents the x-coordinate of the qth boundary point of the mth engineering structure; Each monitoring-agent i needs to observe its own model, observation range, plane position, height and orientation angle, that is, , It represents the angle that monitor i can observe, which is an inherent attribute of monitor i; Indicates the position of monitoring i in the plane; represents the height of monitoring i; represents the direction of monitoring i; Each monitoring agent i needs to observe the positions and parameters of other agents, that is, ; The reward of the monitoring-agent is the feedback of the environment to the monitoring arrangement; The rewards obtained by the monitoring-agent guide the agent to move to a better position and encourage the agent to choose a more suitable device type. In addition, since the goals of the monitoring-agents are common goals and there is no competition between them, the global reward is used as the reward for each monitoring-agent. Of course, the reward setting of the intelligent agent needs to take into account many aspects, as shown in formula (2), where: and Indicates the weights of different factors, determined by the designer or through experimental adjustment: ， in, is the monitoring coverage reward, which is used to guide the agent to cover more areas, as shown in formula (3): ， is the area covered by the monitoring, is the area of ​​the site; is the monitored equipment purchase reward, guiding the agent to use less equipment purchase cost, as shown in formula (4): ， represents the purchase cost of the i-th device, and N represents the number of all devices; is the installation and construction reward for monitoring, guiding the agent to use less installation and construction costs, as shown in formula (5): ， Where N represents the number of all equipment, that is, the fewer the number of equipment, the lower the installation and construction costs; Indicates the number of types of equipment, that is, the fewer the types of equipment, the less complicated the installation, that is, the lower the cost; S3: Build an attention-based intelligent network architecture.

2. A construction site monitoring arrangement method based on deep reinforcement learning according to claim 1, characterized in that: In the agent network architecture based on the attention mechanism, each monitoring-agent has its own set of action networks and evaluation networks. In addition, it can be seen from the observation content of the monitoring-agent that the monitoring-agent observes more content, so the monitoring-agent needs to pay more attention to important content. The network based on the attention mechanism is used as the action network of the agent, and it is also used as the evaluation network of the agent. In the proposed network, Transformer is the final output network.

3. A construction site monitoring arrangement method based on deep reinforcement learning according to claim 2, characterized in that: The Transformer network queries Q ,key K Sum V These are the three core components in the attention mechanism, and the output of the LSTM network is used as the input of the three core components respectively; The inputs of the three LSTMs are observation content A, observation content A, and observation content B; The entire network architecture is expressed as formula (6): ， in: LSTM Q Indicates the formation Q LSTM network of values, d k yes LSTM Q The vector dimensions of the output of Obv A It represents the observed content A, and softmax is the activation function, which represents the ratio of the index of the element to the sum of the indexes of all elements. Similarly, LSTM K Indicates the formation K LSTM network of values, LSTM V Indicates the formation V LSTM network of values, Obv B It indicates the observation content B; In the action network, the parameters As Obv B , other parameters as Obv A ； In the evaluation network, the action to be taken a As Obv B , other parameters as Obv A ； The two networks obtain actions and rewards respectively through the attention mechanism expressed in formula (6).

Images