Intelligent behavior modeling system based on deep reinforcement learning

Abstract

The invention provides an intelligent behavior modeling system based on deep reinforcement learning, and relates to the technical field of data processing, and the method comprises the steps: obtaining historical state data of an intelligent agent from monitoring equipment of the intelligent agent, carrying out the matching of the historical state data according to the monitoring time, obtaining an interaction data set, and carrying out the matching of the historical state data according to the interaction data set; the method comprises the steps of obtaining a multi-dimensional state vector data set and a standard action instruction data set, constructing an environment space and a behavior space of an intelligent agent, calculating execution result values of behaviors in the behavior space executed by the intelligent agent in different environment spaces, constructing an initial intelligent behavior modeling model based on a deep Q network, and according to the environment space, the behavior space and reward feedback values, establishing an initial intelligent behavior modeling model. And training the initial intelligent behavior modeling model to obtain an intelligent behavior modeling model, acquiring real-time state data of the intelligent agent, and inputting the real-time state data into the intelligent behavior modeling model to obtain a decision behavior. The intelligent agent behavior modeling method can perform behavior modeling on the intelligent agent based on the deep reinforcement learning algorithm.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an intelligent behavior modeling system based on deep reinforcement learning. Background Art

[0002] At present, intelligent behavior modeling systems mainly rely on traditional rule-based methods and machine learning methods. Rule-based methods simulate behaviors through predefined rules, but lack flexibility and may not be able to effectively cope with complex and dynamically changing environments. Machine learning methods rely on a large amount of labeled data for training and can make predictions to a certain extent, but may not be able to autonomously explore and optimize the decision-making process without labels.

[0003] Deep reinforcement learning technology has significantly improved the system's adaptability in dynamic environments. However, existing deep reinforcement learning technology usually relies on large-scale training data and a lot of training time, has low training efficiency, and is easily limited by the training environment. For example, in the field of autonomous driving, deep reinforcement learning models may have a long learning convergence time in complex environments, resulting in the system's inability to respond to environmental changes in real time. Summary of the invention

[0004] The purpose of the present invention is to provide an intelligent behavior modeling system based on deep reinforcement learning, aiming to solve the problems mentioned in the background technology.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] An intelligent behavior modeling system based on deep reinforcement learning, the system comprising:

[0007] The data acquisition module is used to obtain the historical state data of the intelligent agent from the monitoring device of the intelligent agent, and the historical state data includes environmental state data, behavior action data and execution result data;

[0008] The data processing module is used to pre-process the historical status data, remove redundant data, and match it according to the monitoring time to obtain the interactive data set;

[0009] The feature extraction module is used to extract the state information features of the environment where the agent is located according to the environmental state data in the interactive data set, and obtain the position information, speed information and environmental information; extract the operation instruction features executed by the agent according to the behavior action data in the interactive data set, and obtain the movement instruction, rotation instruction and adjustment instruction; extract the execution result features of the behavior executed by the agent according to the execution result data in the interactive data set, and obtain the behavior completion ratio, behavior execution time and energy consumption;

[0010] The environment space module is used to vectorize the position information, speed information and environment information to obtain a multi-dimensional state vector data set, and construct the environment space of the intelligent agent based on it;

[0011] The behavior space module is used to discretize movement instructions, rotation instructions, and adjustment instructions to obtain a standard action instruction dataset and construct the agent's behavior space based on it;

[0012] The execution result module is used to calculate the execution result value of the behavior in the behavior space executed by the agent in different environment spaces according to the completion ratio of the behavior, the execution time of the behavior and the energy consumption;

[0013] A model building module, used to build an initial intelligent behavior modeling model based on a deep Q network;

[0014] The model training module is used to set the experience recycling pool according to the environment space, behavior space and reward feedback value, and train the initial intelligent behavior modeling model based on it to obtain the intelligent behavior modeling model;

[0015] The decision-making behavior module is used to obtain the real-time status data of the intelligent agent and input it into the intelligent behavior modeling model to obtain the decision-making behavior.

[0016] Furthermore, according to the environmental state data in the interaction data set, the state information characteristics of the environment in which the agent is located are extracted to obtain the position information, speed information and environmental information, including:

[0017] According to the environmental status data, the relative position between the intelligent agent and the monitoring device is obtained;

[0018] By filtering the relative position for noise, the positioning error caused by environmental interference is eliminated and mapped to a three-dimensional coordinate system to obtain the position information;

[0019] By obtaining the position information of adjacent time points, the speed of the agent is calculated to obtain the speed information;

[0020] According to the environmental status data, obtain the environmental information recorded by the sensor;

[0021] By performing time alignment on the position information, speed information and environment information, it is ensured that the position information, speed information and environment information are synchronized at the same time point.

[0022] Furthermore, according to the behavior action data in the interaction data set, the operation instruction features executed by the agent are extracted to obtain movement instructions, rotation instructions and adjustment instructions, including:

[0023] According to the behavior action data, the timestamp, position change value, speed change value, posture change value and action category of the intelligent agent's execution behavior are extracted to obtain the behavior operation data set;

[0024] According to the preset change value data set and the behavior operation data set, the position change value, speed change value, and posture change value that are less than the preset change value are screened out to obtain significant change data and slight change data;

[0025] Determine the moving distance and moving direction according to the position change value and speed change value in the significant change data, and generate a moving instruction based on the distance and direction;

[0026] Determine the rotation amplitude and rotation direction according to the posture change value in the significant change data, and generate a rotation instruction based on the rotation amplitude and rotation direction;

[0027] The adjustment amplitude is calculated according to the position change value, speed change value and posture change value in the slight change data. When the adjustment amplitude is less than the preset adjustment amplitude, an adjustment instruction is generated according to the adjustment amplitude.

[0028] Furthermore, according to the execution result data in the interaction data set, the execution result characteristics of the agent's execution behavior are extracted to obtain the behavior completion ratio, behavior execution time and energy consumption, including:

[0029] According to the execution result data, the number of executed behaviors is obtained, and the completion ratio of behaviors is obtained;

[0030] The behavior execution time is obtained by recording the time from the start of the behavior execution to the completion of the behavior execution;

[0031] According to the behavior execution time, the energy consumed by executing the behavior is counted to obtain the energy consumption.

[0032] Furthermore, by vectorizing the position information, speed information and environment information, a multi-dimensional state vector data set is obtained, and the environment space of the intelligent agent is constructed based on it, including:

[0033] According to the three-dimensional coordinate system, the position information and the speed information are converted into vectors to obtain a position information vector and a speed information vector;

[0034] By converting environmental information into a multi-dimensional vector, an environmental information vector is obtained;

[0035] By combining the position information vector, the speed information vector and the environment information vector into one vector, a multi-dimensional state vector is obtained;

[0036] By collecting multidimensional state vectors at multiple monitoring time points, a multidimensional state vector data set is obtained;

[0037] The environment space is constructed by converting a multi-dimensional state vector data set into a data point set.

[0038] Furthermore, by discretizing the movement instructions, rotation instructions, and adjustment instructions, a standard action instruction dataset is obtained, and the behavior space of the agent is constructed based on it, including:

[0039] By counting the movement instructions, rotation instructions and adjustment instructions, analyzing the movement range and minimum movement distance, rotation range and minimum rotation angle, adjustment range and minimum adjustment amplitude of the agent, the discrete range and discrete granularity of each instruction are determined, and a discrete standard data set is obtained;

[0040] According to the discrete standard data set, each instruction is divided into a discrete range according to its discrete granularity, and a discrete set of each instruction is determined to obtain a discrete instruction data set;

[0041] According to the discrete instruction data set, one action is determined by three different instructions, and a standard action instruction data set is obtained;

[0042] Based on the standard action instruction data set, each action performed by the agent is determined and the behavior space is constructed.

[0043] Furthermore, according to the behavior completion ratio, behavior execution time and energy consumption, the execution result values ​​of the behaviors in the behavior space executed by the agent in different environment spaces are calculated, including:

[0044] The calculation formula of the execution result value is: ,

[0045] in, For the The execution result value of a behavior, is the index of the behavior, is the behavior completion ratio, It is the preset behavior completion ratio threshold. For the The energy consumption of each behavior, For the The execution time of each behavior, is the coefficient.

[0046] Furthermore, an initial intelligent behavior modeling model is constructed based on the deep Q network, including:

[0047] Configure the input layer to receive the multi-dimensional state vector in the environment space and obtain the environment state;

[0048] A multi-layer hidden layer is configured to determine the number of hidden layers according to the dimension of the environment space, and the environment state is analyzed to extract feature vectors of different dimensions in the environment state to obtain an environment feature vector;

[0049] Configure a fully connected layer to receive the environment feature vector output by multiple hidden layers, map it to the behavior space, and calculate the Q value of different behaviors in the behavior space to obtain a Q value set;

[0050] Configure the output layer to output the Q value set.

[0051] Furthermore, a fully connected layer is configured to receive the environment feature vector output by the multi-layer hidden layer, map it to the behavior space, and calculate the Q value of different behaviors in the behavior space to obtain a Q value set, including:

[0052] According to the dimension of the environment space and the total number of behaviors in the behavior space, the influence of the environment characteristics on different behaviors is identified, and the weight matrix is ​​obtained;

[0053] Perform linear transformation based on the weight matrix and the environment feature vector, map the environment feature vector to the behavior space, ensure that each behavior corresponds to a mapping value, and obtain a linear transformation data set;

[0054] By performing nonlinear activation on the linearly changing data set, the negative numbers in the mapping value are truncated to zero, only the non-negative numbers are retained, and the Q value of the behavior is calculated based on it;

[0055] The calculation formula of the Q value is: ,

[0056] in, In the behavior space The Q value of a behavior, For the The environmental feature vector, is the index of the environmental feature vector, is the total number of environmental feature vectors, In the behavior space The mapping value of each behavior, is the index of the behavior, is the total number of behaviors, In the behavior space The execution result value of a behavior, is the coefficient.

[0057] Furthermore, according to the environment space, the behavior space and the execution result value, an experience recycling pool is set, and the initial intelligent behavior modeling model is trained based on it to obtain an intelligent behavior modeling model, including:

[0058] According to the multi-dimensional state vector in the environment space, the experience replay pool is set;

[0059] Determine the training environment state by randomly selecting a multi-dimensional state vector from the experience replay pool;

[0060] Input the training environment state into the initial intelligent behavior modeling model to obtain a training Q value set;

[0061] According to the training Q value set, the behavior matched by the training Q value with the largest value is selected to obtain the training behavior;

[0062] The intelligent agent executes the training behavior and obtains the training execution result;

[0063] According to the training execution results, the training execution result value is calculated and the next state at the next monitoring time point is recorded;

[0064] The training environment state, training action, training execution result value and next state are combined into a four-tuple and stored in the experience replay pool;

[0065] The training difference is obtained by calculating the difference between the training execution result value and the execution result value;

[0066] According to the training difference, the parameters of the initial intelligent behavior modeling model are adjusted. When the training difference is less than the preset difference, the training is terminated to obtain the intelligent behavior modeling model.

[0067] The above solution of the present invention includes at least the following beneficial effects:

[0068] The present invention extracts the multi-dimensional features of the agent's behavior, including environmental status, behavioral actions, and execution results, by deeply analyzing the interactive data set. This multi-level feature extraction method comprehensively reflects the actual behavioral state of the agent and provides detailed input data for subsequent modeling and optimization. By combining speed information with position information, the system can accurately analyze the movement trajectory of the agent, while the behavior completion ratio and energy consumption directly reflect the efficiency of behavior execution and resource utilization. This refined behavior feature description method makes the input of the intelligent behavior modeling model more accurate, effectively solving the problem of insufficient generalization ability of the intelligent behavior modeling model due to single features.

[0069] The present invention constructs an environment space and a behavior space, and systematically processes the environment state and behavior instructions of the intelligent agent respectively. The design of environmental information vectorization and behavioral instruction discretization simplifies complex input data into standardized data sets, greatly reducing the complexity of model calculation. By vectorizing the environment state, the model can quickly obtain key information such as the position and speed of the intelligent agent in the environment; and the discretized behavioral instructions ensure the efficiency and consistency of model training. This design improves the efficiency of data organization and reduces the computational overhead of the system. In addition, the discretized behavioral instructions can also significantly reduce the dimension of system training and shorten the convergence time of the intelligent behavior modeling model, thereby improving the overall modeling efficiency.

[0070] The present invention converts complex behavioral effects into clear numerical indicators by quantifying the behavior completion ratio, behavior execution time and energy consumption. The precise quantification of this data enables the system to comprehensively consider the balance between efficiency, time and resource utilization in behavior modeling. The behavior completion ratio can intuitively reflect the success rate of the intelligent agent in completing the task, while the energy consumption provides the resource cost information of the behavior. Through the comprehensive analysis of these quantitative indicators, the system can generate more optimized behavioral decisions. Compared with the evaluation method based on rules or a single indicator, this module significantly improves the accuracy and reliability of behavior modeling, and provides a more scientific decision-making basis for the system.

[0071] The present invention can construct an initial intelligent behavior modeling model based on a deep Q network, and realize reinforcement learning in combination with an experience replay pool. This design enables the intelligent agent to continuously optimize its behavior strategy in a multi-dimensional environment. By replaying past experience, the intelligent agent can identify and avoid inefficient behaviors and gradually optimize its decision-making ability. The introduction of the deep Q network enables the system to autonomously explore the optimal solution without clear labeled data. Compared with traditional machine learning that relies on large-scale labeled data, this autonomous learning and optimization capability not only reduces the need for human intervention, but also significantly improves the versatility and adaptability of the intelligent behavior modeling model.

[0072] The present invention greatly improves the adaptability and scalability of the intelligent behavior modeling model through the representation of environmental space and behavior space. This modular design enables the intelligent agent to quickly adapt to different environmental conditions and behavioral requirements. The construction method of the environmental space allows the model to switch seamlessly between different three-dimensional scenes, and the discrete design of the behavior space enables the model to be compatible with a variety of behavior instruction types. Compared with traditional models that rely on specific environments or tasks, this design provides a broader application space for the intelligent behavior modeling model and provides a basic framework for system module expansion and function iteration. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 It is a flowchart of an intelligent behavior modeling system based on deep reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0075] like Figure 1As shown, an embodiment of the present invention proposes an intelligent behavior modeling system based on deep reinforcement learning, and the system includes:

[0076] The data acquisition module is used to obtain the historical state data of the intelligent agent from the monitoring device of the intelligent agent, and the historical state data includes environmental state data, behavior action data and execution result data;

[0077] The data processing module is used to pre-process the historical status data, remove redundant data, and match it according to the monitoring time to obtain the interactive data set;

[0078] The feature extraction module is used to extract the state information features of the environment where the agent is located according to the environmental state data in the interactive data set, and obtain the position information, speed information and environmental information; extract the operation instruction features executed by the agent according to the behavior action data in the interactive data set, and obtain the movement instruction, rotation instruction and adjustment instruction; extract the execution result features of the behavior executed by the agent according to the execution result data in the interactive data set, and obtain the behavior completion ratio, behavior execution time and energy consumption;

[0079] The environment space module is used to vectorize the position information, speed information and environment information to obtain a multi-dimensional state vector data set, and construct the environment space of the intelligent agent based on it;

[0080] The behavior space module is used to discretize movement instructions, rotation instructions, and adjustment instructions to obtain a standard action instruction dataset and construct the agent's behavior space based on it;

[0081] The execution result module is used to calculate the execution result value of the behavior in the behavior space executed by the agent in different environment spaces according to the completion ratio of the behavior, the execution time of the behavior and the energy consumption;

[0082] A model building module, used to build an initial intelligent behavior modeling model based on a deep Q network;

[0083] The model training module is used to set the experience recycling pool according to the environment space, behavior space and reward feedback value, and train the initial intelligent behavior modeling model based on it to obtain the intelligent behavior modeling model;

[0084] The decision-making behavior module is used to obtain the real-time status data of the intelligent agent and input it into the intelligent behavior modeling model to obtain the decision-making behavior.

[0085] In an embodiment of the present invention, historical status data of the intelligent agent is obtained from the monitoring device of the intelligent agent. By comprehensively collecting multi-dimensional data, the system can build a detailed historical behavior database; the historical status data is preprocessed to eliminate redundant data, and it is matched according to the monitoring time to obtain an interactive data set, which significantly improves the quality and structured degree of the data; based on the environmental status data in the interactive data set, the state information characteristics of the environment in which the intelligent agent is located are extracted to obtain position information, speed information and environmental information; based on the behavioral action data in the interactive data set, the operation instruction characteristics executed by the intelligent agent are extracted to obtain movement instructions, rotation instructions and adjustment instructions; based on the execution result data in the interactive data set, the execution result characteristics of the intelligent agent's execution behavior are extracted to obtain the behavior completion ratio, behavior execution time and energy consumption, and the key features contained in the data can be refined in a refined manner.

[0086] The position information, speed information and environmental information are vectorized to obtain a multi-dimensional state vector data set, and the environmental space of the intelligent agent is constructed based on it. By constructing a high-dimensional vectorized representation, an accurate description of the environmental state is provided for the model; the movement instructions, rotation instructions and adjustment instructions are discretized to obtain a standard action instruction data set, and the behavior space of the intelligent agent is constructed based on it, which can effectively regulate the behavior selection of the intelligent agent; according to the behavior completion ratio, behavior execution time and energy consumption, the execution result value of the behavior executed by the intelligent agent in the behavior space in different environmental spaces is calculated, which provides accurate effect feedback for the system's behavioral decision-making and ensures that the model can optimize the behavior strategy based on quantitative indicators.

[0087] An initial intelligent behavior modeling model is constructed based on a deep Q network to provide the system with an initial intelligent behavior modeling model with learning ability. An experience recycling pool is set according to the environment space, behavior space and reward feedback value, and the initial intelligent behavior modeling model is trained based on it to obtain an intelligent behavior modeling model. The reinforcement learning method is used to enable the intelligent agent to continuously optimize the behavior strategy through self-iteration. The real-time state data of the intelligent agent is obtained and input into the intelligent behavior modeling model to obtain the decision-making behavior, thereby realizing the real-time response capability of the intelligent agent and ensuring its flexibility and reliability in complex scenarios.

[0088] The data acquisition module is used to obtain the historical state data of the intelligent agent from the monitoring device of the intelligent agent. The historical state data includes environmental state data, behavior action data and execution result data, specifically including:

[0089] Deploy a variety of sensor devices on the intelligent agent and its surrounding environment, such as cameras, lidars, accelerometers, etc., to monitor the historical status data of the intelligent agent in real time, including environmental status, behavioral actions and execution results; record the behavior trajectory of the intelligent agent, such as position changes, posture changes, etc., through data acquisition equipment, and record environmental status data such as temperature and humidity, light, and obstacle positions in the environment; the collected historical status data is classified and stored by timestamp to ensure the integrity and synchronization of time series data.

[0090] The data processing module is used to pre-process the historical status data, remove redundant data, and match it according to the monitoring time to obtain the interactive data set, which specifically includes:

[0091] The collected historical status data is subjected to noise filtering and redundancy elimination, and the Kalman filtering method is used to reduce the error introduced by sensor noise; the environmental status data, behavioral action data and execution result data are matched by timestamp to form a structured interactive data set; after data cleaning, its units are unified and the interactive data set is saved in a distributed storage method to ensure data access efficiency under high concurrency.

[0092] In a preferred embodiment of the present invention, according to the environmental state data in the interactive data set, the state information characteristics of the environment in which the agent is located are extracted to obtain the position information, speed information and environmental information, including:

[0093] According to the environmental status data, the relative position between the intelligent agent and the monitoring device is obtained;

[0094] By filtering the relative position for noise, the positioning error caused by environmental interference is eliminated and mapped to a three-dimensional coordinate system to obtain the position information;

[0095] By obtaining the position information of adjacent time points, the speed of the agent is calculated to obtain the speed information;

[0096] According to the environmental status data, obtain the environmental information recorded by the sensor;

[0097] By performing time alignment on the position information, speed information and environment information, it is ensured that the position information, speed information and environment information are synchronized at the same time point.

[0098] In an embodiment of the present invention, the relative position between the intelligent agent and the monitoring device is obtained based on the environmental status data to provide basic spatial reference data for the subsequent process; the relative position is noise filtered to eliminate the positioning error caused by environmental interference, and is mapped to a three-dimensional coordinate system to obtain position information, thereby ensuring that the spatial state of the intelligent agent can be consistently represented and analyzed on a global scale; the position information of adjacent time points is obtained to calculate the speed of the intelligent agent to obtain speed information, and by differential calculation of the position information, the system can dynamically evaluate the movement speed of the intelligent agent in the environment; based on the environmental status data, the environmental information recorded by the sensor is obtained to fully understand the working environment of the intelligent agent; the position information, speed information and environmental information are time-aligned to ensure that the position information, speed information and environmental information are synchronized at the same time point, thereby ensuring the temporal consistency of the data, so that each feature can be analyzed and compared at the same time point.

[0099] Among them, by filtering the relative position for noise, eliminating the positioning error caused by environmental interference, and mapping it to a three-dimensional coordinate system, the position information is obtained, including:

[0100] The system will perform noise filtering on the acquired relative position data, remove irrelevant noise information in the original data through filters, and smooth it. After filtering the relative position data, the system will map these data into a three-dimensional coordinate system. Through a geometric transformation algorithm, the local data collected by the sensor is converted into coordinate points in the global coordinate system. This conversion is usually based on known datum points or reference points, and is mapped using relevant mathematical models. Through this mapping, the system can obtain a more unified and standardized spatial representation.

[0101] Among them, according to the environmental status data, the environmental information recorded by the sensor is obtained, which specifically includes:

[0102] The agent's environmental information is also crucial to behavior modeling, such as temperature, humidity, obstacle location, etc. The system obtains real-time environmental status data through various environmental sensors, such as temperature and humidity sensors, visual sensors, lidar, ultrasonic sensors, etc. These data reflect the state of the agent's surrounding environment at a specific point in time, such as the presence of obstacles, climatic conditions, etc. The system combines data from different sensors through data fusion technology to form a more comprehensive and accurate description of the environmental status.

[0103] In a preferred embodiment of the present invention, based on the behavior action data in the interaction data set, the operation instruction features executed by the agent are extracted to obtain the movement instruction, rotation instruction and adjustment instruction, including:

[0104] According to the behavior action data, the timestamp, position change value, speed change value, posture change value and action category of the intelligent agent's execution behavior are extracted to obtain the behavior operation data set;

[0105] According to the preset change value data set and the behavior operation data set, the position change value, speed change value, and posture change value that are less than the preset change value are screened out to obtain significant change data and slight change data;

[0106] Determine the moving distance and moving direction according to the position change value and speed change value in the significant change data, and generate a moving instruction based on the distance and direction;

[0107] Determine the rotation amplitude and rotation direction according to the posture change value in the significant change data, and generate a rotation instruction based on the rotation amplitude and rotation direction;

[0108] The adjustment amplitude is calculated according to the position change value, speed change value and posture change value in the slight change data. When the adjustment amplitude is less than the preset adjustment amplitude, an adjustment instruction is generated according to the adjustment amplitude.

[0109] In an embodiment of the present invention, based on the behavioral action data, the timestamp, position change value, speed change value, posture change value and action category of the intelligent agent's execution of the behavior are extracted to obtain a behavioral operation data set, which provides an accurate data basis for subsequent analysis and feature extraction; based on the preset change value data set and the behavioral operation data set, the position change value, speed change value, and posture change value that are less than the preset change value are screened out to obtain significant change data and slight change data, and the system can accurately identify the key changes in the behavior of the intelligent agent; based on the position change value and speed change value in the significant change data, the moving distance and moving direction are determined, and a moving instruction is generated based on them, and the system can ensure that the movement of the intelligent agent meets the requirements of the actual environment; based on the posture change value in the significant change data, the rotation amplitude and rotation direction are determined, and a rotation instruction is generated based on them, so that the intelligent agent is more flexible when performing rotation operations; based on the position change value, speed change value and posture change value in the slight change data, the adjustment amplitude is calculated, and when the adjustment amplitude is less than the preset adjustment amplitude, an adjustment instruction is generated based on the adjustment amplitude, thereby enhancing the degree of refinement of the intelligent agent in micro-operations.

[0110] Among them, according to the behavior action data, the timestamp, position change value, speed change value, posture change value and action category of the intelligent agent's execution behavior are extracted to obtain the behavior operation data set, which specifically includes:

[0111] Extract timestamps from sensors or monitoring devices to ensure that behavioral operations have clear time records;

[0112] Calculate the position change value through the change of the position information of the intelligent body;

[0113] Calculate the speed change value based on the difference in position information at adjacent time points;

[0114] Use the agent's posture sensor to record its posture change value;

[0115] Based on the action category recorded by the sensor, the action type is labeled, such as move, rotate, or adjust behavior.

[0116] Among them, the adjustment range is calculated according to the position change value, speed change value and posture change value in the slight change data. When the adjustment range is less than the preset adjustment range, an adjustment instruction is generated according to the adjustment range, which specifically includes:

[0117] The change values ​​in the slight change data are accumulated and the required adjustment range is calculated. When the adjustment range is less than the set threshold, an adjustment instruction is generated, such as a 5-degree fine-tuning to the left. The adjustment instruction is input into the control system of the intelligent body to ensure the accurate execution of small adjustment operations.

[0118] In a preferred embodiment of the present invention, based on the execution result data in the interaction data set, the execution result characteristics of the agent's execution behavior are extracted to obtain the behavior completion ratio, behavior execution time and energy consumption, including:

[0119] According to the execution result data, the number of executed behaviors is obtained, and the completion ratio of behaviors is obtained;

[0120] The behavior execution time is obtained by recording the time from the start of the behavior execution to the completion of the behavior execution;

[0121] According to the behavior execution time, the energy consumed by executing the behavior is counted to obtain the energy consumption.

[0122] In an embodiment of the present invention, based on the execution result data, the number of executed behaviors is obtained, and the behavior completion ratio is obtained. By accurately counting the behavior completion ratio, a quantitative execution effect indicator is provided for subsequent analysis; by recording the time from the start of the behavior to the completion of the behavior, the behavior execution time is obtained, and the system can intuitively evaluate the efficiency of the intelligent body; based on the behavior execution time, the energy consumed by the execution of the behavior is counted to obtain the energy consumption, which can measure the execution efficiency of the intelligent body.

[0123] Among them, according to the execution result data, the number of executed behaviors is obtained, and the behavior completion ratio is obtained, which specifically includes:

[0124] The system first obtains the number of recorded behaviors from the agent's execution result data. The number of behaviors includes the total number of behaviors that the agent attempts to execute within a specified time and the number of behaviors that are successfully completed. By comparing the behavior identifiers in the records, such as task ID or action type, the system filters out the behaviors that are successfully completed and counts their proportion.

[0125] Among them, according to the behavior execution time, the energy consumed by the execution behavior is counted to obtain the energy consumption, which specifically includes:

[0126] Based on the energy usage information recorded in the execution result data, the system counts the energy consumed to complete each behavior, reads the energy status of the agent at the beginning and end of the behavior execution, such as the power or fuel remaining, and calculates the consumption through the difference between the two. Energy consumption = initial energy - remaining energy. For the energy consumption calculation of complex tasks, the system decomposes the behavior into multiple independent steps, counts the energy consumption of each part one by one, and finally accumulates them to obtain the total value.

[0127] In a preferred embodiment of the present invention, by vectorizing the position information, speed information and environment information, a multi-dimensional state vector data set is obtained, and the environment space of the intelligent agent is constructed based on the multi-dimensional state vector data set, including:

[0128] According to the three-dimensional coordinate system, the position information and the speed information are converted into vectors to obtain a position information vector and a speed information vector;

[0129] By converting environmental information into a multi-dimensional vector, an environmental information vector is obtained;

[0130] By combining the position information vector, the speed information vector and the environment information vector into one vector, a multi-dimensional state vector is obtained;

[0131] By collecting multidimensional state vectors at multiple monitoring time points, a multidimensional state vector data set is obtained;

[0132] The environment space is constructed by converting a multi-dimensional state vector data set into a data point set.

[0133] In an embodiment of the present invention, according to a three-dimensional coordinate system, position information and speed information are converted into vectors to obtain a position information vector and a speed information vector, thereby effectively improving the accuracy and consistency of the position information and speed information; by converting environmental information into a multidimensional vector to obtain an environmental information vector, through normalization and dimensionality reduction operations, the system can more effectively mine key environmental features and reduce noise interference; by merging the position information vector, the speed information vector and the environmental information vector into one vector, a multidimensional state vector is obtained, thereby ensuring the synchronization and correlation of different data sources; by collecting multidimensional state vectors at multiple monitoring time points, a multidimensional state vector data set is obtained, and through the multidimensional state vector data set, the problem of insufficient training samples can be effectively addressed; by converting the multidimensional state vector data set into a data point set, an environmental space is constructed, providing a clear state representation framework for the intelligent agent.

[0134] The environmental information vector is obtained by converting the environmental information into a multi-dimensional vector, which specifically includes:

[0135] Environmental information includes data recorded by sensors such as temperature, humidity, and light intensity. The system first normalizes the original environmental data and removes outliers to reduce the impact of data deviation on the model. It then maps various environmental parameters into independent numerical vectors and uses principal component analysis to reduce the interference of redundant information and generate a unified multi-dimensional environmental information vector.

[0136] Among them, the environment space is constructed by converting the multidimensional state vector data set into a data point set, which specifically includes:

[0137] The system converts the multidimensional state vector data set into a data point set and constructs an environmental space in the high-dimensional space. It classifies the multidimensional state vectors through a clustering algorithm, identifies the patterns of different environmental states, and constructs sub-areas of the high-dimensional environmental space based on the clustering results. The system further uses dimensionality reduction technology to visualize the high-dimensional space, making the environmental state of the intelligent agent more intuitive.

[0138] In a preferred embodiment of the present invention, by discretizing the movement instructions, rotation instructions and adjustment instructions, a standard action instruction data set is obtained, and the behavior space of the intelligent agent is constructed based on the standard action instruction data set, including:

[0139] By counting the movement instructions, rotation instructions and adjustment instructions, analyzing the movement range and minimum movement distance, rotation range and minimum rotation angle, adjustment range and minimum adjustment amplitude of the agent, the discrete range and discrete granularity of each instruction are determined, and a discrete standard data set is obtained;

[0140] According to the discrete standard data set, each instruction is divided into a discrete range according to its discrete granularity, and a discrete set of each instruction is determined to obtain a discrete instruction data set;

[0141] According to the discrete instruction data set, one action is determined by three different instructions, and a standard action instruction data set is obtained;

[0142] Based on the standard action instruction data set, each action performed by the agent is determined and the behavior space is constructed.

[0143] In the embodiment of the present invention, by statistically analyzing the movement range and minimum movement distance, rotation range and minimum rotation angle, adjustment range and minimum adjustment amplitude of the intelligent body, the discrete range and discrete granularity of each instruction are determined, and a discrete standard data set is obtained, which lays a data foundation for the discretization of instructions; according to the discrete standard data set, each instruction is divided into a discrete range according to its discrete granularity, and a discrete set of each instruction is determined to obtain a discrete instruction data set, which can reduce data complexity while retaining the details of the behavioral characteristics;

[0144] According to the discrete instruction data set, one action is determined by three different instructions, and a standard action instruction data set is obtained, which provides a structured expression for the diversity of the agent's behavior. According to the standard action instruction data set, each action performed by the agent is determined, and a behavior space is constructed, so that the agent's behavior has a clear mathematical representation.

[0145] Among them, by statistically analyzing the movement range and minimum movement distance, rotation range and minimum rotation angle, adjustment range and minimum adjustment amplitude of the agent, the discrete range and discrete granularity of each instruction are determined, and the discrete standard data set is obtained, which specifically includes:

[0146] First, the system conducts a comprehensive statistical analysis of the historical behavior data of the agent in the environment, including the movement range, rotation range and adjustment range, as well as the minimum change amplitude of each instruction. By extracting these data features, the system can form a comprehensive statistical database. For example, the statistics of the movement range can determine the possible minimum and maximum movement distances of the agent by analyzing the change value of the position information; the statistics of the rotation range can extract the extreme values ​​and common change patterns of the rotation angle based on the posture change value; the analysis of the adjustment range is based on the subtle changes in speed, position or posture to determine the precise range of the adjustment amplitude.

[0147] According to the discrete instruction data set, one action is determined by three different instructions to obtain a standard action instruction data set, which specifically includes:

[0148] Based on the discrete instruction data set, the system maps each discrete instruction combination into a standardized action instruction. For example, the combination of moving 1 meter, rotating 5°, and adjusting 0.1 amplitude can be defined as action instruction A1; the combination of moving 2 meters, rotating 10°, and adjusting 0.2 amplitude is defined as A2. In this process, the system needs to set the number and distribution of standard action instructions based on task requirements, and assign a unique code to each action instruction.

[0149] Among them, according to the standard action instruction data set, each action performed by the intelligent agent is determined and the behavior space is constructed, which specifically includes:

[0150] Based on the standard action instruction data set, the system maps each action into a multi-dimensional space to form a complete behavior space. For example, the discretization results of the three instructions can constitute a coordinate system in a three-dimensional space, and each action instruction corresponds to a coordinate point. Through multiple sampling and optimization, the system ensures that the behavior space covers all behavioral requirements of the intelligent agent and eliminates repeated or redundant action points.

[0151] In a preferred embodiment of the present invention, the execution result value of the behavior in the behavior space executed by the agent in different environment spaces is calculated according to the behavior completion ratio, behavior execution time and energy consumption, including:

[0152] The calculation formula of the execution result value is: ,

[0153] in, For the The execution result value of a behavior, is the index of the behavior, is the behavior completion ratio, It is the preset behavior completion ratio threshold. For the The energy consumption of each behavior, For the The execution time of each behavior, is the coefficient.

[0154] In an embodiment of the present invention, the calculation of the execution result value comprehensively considers the multi-dimensional factors of completion ratio, time and energy consumption, and provides a unified and accurate evaluation index for the intelligent agent behavior modeling. By dynamically adjusting the weights of each parameter, the system can adapt to a variety of task scenarios, thereby improving the generalization ability and practicality of behavior modeling.

[0155] in, It is the behavior completion ratio correction factor, which adopts the form of logical function to smoothly adjust the influence of the behavior completion ratio to make it more stable and reasonable. Through the behavior completion ratio correction factor, the system can flexibly adjust the influence of the completion ratio on the execution result. Even if the behavior completion ratio is slightly lower than the threshold, the system can still avoid a sharp drop in the execution result, thereby improving the robustness and adaptability of the model.

[0156] in, The behavior time weight reflects the balance between energy consumption and time, unifies the behavior time and energy consumption values ​​to the same scale, and ensures that the comprehensive evaluation of the execution results is more scientific. By calculating the time weight factor, the system can effectively quantify the relative importance of time and energy consumption. For high-efficiency tasks, it can reduce the impact of energy consumption on the result value, and for high-energy consumption tasks, it increases the impact weight of time, so that the result evaluation is more in line with actual needs.

[0157] in, It is an energy consumption-time decay function, which reflects the negative impact of high-energy consumption and long-time behaviors on the execution results, and gives certain incentives to low-energy consumption and short-time behaviors. It balances the impact of energy consumption and time on the behavior results through exponential decay, and enhances the dynamics and adaptability of the execution result calculation. This decay function can accurately capture the behavior characteristics and improve the rationality of the evaluation results.

[0158] In a preferred embodiment of the present invention, an initial intelligent behavior modeling model is constructed based on a deep Q network, including:

[0159] Configure the input layer to receive the multi-dimensional state vector in the environment space and obtain the environment state;

[0160] A multi-layer hidden layer is configured to determine the number of hidden layers according to the dimension of the environment space, and the environment state is analyzed to extract feature vectors of different dimensions in the environment state to obtain an environment feature vector;

[0161] Configure a fully connected layer to receive the environment feature vector output by multiple hidden layers, map it to the behavior space, and calculate the Q value of different behaviors in the behavior space to obtain a Q value set;

[0162] Configure the output layer to output the Q value set.

[0163] In an embodiment of the present invention, an input layer is configured to receive a multi-dimensional state vector in an environmental space to obtain an environmental state, thereby ensuring high-quality input of data; a multi-layer hidden layer is configured to determine the number of hidden layers according to the dimension of the environmental space, and to analyze the environmental state, extract feature vectors of different dimensions in the environmental state, and obtain an environmental feature vector. The setting of the multi-layer hidden layer allows the model to capture the complex relationship between different dimensions in the environmental space, so that the model can better understand the nonlinear correlation of environmental characteristics; a fully connected layer is configured to receive the environmental feature vector output by the multi-layer hidden layer, map it to the behavior space, and calculate the Q value of different behaviors in the behavior space to obtain a Q value set. The effective mapping of the feature vector and the behavior space is achieved through the weight matrix and the nonlinear activation function, so that the model can perform quantitative analysis on the potential benefits of each behavior; an output layer is configured to output a Q value set. By outputting the Q value set and selecting the optimal behavior based on a greedy strategy, the system can quickly generate the optimal decision of the intelligent agent.

[0164] Among them, multiple hidden layers are configured to determine the number of hidden layers according to the dimension of the environment space, and the environment state is analyzed to extract feature vectors of different dimensions in the environment state to obtain the environment feature vector, which specifically includes:

[0165] By counting the dimensions of the multidimensional state vector in the statistical environment space, as the basis for the design of the hidden layer structure, multiple hidden layers are set. Each layer uses a nonlinear activation function to extract features and nonlinearly map the input data. Each hidden layer maps the high-dimensional input data to a low-dimensional feature representation through matrix operations and activation functions, while retaining the key features. The output data of the hidden layer is weighted initialized and bias calibrated to ensure numerical stability during model training.

[0166] In a preferred embodiment of the present invention, a fully connected layer is configured to receive the environment feature vector output by the multiple hidden layers, map it to the behavior space, and calculate the Q value of different behaviors in the behavior space to obtain a Q value set, including:

[0167] According to the dimension of the environment space and the total number of behaviors in the behavior space, the influence of the environment characteristics on different behaviors is identified, and the weight matrix is ​​obtained;

[0168] Perform linear transformation based on the weight matrix and the environment feature vector, map the environment feature vector to the behavior space, ensure that each behavior corresponds to a mapping value, and obtain a linear transformation data set;

[0169] By performing nonlinear activation on the linearly changing data set, the negative numbers in the mapping value are truncated to zero, only the non-negative numbers are retained, and the Q value of the behavior is calculated based on it;

[0170] The calculation formula of the Q value is: ,

[0171] in, In the behavior space The Q value of a behavior, For the The environmental feature vector, is the index of the environmental feature vector, is the total number of environmental feature vectors, In the behavior space The mapping value of each behavior, is the index of the behavior, is the total number of behaviors, In the behavior space The execution result value of a behavior, is the coefficient.

[0172] In an embodiment of the present invention, based on the dimension of the environmental space and the total number of behaviors in the behavior space, the degree of influence of environmental features on different behaviors is identified to obtain a weight matrix. By generating the weight matrix, the system can clearly understand the importance of environmental features in the execution of different behaviors; a linear transformation is performed based on the weight matrix and the environmental feature vector, and the environmental feature vector is mapped to the behavior space to ensure that each behavior corresponds to a mapping value, thereby obtaining a linear transformation data set, effectively compressing the high-dimensional environmental feature space into the behavior space, and retaining feature information that has a key impact on behavioral decisions; by performing nonlinear activation on the linear change data set, negative numbers in the mapping value are truncated to zero, and only non-negative numbers are retained, and the Q value of the behavior is calculated based on it, thereby ensuring that the Q value of the behavior space only contains positive information and avoiding interference of negative values ​​on model optimization.

[0173] Among them, according to the dimension of the environmental space and the total number of behaviors in the behavioral space, the influence of environmental characteristics on different behaviors is identified, and the weight matrix is ​​obtained, which specifically includes:

[0174] First, the correlation between environmental features and behaviors is analyzed through the dimensions of the environmental space and the total number of behaviors in the behavioral space. The correlation is quantified by counting the influence of environmental states such as position information, speed information and environmental information on the success rate of each behavior such as movement, rotation and adjustment. For each dimensional feature in the environmental state, its weight value when different behaviors are successfully executed is calculated. This can be done through correlation coefficient analysis or sensitivity analysis based on gradient back propagation. Finally, these weight values ​​are organized into a matrix form, where the rows of the matrix represent the dimensions of environmental features, the columns represent different behaviors, and each element of the matrix is ​​the weight of the corresponding environmental feature on the behavior.

[0175] Among them, a linear transformation is performed according to the weight matrix and the environmental feature vector, and the environmental feature vector is mapped to the behavior space to ensure that each behavior corresponds to a mapping value, and a linear transformation data set is obtained, which specifically includes:

[0176] According to the weight matrix, each environmental feature vector is linearly transformed, and the dot product operation is performed on the environmental feature vector and the column vector of the weight matrix one by one to generate the initial mapping value corresponding to each behavior. The linearly transformed data is stored as a linear transformation data set, in which each behavior corresponds to a mapping value.

[0177] In a preferred embodiment of the present invention, an experience recovery pool is set according to the environment space, the behavior space and the execution result value, and the initial intelligent behavior modeling model is trained according to the experience recovery pool to obtain the intelligent behavior modeling model, including:

[0178] According to the multi-dimensional state vector in the environment space, the experience replay pool is set;

[0179] Determine the training environment state by randomly selecting a multi-dimensional state vector from the experience replay pool;

[0180] Input the training environment state into the initial intelligent behavior modeling model to obtain a training Q value set;

[0181] According to the training Q value set, the behavior matched by the training Q value with the largest value is selected to obtain the training behavior;

[0182] The intelligent agent executes the training behavior and obtains the training execution result;

[0183] According to the training execution results, the training execution result value is calculated and the next state at the next monitoring time point is recorded;

[0184] The training environment state, training action, training execution result value and next state are combined into a four-tuple and stored in the experience replay pool;

[0185] The training difference is obtained by calculating the difference between the training execution result value and the execution result value;

[0186] According to the training difference, the parameters of the initial intelligent behavior modeling model are adjusted. When the training difference is less than the preset difference, the training is terminated to obtain the intelligent behavior modeling model.

[0187] In an embodiment of the present invention, an experience replay pool is set according to the multidimensional state vector in the environment space, and by centrally storing historical experience data, resource waste of repeated collection of training samples is avoided, thereby improving data utilization; the training environment state is determined by randomly selecting the multidimensional state vector in the experience replay pool, and the random selection method reduces the time correlation between samples and improves the diversity of training data; the training environment state is input into the initial intelligent behavior modeling model to obtain a training Q value set, and the environment state is mapped to a high-dimensional feature space through processing by the initial intelligent behavior modeling model, and corresponds to the behavior space; according to the training Q value set, the behavior matched with the training Q value with the largest numerical value is selected to obtain the training behavior, and selecting the behavior with the largest Q value can guide the intelligent agent to execute the current optimal decision.

[0188] The training behavior is executed by the intelligent agent to obtain the training execution result, and the training behavior is executed and the execution result is collected, which provides feedback data in the real environment for model optimization; according to the training execution result, the training execution result value is calculated, and the next state at the next monitoring time point is recorded. The training execution result value is convenient for the target quantification during model optimization, and the recording of the next state provides the necessary dynamic environment information for subsequent training samples; the training environment state, training action, training execution result value and next state are formed into a quaternary group and stored in the experience replay pool, so that the initial intelligent behavior modeling model can fully reproduce the entire process from environmental state to behavior decision and then to result feedback during training; according to the training difference, the parameters of the initial intelligent behavior modeling model are adjusted. When the training difference is less than the preset difference, the training is terminated to obtain the intelligent behavior modeling model. The parameters of the initial intelligent behavior modeling model are optimized through the gradient descent algorithm. The system can continuously approach the global optimal solution, improve the accuracy and efficiency of behavior modeling, and combine with reasonable setting of termination conditions to ensure that the model achieves the expected performance while training efficiently.

[0189] Among them, according to the multidimensional state vector in the environment space, the experience replay pool is set, specifically including:

[0190] Based on the multidimensional state vector in the environment space, a storage structure such as a circular queue is created to record the environment state, behavioral actions, execution results and next state data of the agent at different time points. The capacity of the experience replay pool can be set to a fixed value based on computing resources and task complexity. When new data is added, the part exceeding the capacity is replaced on a first-in-first-out basis.

[0191] Among them, according to the training difference, the parameters of the initial intelligent behavior modeling model are adjusted. When the training difference is less than the preset difference, the training is terminated to obtain the intelligent behavior modeling model, which specifically includes:

[0192] Through the back propagation algorithm of deep reinforcement learning, the weights and bias parameters of the initial intelligent behavior modeling model are adjusted according to the training difference. The training difference calculation is based on the mean square error, and the optimization algorithm is used to update the parameters. When the error is less than the set threshold, the training ends and the optimized intelligent behavior modeling model is output. The training process sets the termination condition according to the error size. When the condition is met, the training is stopped and the final optimized intelligent behavior modeling model is output for practical application.

[0193] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An intelligent behavior modeling system based on deep reinforcement learning, characterized in that: The system comprises: The data acquisition module is used to obtain the historical state data of the intelligent agent from the monitoring device of the intelligent agent, and the historical state data includes environmental state data, behavior action data and execution result data; The data processing module is used to pre-process the historical status data, remove redundant data, and match it according to the monitoring time to obtain the interactive data set; The feature extraction module is used to extract the state information features of the environment where the agent is located according to the environmental state data in the interactive data set, and obtain the position information, speed information and environmental information; extract the operation instruction features executed by the agent according to the behavior action data in the interactive data set, and obtain the movement instruction, rotation instruction and adjustment instruction; extract the execution result features of the behavior executed by the agent according to the execution result data in the interactive data set, and obtain the behavior completion ratio, behavior execution time and energy consumption; The environment space module is used to vectorize the position information, speed information and environment information to obtain a multi-dimensional state vector data set, and construct the environment space of the intelligent agent based on it; The behavior space module is used to discretize movement instructions, rotation instructions, and adjustment instructions to obtain a standard action instruction dataset and construct the agent's behavior space based on it; The execution result module is used to calculate the execution result value of the behavior in the behavior space executed by the agent in different environment spaces according to the completion ratio of the behavior, the execution time of the behavior and the energy consumption; A model building module, used to build an initial intelligent behavior modeling model based on a deep Q network; The model training module is used to set the experience recycling pool according to the environment space, behavior space and reward feedback value, and train the initial intelligent behavior modeling model based on it to obtain the intelligent behavior modeling model; The decision-making behavior module is used to obtain the real-time status data of the intelligent agent and input it into the intelligent behavior modeling model to obtain the decision-making behavior.

2. The intelligent behavior modeling system based on deep reinforcement learning according to claim 1, characterized in that: According to the environmental state data in the interaction data set, the state information characteristics of the environment where the agent is located are extracted to obtain the position information, speed information and environmental information, including: According to the environmental status data, the relative position between the intelligent agent and the monitoring device is obtained; By filtering the relative position for noise, the positioning error caused by environmental interference is eliminated and mapped to a three-dimensional coordinate system to obtain the position information; By obtaining the position information of adjacent time points, the speed of the agent is calculated to obtain the speed information; According to the environmental status data, obtain the environmental information recorded by the sensor; By performing time alignment on the position information, speed information and environment information, it is ensured that the position information, speed information and environment information are synchronized at the same time point.

3. The intelligent behavior modeling system based on deep reinforcement learning according to claim 2, characterized in that: According to the behavior data in the interaction dataset, the characteristics of the operation instructions executed by the agent are extracted to obtain movement instructions, rotation instructions and adjustment instructions, including: According to the behavior action data, the timestamp, position change value, speed change value, posture change value and action category of the intelligent agent's execution behavior are extracted to obtain the behavior operation data set; According to the preset change value data set and the behavior operation data set, the position change value, speed change value, and posture change value that are less than the preset change value are screened out to obtain significant change data and slight change data; Determine the moving distance and moving direction according to the position change value and speed change value in the significant change data, and generate a moving instruction based on the distance and direction; Determine the rotation amplitude and rotation direction according to the posture change value in the significant change data, and generate a rotation instruction based on the rotation amplitude and rotation direction; The adjustment amplitude is calculated according to the position change value, speed change value and posture change value in the slight change data. When the adjustment amplitude is less than the preset adjustment amplitude, an adjustment instruction is generated according to the adjustment amplitude.

4. The intelligent behavior modeling system based on deep reinforcement learning according to claim 3, characterized in that: According to the execution result data in the interaction data set, the execution result characteristics of the agent's execution behavior are extracted to obtain the behavior completion ratio, behavior execution time and energy consumption, including: According to the execution result data, the number of executed behaviors is obtained, and the completion ratio of behaviors is obtained; The behavior execution time is obtained by recording the time from the start of the behavior execution to the completion of the behavior execution; According to the behavior execution time, the energy consumed by executing the behavior is counted to obtain the energy consumption.

5. The intelligent behavior modeling system based on deep reinforcement learning according to claim 4, characterized in that: By vectorizing the position information, speed information and environment information, a multi-dimensional state vector data set is obtained, and the environment space of the intelligent agent is constructed based on it, including: According to the three-dimensional coordinate system, the position information and the speed information are converted into vectors to obtain a position information vector and a speed information vector; By converting environmental information into a multi-dimensional vector, an environmental information vector is obtained; By combining the position information vector, the speed information vector and the environment information vector into one vector, a multi-dimensional state vector is obtained; By collecting multidimensional state vectors at multiple monitoring time points, a multidimensional state vector data set is obtained; The environment space is constructed by converting a multi-dimensional state vector data set into a data point set.

6. The intelligent behavior modeling system based on deep reinforcement learning according to claim 5, characterized in that: By discretizing movement instructions, rotation instructions, and adjustment instructions, we obtain a standard action instruction dataset, and based on it, we construct the agent’s behavior space, including: By counting the movement instructions, rotation instructions and adjustment instructions, analyzing the movement range and minimum movement distance, rotation range and minimum rotation angle, adjustment range and minimum adjustment amplitude of the agent, the discrete range and discrete granularity of each instruction are determined, and a discrete standard data set is obtained; According to the discrete standard data set, each instruction is divided into a discrete range according to its discrete granularity, and a discrete set of each instruction is determined to obtain a discrete instruction data set; According to the discrete instruction data set, one action is determined by three different instructions, and a standard action instruction data set is obtained; Based on the standard action instruction data set, each action performed by the agent is determined and the behavior space is constructed.

7. The intelligent behavior modeling system based on deep reinforcement learning according to claim 6, characterized in that: According to the behavior completion ratio, behavior execution time and energy consumption, the execution result value of the behavior in the behavior space executed by the agent in different environment spaces is calculated, including: The calculation formula of the execution result value is: ， in, For the The execution result value of each behavior, is the index of the behavior, is the behavior completion ratio, It is the preset behavior completion ratio threshold. For the The energy consumption of each behavior, For the The execution time of each behavior, is the coefficient.

8. The intelligent behavior modeling system based on deep reinforcement learning according to claim 7, characterized in that: Construct an initial intelligent behavior model based on a deep Q network, including: Configure the input layer to receive the multi-dimensional state vector in the environment space and obtain the environment state; A multi-layer hidden layer is configured to determine the number of hidden layers according to the dimension of the environment space, and the environment state is analyzed to extract feature vectors of different dimensions in the environment state to obtain an environment feature vector; Configure a fully connected layer to receive the environment feature vector output by multiple hidden layers, map it to the behavior space, and calculate the Q value of different behaviors in the behavior space to obtain a Q value set; Configure the output layer to output the Q value set.

9. The intelligent behavior modeling system based on deep reinforcement learning according to claim 8, characterized in that: Configure a fully connected layer to receive the environment feature vector output by the multi-layer hidden layer, map it to the behavior space, and calculate the Q value of different behaviors in the behavior space to obtain a Q value set, including: According to the dimension of the environment space and the total number of behaviors in the behavior space, the influence of the environment characteristics on different behaviors is identified, and the weight matrix is ​​obtained; Perform linear transformation based on the weight matrix and the environment feature vector, map the environment feature vector to the behavior space, ensure that each behavior corresponds to a mapping value, and obtain a linear transformation data set; By performing nonlinear activation on the linearly changing data set, the negative numbers in the mapping value are truncated to zero, only the non-negative numbers are retained, and the Q value of the behavior is calculated based on it; The calculation formula of the Q value is: ， in, In the behavior space The Q value of a behavior, For the The environmental feature vector, is the index of the environmental feature vector, is the total number of environmental feature vectors, In the behavior space The mapping value of each behavior, is the index of the behavior, is the total number of behaviors, In the behavior space The execution result value of each behavior, is the coefficient.

10. The intelligent behavior modeling system based on deep reinforcement learning according to claim 9, characterized in that: According to the environment space, behavior space and execution result value, an experience recycling pool is set, and the initial intelligent behavior modeling model is trained based on it to obtain an intelligent behavior modeling model, including: According to the multi-dimensional state vector in the environment space, the experience replay pool is set; Determine the training environment state by randomly selecting a multi-dimensional state vector from the experience replay pool; Input the training environment state into the initial intelligent behavior modeling model to obtain a training Q value set; According to the training Q value set, the behavior matched by the training Q value with the largest value is selected to obtain the training behavior; The intelligent agent executes the training behavior and obtains the training execution result; According to the training execution results, the training execution result value is calculated and the next state at the next monitoring time point is recorded; The training environment state, training action, training execution result value and next state are combined into a four-tuple and stored in the experience replay pool; The training difference is obtained by calculating the difference between the training execution result value and the execution result value; According to the training difference, the parameters of the initial intelligent behavior modeling model are adjusted. When the training difference is less than the preset difference, the training is terminated to obtain the intelligent behavior modeling model.

Images