AI agent generation method, platform system, electronic device and storage device

Abstract

The present invention provides an AI agent generation method, platform system, electronic device and storage device, including: obtaining a standardized task data set; extracting features from the standardized task data set to construct task feature data; using a hierarchical distributed architecture, combining a genetic algorithm to optimize the initial parameters of the agent, determining the basic structure and initial state of the agent, and obtaining an initialized agent architecture; performing reinforcement learning training and optimization on the initialized agent architecture based on the task feature data to generate an AI agent; wherein, using a proximal strategy optimization algorithm, using the task feature data to repeatedly execute the agent architecture in a simulated environment, and continuously adjusting the policy network parameters of the agent architecture according to the reward feedback of task completion to obtain a trained and optimized AI agent. In the present invention, the trained agent can quickly adapt to changes in the environment and adjust its own strategy in a timely manner.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an AI agent generation method, a platform system, an electronic device and a storage device. Background Art

[0002] AI agents have shown great application potential in many fields, such as industrial automation, intelligent transportation, medical diagnosis, and financial risk assessment.

[0003] In the training and optimization stage of the intelligent agent, commonly used training algorithms such as Q-learning have defects such as slow convergence speed and low sample utilization efficiency when dealing with tasks in high-dimensional state space and continuous action space. In addition, traditional training methods lack effective use of dynamic feedback in the process of interaction between the intelligent agent and the environment, which makes it difficult for the intelligent agent to quickly adapt to environmental changes in actual applications and cannot adjust its own strategy in time to deal with various complex situations, thus affecting the quality and efficiency of the intelligent agent's task completion. Summary of the invention

[0004] The main purpose of the present invention is to provide an AI agent generation method, platform system, electronic device and storage device, aiming to overcome the defects that the agent is difficult to quickly adapt to environmental changes and cannot adjust its own strategy in time in practical applications.

[0005] To achieve the above object, the present invention provides an AI agent generation method, comprising the following steps:

[0006] Acquire a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data;

[0007] Extracting features from the normalized task data set to construct task feature data;

[0008] Adopting a hierarchical distributed architecture and combining genetic algorithms to optimize the initial parameters of the agent, determine the basic structure and initial state of the agent, and obtain the initialized agent architecture;

[0009] Based on the task feature data, the initialized intelligent agent architecture is subjected to reinforcement learning training and optimization to generate an AI intelligent agent; wherein, a proximal strategy optimization algorithm is used to repeatedly execute the intelligent agent architecture in a simulation environment using the task feature data, and the strategy network parameters of the intelligent agent architecture are continuously adjusted according to reward feedback from task completion to obtain a trained and optimized AI intelligent agent.

[0010] Furthermore, the step of obtaining a standardized task data set includes:

[0011] Preprocess the historical task data and the corresponding execution result data to remove invalid and erroneous data, and perform standardized conversion on data in different formats to obtain a normalized task data set.

[0012] Furthermore, a hierarchical distributed architecture is used to optimize the initial parameters of the agent in combination with a genetic algorithm, determine the basic structure and initial state of the agent, and obtain the initialized agent architecture, including:

[0013] The intelligent agent is divided into a perception layer, a decision layer, and an execution layer. Various types of sensor data fusion models are configured in the perception layer to receive data.

[0014] A decision model based on a neural network is constructed at the decision layer, and a genetic algorithm is used to encode the weights, thresholds, and neuron connection structures of the decision model to generate an initial population;

[0015] Setting a fitness function, which is constructed based on the agent's execution efficiency, resource consumption, and task completion accuracy in a predefined task scenario;

[0016] The initial population is iteratively evolved through the selection, crossover and mutation operations of the genetic algorithm. In each generation of evolution, the quality of individuals is evaluated according to the fitness function, and excellent individuals are selected to enter the next generation of population. The selected individuals are crossover and mutation operations are performed to explore better parameter combinations.

[0017] When the predetermined number of iterations is reached or the convergence condition is met, the individual with the highest fitness value is selected from the final population, and its corresponding parameters are decoded as the optimization parameters of the decision layer. Combined with the preset standard configuration of the perception layer and the execution layer, the basic structure and initial state of the intelligent agent are determined to obtain the initialized intelligent agent architecture.

[0018] Further, based on the task feature data, the initialized intelligent agent architecture is subjected to reinforcement learning training and optimization to generate an AI intelligent agent, including:

[0019] Initialize the policy network parameters and value network parameters of the agent architecture. The policy network is used to generate action decisions, and the value network is used to evaluate benefits.

[0020] Using the proximal policy optimization algorithm, based on the task feature data, the intelligent agent architecture obtains rewards and new states after execution in the simulation environment, and stores them in the experience replay buffer; after the experience replay buffer is full, data is sampled to update the policy network parameters and value network parameters;

[0021] After completing the predetermined number of training rounds, the resulting agent is the generated AI agent.

[0022] Furthermore, after the AI ​​agent is generated, it also includes:

[0023] Obtaining the terminal type of each terminal on which the AI ​​agent is to be deployed;

[0024] Generate a basic deployment code based on each terminal type;

[0025] For each terminal, generate a terminal deployment code based on its corresponding terminal type;

[0026] When deploying the AI ​​agent for each terminal, the basic deployment code is combined with the terminal deployment code as the permission code for deploying the AI ​​agent for each terminal;

[0027] When the AI ​​agent is sent to each terminal, the deployment authority of the AI ​​agent is configured based on the corresponding authority code.

[0028] Furthermore, the generating of a basic deployment code based on each terminal type includes:

[0029] Encode and combine each terminal type respectively to obtain a coding combination;

[0030] Adding the characters in the coding combination one by one to a matrix of multiple rows and columns to generate a character matrix;

[0031] Obtain graphics corresponding to each terminal type in the database;

[0032] The acquired graphics are digitized, the coordinates of key feature points of each graphic are extracted, and the coordinates of each key feature point are arranged into a coordinate sequence according to a predetermined order;

[0033] Performing singular value decomposition on the character matrix to obtain multiple singular values, selecting the first several singular values ​​to fuse with the coordinate sequence to obtain a fused numerical sequence;

[0034] The fused numerical sequence is encrypted to serve as the basic deployment code.

[0035] Furthermore, the generating of a basic deployment code based on each terminal type includes:

[0036] According to different terminal types, it is encoded into a binary code string of a specific length;

[0037] Perform XOR operation on all binary code strings to obtain an initial fusion code;

[0038] The initial fusion code is used as a coefficient sequence of a polynomial, and error correction coding is performed on it through a coding algorithm to add redundant information to obtain a coding polynomial;

[0039] Performing an evaluation operation on the coding polynomial on a finite field, and selecting evaluation results at a number of specific positions to form a numerical sequence;

[0040] Using the values ​​in the numerical sequence as control points, and using a Bezier curve generation algorithm to draw a Bezier curve on a two-dimensional plane;

[0041] The Bezier curve is uniformly sampled to obtain coordinate values ​​of multiple sampling points, and the coordinate value encoding is converted into a string of characters as the basic deployment code.

[0042] The present invention also provides a platform system for generating an AI agent, comprising:

[0043] An acquisition unit, used to acquire a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data;

[0044] An extraction unit, used for extracting features from the normalized task data set to construct task feature data;

[0045] An initialization unit is used to optimize the initial parameters of the intelligent agent by adopting a hierarchical distributed architecture combined with a genetic algorithm, determine the basic structure and initial state of the intelligent agent, and obtain an initialized intelligent agent architecture;

[0046] A generation unit is used to perform reinforcement learning training and optimization on the initialized intelligent agent architecture based on the task feature data to generate an AI intelligent agent; wherein, a proximal strategy optimization algorithm is used to repeatedly execute the intelligent agent architecture in a simulation environment using the task feature data, and the policy network parameters of the intelligent agent architecture are continuously adjusted according to the reward feedback of task completion to obtain the trained and optimized AI intelligent agent.

[0047] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0048] The present invention also provides a storage device on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0049] The AI ​​agent generation method, platform system, electronic device and storage device provided by the present invention include: obtaining a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data; extracting features from the standardized task data set to construct task feature data; using a hierarchical distributed architecture, combining a genetic algorithm to optimize the initial parameters of the agent, determining the basic structure and initial state of the agent, and obtaining an initialized agent architecture; performing reinforcement learning training and optimization on the initialized agent architecture based on the task feature data to generate an AI agent; wherein, using a proximal strategy optimization algorithm, using the task feature data to repeatedly execute the agent architecture in a simulated environment, and continuously adjusting the policy network parameters of the agent architecture according to the reward feedback of task completion, to obtain the trained and optimized AI agent. In the present invention, by using a proximal strategy optimization algorithm, repeatedly executing the agent architecture in a simulated environment, and continuously adjusting the policy network parameters of the agent architecture according to the reward feedback of task completion, the trained agent can quickly adapt to changes in the environment and adjust its own strategy in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a schematic diagram of the steps of a method for generating an AI agent in one embodiment of the present invention;

[0051] Figure 2 It is a block diagram of the platform system structure of AI agent generation in one embodiment of the present invention;

[0052] Figure 3 It is a schematic block diagram of the structure of an electronic device according to an embodiment of the present invention.

[0053] The implementation, functional features and advantages of the present invention will be further described in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] Reference Figure 1 In one embodiment of the present invention, a method for generating an AI agent is provided, comprising the following steps:

[0056] Step S1, obtaining a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data;

[0057] Step S2, extracting features from the normalized task data set to construct task feature data;

[0058] Step S3, using a hierarchical distributed architecture and combining a genetic algorithm to optimize the initial parameters of the agent, determine the basic structure and initial state of the agent, and obtain an initialized agent architecture;

[0059] Step S4, based on the task feature data, the initialized intelligent agent architecture is subjected to reinforcement learning training and optimization to generate an AI intelligent agent; wherein, a proximal strategy optimization algorithm is used to repeatedly execute the intelligent agent architecture in a simulation environment using the task feature data, and the strategy network parameters of the intelligent agent architecture are continuously adjusted according to reward feedback for task completion to obtain a trained and optimized AI intelligent agent.

[0060] In this embodiment, as described in step S1 above, the historical task data includes various information about the tasks executed in the past, such as the specific content of the tasks, various parameters during the execution process, environmental conditions, etc., and the corresponding execution result data reflects the completion of the tasks under these conditions, including the identification of success or failure, the efficiency index of completing the tasks, the benefits or losses generated, etc. By associating the two, the intelligent agent can learn the causal relationship and effective mode in the task execution process.

[0061] First, it is necessary to collect historical task data and its execution result data from multiple data sources, which can be business databases, sensor records, log files, etc. of enterprises or institutions. The collected data may have different formats, precisions, and data quality, so preprocessing operations are required. Data cleaning technology is used to remove noise data in the data, such as outliers caused by sensor failures, records that obviously do not conform to business logic, etc. At the same time, duplicate data is eliminated to avoid unnecessary interference and deviations in the training of intelligent agents.

[0062] For data in different formats, such as structured database table data and unstructured text description data, format conversion and standardization are performed to unify them into a format suitable for subsequent analysis and processing. For example, all data can be converted into numerical vectors or specific data structures to ensure that each data point has a clear and consistent meaning and representation.

[0063] As described in step S2 above, feature information that has a key impact on the agent's decision and behavior is extracted from the original task data set, reducing the data dimension and the amount of calculation, while improving the efficiency and accuracy of the agent's learning. The original data set contains a large amount of redundant or irrelevant information. Directly using this data for agent training will increase the computational complexity and time cost, causing the agent to learn irrelevant patterns and fail to focus on the truly important task features.

[0064] Using a variety of feature extraction techniques, such as principal component analysis (PCA), it projects the original data into a new low-dimensional space by performing eigenvalue decomposition on the data covariance matrix, and selects principal components with larger variance as the main features. These principal components can retain most of the information of the data, thereby achieving data dimensionality reduction. Combined with the autoencoder network in deep learning, the autoencoder can learn the low-dimensional representation of the data by encoding the input data into a low-dimensional vector and then decoding it back to a form that approximates the original data. The intermediate layer feature vectors extracted during the encoding process can effectively capture the key features of the data, and have stronger expressiveness and adaptability, and can handle complex data distributions and nonlinear relationships.

[0065] According to the specific nature of the task and domain knowledge, specific feature engineering methods can also be used. For example, in image recognition tasks, image edges, textures, color histograms and other features can be extracted; in financial risk assessment tasks, risk-related financial indicators, market volatility parameters, etc. are selected as key features to further enhance the pertinence and effectiveness of feature extraction, and ultimately construct task feature data that can accurately reflect the essence of the task.

[0066] As described in step S3 above, a reasonable, efficient and scalable intelligent agent architecture is designed, and a set of optimized initial parameters are determined for it to improve the performance and convergence speed of the intelligent agent in the subsequent training process and avoid falling into the local optimal solution. The hierarchical distributed architecture helps to decompose complex intelligent agent functions into multiple levels and modules, realize modular management and efficient collaboration of functions, and at the same time, the genetic algorithm can search for a better initial parameter combination in a huge parameter space, providing a good starting point for the training of the intelligent agent.

[0067] As described in step S4 above: by allowing the agent to continuously interact with the environment in the simulation environment, the optimal behavior strategy is learned based on the reward feedback of task completion, thereby continuously improving the performance and decision-making ability of the agent, enabling it to make efficient and accurate decisions in actual tasks, and finally generating an AI agent with high adaptability and intelligence. In the present invention, by using the proximal policy optimization algorithm, the agent architecture is repeatedly executed in the simulation environment, and the policy network parameters of the agent architecture are continuously adjusted based on the reward feedback of task completion, so that the trained agent can quickly adapt to changes in the environment and adjust its own strategy in time.

[0068] In one embodiment, first, the task feature data is divided into a training set, a validation set, and a test set according to a preset ratio. The training set is used in the learning and training process of the intelligent agent, the validation set is used to regularly evaluate the performance of the intelligent agent during the training process, and to adjust the hyperparameters in the training process according to the evaluation results, such as the learning rate, discount factor, etc., to prevent overfitting and improve generalization ability, and the test set is used to conduct a comprehensive and objective evaluation of the performance of the final generated AI intelligent agent after the entire training process is completed to ensure that it meets the requirements of the actual task.

[0069] Initialize the policy network and value network parameters of the agent architecture. The policy network is used to generate action decisions based on the input task feature data, such as determining the agent's movement direction, attack behavior, etc. in the game scene. The value network is used to evaluate the expected benefits of taking the action decision under the current state, that is, to judge the contribution of the action to achieving the final task goal.

[0070] Using the proximal policy optimization algorithm (PPO), in each training iteration, the agent samples a batch of state information corresponding to the task feature data from the training set, and generates corresponding actions based on the current policy network. After executing these actions in the simulated environment, it obtains the reward signal of the environment feedback and the next state information. The reward signal is designed according to the goals and requirements of the task. For example, in a robot navigation task, if the agent successfully reaches the target position, it will be given a higher positive reward, and if it collides with an obstacle, it will be given a negative reward. In this way, the agent is guided to learn the correct behavior strategy.

[0071] The sampled state, action, reward, and next state information are stored in the experience replay buffer. When the buffer reaches a certain capacity, a batch of data is randomly sampled from it to update the policy network and value network. When updating the policy network, by calculating the ratio of the current policy to the old policy, and combining the advantage function and the estimated value of the value network, the policy network is updated by gradient ascent using the objective function of the proximal policy optimization algorithm to improve the performance of the actions generated by the policy network in long-term returns, so that the agent is more inclined to choose actions that can bring higher cumulative rewards. At the same time, the mean square error loss function is used to update the value network to more accurately estimate the value in the current state, thereby better guiding the learning and decision-making of the policy network.

[0072] During the training process, the validation set is used regularly to evaluate the performance of the agent, and the hyperparameters of the proximal policy optimization algorithm, such as the learning rate and discount factor, are adjusted based on the evaluation results to balance the learning speed and generalization ability of the agent. When the performance indicators of the agent on the validation set reach the preset convergence threshold, or after completing the predetermined number of training rounds, the test set is used to perform a final performance test on the agent. If the predetermined performance requirements are met, the currently trained agent is used as the final AI agent, which can make decisions and perform tasks autonomously in the actual task environment based on the learned strategies and experience to achieve the best task completion effect.

[0073] In one embodiment, the step of obtaining a normalized task data set includes:

[0074] Preprocess the historical task data and the corresponding execution result data to remove invalid and erroneous data, and perform standardized conversion on data in different formats to obtain a normalized task data set.

[0075] In one embodiment, a hierarchical distributed architecture is used to optimize the initial parameters of the agent in combination with a genetic algorithm, determine the basic structure and initial state of the agent, and obtain an initialized agent architecture, including:

[0076] The intelligent agent is divided into a perception layer, a decision layer, and an execution layer. Various types of sensor data fusion models are configured in the perception layer to receive data.

[0077] A decision model based on a neural network is constructed at the decision layer, and a genetic algorithm is used to encode the weights, thresholds, and neuron connection structures of the decision model to generate an initial population;

[0078] Setting a fitness function, which is constructed based on the agent's execution efficiency, resource consumption, and task completion accuracy in a predefined task scenario;

[0079] The initial population is iteratively evolved through the selection, crossover and mutation operations of the genetic algorithm. In each generation of evolution, the quality of individuals is evaluated according to the fitness function, and excellent individuals are selected to enter the next generation of population. The selected individuals are crossover and mutation operations are performed to explore better parameter combinations.

[0080] When the predetermined number of iterations is reached or the convergence condition is met, the individual with the highest fitness value is selected from the final population, and its corresponding parameters are decoded as the optimization parameters of the decision layer. Combined with the preset standard configuration of the perception layer and the execution layer, the basic structure and initial state of the intelligent agent are determined to obtain the initialized intelligent agent architecture.

[0081] In this embodiment, the intelligent agent is clearly divided into a perception layer, a decision layer, and an execution layer according to its function. The perception layer is at the forefront of direct contact with the external environment, and its core function is to obtain external data. For example, in an autonomous driving scenario, the perception layer will be equipped with a variety of sensors such as cameras, radars, and lidars. The camera is responsible for capturing visual image information, the radar can detect the distance and speed of surrounding objects, and the lidar can build a three-dimensional model of the surrounding environment with high precision.

[0082] Configure the corresponding data fusion model for the data obtained by different sensors. Common data fusion methods include weighted average method, Kalman filter method, fusion method based on evidence theory, etc. Taking the weighted average method as an example, if there are multiple temperature sensors to obtain ambient temperature data, different weights can be assigned to them according to factors such as the accuracy of each sensor, and then a weighted average calculation is performed to obtain a relatively more accurate fusion temperature value. In more complex multimodal sensor scenarios, such as the above-mentioned autonomous driving, Kalman filtering may be used to fuse the object position information identified by the camera image with the distance and speed information detected by the radar, so that the intelligent body can perceive the state of surrounding vehicles, pedestrians and other objects more accurately and comprehensively, so that it can be subsequently passed to the decision-making layer as reliable input information.

[0083] A neural network model framework with strong learning and decision-making capabilities is built at the decision-making layer, and a genetic algorithm is used to explore the appropriate initial parameter combination for it to form an initial population, laying the foundation for finding the optimal parameters through subsequent evolutionary search. The neural network itself has a high degree of nonlinear fitting ability and can handle complex input-output relationships, but the setting of initial parameters has a significant impact on its final performance. Genetic algorithms can effectively search in a wide parameter space.

[0084] First, according to the specific task requirements and data characteristics, select the appropriate neural network structure to build a decision model. For example, for image classification tasks, you can choose a convolutional neural network (CNN), and when processing sequence data, choose a recurrent neural network (RNN) or its variants (such as long short-term memory network LSTM, gated recurrent unit GRU, etc.). Taking CNN as an example, determine the basic architectural parameters such as the number of network layers, the size of each layer's convolution kernel, the step size, the padding method, and the number of neurons in the fully connected layer, and build an initial neural network model framework. The model will use the fused data transmitted from the perception layer as input and output the corresponding decision results, such as determining the next action strategy of the intelligent agent in the game scene.

[0085] The key parameters in the decision model, namely weights (weight values ​​of connections between neurons), thresholds (threshold parameters of neuron activation functions), and neuron connection structures (which neurons are connected), are encoded and converted into chromosome forms that can be processed by genetic algorithms. Common encoding methods include binary encoding and real number encoding. For example, when binary encoding is used, each weight value, threshold, etc. is represented by a binary string of a certain length. Multiple such binary strings are arranged in a certain order to form a chromosome, representing a set of parameter combinations. By randomly generating multiple sets of such chromosomes, an initial population is formed to prepare for subsequent evolutionary iterations.

[0086] Furthermore, some representative and operational task scenarios are pre-designed according to the actual application areas and target tasks of the agent. For example, for the intelligent warehouse robot agent, the cargo handling task scenario can be defined, and different warehouse layouts, cargo storage locations, handling starting points and other conditions can be set to simulate the actual warehouse operation environment.

[0087] The fitness function is constructed by comprehensively considering multiple performance indicators. In terms of execution efficiency, the time it takes for the agent to complete the task can be measured. The shorter the time, the better the performance on this indicator; resource consumption can include computing resources (such as CPU usage, memory usage, etc.) and energy consumption (such as power consumption). The lower the resource consumption, the better; the task completion accuracy focuses on whether the agent has completed the task accurately as required, such as whether the goods are accurately transported to the designated location in the transport task. The higher the accuracy, the higher the score of this indicator. These indicators are combined according to a certain weight relationship to construct a comprehensive fitness function.

[0088] By utilizing the core operating mechanism of genetic algorithms and simulating the biological evolution process in nature, the parameter combinations in the population are continuously optimized, and the parameter space regions that are more suitable for the intelligent agent to achieve high performance in a given task scenario are gradually explored. The problem that traditional parameter initialization methods are prone to falling into local optimal solutions is overcome, and the probability of finding the global optimal solution or a solution close to the global optimal solution is increased.

[0089] In each generation of the population, the fitness value of each individual is calculated according to the fitness function, and relatively excellent individuals are selected to enter the next generation of the population according to a certain selection strategy. Common selection strategies include roulette selection and tournament selection. Taking roulette selection as an example, the proportion of each individual's fitness value to the total fitness value of the population is regarded as its area share on the roulette wheel. By randomly rotating the roulette wheel (simulating the random sampling process), the individuals with higher fitness values ​​have a greater probability of being selected, so that better individuals have a greater chance of entering the next generation, ensuring that the population as a whole develops in the direction of higher fitness.

[0090] The selected excellent individuals are paired in pairs, and some gene fragments in their chromosomes are exchanged according to the preset crossover probability and crossover method to generate new individuals. For example, under binary coding, single-point crossover or multi-point crossover can be used. Single-point crossover is to randomly select a crossover point on the chromosome, exchange the gene fragments before and after the point of the paired individuals, and generate two new offspring individuals. Through the crossover operation, the excellent genes of different excellent individuals can be combined, which may produce new parameter combinations with better performance, broaden the search space, and explore new and better solution areas.

[0091] Randomly change certain genes on individual chromosomes in the population with a preset mutation probability. For example, in binary coding, change 0 on a gene position to 1 or 1 to 0. The mutation operation can introduce new genetic diversity to the population, prevent the population from converging to the local optimal solution too early, and enable the algorithm to jump out of the local optimal "trap" and continue to search for possible better parameter combinations in a wider parameter space, increasing the chance of finding the global optimal solution.

[0092] By continuously repeating the above selection, crossover and mutation operations, the population is updated and evolved. Each iteration will evaluate the quality of the newly generated individuals according to the fitness function, continuously develop in the direction of improving the overall fitness of the population, and gradually explore better parameter combinations.

[0093] End the iterative search process of the genetic algorithm, obtain the optimal parameter combination currently searched according to the termination criterion, and apply it to the decision-making layer model. At the same time, combined with the standard configuration of the perception layer and execution layer that have been set, completely determine the initial architecture and state of the intelligent agent, and provide a good initial starting point for subsequent reinforcement learning training and other operations based on this architecture. It is expected that it can converge quickly in subsequent training and achieve better performance.

[0094] The predetermined number of iterations is a common way to set the termination condition. For example, the genetic algorithm is pre-set to perform a maximum of 100 iterations. When this number is reached, the iteration stops regardless of whether it converges. The convergence condition is usually determined based on the fitness change of the population. For example, if the average fitness change of the population for several consecutive generations is less than a certain set threshold, it means that it is difficult for the population to have a significant performance improvement. At this time, the algorithm can be considered to have converged and the iteration stops.

[0095] When the termination condition is met, the individual with the highest fitness value is found in the final population, because the parameter combination corresponding to this individual shows the relatively optimal performance in the entire evolution process. Then, according to the encoding rules used in the previous encoding, the parameters represented by the chromosome of the individual are decoded to restore the actual decision model weights, thresholds, neuron connection structures and other parameters, and these parameters are assigned to the neural network model constructed by the decision layer to complete the parameter optimization of the decision layer.

[0096] Combined with the sensor data fusion model pre-configured by the perception layer and the standard execution mechanism set by the execution layer (for example, how the execution layer converts the decision instructions output by the decision layer into actual physical actions, which may involve specific operations such as motor control in the robot scenario), the basic structure and initial state of the intelligent agent are jointly determined. At this point, the initialized intelligent agent architecture is obtained. This architecture has a reasonable hierarchical structure and optimized decision-making layer parameters, laying the foundation for subsequent training and practical applications.

[0097] In one embodiment, the initialized agent architecture is subjected to reinforcement learning training and optimization based on the task feature data to generate an AI agent, including:

[0098] Initialize the policy network parameters and value network parameters of the agent architecture. The policy network is used to generate action decisions, and the value network is used to evaluate benefits.

[0099] Using the proximal policy optimization algorithm, based on the task feature data, the intelligent agent architecture obtains rewards and new states after execution in the simulation environment, and stores them in the experience replay buffer; after the experience replay buffer is full, data is sampled to update the policy network parameters and value network parameters;

[0100] After completing the predetermined number of training rounds, the resulting agent is the generated AI agent.

[0101] In this embodiment, the parameters of the policy network can be initialized by random initialization or initialization based on a specific distribution (such as normal distribution) to provide an initial decision-making ability basis for it, so that it can generate preliminary action decisions based on the input task feature data, such as determining the moving direction and action selection of the agent in the game scene. The parameters of the value network are also initialized to make an initial estimate of the expected benefits brought by the actions taken by the agent in different states. Its initialization method can also adopt a similar random or distributed initialization method, and these initial estimates are continuously corrected through subsequent training so that they can accurately reflect the actual value situation.

[0102] The agent operates in the simulated environment according to the action decisions generated by the current policy network. After execution, the environment feedbacks reward signals and new state information. The reward signal is set according to the task goal. For example, in the robot navigation task, a high reward is given for successfully reaching the target position, and a negative reward is given for colliding with obstacles, so as to guide the agent to learn the correct behavior. These experience data (state, action, reward, new state) are stored in the experience replay buffer.

[0103] When the experience replay buffer accumulates enough data (i.e., the buffer is full), a batch of data is randomly sampled from it. Using the proximal policy optimization algorithm, by calculating the ratio of the current policy to the old policy, and combining the advantage function and the estimated value of the value network, the parameters of the policy network are updated by gradient ascent, so that the policy network is more inclined to choose actions that can bring higher cumulative rewards. At the same time, using the mean square error loss function, according to the difference between the actual reward in the sampled data and the predicted value of the value network, the value network is updated so that it can more accurately evaluate the value in each state, thereby better guiding the learning of the policy network.

[0104] During the training process, the training is repeated according to the pre-set number of training rounds. Each round includes the above-mentioned steps of agent-simulation environment interaction, data collection, and network parameter update. As the number of training rounds increases, the agent's policy network and value network are continuously optimized, and its decision-making ability and adaptability to tasks are gradually improved. After completing the predetermined number of training rounds, the agent is considered to have learned enough knowledge and strategies to perform well in actual tasks, and is thus determined to be the final generated AI agent, which can be deployed in actual application scenarios to perform corresponding tasks, such as automated control in industrial production, dialogue processing in intelligent customer service systems, and other task areas.

[0105] In one embodiment, after generating the AI ​​agent, the method further includes:

[0106] Obtaining the terminal type of each terminal on which the AI ​​agent is to be deployed;

[0107] Generate a basic deployment code based on each terminal type;

[0108] For each terminal, generate a terminal deployment code based on its corresponding terminal type;

[0109] When deploying the AI ​​agent for each terminal, the basic deployment code is combined with the terminal deployment code as the permission code for deploying the AI ​​agent for each terminal;

[0110] When the AI ​​agent is sent to each terminal, the deployment authority of the AI ​​agent is configured based on the corresponding authority code.

[0111] In this embodiment, the terminal type information can be obtained in a variety of ways. For example, a lightweight detection program is installed on the terminal device, which automatically collects the terminal's hardware information (such as CPU model, memory size, graphics card type, etc.) and operating system information (such as operating system version, kernel type, etc.) when it is started, and sends this information back to the server or management center. In addition, when the terminal is connected to a specific network, the data packet containing its type information sent by the terminal can be obtained through the information interaction process in the network authentication protocol, and then the data can be parsed and extracted to determine the terminal type.

[0112] The basic deployment code is one of the core components of the entire deployment permission system. It serves as a unified identification and security verification basis for different terminal types. By combining the key features of the terminal type to generate a unique basic deployment code, it provides the key information basis for the subsequent combination with the terminal deployment code to generate the final permission code. It also increases the security and controllability of the deployment process and prevents illegal tampering and forgery.

[0113] First, extract and quantify the key information of the terminal type. For example, for hardware information, the CPU model can be converted into the corresponding digital code, the memory size can be converted into a numerical value according to certain quantization rules, and the operating system version can also be converted into a similar code. Then, these quantized values ​​are processed through a specific encryption algorithm, such as using a hash function (such as SHA-256) to perform a hash operation on the string composed of these values ​​to obtain a hash value of a fixed length. Then, the hash value is further transformed, such as using a symmetric encryption algorithm (such as AES) to encrypt the hash value with a pre-set key to obtain the encrypted ciphertext, which is the basic deployment code. The basic deployment code generated in this way not only contains the key information of the terminal type, but also has been encrypted, and has high security and uniqueness.

[0114] The terminal deployment code is a personalized code generated for each specific terminal. It further refines the identification and authorization management of a single terminal based on the basic deployment code. By combining the specific attributes of the terminal to generate the terminal deployment code, it is possible to achieve accurate authorization and control of each terminal, ensuring that the agent can only be deployed and run on authorized specific terminals, improving the security and flexibility of deployment, and preventing the agent from being illegally copied to other unauthorized terminals of the same type.

[0115] In addition to considering the general information of the terminal type, some unique identification information of the terminal can also be included, such as the MAC address of the network card, the serial number of the hard disk and other hardware unique identifiers. This information is integrated with the terminal type information, for example, they are first connected into a string, and then the string is encrypted using an encryption method similar to the one used to generate the basic deployment code, but with different keys or encryption parameters, to obtain the terminal deployment code. In this way, even for terminals of the same type, the generated terminal deployment codes are different due to their different hardware unique identifiers, thereby realizing personalized authorization management for each terminal.

[0116] The generated basic deployment code and terminal deployment code are concatenated in a predetermined order to form a new string, which is the final permission code. For example, the basic deployment code can be placed in front and the terminal deployment code in the back, and then some simple verification processing is performed on the concatenated string, such as calculating its CRC (cyclic redundancy check) value and appending the CRC value to the end of the string to ensure the integrity and accuracy of the permission code during transmission and storage.

[0117] According to the permission code of each terminal, the deployment permissions of the agent are precisely configured to ensure that the functional availability and data access permissions of the agent on different terminals meet the expected security and business requirements. This step realizes the mapping from permission codes to actual deployment permissions, so that the operation of the agent on different terminals is reasonably restricted and managed, which can not only ensure that the agent can function normally on legally authorized terminals, but also prevent it from performing illegal operations or data leakage on unauthorized or insufficiently authorized terminals.

[0118] Specifically, a permission verification and configuration module is built into the deployment package of the agent. When the agent is sent to the terminal and the installation is started, the module first reads the permission code provided by the terminal and sends it back to the server or management center for verification. The server determines the specific deployment permissions of the terminal for the agent based on the correspondence between the pre-stored permission code and the deployment permissions, such as the scope of local resources allowed to be accessed, the network functions that can be used, the specific task modules that can be executed, etc., and sends these permission configuration information back to the terminal. The agent sets and restricts its own operating parameters and functional modules on the terminal according to the received permission configuration information, thereby realizing the deployment permission configuration based on the permission code.

[0119] In one embodiment, generating a basic deployment code based on each terminal type includes:

[0120] Encode and combine each terminal type respectively to obtain a coding combination;

[0121] Adding the characters in the coding combination one by one to a matrix of multiple rows and columns to generate a character matrix;

[0122] Obtain graphics corresponding to each terminal type in the database;

[0123] The acquired graphics are digitized, the coordinates of key feature points of each graphic are extracted, and the coordinates of each key feature point are arranged into a coordinate sequence according to a predetermined order;

[0124] Performing singular value decomposition on the character matrix to obtain multiple singular values, selecting the first several singular values ​​to fuse with the coordinate sequence to obtain a fused numerical sequence;

[0125] The fused numerical sequence is encrypted to serve as the basic deployment code.

[0126] In this embodiment, first, the key information covered by the terminal type is analyzed, which may include hardware aspects (such as CPU architecture, memory type, storage capacity, etc.), operating system related features (such as operating system name, version number, kernel version, etc.) and network configuration (such as network interface type, supported network protocol, etc.). For each type of information, a corresponding coding rule is formulated. For example, for the CPU architecture, specific digital codes can be assigned according to several common architecture types on the market, such as Intel's x86 architecture code 01, ARM architecture code 02, etc.; the operating system version number can directly use its digital representation as the code, or map it to different code values ​​according to a certain version interval. Then, these codes generated according to different dimensional information are combined in a predetermined order, such as hardware-related codes first, then operating system codes, and finally network configuration codes, which are connected in sequence to form a complete coding combination. This combination can uniquely correspond to a terminal type and contains the key feature descriptions of the terminal type in many aspects.

[0127] The number of rows and columns of the matrix is ​​determined according to the preset rules. This rule can be formulated based on factors such as security requirements and the number of terminal types. For example, in order to accommodate more terminal types and ensure sufficient security, a larger matrix dimension can be set, such as 10 rows and 20 columns. Then, the characters in the coding combination are filled into each element position of the matrix in sequence. If the number of characters in the coding combination is less than the total number of matrix elements, a specific filling method can be used, such as filling with default characters (such as 0); if the number of characters is more than the total number of matrix elements, the previous part of the characters can be selected for filling, ensuring that each coding combination can generate a corresponding character matrix, so that the terminal type information is presented in a new two-dimensional structure, ready for subsequent processing such as combining with graphic features.

[0128] Graphic elements are introduced as additional feature information to further enrich the data source for the generation of basic deployment codes. Graphics themselves have unique geometric features and visual patterns, which can provide more changes and complexity compared to simple text encoding. In addition, different terminal types correspond to different graphics, which makes the final generated basic deployment code more closely related to the terminal type and more unique. At the same time, it also increases the difficulty of forging the basic deployment code and improves the security of intelligent agent deployment authority management.

[0129] A mapping table between terminal types and graphics is pre-built in the database. In this table, a specific graphic is associated with each terminal type. These graphics can be generated by a specific algorithm based on certain key characteristics of the terminal type. For example, a unique irregular geometric figure is generated using a graphics generation algorithm (such as an algorithm based on fractal geometry principles) based on the hash value of the terminal's hardware architecture as a seed; or a graphic with a specific shape and feature is drawn through mathematical transformation based on some key parameters of the operating system. When a basic deployment code needs to be generated for a certain terminal type, the corresponding graphic is queried and obtained in the database based on its terminal type information for subsequent digital processing steps.

[0130] The information in the visual form of graphics is converted into a numerical sequence form that can be processed and calculated by computers, so that it can be integrated with the character matrix information generated based on the terminal type code. By extracting the coordinates of key feature points, the core features of the graphics can be captured, and the main identification information of the graphics can be retained while ensuring the simplicity of the data. The final generated basic deployment code not only integrates the multi-dimensional information of the terminal type, but also has the uniqueness and complexity brought by the graphics.

[0131] Using graphics digitization algorithms, for example, for simple two-dimensional graphics, edge detection algorithms can be used to identify the outline of the graphics, and then representative key feature points can be selected on the outline. These points can be turning points of the outline, points with large changes in curvature, etc.; for more complex three-dimensional graphics, key feature points can be determined through voxel processing, surface meshing and other technologies. After determining the key feature points, a certain coordinate system (such as a two-dimensional rectangular coordinate system or a three-dimensional Cartesian coordinate system) is used as a reference to record the coordinate value of each feature point, and the coordinates of all feature points are arranged in a predetermined order (such as arranged along the outline of the graphics in a clockwise or counterclockwise direction) to form a coordinate sequence. This sequence becomes the key information representation after the graphics are digitized, which can be used for subsequent fusion operations with character matrix information.

[0132] Singular value decomposition (SVD) is an important matrix analysis technology. By performing SVD on the character matrix, its main and representative feature information (i.e., singular values) can be extracted, and some relatively minor information can be discarded to achieve data dimension reduction and feature extraction. By fusing these extracted key singular values ​​with the coordinate sequence of the graphics, the core information from both the terminal type code and the graphic features can be organically combined to generate a fused numerical sequence. This sequence integrates the key elements of information from two different sources, providing a key data foundation for generating a unique and highly secure basic deployment code.

[0133] Specifically, the generated character matrix is ​​subjected to a singular value decomposition algorithm, which decomposes the matrix into the product of three matrices, which contains a set of singular values. The singular values ​​reflect the importance of the information contained in the matrix, and are generally arranged in order from large to small. According to actual needs and experience, the first several larger singular values ​​are selected (for example, the first 5 or the first 10, the specific number can be determined according to the overall security and data processing complexity requirements), and then fused with the coordinate sequence through a certain fusion method. The fusion method can be a simple splicing, that is, the selected singular values ​​are arranged in sequence in front of or behind the coordinate sequence to form a new sequence; or a more complex fusion method such as weighted summation can be used, such as assigning different weights to the singular values ​​and coordinate values, and then multiplying and adding the corresponding elements to obtain the fused values, and finally forming a fused numerical sequence, which combines the key information of the terminal type code and the graphic features, and the feature extraction after singular value decomposition is more refined and representative.

[0134] Encryption processing is the last key step to ensure the security of the basic deployment code. By encrypting the fused numerical sequence, the key information contained in it is obfuscated and protected. Even if the basic deployment code is obtained externally, it is difficult to reversely parse the original information related to the terminal type and graphics, etc., to ensure that only terminals with the correct decryption key and authorization can use the basic deployment code in subsequent deployment processes, thereby effectively preventing the intelligent agent from being illegally deployed on unauthorized terminals and protecting the intellectual property rights and usage security of the intelligent agent.

[0135] In one embodiment, generating a basic deployment code based on each terminal type includes:

[0136] According to different terminal types, it is encoded into a binary code string of a specific length;

[0137] Perform XOR operation on all binary code strings to obtain an initial fusion code;

[0138] The initial fusion code is used as a coefficient sequence of a polynomial, and error correction coding is performed on it through a coding algorithm to add redundant information to obtain a coding polynomial;

[0139] Performing an evaluation operation on the coding polynomial on a finite field, and selecting evaluation results at a number of specific positions to form a numerical sequence;

[0140] Using the values ​​in the numerical sequence as control points, and using a Bezier curve generation algorithm to draw a Bezier curve on a two-dimensional plane;

[0141] The Bezier curve is uniformly sampled to obtain coordinate values ​​of multiple sampling points, and the coordinate value encoding is converted into a string of characters as the basic deployment code.

[0142] In this embodiment, first, the key elements involved in the terminal type are analyzed in detail, such as the CPU performance level, memory size, storage device type in hardware, the operating system type, version number, and network configuration such as network speed, connection method, etc. in software. Then, a unique binary encoding rule is formulated for each element. For example, several common CPU performance levels correspond to different 3-bit binary codes, and the memory size is divided into specific ranges and corresponds to different 4-bit binary codes. Then, these binary codes for each element are combined in a predetermined order to form a binary code string of a specific length representing the terminal type, ensuring that each terminal type has a unique corresponding standardized binary code representation, so as to be uniformly processed and identified in subsequent steps.

[0143] By combining the binary code strings of different terminal types through XOR operations, we can not only integrate some characteristics of each terminal type, but also use the characteristics of XOR operations to confuse the original information, making it difficult for outsiders to directly infer the specific codes of each terminal type from the initial fusion code, thereby enhancing the security and confidentiality of the basic deployment code generation process and laying the foundation for the subsequent generation of unique basic deployment codes.

[0144] Specifically, take out the binary code string of each terminal type in turn, and perform XOR operation on each bit starting from the lowest bit. For example, there are two binary code strings of terminal types, namely "10101" and "01100". First, perform XOR operation on the lowest bit "1" and "0" to obtain "1", then perform XOR operation on the second lowest bit "0" and "1" to obtain "1", and so on. After completing the XOR operation of all bits, the result "11001" is the initial fusion code. In actual operation, such a bit-by-bit XOR operation will be performed on all terminal type code strings that need to be processed to ensure that all terminal type information is involved in the fusion process, and generate an initial fusion code that integrates multiple terminal type features and has been obfuscated.

[0145] Error correction coding technology is used to add redundant information to the initial fusion code to improve the reliability and stability of the basic deployment code. During data transmission and storage, various interferences may occur, causing data errors. The coding polynomial after error correction coding can detect and correct these errors to a certain extent, ensuring the integrity and accuracy of the basic deployment code, ensuring the stable operation of the intelligent agent deployment permission management, and preventing permission management chaos caused by data errors.

[0146] By evaluating the coding polynomial with the help of the special mathematical properties of the finite field, the data information can be further transformed and hidden, making the basic deployment code more confidential and unique. By selecting the evaluation results at specific positions to form a numerical sequence, the key feature information in the coding polynomial is extracted. This information is related to the original terminal type and is encrypted through complex mathematical operations, making it difficult to crack and forge, providing a key data foundation for the subsequent generation of secure and reliable basic deployment codes.

[0147] The introduction of the graphic element of Bezier curve presents the information in the numerical sequence in the form of visual geometric graphics, making the basic deployment code more complex and unique. The shape of the Bezier curve is determined by the control points. By using the values ​​in the numerical sequence as control points, a unique curve is generated. This curve contains the terminal type related information after the previous steps. Moreover, its graphic form is complex and difficult to be reverse analyzed, which greatly improves the security and innovation of the basic deployment code. At the same time, it also provides a rich source of geometric data for subsequent sampling processing.

[0148] By uniformly sampling the Bezier curve, the complex geometric information contained in the curve is converted into discrete coordinate value data, and then these coordinate value codes are converted into character form to finally generate the basic deployment code. Such a basic deployment code integrates various information contained in multiple steps of complex processing starting from the terminal type, and is presented in the form of a unique character sequence. It has high security, uniqueness and unpredictability, can effectively prevent illegal acquisition and forgery, and provide a reliable and difficult-to-crack authorization identification for the deployment authority management of the intelligent agent on different terminals, ensuring the security and standardization of the intelligent agent deployment process.

[0149] Specifically, the number of sampling points and the sampling interval are determined according to factors such as the required length of the basic deployment code, security requirements, and expectations for data accuracy. For example, a sampling point is selected on the Bezier curve every 0.1 arc length, and the coordinate values ​​of these sampling points in the two-dimensional plane rectangular coordinate system are calculated using mathematical methods, such as (0.1, 0.3), (0.2, 0.45), etc. Then, these coordinate values ​​are encoded and converted in a variety of ways, such as converting the values ​​of the horizontal and vertical coordinates into hexadecimal representations, and then concatenating them into a string in the order of the horizontal coordinate first and the vertical coordinate second; or performing some custom mapping encoding on the coordinate values ​​to convert them into ASCII code characters, etc. The final string of characters is the basic deployment code, which can be used as a unique identifier in the intelligent agent deployment permission management system to authorize the terminal to legally deploy the intelligent agent, ensuring the security and controllability of the entire deployment process.

[0150] In one embodiment, generating a basic deployment code based on each terminal type includes:

[0151] Encoding is performed based on each terminal type to obtain a feature value set.

[0152] The correlation matrix is ​​constructed based on the characteristic value set, and the correlation matrix is ​​decomposed into singular values, and the first N singular values ​​are combined into a fused eigenvector.

[0153] Generate 3D curves and 2D graphics based on fused feature vectors;

[0154] Perform Fourier transform on the three-dimensional curve to obtain the spectrum feature vector, calculate the geometric moment feature vector for the two-dimensional graph, and concatenate the spectrum feature vector and the geometric moment feature vector to obtain the final feature vector;

[0155] The final feature vector is encrypted based on the elliptic curve cryptography to obtain the basic deployment code.

[0156] In this embodiment, the diverse and characteristic information contained in different terminal types is converted into a standardized numerical form that can be used for subsequent mathematical operations and processing. By encoding to form a characteristic numerical set, the differences between terminal types can be clearly reflected through the size, distribution and other characteristics of the numerical values, laying the foundation for the subsequent construction of an association matrix and a series of complex processing, facilitating the extraction of key features, and then generating a unique and secure basic deployment code.

[0157] Constructing an association matrix can integrate the feature information of each terminal in a structured way, clearly showing the relationship and differences between different terminals in each feature dimension. Singular value decomposition (SVD) is a powerful matrix analysis tool that can extract the main feature information contained in the association matrix, discard relatively minor information, and achieve data dimensionality reduction and feature extraction. Taking the first N singular values ​​to combine into a fused feature vector can make the data more compact and representative while retaining key features, making it easier to use it to generate graphics and other further processing to generate the final basic deployment code.

[0158] The constructed correlation matrix is ​​subjected to the singular value decomposition algorithm, which decomposes the matrix into the product of three matrices, from which a set of singular values ​​and corresponding singular vectors are obtained. The singular values ​​are arranged in order from large to small, and their size reflects the importance of the corresponding feature information in the data represented by the entire matrix. The larger the value, the more critical the feature is and the more information it contains.

[0159] The number of singular values ​​N to be selected is determined based on actual conditions and experience. Usually, the first N larger singular values ​​are selected because they carry the main information in the matrix. Then, the singular vectors corresponding to these N singular values ​​are combined according to certain rules, such as simple splicing or weighted combination according to specific weights, and finally a fused feature vector is formed. This vector integrates the key information of each terminal type feature in the association matrix, and after dimensionality reduction and refinement, it is easier to use it in subsequent processing steps to generate representative and unique graphics, curves and other elements.

[0160] The abstract, numerical terminal type feature information contained in the fused feature vector is visualized in the form of intuitive graphics (two-dimensional graphics) and curves (three-dimensional curves). On the one hand, the rich geometric features of the graphics and curves themselves can be used to further carry and fuse the terminal type information, increasing the complexity and uniqueness of the data; on the other hand, more in-depth features can be extracted by mathematically processing these graphics and curves, providing a richer data source for generating highly secure and unique basic deployment codes, increasing the difficulty of cracking the basic deployment codes, and ensuring the security of intelligent body deployment and the effectiveness of permission management.

[0161] A variety of methods can be used to generate three-dimensional curves based on fused feature vectors, such as using a cubic spline interpolation algorithm. The elements in the fused feature vector are assigned to coordinate point information in three-dimensional space in a certain order and rule, such as taking the first few elements as x-coordinate components, the middle few elements as y-coordinate components, and the remaining elements as z-coordinate components to determine a series of coordinate points. Then, based on these coordinate points, a smooth curve in three-dimensional space is calculated through a cubic spline interpolation algorithm, so that the curve can better fit the information represented by these coordinate points. The shape, trend and other geometric features of the curve are closely related to the terminal type features contained in the fused feature vector, and become an important data basis for subsequent feature extraction.

[0162] For the generation of two-dimensional graphics, the key parameters of the graphics can be determined based on some elements in the fused feature vector. For example, select several elements in the vector as the vertex coordinates of the polygon, and connect these vertices in order to form a closed two-dimensional polygonal figure, such as a triangle or quadrilateral; or use vector elements to determine the major semi-axis, minor semi-axis, center coordinates and other parameters of the ellipse to draw an ellipse. The geometric features of different two-dimensional graphics, such as shape, size, and angle, all contain information related to the terminal type from the fused feature vector, and together with the three-dimensional curve, they constitute a multi-dimensional data representation with rich geometric characteristics.

[0163] By performing Fourier transform on the three-dimensional curve, it is possible to convert from the time domain (the spatial morphology of the curve) to the frequency domain, extract the spectrum characteristics such as the energy distribution of the curve at different frequency components, and convert the complex geometric information contained in the curve into numerical features in the frequency domain, so as to further mine and refine the key information related to the terminal type. By calculating the geometric moment feature vector of the two-dimensional graphics, representative numerical features can be extracted from the perspective of geometric properties such as the shape, position, and direction of the graphics. The feature vectors extracted from these two different dimensions and angles are spliced ​​to form the final feature vector, which can integrate the core information from the three-dimensional curve and the two-dimensional graphics, so that the final feature vector integrates multiple key features, more comprehensively and accurately reflects the terminal type related information, and provides strong data support for the subsequent encryption to generate a secure and reliable basic deployment code.

[0164] Apply the Fourier transform algorithm to the generated three-dimensional curve, convert the coordinate point sequence of the curve in three-dimensional space (which can be regarded as a time domain signal representation) into a frequency domain representation, and obtain the spectrum information of the curve. The spectrum information contains data such as the amplitude and phase corresponding to different frequency components. The key information such as the main frequency components and their amplitudes are extracted and formed into a spectrum feature vector in a certain order.

[0165] For the generated two-dimensional graphics, calculate its geometric moment features. Geometric moments include moments of different orders such as zero-order moment, first-order moment, and second-order moment. The zero-order moment can reflect the area of ​​the graphics, the first-order moment is related to the center of gravity of the graphics, and the second-order moment involves the direction, shape and other attributes of the graphics. By calculating these geometric moment values ​​of different orders, they are combined into a geometric moment feature vector. This vector extracts the key features of the two-dimensional graphics from the perspective of geometric properties and also carries information related to the terminal type.

[0166] The obtained spectral feature vector and geometric moment feature vector are concatenated in a predetermined order, for example, the elements of the spectral feature vector are arranged in the front, and the elements of the geometric moment feature vector are followed closely to form a new, longer vector, which is the final feature vector. It combines the core feature information extracted from the three-dimensional curve and the two-dimensional graphics after different dimensional processing, more comprehensively reflects the comprehensive characteristics of the terminal type, and is presented in a compact numerical vector form, which is convenient for subsequent encryption and other operations.

[0167] The final feature vector is encrypted using the elliptic curve cryptography (ECC) system to ensure the security and confidentiality of the basic deployment code. ECC is a public key encryption system with high security and relatively short key length advantages. Through encryption operations, the key information related to the terminal type contained in the final feature vector is obfuscated and protected, so that even if the basic deployment code is obtained externally, the original information such as the terminal type contained therein cannot be parsed without the correct decryption key, thereby effectively preventing the intelligent agent from being illegally deployed on unauthorized terminals, ensuring the safe and reliable permission management of the intelligent agent deployment, and protecting the intellectual property rights and application security of the intelligent agent.

[0168] When it is necessary to deploy the intelligent agent and verify the terminal authority, only the authorized terminal with the corresponding private key can decrypt the basic deployment code (ciphertext) and restore the final feature vector, and then confirm whether the terminal has legal deployment authority through a series of reverse verification processes, thereby ensuring the security and controllability of the intelligent agent deployment process.

[0169] Reference Figure 2 In one embodiment of the present invention, a platform system for generating an AI agent is provided, comprising:

[0170] An acquisition unit, used to acquire a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data;

[0171] An extraction unit, used for extracting features from the normalized task data set to construct task feature data;

[0172] An initialization unit is used to optimize the initial parameters of the intelligent agent by adopting a hierarchical distributed architecture combined with a genetic algorithm, determine the basic structure and initial state of the intelligent agent, and obtain an initialized intelligent agent architecture;

[0173] A generation unit is used to perform reinforcement learning training and optimization on the initialized intelligent agent architecture based on the task feature data to generate an AI intelligent agent; wherein, a proximal strategy optimization algorithm is used to repeatedly execute the intelligent agent architecture in a simulation environment using the task feature data, and the policy network parameters of the intelligent agent architecture are continuously adjusted according to the reward feedback of task completion to obtain the trained and optimized AI intelligent agent.

[0174] In this embodiment, for the specific implementation of each unit in the above system embodiment, please refer to the above method embodiment, which will not be repeated here.

[0175] Reference Figure 3 In an embodiment of the present invention, an electronic device is also provided. The electronic device may be a server, and its internal structure may be as follows: Figure 3 As shown. The electronic device includes a processor, a memory, a display screen, an input device, a network interface and a database connected via a system bus. Among them, the computer-designed processor is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store the corresponding data in this embodiment. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above method is implemented.

[0176] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device to which the solution of the present invention is applied.

[0177] An embodiment of the present invention further provides a storage device on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the storage device in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0178] In summary, the AI ​​agent generation method, platform system, electronic device and storage device provided in the embodiments of the present invention include: obtaining a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data; extracting features from the standardized task data set to construct task feature data; using a hierarchical distributed architecture, combining a genetic algorithm to optimize the initial parameters of the agent, determining the basic structure and initial state of the agent, and obtaining an initialized agent architecture; based on the task feature data, performing reinforcement learning training and optimization on the initialized agent architecture to generate an AI agent; wherein, using a proximal strategy optimization algorithm, using the task feature data to repeatedly execute the agent architecture in a simulated environment, and continuously adjusting the policy network parameters of the agent architecture according to the reward feedback of task completion, to obtain the trained and optimized AI agent. In the present invention, by using a proximal strategy optimization algorithm, repeatedly executing the agent architecture in a simulated environment, and continuously adjusting the policy network parameters of the agent architecture according to the reward feedback of task completion, the trained agent can quickly adapt to changes in the environment and adjust its own strategy in a timely manner.

[0179] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0180] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0181] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for generating an AI agent, characterized in that: The following steps are involved: Acquire a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data; Extracting features from the normalized task data set to construct task feature data; Adopting a hierarchical distributed architecture and combining genetic algorithms to optimize the initial parameters of the agent, determine the basic structure and initial state of the agent, and obtain the initialized agent architecture; Based on the task feature data, the initialized intelligent agent architecture is subjected to reinforcement learning training and optimization to generate an AI intelligent agent; wherein, the intelligent agent architecture is repeatedly executed in a simulation environment using the task feature data using a proximal strategy optimization algorithm, and the policy network parameters of the intelligent agent architecture are continuously adjusted according to reward feedback of task completion to obtain a trained and optimized AI intelligent agent; Obtaining the terminal type of each terminal on which the AI ​​agent is to be deployed; Generate a basic deployment code based on each terminal type; For each terminal, generate a terminal deployment code based on its corresponding terminal type; When deploying the AI ​​agent for each terminal, the basic deployment code is combined with the terminal deployment code as the permission code for deploying the AI ​​agent for each terminal; When sending the AI ​​agent to each terminal, configuring the deployment authority of the AI ​​agent based on the corresponding authority code; The method generates a basic deployment code based on each terminal type, including: Encoding is performed based on each terminal type to obtain a feature value set; Construct an association matrix based on the feature value set, perform singular value decomposition on the association matrix, and take the first N singular values ​​to combine into a fused feature vector; Generate 3D curves and 2D graphics based on fused feature vectors; Perform Fourier transform on the three-dimensional curve to obtain the spectrum feature vector, calculate the geometric moment feature vector for the two-dimensional graph, and concatenate the spectrum feature vector and the geometric moment feature vector to obtain the final feature vector; The final feature vector is encrypted based on the elliptic curve cryptography to obtain the basic deployment code.

2. The AI ​​agent generation method according to claim 1, characterized in that: The step of obtaining a standardized task data set includes: Preprocess the historical task data and the corresponding execution result data to remove invalid and erroneous data, and perform standardized conversion on data in different formats to obtain a normalized task data set.

3. The AI ​​agent generation method according to claim 1, characterized in that: A hierarchical distributed architecture is used to optimize the initial parameters of the agent in combination with a genetic algorithm, determine the basic structure and initial state of the agent, and obtain the initialized agent architecture, including: The intelligent agent is divided into a perception layer, a decision layer, and an execution layer. Various types of sensor data fusion models are configured in the perception layer to receive data. A decision model based on a neural network is constructed at the decision layer, and a genetic algorithm is used to encode the weights, thresholds, and neuron connection structures of the decision model to generate an initial population; Setting a fitness function, which is constructed based on the agent's execution efficiency, resource consumption, and task completion accuracy in a predefined task scenario; The initial population is iteratively evolved through the selection, crossover and mutation operations of the genetic algorithm. In each generation of evolution, the quality of individuals is evaluated according to the fitness function, and excellent individuals are selected to enter the next generation of population. The selected individuals are crossover and mutation operations are performed to explore better parameter combinations. When the predetermined number of iterations is reached or the convergence condition is met, the individual with the highest fitness value is selected from the final population, and its corresponding parameters are decoded as the optimization parameters of the decision layer. Combined with the preset standard configuration of the perception layer and the execution layer, the basic structure and initial state of the intelligent agent are determined to obtain the initialized intelligent agent architecture.

4. The AI ​​agent generation method according to claim 1, characterized in that: Based on the task feature data, the initialized agent architecture is subjected to reinforcement learning training and optimization to generate an AI agent, including: Initialize the policy network parameters and value network parameters of the agent architecture. The policy network is used to generate action decisions, and the value network is used to evaluate benefits. Using the proximal policy optimization algorithm, based on the task feature data, the intelligent agent architecture obtains rewards and new states after execution in the simulation environment, and stores them in the experience replay buffer; after the experience replay buffer is full, data is sampled to update the policy network parameters and value network parameters; After completing the predetermined number of training rounds, the resulting agent is the generated AI agent.

5. The AI ​​agent generation method according to claim 1, characterized in that: The method generates a basic deployment code based on each terminal type, including: Encode and combine each terminal type respectively to obtain a coding combination; Adding the characters in the coding combination one by one to a matrix of multiple rows and columns to generate a character matrix; Obtain graphics corresponding to each terminal type in the database; The acquired graphics are digitized, the coordinates of key feature points of each graphic are extracted, and the coordinates of each key feature point are arranged into a coordinate sequence according to a predetermined order; Performing singular value decomposition on the character matrix to obtain multiple singular values, selecting the first several singular values ​​to fuse with the coordinate sequence to obtain a fused numerical sequence; The fused numerical sequence is encrypted to serve as the basic deployment code.

6. A platform system for generating AI agents, characterized in that: include: An acquisition unit, used to acquire a standardized task data set; wherein the task data set includes historical task data and corresponding execution result data; An extraction unit, used for extracting features from the normalized task data set to construct task feature data; An initialization unit is used to optimize the initial parameters of the intelligent agent by adopting a hierarchical distributed architecture combined with a genetic algorithm, determine the basic structure and initial state of the intelligent agent, and obtain an initialized intelligent agent architecture; A generating unit, configured to perform reinforcement learning training and optimization on the initialized agent framework based on the task feature data to generate an AI agent; wherein, a proximal strategy optimization algorithm is used to repeatedly execute the agent framework in a simulation environment using the task feature data, and the policy network parameters of the agent framework are continuously adjusted according to reward feedback of task completion to obtain a trained and optimized AI agent; Obtaining the terminal type of each terminal on which the AI ​​agent is to be deployed; Generate a basic deployment code based on each terminal type; For each terminal, generate a terminal deployment code based on its corresponding terminal type; When deploying the AI ​​agent for each terminal, the basic deployment code is combined with the terminal deployment code as the permission code for deploying the AI ​​agent for each terminal; When sending the AI ​​agent to each terminal, configuring the deployment authority of the AI ​​agent based on the corresponding authority code; The method generates a basic deployment code based on each terminal type, including: Encoding is performed based on each terminal type to obtain a feature value set; Construct an association matrix based on the feature value set, perform singular value decomposition on the association matrix, and take the first N singular values ​​to combine into a fused feature vector; Generate 3D curves and 2D graphics based on fused feature vectors; Perform Fourier transform on the three-dimensional curve to obtain the spectrum feature vector, calculate the geometric moment feature vector for the two-dimensional graph, and concatenate the spectrum feature vector and the geometric moment feature vector to obtain the final feature vector; The final feature vector is encrypted based on the elliptic curve cryptography to obtain the basic deployment code.

7. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A storage device having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images