A reinforcement learning-based adaptive geometric envelope method and system for irregular obstacles

Abstract

This invention provides a method and system for adaptive geometric envelope of irregular obstacles based on reinforcement learning, applied in the field of autonomous navigation and safety control technology for mobile robots. The method includes extracting geometric features of the obstacle contour and constructing a normalized real-time state vector; inputting the real-time state vector into a trained reinforcement learning agent and outputting a reference radius for generating the geometric envelope; wherein the reinforcement learning agent is trained using the PPO algorithm; and constructing a continuously differentiable envelope curve based on the reference radius and geometric features to enclose the obstacle. The advantages of this invention are: by extracting the geometric features of the obstacle contour and constructing a normalized real-time state vector, combined with the reference radius output by the reinforcement learning agent, this invention can generate a compact and smooth envelope curve, improving the accuracy of obstacle geometric representation.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation and safety control technology for mobile robots, and in particular to a method and system for adaptive geometric envelope of irregular obstacles based on reinforcement learning. Background Technology

[0002] In navigation and obstacle avoidance of autonomous systems such as mobile robots and drones, accurate and efficient obstacle geometry representation is the foundation of safety planning. To simplify calculations and meet the differentiability requirements of subsequent optimizations (such as optimization controllers based on the control obstacle function CBF), existing technologies often approximate irregular obstacles using simple convex shapes such as circumcircle or circumcircle rectangle.

[0003] Existing technologies also employ more refined methods (such as multi-circle decomposition) to fit obstacles more closely. However, key parameters of these methods (such as the number, position, and radius of circles) often rely on human experience or complex offline optimization, making it impossible to adjust them online and adaptively based on the specific geometric features of the obstacles perceived in real time. This results in either overly conservative envelopes (wasting passage space) or poor fitting performance due to improper parameter settings in complex and variable unstructured environments.

[0004] Specifically, existing technologies have the following problems:

[0005] (1) Inaccurate geometric fitting: The envelope of a fixed shape (such as a single circumscribed circle) will inevitably generate a large amount of redundant space, compressing the feasible path of the robot; and if the parameters of the multi-circle method are not set properly, a tight fit cannot be achieved.

[0006] (2) Lack of online adaptability: The parameters of the existing methods cannot be optimized and adjusted in real time and automatically according to the unique shape of the current obstacle (such as concavity and convexity, aspect ratio, irregularity), resulting in low intelligence.

[0007] (3) It requires manual or offline calculation, the parameter tuning process is cumbersome, and it cannot be integrated into online navigation systems that require real-time response.

[0008] (4) It cannot be guaranteed that the generated envelope function is continuously differentiable, thus it cannot be integrated into modern optimization control frameworks (such as CBF) that require gradient information.

[0009] Therefore, there is an urgent need to develop an adaptive geometric envelope method and system based on reinforcement learning for irregular obstacles to solve the above problems. Summary of the Invention

[0010] This invention provides a reinforcement learning-based adaptive geometric envelope method and system for irregular obstacles to solve the above problems, addressing the issue that navigation and obstacle avoidance cannot make online, adaptive adjustments based on the specific geometric features of obstacles perceived in real time.

[0011] According to a first aspect of the present invention, a reinforcement learning-based adaptive geometric envelope method for irregular obstacles is provided. The method includes: extracting geometric features of the obstacle contour and constructing a normalized real-time state vector;

[0012] The real-time state vector is input into the trained reinforcement learning agent, and the output is the reference radius used to generate the geometric envelope; wherein the reinforcement learning agent is trained using the PPO algorithm;

[0013] A continuous and differentiable envelope curve is constructed based on the reference radius and geometric features to enclose the obstacle.

[0014] Furthermore, the state vector includes multiple normalized geometric features; specifically, it includes the normalized area value of the obstacle, the normalized maximum semi-polar radius value of the obstacle, the normalized minimum semi-polar radius value of the obstacle, the number of vertices of the obstacle profile, and the convexity of the obstacle profile.

[0015] Furthermore, the reinforcement learning agent employs a policy network based on the PPO algorithm. This policy network outputs a continuous real-time action value and maps this real-time action value to obtain a baseline radius, as shown in the following formula:

[0016] ,

[0017] in, Indicates the reference radius. Indicates real-time action value. Indicates the upper bound of the mapping. Indicates the lower bound of the mapping. It is a contractile factor. This represents the minimum distance from the centroid of the obstacle to the vertex of its profile.

[0018] Furthermore, the reinforcement learning agent is trained using the PPO algorithm, specifically including the following steps:

[0019] The reinforcement learning agent obtains empirical data based on real-time state vectors; the empirical data includes the previous state vector, real-time action value, and reward function value.

[0020] Based on the reward function value in the empirical data, the real-time advantage function value is calculated, which is used to evaluate the merits of the real-time action value relative to the average action value.

[0021] Based on the real-time advantage function value, the objective function of the PPO algorithm is optimized to update the parameters of the policy network.

[0022] Furthermore, the reward function value is obtained based on a preset reward function, which is used to quantify the quality of real-time action values;

[0023] The reward function comprises a weighted sum of space utilization and shape regularity;

[0024] The space utilization rate is directly proportional to the ratio of the actual area of ​​the obstacle to the area enclosed by the generated envelope curve;

[0025] The shape regularity is proportional to the ratio of the circumference of the reference circle to the circumference of the generated envelope curve.

[0026] Furthermore, the real-time advantage function value Calculated using the following formula:

[0027] ,

[0028] in, V(·) represents the time-series difference error at time t, and V(·) represents the state-value function. This represents the time-series difference error at time t+1. This represents the reward function value at time t. Let t represent the state vector at time t, and l represent the number of time steps forward. Indicates the discount factor. A parameter representing the tradeoff between bias and variance.

[0029] Constructing a continuously differentiable envelope curve based on the aforementioned reference radius and geometric features includes:

[0030] A reference circle is established with the centroid of the obstacle's outline as the center and the reference radius as the radius. The obstacle's outline is scanned in polar coordinates to identify the portion that exceeds the boundary of the reference circle and the portion that exceeds the boundary of the reference circle is clustered into several independent protruding regions.

[0031] Determine the local parameters of the local envelope curve corresponding to each convex region being covered, and construct a globally continuous and differentiable closed envelope curve segmentally based on the reference radius and local parameters.

[0032] According to a second aspect of the present invention, a reinforcement learning-based adaptive geometric envelope system for irregular obstacles is provided, comprising:

[0033] The acquisition module is used to extract the geometric features of the obstacle contours and construct a normalized real-time state vector.

[0034] A radius determination module, connected to the acquisition module, is used to input the real-time state vector into the trained reinforcement learning agent and output a reference radius for generating the geometric envelope; wherein the reinforcement learning agent is trained using the PPO algorithm.

[0035] An envelope generation module, connected to the radius determination module, is used to construct a continuous and differentiable envelope curve based on the reference radius and geometric features to surround the obstacle.

[0036] According to a third aspect of the present invention, an electronic device is provided, including a memory, a processor, and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement a reinforcement learning-based adaptive geometric envelope method for irregular obstacles.

[0037] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, characterized in that a computer program is stored thereon; the computer program is executed by a processor to implement the method as described in the first aspect.

[0038] The beneficial effects of this invention are:

[0039] This invention extracts the geometric features of the obstacle contour and constructs a normalized real-time state vector. Combined with the baseline radius output by the reinforcement learning agent, it can generate a compact and smooth envelope curve, thereby improving the accuracy of obstacle geometric representation.

[0040] Among them, the PPO algorithm is used to train the reinforcement learning agent, enabling it to make online and adaptive adjustments based on the specific geometric features of obstacles perceived in real time, thereby generating more accurate envelope curves; during the training process, the reinforcement learning agent gradually fits a high-dimensional nonlinear mapping relationship from the topological features of obstacles to the optimal parameters through continuous trial and error and optimization.

[0041] By setting a reward function that includes space utilization and shape regularity, redundant space is minimized while safely enclosing obstacles;

[0042] When constructing the envelope curve, the continuous differentiability of the envelope curve in the global domain is ensured by defining a piecewise raised cosine function with respect to the azimuth angle and determining the local parameter set based on the reference radius and geometric features. This enables it to be integrated into modern optimization control frameworks that require gradient information.

[0043] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0044] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the invention. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0045] Figure 1 The flowchart illustrates an adaptive geometric envelope method for irregular obstacles based on reinforcement learning, provided by an embodiment of the present invention.

[0046] Figure 2 A simplified flowchart of an adaptive geometric envelope method for irregular obstacles based on reinforcement learning, provided by an embodiment of the present invention, is shown.

[0047] Figure 3 This diagram illustrates a framework of an adaptive geometric envelope system for irregular obstacles based on reinforcement learning, provided by an embodiment of the present invention.

[0048] Figure 4 A block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0051] Based on the above considerations, this invention provides a reinforcement learning-based adaptive geometric envelope method for irregular obstacles, see [link to relevant documentation]. Figures 1 to 2 The method specifically includes the following steps:

[0052] S1. Extract the geometric features of the obstacle contours and construct a normalized real-time state vector;

[0053] The geometric features of an obstacle are determined based on its contour point cloud or other boundary data, including its area, number of vertices, and semi-polar diameter.

[0054] Specifically, this invention defines a state vector. The geometric features are normalized to eliminate the influence of dimensions, which is expressed by the following formula:

[0055] ,

[0056] in, / Indicates the actual area of ​​the obstacle The ratio of the area of ​​an obstacle to the maximum area of ​​the obstacle in the dataset (a large number of randomly generated polygons), i.e., the normalized area of ​​the obstacle; The maximum semipolar radius of the obstacle profile and the maximum circumcircle radius of the obstacle in the dataset represent the two largest diameters of the obstacle profile. The ratio of the two, i.e., the normalized value of the maximum half-polar radius of the obstacle; The minimum semipolar radius representing the obstacle profile and the maximum circumcircle radius of the obstacles in the dataset. The ratio of the minimum half-radius radius of the obstacle to the normalized value; The number of vertices representing the outline of the obstacle; The convexity of an obstacle's profile is defined as the actual area of ​​the obstacle. The ratio of its convex hull area. When When the value approaches 1, it indicates that the obstacle is close to a convex shape; conversely, it indicates that there is a deep, non-convex shape.

[0057] By defining a state vector, the parameter constraints with different physical properties in the geometric features are mapped to the same numerical scale, eliminating the dimensional differences in the original data and effectively avoiding the problem of numerical sensitivity of gradient updates caused by the large differences in the range of feature values.

[0058] S2. Input the real-time state vector into the trained reinforcement learning agent and output the reference radius for generating the geometric envelope; wherein, the reinforcement learning agent is trained using the PPO algorithm;

[0059] This invention utilizes a trained reinforcement learning agent and calculates the optimal envelope curve baseline radius based on the calculated obstacle state.

[0060] Specifically, the reinforcement learning agent uses a policy network based on the Proximal Policy Optimization (PPO) algorithm. The policy network outputs a continuous action value, which is then mapped to obtain a baseline radius.

[0061] The reference radius is used to determine the basic shape of the final envelope curve; therefore, during the envelope curve generation process, the reference radius... It requires high precision. Considering that discrete motion spaces inevitably introduce quantization errors, such as discretizing the reference radius into a fixed length, the generated obstacle envelope curve cannot achieve a dynamic balance between compactness and smoothness. Therefore, this invention constructs a domain defined as... The continuous action space, which contains the continuous action values ​​output by the policy network. The reference radius is obtained by performing a linear mapping. Please refer to the following formula for details:

[0062] ,

[0063] in, The upper bound of the mapping is the radius of the circumcircle of the obstacle. ; ( (where is the contraction factor, with a value slightly greater than 1) indicates the lower bound of the mapping. This represents the minimum distance from the centroid of the obstacle to the vertex of its profile, approximately the radius of its inscribed circle. The reference radius should be greater than [missing information]. .

[0064] This invention improves the overall convergence efficiency and numerical stability of the algorithm by constraining the exploration range of the intelligent agent.

[0065] To ensure the policy iteration process remains monotonically constant, the PPO algorithm introduces a truncated objective function. By limiting the KL divergence between the distributions of the new and old policies, the update step size is constrained within a safe region. That is, through extensive training with obstacles of different shapes, the reinforcement learning agent learns the mapping relationship from the state vector to the reference radius. See the following formula for details:

[0066] ,

[0067] in, Represents network parameters, This represents the probability ratio between the old and new strategies. Indicates the current state of the policy network. Take action below The probability, Indicates the old policy network in state Take action below The probability, This represents the cutoff factor, which can be set to 0.2; This represents the expected value based on experience (i.e., the average value). This represents a truncation function that restricts the input value to the range (1-ɛ, 1+ɛ). This represents the real-time advantage function value, used to update the objective function.

[0068] The above steps are implemented in the policy network, and a baseline radius is obtained. Further evaluation is needed to determine its performance.

[0069] This invention establishes a reward function to find the optimal balance between safely enclosing obstacles and minimizing redundancy in the envelope curve, thereby judging the quality of the obtained baseline radius. Considering the multi-constraint characteristics of the envelope fitting task, the reward function includes a space utilization term and a shape regularity term. See the following formula:

[0070] ,

[0071] in, and All are weighting coefficients; This represents space utilization rate, numerically equal to the actual area of ​​the obstacle. The area enclosed by the generated envelope curve The ratio of the generated envelope area to the generated envelope shape is such that when the generated envelope shape is more compact, the generated envelope area is larger. Reduce space utilization Increase the size of the boundary to incentivize the reinforcement learning agent to shrink the boundary as much as possible while safely avoiding obstacles; This indicates shape regularity, which is proportional to the ratio of the circumference of the reference circle to the circumference of the generated envelope curve. This avoids generating envelope curves with too many convex areas. See the following formula for details:

[0072] ,

[0073] in, Indicates the circumference of the reference circle. This represents the actual perimeter of the generated envelope curve. Used to evaluate the geometric smoothness of the envelope curve, when When the perimeter of the envelope is too small, resulting in numerous elongated protrusions, the perimeter of the envelope... It will increase, which in turn leads to Numerical reduction. This invention introduces shape regularity to suppress geometric distortion, resulting in the generation of envelope curves with smooth and concise contours.

[0074] In summary, while real-time evaluation of the quality of each action has been achieved, it is still necessary to further transform the reward function value into a dominance function value to perform time-series cumulative evaluation based on each action, thereby obtaining a reliable optimization signal. See the following steps for details:

[0075] 1. Value network output state value V( )

[0076] For each time t, the value network is based on the current state vector. Output scalar value V( ), which represents the cumulative reward expected to be obtained in this state.

[0077] 2. Calculate the timing difference error

[0078] It is used to measure the gap between actual rewards received and future expectations and current expectations.

[0079] if A value greater than 0 indicates that the actual result was better than expected. It should be increased;

[0080] if < 0 indicates that the actual result is worse than expected, V( It should be lowered.

[0081] It should be noted that when When =0, it means that the prediction of the value network is consistent with the subsequent actual return, and there is no prediction error.

[0082] 3. The generalized dominance estimation algorithm is used to calculate the real-time dominance function value. Please refer to the following formula for details:

[0083] ,

[0084] in, V(·) represents the time-series difference error at time t, and V(·) represents the state-value function. This represents the time-series difference error at time t+1. Let t represent the reward function value obtained at time t, and l represent the number of time intervals to follow. Indicates the discount factor. A parameter representing the tradeoff between bias and variance.

[0085] The real-time advantage function value is a weighted sum that takes into account the time-series difference error over multiple future steps, i.e., it is determined at... Take action value Advantages or disadvantages relative to the average level.

[0086] > 0: This action is better than average; the probability of this action value should be increased.

[0087] < 0: This action is worse than average, and the probability of this action value should be reduced;

[0088] It should be noted that If the value is 0, the action contributes nothing to the objective function and will not cause the policy network to increase or decrease the probability of the action. This indicates that the policy is close to the optimal level or that the current action is exactly at the baseline level.

[0089] 4. Training the value network

[0090] The value network is updated by minimizing the following loss function, i.e., obtaining V( The predicted values ​​are getting closer and closer to the actual observed values.

[0091] = ( or = (V( )-( + V( )) ,

[0092] in, Represents the value network loss function.

[0093] During training, a parallel vector environment is constructed using multi-process methods to collect multiple trajectory data for mini-batch stochastic gradient descent. Furthermore, since reinforcement learning agents may get stuck in local optima, an entropy regularization term can be added to the objective function to drive the policy network to repeatedly interact with a large number of geometric samples, gradually fitting a high-dimensional nonlinear mapping from obstacle topological features to optimal parameters.

[0094] In summary, the reward function is used to measure the quality of the calculated baseline radius, i.e. whether the envelope curve is tight and smooth. However, it cannot provide the contribution of each baseline radius to the envelope curve. Therefore, the advantage function value can provide a more accurate evaluation to guide the policy network to perform more efficient parameter updates, thereby outputting a better baseline radius.

[0095] The above process can be summarized into the following steps:

[0096] The reinforcement learning agent obtains empirical data based on real-time state vectors; the empirical data includes the previous state vector, real-time action value, and reward function value.

[0097] Based on the reward function value in the empirical data, the real-time advantage function value is calculated, which is used to evaluate the merits of the real-time action value relative to the average action value.

[0098] Based on the real-time advantage function value, the objective function of the PPO algorithm is optimized to update the parameters of the policy network.

[0099] The reward function value is obtained based on a preset reward function, which is used to quantify the quality of real-time action values.

[0100] In one specific embodiment, the Actor-Critic framework based on the PPO algorithm is used as the reinforcement learning agent architecture to achieve a better balance between sample utilization and training stability by utilizing the truncation objective function mechanism unique to the Actor-Critic framework.

[0101] In one specific embodiment, the reinforcement learning agent sequentially comprises an input layer, a feature extraction layer, a policy network, a value network, and an output layer. Both the policy network and the value network utilize a multilayer perceptron to process abstract geometric features. The input layer directly receives the preprocessed dimensionless state vector. The feature extraction layer includes two fully connected hidden layers, each containing 128 neurons. Considering the gradient smoothness requirements of continuous control tasks, the hyperbolic tangent activation function is chosen. The output layer is designed as a two-stream structure: the mean of the action values ​​output by the policy network is... and standard deviation Construct a Gaussian distribution To support stochastic exploration in continuous space; the value network outputs a scalar. This is used to estimate the expected effect of the current geometric features, providing a benchmark for the calculation of the dominance function value.

[0102] S3. Construct a continuous and differentiable envelope curve based on the reference radius and geometric features to surround the obstacle.

[0103] A reference circle is established with the centroid of the obstacle's outline as the center and the reference radius as the radius. The obstacle's outline is scanned in polar coordinates to identify the portion that exceeds the boundary of the reference circle and the portion that exceeds the boundary of the reference circle is clustered into several independent protruding regions.

[0104] Determine the local parameters of the local envelope curve corresponding to each convex region being covered, and construct a globally continuous and differentiable closed envelope curve segmentally based on the reference radius and local parameters.

[0105] First, set the reference circle. To establish a polar coordinate system, ( Using the coordinates of the center of the reference circle as an example, scan and acquire the outline of the obstacle. This identifies all protruding regions that extend beyond the reference circle. To avoid high-frequency noise introduced by local rough textures on the obstacle surface, this invention introduces an angular neighborhood clustering strategy. That is, for any two adjacent protruding regions, if the angular interval between their boundary points is... Then it is considered as a raised area. The preset value is used to merge two closely spaced raised areas. When the angle between two adjacent raised areas is less than the preset value, they are treated as the same raised area.

[0106] For each of the identified protruding regions, define the local parameter set of the corresponding local envelope curve. This describes the geometric features of the local envelope curve covering the i-th convex region. This indicates the position where the curve begins to rise. This indicates the point where the curve returns to the reference circle. This indicates the height by which the highest point of the protruding area exceeds the reference radius. Among them, This represents the interval where the amplitude of the envelope curve remains constant, and within this interval, the radius of the envelope curve remains constant. To ensure safe crossing of the obstacle apex, the ascent transition angle width is defined as... The width of the descent transition angle is defined as It is used to create a smooth transition region.

[0107] Based on the above parameters, construct a method for azimuth angles. Piecewise raised cosine function The azimuth angle is obtained by rotating counterclockwise from the center of the reference circle as the pole, as shown in the following formula:

[0108] ,

[0109] in, Indicates the local angle of the rising segment. This represents the local angle of the descending segment. When the tip of the polygon extends beyond the reference circle, it needs to be processed using a piecewise raised cosine function to generate the envelope profile, such as from... Starting from the corresponding angle, the envelope profile rises from the reference circumference, ascending to... Stop ascending after reaching the corresponding angle. This indicates a deviation during the process. The angle.

[0110] Based on the above formula, the envelope curve is guaranteed to be continuous and differentiable in the global range, which is suitable for gradient optimization. In gradient-based quadratic programming, it avoids control variable jitter and improves motion stability.

[0111] Based on the above technical solutions, the present invention has the following advantages:

[0112] This invention extracts the geometric features of the obstacle contour and constructs a normalized real-time state vector. Combined with the baseline radius output by the reinforcement learning agent, it can generate a compact and smooth envelope curve, thereby improving the accuracy of obstacle geometric representation.

[0113] Among them, the PPO algorithm is used to train the reinforcement learning agent, enabling it to make online and adaptive adjustments based on the specific geometric features of obstacles perceived in real time, thereby generating more accurate envelope curves; during the training process, the reinforcement learning agent gradually fits a high-dimensional nonlinear mapping relationship from the topological features of obstacles to the optimal parameters through continuous trial and error and optimization.

[0114] By setting a reward function that includes space utilization and shape regularity, redundant space is minimized while safely enclosing obstacles;

[0115] When constructing the envelope curve, the continuous differentiability of the envelope curve in the global domain is ensured by defining a piecewise raised cosine function with respect to the azimuth angle and determining the local parameter set based on the reference radius and geometric features. This enables it to be integrated into modern optimization control frameworks that require gradient information.

[0116] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0117] The acquisition, storage, and application of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0118] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing embodiments, and will not be repeated here.

[0119] This invention also provides a reinforcement learning-based adaptive geometric envelope system for irregular obstacles 300, see [link to relevant documentation]. Figure 3 Specifically, it includes:

[0120] The acquisition module 310 is used to extract the geometric features of the obstacle contour and construct a normalized real-time state vector.

[0121] The radius determination module 320, connected to the acquisition module 310, is used to input the real-time state vector into the trained reinforcement learning agent and output a reference radius for generating the geometric envelope; wherein, the reinforcement learning agent is trained using the PPO algorithm;

[0122] The envelope generation module 330, connected to the radius determination module 320, is used to construct a continuous and differentiable envelope curve based on the reference radius and geometric features to surround the obstacle.

[0123] Please refer to the previous methods for other details, which will not be repeated here.

[0124] Figure 4 A schematic block diagram of an electronic device 400 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0125] Electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in ROM 402 or a computer program loaded into RAM 403 from storage unit 408. RAM 403 can also store various programs and data required for the operation of electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O interface 405 is also connected to bus 404.

[0126] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 405, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0127] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various processes described above. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409.

[0128] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code used to implement the embodiments of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0132] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0133] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0134] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0135] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A reinforcement learning-based adaptive geometric envelope method for irregular obstacles, characterized in that, Includes the following steps: Extract the geometric features of the obstacle contours and construct a normalized real-time state vector; The real-time state vector is input into the trained reinforcement learning agent, and the reference radius used to generate the geometric envelope is output; wherein, the reinforcement learning agent is trained using the PPO algorithm; A continuous and differentiable envelope curve is constructed based on the reference radius and geometric features to enclose the obstacle; The reinforcement learning agent uses a policy network based on the PPO algorithm. The policy network outputs a continuous real-time action value and maps the real-time action value to obtain a baseline radius. The reinforcement learning agent is trained using the PPO algorithm, specifically including the following steps: The reinforcement learning agent obtains empirical data based on real-time state vectors; the empirical data includes the previous state vector, real-time action value, and reward function value. Based on the reward function value in the empirical data, a real-time advantage function value is calculated, which is used to evaluate the merits of the real-time action value relative to the average action value. Based on the real-time advantage function value, the objective function of the PPO algorithm is optimized to update the parameters of the policy network; The reward function value is obtained based on a preset reward function, which is used to quantify the quality of real-time action values. The reward function comprises a weighted sum of space utilization and shape regularity; The space utilization rate is directly proportional to the ratio of the actual area of ​​the obstacle to the area enclosed by the generated envelope curve; The shape regularity is proportional to the ratio of the circumference of the reference circle to the circumference of the generated envelope curve; Constructing a continuously differentiable envelope curve based on the aforementioned reference radius and geometric features includes: A reference circle is established with the centroid of the obstacle's outline as the center and the reference radius as the radius. The obstacle's outline is scanned in polar coordinates to identify the portion that exceeds the boundary of the reference circle and the portion that exceeds the boundary of the reference circle is clustered into several independent protruding regions. Determine the local parameters of the local envelope curve corresponding to each convex region being covered, and construct a globally continuous and differentiable closed envelope curve segmentally based on the reference radius and local parameters.

2. The adaptive geometric envelope method for irregular obstacles based on reinforcement learning according to claim 1, characterized in that, The state vector includes multiple normalized geometric features; specifically, it includes the normalized area value of the obstacle, the normalized maximum semi-polar radius value of the obstacle, the normalized minimum semi-polar radius value of the obstacle, the number of vertices of the obstacle profile, and the convexity of the obstacle profile.

3. The adaptive geometric envelope method for irregular obstacles based on reinforcement learning according to claim 1, characterized in that, The reference radius is obtained by mapping the real-time action value, as shown in the following formula: ， in, Indicates the reference radius. Indicates real-time action value. Indicates the upper bound of the mapping. Indicates the lower bound of the mapping. It is a contractile factor. This represents the minimum distance from the centroid of the obstacle to the vertex of its profile.

4. The adaptive geometric envelope method for irregular obstacles based on reinforcement learning according to claim 1, characterized in that, The real-time advantage function value Calculated using the following formula: ， in, V(·) represents the time-series difference error at time t, and V(·) represents the state-value function. This represents the time-series difference error at time t+1. This represents the reward function value at time t. Let t represent the state vector at time t, and l represent the number of time steps forward. Indicates the discount factor. A parameter representing the tradeoff between bias and variance.

5. A reinforcement learning-based adaptive geometric envelope system for irregular obstacles, used to implement the reinforcement learning-based adaptive geometric envelope method for irregular obstacles as described in any one of claims 1-4, characterized in that, The system includes: The acquisition module is used to extract the geometric features of the obstacle contours and construct a normalized real-time state vector. A radius determination module, connected to the acquisition module, is used to input the real-time state vector into the trained reinforcement learning agent and output a reference radius for generating the geometric envelope; wherein the reinforcement learning agent is trained using the PPO algorithm. An envelope generation module, connected to the radius determination module, is used to construct a continuous and differentiable envelope curve based on the reference radius and geometric features to surround the obstacle.

6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement a reinforcement learning-based adaptive geometric envelope method for irregular obstacles as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores a computer program thereon; the computer program is executed by a processor to implement the method as described in any one of claims 1 to 4.

Images