A reinforcement learning environment perception method and system based on visual prior probability map guidance

Abstract

This invention provides a reinforcement learning environment perception method guided by visual prior probability graphs, relating to the fields of machine vision and reinforcement learning. The method includes: acquiring multimodal information using an industrial sensing terminal; constructing a visual prior feature space through multi-scale feature extraction operators; defining features as logical nodes and establishing conditional probability factors to form a logical reasoning network; using a confidence propagation algorithm to solve the probability distribution of the contribution of each visual region to the task objective; embedding the probability distribution into the reinforcement learning state space, suppressing redundant data and pruning the action space to guide heuristic exploration; and finally establishing a visual confidence feedback loop to achieve online fusion of prior logic and reinforcement learning experience. This invention effectively solves the problems of low sample efficiency and insufficient feature understanding when reinforcement learning processes high-dimensional visual input, significantly improving the robot's environmental adaptability and perception accuracy under visual guidance.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and reinforcement learning technology, specifically to a reinforcement learning environment perception method and system based on prior probability graph guidance, for robots to perform visual semantic understanding, high-dimensional feature completion, and heuristic task decision-making in unstructured environments. Background Technology

[0002] With the rapid development of intelligent manufacturing and embodied intelligence technologies, industrial robots have been widely applied in precision assembly, workpiece handling, and automated quality inspection. In these scenarios, how to enable robots to possess "visual intuition" similar to humans—that is, to quickly identify key operational features and make reasonable decisions in complex visual backgrounds—is a current research hotspot. Current robot visual perception and decision-making solutions mainly face the following technical bottlenecks:

[0003] 1. Insufficient robustness of visual perception: In real industrial environments, reflections, shadows, occlusions, and even minor changes in workpiece pose can lead to the loss or distortion of extracted visual features. Traditional end-to-end deep learning models lack logical reasoning capabilities, and once the quality of visual input deteriorates, the strategy is prone to failure, lacking stability for environmental adaptation.

[0004] 2. Low sample efficiency in reinforcement learning: Traditional deep reinforcement learning (DRL) methods often operate in a state of "blind exploration" when dealing with high-dimensional visual input. The agent needs to make random attempts in a huge action space, resulting in extremely slow training convergence and difficulty in handling the complex problem of dual coupling between action space and visual state space in industrial scenarios.

[0005] 3. Expert experience is difficult to effectively translate: Skilled operators usually have "visual intuition" based on physical logic (e.g., predicting the gripping position based on the edge direction of the workpiece), but current reinforcement learning frameworks have difficulty effectively translating this highly structured prior knowledge into perceptual constraints for robots.

[0006] 4. Catastrophic forgetting and generalization bottleneck: When robots learn new visual environments or tasks, they often lose their understanding of existing visual logic. Furthermore, when faced with unstructured and complex workpieces, the generalization ability of the algorithm is limited, and effective knowledge transfer cannot be achieved.

[0007] In summary, existing robot trajectory planning or environmental perception methods lack a mechanism that can deeply couple human visual logic priors with reinforcement learning's self-evolutionary capabilities, resulting in the system's intelligence level and operational reliability failing to meet the requirements of precision manufacturing under complex industrial conditions. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a reinforcement learning-based environment perception method and system guided by visual prior probability graphs, aiming to improve the logical rigor, feature completion capability, and convergence efficiency of strategy training for robots in complex industrial scenarios.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] This invention provides a reinforcement learning-based environment perception method guided by visual prior probability graphs, comprising:

[0011] Step S1: Use industrial sensing terminals to acquire RGB-D multimodal visual information of the target object to be operated, and use multi-scale feature extraction operators to identify visual semantic key points and geometric topological features of the target object to construct a visual prior feature space.

[0012] Step S2: Using a priori probabilistic graphical model, the visual semantic key points and geometric topological features of the target object described in step S1 are defined as logical nodes. Conditional probability factors between nodes are established based on the physical constraints of the target object and the physical logic of the working environment, forming a logical reasoning network that can characterize the geometric topological features of the target object.

[0013] Step S3: Using the confidence propagation algorithm, the prior probability graphical model established in step S2 is solved globally for consistency, and the probability distribution of the contribution of each visual region of the target object to the current task target is calculated.

[0014] Step S4: Embed the contribution probability distribution described in step S3 into the reinforcement learning state space, use a probability-guided selective mechanism to suppress redundant data in the policy network, and perform geometric constraint pruning on the action space to guide the agent to perform heuristic exploration in visually salient regions.

[0015] Step S5: Establish a feedback loop based on visual confidence, and adjust the parameters in the probabilistic graphical model in real time according to the visual feedback value after the strategy is executed, so as to realize the online fusion of visual prior logic and reinforcement learning exploration experience.

[0016] Furthermore, the identification of the target object through the multi-scale feature extraction operator in step S1 includes:

[0017] Step S1.1: Extract the visual semantic key point heatmap of the target object using a deep residual network;

[0018] Step S1.2: Extract the surface curvature distribution and normal pointing tensor using point cloud operators.

[0019] Furthermore, the method for acquiring multimodal visual information of the target object to be operated using an industrial sensing terminal in step S1 includes: the industrial sensing terminal introduces an adaptive sampling mechanism to automatically increase the point cloud sampling density and calculate multi-scale geometric features for large curvature edges or narrow deep groove areas in complex workpieces, ensuring that the generated feature space can capture microscopic topological changes and provide high-fidelity surface detail support for subsequent probabilistic inference.

[0020] Furthermore, the conditional probability factor mentioned in step S2 can define the coupling strength between random variable nodes by pre-setting the geometric relationship between key points and the physical logic of the work environment, thereby transforming human work intuition into calculable prior probability constraints.

[0021] Furthermore, the prior probability graphical model described in step S2 adopts a factor graph architecture. Its conditional probability factors not only utilize the prior space construction unit to perform tensor mapping, but also configure potential field constraints based on normal smoothness. In the region of perceived data distortion or missing data, probability weights are used to infer the geometric orientation that conforms to physical common sense, thereby achieving logical completion at the feature level.

[0022] Furthermore, the calculation of the probability distribution of the contribution of each visual region of the target object to the current task target in step S3 includes the following steps:

[0023] Step S3.1: The confidence propagation algorithm is used to solve the global consistency constraint of the prior probability graphical model in step S2, and the local geometric features of the target object acquired in real time are transformed into the state posterior distribution of the contribution to the task.

[0024] Step S3.2: Iteratively optimize the geometric topological feature nodes of the target object through the confidence propagation algorithm to eliminate noise interference in the visual perception of the industrial sensing terminal, and calculate the probabilistic weight matrix covering the entire surface of the target object.

[0025] Step S3.3: Based on the mapping relationship between the probabilistic weight matrix and the physical surface of the target object, the weight of visual features on the decision is calibrated, and a heat map matrix guiding the decision with the same dimension as the original visual image is output.

[0026] Furthermore, in step S3, the confidence propagation algorithm introduces a task-oriented potential function when performing probability inference. This function quantifies the sensitivity of the geometric feature normal deviation of the target object to the final task environment, transforming ordinary geometric features into significant weights with task value, and can accurately extract key geometric feature points of the task environment from the global point cloud.

[0027] Furthermore, the selective mechanism described in step S4 includes:

[0028] Step S4.1: Multiply the state input vector of reinforcement learning with the probability distribution map generated in step S3 element-wise to filter out redundant visual information.

[0029] Step S4.2: Weighted sampling of the action output of the policy network is performed according to the probability distribution to filter out low-probability actions that violate physical logic and improve the sample utilization efficiency of reinforcement learning.

[0030] Furthermore, the closed-loop feedback in step S5 uses visual reconstruction error or discriminator score to evaluate the confidence level of the current perception state. When the visual confidence level is lower than a preset threshold, the weight of the prior probability map in decision guidance is automatically increased to compensate for feature loss due to environmental interference.

[0031] A reinforcement learning-based environment perception system guided by visual prior probability graphs, comprising:

[0032] Multi-dimensional surface feature extraction module: It is used to perform intrinsic parameter mapping between high-precision point clouds collected by multi-source industrial sensing terminals and multi-scale textures by integrating spatial coordinate alignment units, and multi-scale feature extraction operators to identify visual semantic key points and geometric topological features of target objects, and construct visual prior feature space.

[0033] Prior probability graph encoding module: used to build a probability graph structure containing variable nodes and factor sets according to the task objective; to perform tensor mapping using prior space construction units and configure potential field constraints based on normal smoothness; to infer geometric orientation that conforms to physical common sense in the region of perceived data distortion or missing data using probability weights, providing the system with consistent spatial pose prior;

[0034] Task-oriented contribution inference module: used to integrate the confidence propagation inference operator to perform message passing iteration on the probabilistic graph, aggregate global topological constraints to eliminate random input noise and achieve feature completion; map the probabilistic weight matrix to the physical surface of the target object, calibrate the weight of visual features on the decision, and generate a heatmap matrix to guide the decision.

[0035] Heuristic Perception Reinforcement Decision Module: It is used to receive the contribution distribution map, use the mask operator to perform geometric constraint pruning on the action space of the policy network, and shield invalid actions that violate geometric logic; at the same time, it performs feature reweighting to guide the agent to perform heuristic exploration in visually salient regions;

[0036] Closed-loop adaptive evolution module: It is used to evaluate perceptual determinism in real time using geometric confidence evaluation operators, dynamically adjust prior guidance weights, and trigger backoff logic when the entropy value is abnormal; at the same time, this module feeds back reinforcement learning reward signals to the probabilistic graphical model, realizing the online fusion of visual prior logic and reinforcement learning exploration experience.

[0037] Compared with the prior art, the technical solution disclosed in this invention has the following beneficial effects:

[0038] 1. This invention constructs a surface prior feature space and encodes a prior probability map, defines surface feature points as probability map nodes and establishes conditional probability factors, and realizes the inference of the surface direction of the missing region based on the observed features when there is occlusion or data distortion, thus completing the logical completion of the feature level.

[0039] 2. This invention infers the probability distribution of the contribution of each region of the surface to the task through the confidence propagation algorithm, and by embedding the probability distribution into the reinforcement learning feature encoding layer, it uses the attention mechanism to suppress redundant data, prunes the action space, and labels the weight of visual features on decision-making, thereby improving the training convergence speed to several times that of traditional RL algorithms.

[0040] 3. This invention encodes the continuity constraints of differential geometric manifolds into conditional probability factors of probabilistic graphs, thereby transforming human cognitive logic of the physical geometry of curved surfaces into perceptual constraints that can be executed by robots. This ensures that each perceptual action is supported by geometric logic and no longer relies on black-box end-to-end learning.

[0041] 4. This invention establishes a feedback loop driven by geometric confidence to evaluate perception accuracy in real time, automatically enhances prior guidance when confidence decreases, and optimizes probabilistic graph parameters online using reinforcement learning reward signals. This enables cross-object adaptation without retraining by loading different geometric prior models, absorbing experience from new target objects while suppressing catastrophic forgetting. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This invention provides a reinforcement learning environment perception method based on visual prior probability graphs. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] To make the above-mentioned objects, features and advantages of the present invention more readily understood, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] This invention provides a reinforcement learning-based environment perception method and system guided by visual prior probabilistic graphs. By constructing a five-layer architecture comprising multidimensional surface feature extraction, prior probabilistic graph visual intuition encoding, visual contribution probability inference, probability-guided heuristic policy learning, and geometric confidence-driven feedback evolution, the geometrically aligned prior acquired by industrial sensing terminals serves as the basis for logical reasoning. Combined with the confidence propagation mechanism of the probabilistic graphical model and the autonomous exploration capability of the reinforcement learning agent, a highly convergent, geometrically robust, and logic-driven environment perception architecture is constructed. This invention enables the agent to effectively supplement feature gaps in complex surfaces (such as alternating surfaces and narrow cavities) through logical guidance from prior probabilistic graphs, and to achieve heuristic search and computational resource optimization of high-dimensional surface data through probabilistic pruning mechanisms, thereby achieving accurate perception of the edges and curvature abrupt changes of complex workpieces.

[0047] like Figure 1 As shown, this embodiment of the invention provides a reinforcement learning environment perception method guided by a visual prior probability graph, specifically including the following steps:

[0048] Step S1, Multidimensional surface feature extraction and geometric prior space construction:

[0049] In this embodiment, an adaptive sampling mechanism is introduced using an industrial sensing terminal (such as a high-precision line laser or binocular structured light camera). For large curvature edges or narrow deep groove regions in complex workpieces, the point cloud sampling density is automatically increased and multi-scale geometric features are calculated to ensure that the generated feature space can capture microscopic topological changes, thereby obtaining the original point cloud of the complex workpiece surface. .

[0050] In this embodiment, to characterize the surface topology, a differential geometric operator is introduced to extract local manifold features, including the following two steps:

[0051] 1. Geometric feature tensor calculation: For any sampling point on the surface... Calculate its local covariance matrix C and solve for its eigenvalues. This defines the surface variation rate at that point. Used to quantitatively describe curvature abrupt changes:

[0052]

[0053] 2. Multimodal Alignment Representation: Constructing Feature Vectors ,in RGB information The unit normal vector is used. By mapping to a high-dimensional feature sequence through linear projection, a unified vectorized modeling of geometric topology and visual semantics is achieved.

[0054] Step S2, prior probability graphical encoding based on complex surface topology:

[0055] A visual intuitive encoder based on the Prior Probabilistic Graph Model (PPGM) is constructed. Feature mutation points, surface boundaries, and critical paths on the workpiece surface are defined as random variable nodes. Conditional probability factors between nodes are established based on the geometric continuity constraints of the surface and the physical processing logic, forming an inference network that can represent the topological logic of complex surfaces.

[0056] In this embodiment, the prior probability graph consists of a set of nodes. and factor set Composition. For each pair of adjacent surface feature nodes, its underlying logic follows the joint probability distribution:

[0057] (2)

[0058] in:

[0059] These represent key points, edge lines, and robot trajectory motion nodes on the curved surface.

[0060] It represents the joint probability of all variable nodes in the system;

[0061] Representative factor set Elements in;

[0062] The potential function is used to measure the degree to which a combination of variable nodes satisfies the constraints.

[0063] Z represents the normalization function;

[0064] Representative and Factor , neighboring nodes;

[0065] To achieve an effective representation of the complex surface topology by the aforementioned joint probability distribution, this embodiment uses the potential function in equation (2). Further specified as geometric constraint factors By introducing a potential function based on an exponential mapping, neighboring nodes are constrained. and Normal smoothness:

[0066]

[0067] in:

[0068] This represents the smoothness sensitivity coefficient, used to adjust the model's tolerance to drastic changes in surface curvature;

[0069] It represents the dot product of adjacent normal vectors.

[0070] S3. Inference of the probability distribution of contribution of surface operations:

[0071] The Belief Propagation (BP) algorithm is employed to solve the global consistency constraint of the prior probabilistic graphical model, transforming dynamically acquired local surface topography features into a posterior state distribution contributing to the task. This process eliminates noise interference in visual perception by iteratively message-passing through surface topology nodes, calculating a probabilistic weight matrix covering the entire workpiece surface. Through the mapping relationship between this matrix and the workpiece's physical surface, the system autonomously calibrates key geometric singularities and feature-change points in complex surfaces, guiding the system to focus its perceptual computing power on high-contribution regions. Specifically:

[0072] 1. Message Passing and State Inference: Iterative message passing is performed during this process to aggregate global topology constraints. Nodes To its neighboring factors New messages sent It has the following forms:

[0073] (4)

[0074] in:

[0075] Representative node Reliability of geometric state estimation;

[0076] Remove node i Other than the nearest neighbor set;

[0077] Representative factor The confidence message fed back to node i;

[0078] This mechanism eliminates local noise introduced by the visual sensor through repeated iterations on the graph structure, and achieves full logical completion of surface features.

[0079] 2. Contribution weight generation driven by task objectives: This involves obtaining the marginal posterior probability of each node. Then, the objective cost function of the task is introduced. This transforms pure geometric probabilities into task-related weights, generating a contribution heatmap matrix. :

[0080]

[0081] in:

[0082] Represents the weight matrix The first in The salience weight of each element in the current task;

[0083] Represents the marginal posterior probability of node i;

[0084] The cost function represents the objective of the task, where For the positional deviation of the operation

[0085] Represents the gradient operator, used for quantizing nodes. Sensitivity to the impact of state changes on the final operation error;

[0086] S4 probability-guided selective enhancement of surface features:

[0087] A G-DRL framework is constructed, embedding the contribution probability distribution output by S3 into the agent's visual feature encoding layer. A probability-guided selection mechanism is used to suppress redundant data in complex surfaces, forcing the model to focus on high curvature changes and overlapping regions. Based on this, the action space of reinforcement learning is geometrically pruned to achieve heuristic exploration. In this embodiment, the probability weights described in S3 are... The decision-making chain deeply embedded in the intelligent agent enables a heuristic mapping from raw data to high-quality action instructions, which is specifically divided into two steps:

[0088] 1. Use the Hadamard Product on the original features. Perform dynamic perception weighting:

[0089]

[0090] in:

[0091] Representing the intelligent agent in The logical enhancement features received at any given time;

[0092] The Hadamard product represents the element-wise multiplication of two tensors.

[0093] Represents the temperature coefficient, used to adjust the smoothness of the probability distribution;

[0094] Represents the normalized exponential function;

[0095] Equation (6) forces the policy network to focus its limited computing resources on highly contributing surface abrupt changes or overlapping regions through the "spatial filtering" effect of probability weights, which can effectively solve the problem of feature extraction ambiguity caused by data redundancy in complex surface environments.

[0096] 2. Geometric logic-driven action space pruning: This involves performing sampling actions on the agent. At that time, a mask operator based on prior probabilistic graphical logic is introduced. Hard-constrained filtering is applied to the original output distribution of the policy network:

[0097]

[0098] in:

[0099] The action probability distribution represented by the parameter policy network output to be optimized;

[0100] This represents an indicator function that takes the value 1 if the condition within the parentheses is met, and 0 otherwise.

[0101] Represents a geometrical feasibility mask generated based on the contribution matrix;

[0102] This represents the pruning threshold, used to filter out invalid action spaces that contribute too little to the target or violate geometric logic;

[0103] Equation (7) ensures that the agent always performs heuristic exploration within a physically feasible and geometrically reasonable subspace, which greatly reduces the search space and suppresses random and ineffective trial and error in reinforcement learning in complex industrial scenarios.

[0104] S5 Geometric Confidence-Driven Perceptual Feedback Closed Loop:

[0105] This embodiment establishes a model evolution mechanism based on geometric consistency. The matching accuracy between the currently perceived surface model and the ideal prior model is evaluated in real time. When faced with extreme irregular regions that cause a decrease in perception confidence, the logical guidance of the prior probability map is automatically enhanced. Geometric logic is used to "predict" the surface trend of occluded or distorted regions, and the probability map factors are optimized online using the reward signal from reinforcement learning, achieving adaptive evolution for the perception of complex surfaces. Specifically, this involves the following two steps:

[0106] 1. Confidence Evaluation of Multidimensional Perception: To quantify the system's understanding of the current complex workpiece morphology, a perception confidence index is defined. This indicator is comprehensively evaluated by calculating the information entropy of the marginal distribution of the probability graph.

[0107]

[0108] in:

[0109] This represents the overall perception confidence index, used to measure the certainty of the system's understanding of the current environment;

[0110] It represents information entropy, which quantifies the uncertainty of probability distribution;

[0111] 2. Online Policy Update: Introducing KL divergence as a constraint term, and adjusting prior probability graphical parameters. In this process, ensure a smooth transition between new knowledge and old logic:

[0112]

[0113] in:

[0114] This represents the loss of rewards for reinforcement learning tasks;

[0115] This represents the balance coefficient, used to adjust the weight between the objective benefit of the task and the stability of the strategy;

[0116] () represents KL divergence, which is used to quantitatively measure the logistic deviation distance between the old and new probabilities;

[0117] Equation (9) ensures that the system does not disrupt existing basic physical knowledge and motion logic when learning the geometric characteristics of new workpieces, thus effectively suppressing the "catastrophic forgetting" phenomenon in deep learning. This closed-loop architecture enables the perception system to continuously fine-tune the conditional factors of the probability map using task feedback, achieving continuous autonomous growth and evolution of its ability to perceive complex surfaces.

[0118] This invention also provides a reinforcement learning environment perception system guided by visual prior probability graphs, used to run the perception method described above. The system includes: a multi-dimensional feature extraction module, a probabilistic graphical encoding module, a contribution inference module, a heuristic decision enhancement module, and a closed-loop adaptive evolution module.

[0119] Multidimensional surface feature extraction module: In this embodiment, the module is responsible for manifold representation mapping of unstructured point clouds, and is specifically divided into two units:

[0120] Geometric topological feature calculation unit: Extracting sampling points using differential geometric operators. Local surface change rate This unit constructs the local covariance matrix and performs eigenvalue decomposition. The calculated Used to quantify the singularity of surface topology. This unit provides the system with a geometrically invariant characteristic basis against sensor noise.

[0121] Multimodal manifold embedding unit: responsible for constructing feature vectors Furthermore, visual semantics and geometric manifolds are mapped to a unified high-dimensional tensor space through linear embedding operators. .

[0122] Prior probability graphical encoding module: In this embodiment, the module is used to parameterize the human physical intuition about surface continuity and operational logic. This module specifically includes the following two units:

[0123] Probabilistic graph topology building unit: Establishes a set of variable nodes based on the task objective. With factor set The probabilistic graphical structure. This unit models the perception task as a problem of solving a globally joint probabilistic distributed (2) problem.

[0124] Geometric continuity constraint element: configuring a potential function based on normal smoothness. This unit constrains the topological trend of adjacent regions through equation (3), then senses the data missing regions, and uses probability weights to infer the geometric orientation that conforms to physical common sense.

[0125] Task-Oriented Contribution Inference Module: In this embodiment, the module is responsible for performing confidence propagation and identifying key task features. This module includes the following two units:

[0126] BP Iteration Unit: This unit performs iteration based on message passing mechanism using equation (4) on the probabilistic graph. It eliminates local random input noise by aggregating global topological constraints and outputs the marginal posterior probability of each node's state. .

[0127] Contribution mapping generation unit: used to generate heatmap matrix This unit uses equation (5) to couple the nodes with the gradient flow of the job cost function, thus clearly identifying the key singular regions that contribute the most to the job accuracy.

[0128] Heuristic Perception-Enhanced Decision-Making Module: In this embodiment, the module is responsible for embedding logical weights into the agent's decision-making chain to achieve end-to-end mapping between perception and decision-making. This module comprises the following two units:

[0129] Attention feature enhancement unit: The input features are dynamically weighted using Equation (6), which forces the policy network to focus the computation operators on highly saliency regions such as high curvature or feature overlap.

[0130] Action space mask pruning unit: Configure mask operators based on indicator functions The unit filters the action space according to equation (7). By shielding actions that violate set logic, it ensures that the agent clock performs heuristic exploration within a physically reasonable manifold.

[0131] Closed-loop adaptive evolution module: In this embodiment, the module is responsible for constructing a closed loop for the self-evaluation and knowledge growth of the perception system. This module specifically includes the following two units:

[0132] Uncertainty monitoring and evaluation unit: Based on the information entropy theory of formula (8), the perceived confidence index is calculated to evaluate the certainty of the system's understanding of the current environment in real time. When the entropy value exceeds the safety threshold, the system's strategy fallback logic is automatically triggered.

[0133] Knowledge transfer strategy update unit: Introducing KL divergence as a regularization constraint, the parameters are corrected through the objective function of optimization formula (9). While absorbing the perception experience of new workpieces, the balance coefficient is used. Limit the deviation between old and new logic and suppress catastrophic forgetting in the perceptual system.

[0134] Furthermore, the multimodal manifold embedding unit introduces a local neighborhood weighting mechanism based on geodesic distance when performing feature aggregation. The system defines the neighborhood category based on the connectivity of the point cloud surface manifold rather than Euclidean distance, ensuring that when processing complex surfaces with thin-walled structures or deep cavity joints, the extracted feature vectors can accurately reflect the physical reality of the topological structure and avoid feature contamination across space.

[0135] Furthermore, in the prior probability graphical encoding module, the system introduces a non-uniform anisotropic potential field mechanism. The potential function instantiation unit dynamically adjusts the smoothness coefficient based on the local Gaussian curvature of the sampling points. Increase in the overlapping region where curvature fluctuations are severe. Weights are used to capture fine textures and are reduced in flat areas. To enhance global smoothness, thereby enabling adaptive allocation of perceptual resolution in heterogeneous regions of the surface.

[0136] Furthermore, in the task-oriented contribution inference module, the contribution mapping generation unit introduces an anisotropic sensitivity evaluation mechanism. The system uses the Jacobian matrix J to calculate the motion operability of the robot's end effector in different degrees of freedom directions in Cartesian space, and projects the pose deviation e to the direction with the highest sensitivity. Specifically, the gradient operator... Based on the distribution of the major and minor axes of the maneuverability ellipsoid, the contribution heatmap W is asymmetrically corrected: in directions where movement is restricted, the contribution weight of the surface features in that region is automatically increased. This mechanism ensures that the contribution matrix not only reflects the geometric characteristics of the surface but also deeply integrates the robot's own physical motion capabilities, achieving collaborative calibration of "perception and mechanism".

[0137] Furthermore, in the heuristic perception-enhanced decision-making module, the action space mask pruning unit supports a temporal continuity feedforward masking strategy. This strategy monitors the smoothness of the decision sequence in joint space by calculating the action envelope over k consecutive sampling times. When the mask operator M determines that the current output action may cause a sudden change in joint torque or approach the physical limit, the pruning threshold is adjusted. The action sampling range of the reinforcement learning agent is dynamically limited, forcing the decision distribution to shift towards a subspace that satisfies the continuity of higher-order derivatives. This mechanism, combined with a state reshaping unit, effectively solves the end-point jitter and trajectory overshoot problems that exist in traditional reinforcement learning decision-making when dealing with complex irregular curved surfaces.

[0138] Furthermore, in the closed-loop adaptive evolution module, the incremental strategy optimization unit introduces an uncertainty-weighted update mechanism based on risk perception. The system updates the update based on the confidence index output by the uncertainty monitoring and evaluation unit. Dynamically adjust the penalty coefficient of the KL divergence constraint term. When entering a completely new work environment leads to... During periods of drastic fluctuation, the system automatically increases... It also locks the core physical logic parameters, allowing the system to perform exploratory updates only in specific areas; as the confidence level stabilizes with job iterations, it is gradually reduced. This unlocks the learning potential. This "conservative first, then progressive" parameter evolution strategy ensures that the system can achieve autonomous and safe growth in perception accuracy and decision-making efficiency while preserving the stability of the underlying motion logic when dealing with complex task flows of varying topology and size.

[0139] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0140] The block diagrams of devices, apparatuses, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0141] It should also be noted that in the apparatus, device, and method of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of the present invention.

[0142] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0143] It should be understood that the qualifying terms "first", "second", "third", "fourth", "fifth" and "sixth" used in the description of the embodiments of the present invention are only used to more clearly illustrate the technical solutions and are not intended to limit the scope of protection of the present invention.

[0144] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A reinforcement learning-based environment perception method guided by visual prior probability graphs, characterized in that, include: Step S1: Use industrial sensing terminals to acquire multimodal visual information of the target object to be operated, identify the visual semantic key points and geometric topological features of the target object through multi-scale feature extraction operators, and construct a visual prior feature space. Step S2: Using a priori probabilistic graphical model, the visual semantic key points and geometric topological features of the target object described in step S1 are defined as logical nodes. Conditional probability factors between nodes are established based on the physical constraints of the target object and the physical logic of the working environment, forming a logical reasoning network that can characterize the geometric topological features of the target object. Step S3: Using the confidence propagation algorithm, the prior probability graphical model established in step S2 is solved globally for consistency, and the probability distribution of the contribution of each visual region of the target object to the current task target is calculated. Step S4: Embed the contribution probability distribution described in step S3 into the reinforcement learning state space, use a probability-guided selective mechanism to suppress redundant data in the policy network, and perform geometric constraint pruning on the action space to guide the agent to perform heuristic exploration in visually salient regions. Step S5: Establish a feedback loop based on visual confidence, and adjust the parameters in the probabilistic graphical model in real time according to the visual feedback value after the strategy is executed, so as to realize the online fusion of visual prior logic and reinforcement learning exploration experience.

2. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, The step S1, which involves identifying the target object using a multi-scale feature extraction operator, includes: Step S1.1: Extract the visual semantic key point heatmap of the target object through a deep residual network; Step S1.2: Extract the surface curvature distribution and normal pointing tensor using point cloud operators.

3. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, The step S1, which involves using an industrial sensing terminal to acquire multimodal visual information of the target object to be worked on, includes: the industrial sensing terminal introducing an adaptive sampling mechanism to automatically increase the point cloud sampling density and calculate multi-scale geometric features for large curvature edges or narrow deep groove areas in complex workpieces, ensuring that the generated feature space can capture microscopic topological changes and provide high-fidelity surface detail support for subsequent probabilistic inference.

4. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, The conditional probability factor mentioned in step S2 defines the coupling strength between random variable nodes by pre-setting the geometric relationship between key points and the physical logic of the work environment, thereby transforming human work intuition into calculable prior probability constraints.

5. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, The prior probability graphical model described in step S2 adopts a factor graph architecture. Its conditional probability factors not only utilize the prior space construction unit to perform tensor mapping, but also configure potential field constraints based on normal smoothness. In the region of perceived data distortion or missing data, probability weights are used to infer the geometric orientation that conforms to physical common sense, thereby realizing logical completion at the feature level.

6. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, Step S3, which involves calculating the probability distribution of the contribution of each visual region of the target object to the current task target, includes the following steps: Step S3.1: The confidence propagation algorithm is used to solve the global consistency constraint of the prior probability graphical model in step S2, and the local geometric features of the target object acquired in real time are transformed into the state posterior distribution of the contribution to the task. Step S3.2: Iteratively optimize the geometric topological feature nodes of the target object through the confidence propagation algorithm to eliminate noise interference in the visual perception of the industrial sensing terminal, and calculate the probabilistic weight matrix covering the entire surface of the target object. Step S3.3: Based on the mapping relationship between the probabilistic weight matrix and the physical surface of the target object, the weight of visual features on the decision is calibrated, and a heat map matrix for guiding decision-making with the same dimensions as the original visual image is output.

7. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, In step S3, the confidence propagation algorithm introduces a task-oriented potential function when performing probability inference. This potential function quantifies the sensitivity of the geometric feature normal deviation of the target object to the final task environment, transforming ordinary geometric features into significant weights with task value, and accurately extracting key geometric feature points of the task environment from the global point cloud.

8. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, The selective mechanism described in step S4 includes: Step S4.1: Multiply the state input vector of reinforcement learning with the probability distribution map generated in step S3 element-wise to filter out redundant visual information. Step S4.2: Weighted sampling of the action output of the policy network is performed according to the probability distribution to filter out low-probability actions that violate physical logic and improve the sample utilization efficiency of reinforcement learning.

9. The reinforcement learning environment perception method based on visual prior probability graphs as described in claim 1, characterized in that, The closed-loop feedback in step S5 uses visual reconstruction error or discriminator score to evaluate the confidence level of the current perception state. When the visual confidence level is lower than a preset threshold, the weight of the prior probability map in decision guidance is automatically increased to compensate for feature loss due to environmental interference.

10. A reinforcement learning environment perception system guided by visual prior probability graphs, characterized in that, include: Multi-dimensional surface feature extraction module: It is used to perform intrinsic parameter mapping between high-precision point clouds collected by multi-source industrial sensing terminals and multi-scale textures by integrating spatial coordinate alignment units, and multi-scale feature extraction operators to identify visual semantic key points and geometric topological features of target objects, and construct visual prior feature space. Prior probability graph encoding module: used to build a probability graph structure containing variable nodes and factor sets according to the task objective; to perform tensor mapping using prior space construction units and configure potential field constraints based on normal smoothness; to infer geometric orientation that conforms to physical common sense in the region of perceived data distortion or missing data using probability weights, providing the system with consistent spatial pose prior; Task-oriented contribution inference module: used to integrate the confidence propagation inference operator to perform message passing iteration on the probabilistic graph, aggregate global topological constraints to eliminate random input noise and achieve feature completion; map the probabilistic weight matrix to the physical surface of the target object, calibrate the weight of visual features on the decision, and generate a heatmap matrix to guide the decision. Heuristic Perception Reinforcement Decision Module: It is used to receive the contribution distribution map, use the mask operator to perform geometric constraint pruning on the action space of the policy network, and shield invalid actions that violate geometric logic; at the same time, it performs feature reweighting to guide the agent to perform heuristic exploration in visually salient regions; Closed-loop adaptive evolution module: It is used to evaluate perceptual determinism in real time using geometric confidence evaluation operators, dynamically adjust prior guidance weights, and trigger backoff logic when the entropy value is abnormal; at the same time, this module feeds back reinforcement learning reward signals to the probabilistic graphical model, realizing the online fusion of visual prior logic and reinforcement learning exploration experience.

Images