A behavior prediction method, apparatus, device, and medium

Abstract

The application discloses a behavior prediction method, system, device and medium, the method comprises the following steps: based on the object geometric model library, the physical property database and the dynamics simulation prior of the typical operation scene, a symbolic reasoning prior knowledge base is constructed; the continuous behavior data of the sample action object in the preset time window is extracted, and the continuous behavior data is associated with the symbolic reasoning prior knowledge base for associated retrieval, matched associated physical prior information is obtained and embedded in the continuous behavior data, and an enhanced behavior sequence is generated; the enhanced behavior sequence is input into a multimodal large model, the multimodal large model is guided to reason by a prompt template containing step-by-step reasoning guide instructions, and a generative labeling result is output; a labeling data set composed of a plurality of generative labeling results is used to train a behavior prediction model; real-time behavior data of an action object to be predicted is input into the trained behavior prediction model, and an abnormal operation risk assessment report for the action object to be predicted in an unstructured environment is output.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and behavior analysis technology, and in particular to a behavior prediction method, system, device and medium. Background Technology

[0002] Behavior prediction is one of the core technologies in the fields of artificial intelligence and behavior analysis. Its core task is to logically reason and generate future action sequences based on the historical behavioral trajectories of specific objects (such as intelligent agents and robots). As an important direction of robotics research, behavior prediction has wide application value in fields such as intelligent prosthetics, industrial assembly, service robots, and special operations. In recent years, with the evolution of multimodal large models and deep learning technologies, behavior prediction has gradually shifted from traditional inverse behavior analysis and trajectory interpolation methods to understanding complex operational intentions and modeling the semantics of physical interactions, greatly promoting the research progress of adaptive operation technology in unstructured environments in the field of robotics.

[0003] Currently, behavior prediction technology mainly employs deep neural network architectures, learning from massive amounts of behavioral data to probabilistically assess future joint angles or contact force sequences. Common prediction methods include temporal modeling methods based on recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and Transformer architectures. These methods have achieved certain results in handling nonlinear behavioral features and long-term dependencies. Especially in the field of robot grasping and manipulation, existing technologies attempt to construct detection and early warning models for abnormal operations such as grasping slippage and collisions by extracting visual features, tactile features, and joint proprioceptive features.

[0004] Despite the progress made by existing technologies, significant challenges remain in practical applications, especially in predicting behavior in dynamic and unstructured environments. First, existing models heavily rely on extracting surface behavioral features, lacking an understanding of deeper operational intentions. In warnings of abnormal operations such as grasping slippage and loss of contact control, unstructured disturbances such as slippery object surfaces, abrupt shape changes, or accidental collisions often make it difficult for traditional models to accurately predict sudden changes in contact states. Second, observational isolation and the lack of physical context limit prediction accuracy. Most existing prediction methods are based on visual observations at the current moment or within a short window, ignoring the physical properties of objects (mass, coefficient of friction, geometric constraints) and dynamic changes in the environment during the operation. This leads to problems such as jittery movements, high slippage rates, and poor generalization ability when facing unfamiliar objects or sudden disturbances.

[0005] The core of behavior prediction lies in predicting the future behavioral trends of a robotic arm through analysis of specific physical interaction environments and multi-dimensional sensor features. While this process shares similarities with traditional robot trajectory planning, there are significant differences in their fundamental nature: trajectory planning focuses primarily on solving the geometric path given a target pose, rather than predicting the actions of a moving object in a non-structured environment. Relying solely on single visual observations or historical joint angles often fails to provide accurate judgments. For example, a moving object may exhibit stable gripping force control when grasping a conventional rigid object, but its operational logic may rapidly deteriorate into slipping or loss of control when grasping objects with slippery surfaces or uneven mass distributions. This change in intent due to abrupt changes in the physical environment cannot be captured by simply analyzing static behavioral features. This indicates that compared to general behavior prediction tasks, behavior prediction in the realm of moving object manipulation places extremely high demands on the model's ability to analyze physical semantic relationships and model deep operational intent. It requires not only identifying explicit features in the behavioral trajectory representing contact instability or force anomalies but also a deep understanding of the physical logic and environmental catalytic factors behind the operation. However, existing technologies often struggle to construct effective nonlinear prediction functions when dealing with complex feature mappings in such high-dimensional spaces, resulting in unsatisfactory prediction accuracy for covert contact risks (such as early signals of minute slippage).

[0006] In practical applications, behavior prediction technologies for object manipulation still face significant challenges due to strong adversarial nature and data quality bottlenecks. Taking the grasping of unknown flexible or fragile objects as an example, non-ideal conditions such as slippery surfaces, elastic deformation, or abrupt changes in mass distribution resemble "adversarial tactics" in a network environment. Existing models, lacking generalization ability and robustness to physical disturbances, struggle to penetrate these "physically packaged" surface features to perceive true contact stability. Simultaneously, the scarcity of high-quality unstructured operation annotation data is a key factor restricting technological development. In particular, the difficulty in collecting early instability samples with leading characteristics (such as force feature changes before minute slippage) leads to a widespread problem of extreme imbalance in the distribution of positive and negative samples in existing datasets. This makes models prone to overfitting massive amounts of stable grasping behavior during training, while remaining insensitive to highly concealed pre-slippage and pre-collision states. Furthermore, the complexity of the physical interaction environment, the diversity of objects, and the mixing of real and false sensor noise make it extremely difficult to build a behavior prediction benchmark that can cover the entire operation path. The complex phenomena in the data, such as tactile noise, visual occlusion, and non-ideal contact, further exacerbate the difficulty of model parameter correction and operation risk assessment. In addition, the millisecond-level response pressure brought by real-time control means that existing technical solutions still have obvious technical shortcomings in ensuring prediction accuracy, physical interpretability, and response timeliness.

[0007] To address these issues, researchers have attempted to introduce object knowledge graphs and physical simulation information to assist in action decision-making. However, current knowledge integration methods are mostly static and struggle to handle real-time changing physical interaction scenarios. Furthermore, there are still technical bottlenecks in deeply integrating massive amounts of structured physical prior knowledge with the logical reasoning capabilities of large-scale pre-trained models, making it difficult to achieve accurate early warning of covert contact risks in unstructured environments. Summary of the Invention

[0008] This invention addresses the problems of existing behavior prediction technologies in unstructured environments, such as lack of physical context, insufficient understanding of deep operational intentions, and difficulty in integrating physical priors with large model reasoning capabilities. It provides a behavior prediction method, device, and storage medium based on symbolic reasoning priors.

[0009] Specifically, this invention integrates a library of object geometric models, a database of physical properties, and prior dynamic simulations of typical operational scenarios to semantically enhance and physically complete the historical behavioral trajectories of objects. Based on this, a multimodal large model is used to perform in-depth deduction and generative prediction of the actions to be predicted, incorporating prior physical knowledge. This allows the prediction model to overcome the limitations of single behavioral features or visual observations, combining real-time physical feedback and contact mechanics logic to gain a deeper understanding and more accurate prediction of whether objects exhibit abnormal behaviors that affect operational stability, such as slippage, collisions, or loss of contact control. Ultimately, this achieves efficient identification and early warning of operational risks in unstructured environments.

[0010] To achieve the above objectives, a first aspect of the present invention provides a behavior prediction method, comprising the following steps:

[0011] S1: Construct a multi-dimensional symbolic reasoning prior knowledge base based on the object geometric model library, physical property database, and dynamic simulation priors of typical operation scenarios;

[0012] S2: Extract continuous behavior data of the sample action object within a preset time window, and perform association retrieval with the symbolic reasoning prior knowledge base to obtain matching associated physical prior information. Embed the associated physical prior information into the continuous behavior data to generate an enhanced behavior sequence.

[0013] S3: Input the enhanced behavior sequence into the multimodal large model, guide the multimodal large model to perform reasoning through the prompt template containing step-by-step reasoning guidance instructions, and generate generative annotation results containing operation intention evaluation, contact stability score and instability risk level;

[0014] S4: Train the behavior prediction model using the labeled dataset composed of multiple generative annotation results;

[0015] S5: Input the real-time behavior data of the target to be predicted into the trained behavior prediction model, and output an abnormal operation risk assessment report for the target to be predicted in an unstructured environment.

[0016] Furthermore, step S1 also includes:

[0017] S11: Construct the object geometric model library, obtain a set of three-dimensional geometric models of various operation target objects, and use a 3D feature extractor to map the set of three-dimensional geometric models to a high-dimensional vector space to generate geometric feature vectors;

[0018] S12: Construct the physical attribute database, obtain the physical attribute set of various operation target objects, perform weighted mapping and hash sequence generation on the physical attribute set, obtain binary fingerprints through binarization and construct hash index;

[0019] S13: Construct a dynamic simulation prior for the typical operation scenario, simulate the typical operation scenario through a physics engine, and extract the action-contact mapping set; use a time-series state graph to store the operation evolution path of the action-contact mapping set, define operation state nodes and quantify contact stability; use the action-contact mapping set, the operation state nodes and the contact stability quantification results as abstract physical state information and convert them into natural language descriptors.

[0020] Furthermore, step S2 also includes:

[0021] S21: Perform behavioral feature extraction and state segmentation on the continuous behavioral data, and use bidirectional temporal coding technology to extract the behavioral patterns, contact states and force features of each behavioral node;

[0022] S22: The extracted features are vector-weighted and merged to generate the physical fingerprint of the behavior sequence to be tested. Hash collision retrieval is performed in the symbolic reasoning prior knowledge base to calculate the Hamming distance. If the Hamming distance meets the preset threshold, the corresponding associated physical prior information is hit and the associated physical prior information is converted into explicit physical constraint labels.

[0023] S23: Using the explicit physical constraint label and the continuous behavior data as input, calculate the similarity between the global behavior semantic vector and each operation state node in the symbolic reasoning prior knowledge base, select the operation state nodes that meet the similarity conditions, and synthesize the physical state information in the selected operation state nodes with the explicit physical constraint label to generate the enhanced behavior sequence.

[0024] Furthermore, step S3 also includes:

[0025] S31: Construct the prompt template, which includes character settings, task description, physical knowledge background, and the step-by-step reasoning guidance instructions;

[0026] S32: Fill the enhanced behavior sequence into the prompt template to guide the multimodal large model to perform reasoning;

[0027] S33: The multimodal large model outputs the generative annotation results, which also include inference path descriptions and predicted future action evolution probabilities.

[0028] Furthermore, step S4 also includes:

[0029] S41: The original behavioral features in the enhanced behavioral sequence, the features corresponding to the associated physical prior information, and the dynamic environment features corresponding to the symbolic reasoning prior knowledge base are vectorized and aligned, and the nonlinear integration of multiple features is achieved using a gated fusion unit;

[0030] S42: Construct a multi-layer bidirectional Transformer encoder architecture, introduce relative position encoding, and extract behavioral trajectory features through multi-layer self-attention blocks;

[0031] S43: Using the generative annotation results as soft labels, the parameters of the behavior prediction model are fine-tuned through knowledge distillation technology.

[0032] Furthermore, step S5 also includes:

[0033] S51: Access the real-time behavior data of the object to be predicted, associate and retrieve the real-time physical feedback and object attribute prior with the symbolic reasoning prior knowledge base, obtain matching associated physical prior information, embed the associated physical prior information, map it to the feature vector space, and generate the feature matrix to be tested.

[0034] S52: Input the feature matrix to be tested into the trained behavior prediction model, use the sliding window mechanism to extract continuous behavior segments, output the probability distribution of instability risk at each time step, and calculate the risk severity index. When the risk severity exceeds the preset mutation threshold, it is identified as an operational state change.

[0035] S53: When the probability of instability exceeds the preset risk threshold, the abnormal operation risk assessment report is automatically generated. The abnormal operation risk assessment report includes action evolution path prediction and control strategy adjustment suggestions.

[0036] In step S12, the physical attribute set is weighted and mapped and hash sequence is generated. After binarization, a binary fingerprint is obtained and a hash index is constructed. Specifically, the physical attribute set is dimension normalized and the weight factor of each attribute dimension is calculated. A sequence is generated by weighted accumulation and then binarized to obtain a binary fingerprint. The binary fingerprint and the associated physical prior information are stored in the hash index table in the form of key-value pairs.

[0037] The operation status node includes joint angle status, contact status quantity, and operation stability score. The contact stability is quantified by the contact force distribution function, and the sum of the weights of each component is 1.

[0038] A second aspect of the present invention provides a behavior prediction device, comprising:

[0039] The symbolic reasoning prior knowledge base construction module is used to build a multi-dimensional symbolic reasoning prior knowledge base based on the object geometric model library, physical property database, and dynamic simulation priors of typical operation scenarios.

[0040] The behavior enhancement module is used to extract continuous behavior data of sample action objects within a preset time window, and to perform association retrieval of the continuous behavior data with the symbolic reasoning prior knowledge base to obtain matching associated physical prior information. The associated physical prior information is then embedded into the continuous behavior data to generate an enhanced behavior sequence.

[0041] The multimodal reasoning module is used to input the enhanced behavior sequence into the multimodal large model, guide the multimodal large model to reason through a prompt template containing step-by-step reasoning guidance instructions, and generate generative annotation results including operation intention evaluation, contact stability score and instability risk level;

[0042] The behavior prediction model training module is used to train the behavior prediction model using a labeled dataset composed of multiple generative annotation results.

[0043] The output module is used to input the real-time behavior data of the target action object into the trained behavior prediction model and output an abnormal operation risk assessment report for the target action object in an unstructured environment.

[0044] A third aspect of the present invention provides an electronic device, comprising: a processor and a memory, the memory being used to store a computer program; when the computer program is executed by the processor, it implements the behavior prediction method as described above.

[0045] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the behavior prediction method described above.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] By dynamically supplementing the symbolic reasoning prior knowledge base, the previously fragmented action object behavior trajectories gain deeper support from the object's physical properties, contact mechanics logic, and operational scenario dynamics. This enables the model to perceive and identify the real instability risks of operational trajectories under specific physical environments, much like a human expert, significantly improving the accuracy of identifying abnormal operations such as slippage and collisions. Secondly, the step-by-step reasoning capability of the multimodal large model makes the prediction process of complex actions more intelligent, effectively addressing the challenges posed by unstructured disturbances (such as slippery surfaces, sudden mass changes, and irregular shapes), and greatly reducing the risk of missed and false alarms in real-time control. Furthermore, this invention deeply integrates the generalization capability of the multimodal large model with the accuracy of the professional symbolic reasoning prior knowledge base, not only improving the real-time performance and physical interpretability of behavior prediction but also providing strong technical support for building a safe and stable intelligent control system for action objects. Attached Figure Description

[0048] Figure 1 A flowchart of the behavior prediction method provided in the first embodiment of the present invention;

[0049] Figure 2 for Figure 1 Flowchart of step S1;

[0050] Figure 3 Another flowchart of the behavior prediction method provided in the first embodiment of the present invention;

[0051] Figure 4 This is a diagram illustrating the architecture of the symbolic reasoning prior knowledge base in the first embodiment of the present invention.

[0052] Figure 5 This is a diagram of the symbolic reasoning prior knowledge embedding logic architecture in the first embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram illustrating the operational risk simulation based on a multimodal large model in the first embodiment of the present invention;

[0054] Figure 7 for Figure 1 Flowchart of step S2;

[0055] Figure 8 for Figure 1 Flowchart of step S3;

[0056] Figure 9 for Figure 1 Flowchart of step S4;

[0057] Figure 10 for Figure 1Flowchart of step S5;

[0058] Figure 11 This is a schematic diagram of the behavior prediction device provided in the second embodiment of the present invention;

[0059] Figure 12 for Figure 11 A schematic diagram of the structure of the symbolic reasoning prior knowledge base construction module;

[0060] Figure 13 for Figure 11 A schematic diagram of the structure of the central bank's enhancement module;

[0061] Figure 14 for Figure 11 A schematic diagram of the structure of the multimodal reasoning module;

[0062] Figure 15 for Figure 11 A schematic diagram of the structure of the training module for the behavioral prediction model;

[0063] Figure 16 for Figure 11 A schematic diagram of the output module;

[0064] In the attached figures, the following labels are used:

[0065] 2-Behavior prediction device;

[0066] 20 - Module for constructing a prior knowledge base for symbolic reasoning;

[0067] 200 - First building block;

[0068] 201 - Second building block;

[0069] 202 - Third building block;

[0070] 21-Behavior Enhancement Module;

[0071] 210 - First Feature Extraction Unit;

[0072] 211 - First search matching unit;

[0073] 212 - Fusion Enhancement Unit;

[0074] 22-Multimodal Reasoning Module;

[0075] 220 - Prompt Template Building Unit;

[0076] 221 - Fill the guide unit;

[0077] 222 - Annotation result generation unit;

[0078] 23-Behavior prediction model training module;

[0079] 230 - Integration Unit;

[0080] 231 - Second Feature Extraction Unit;

[0081] 232-Adjustment unit;

[0082] 24-Output Module;

[0083] 240 - Second search matching unit;

[0084] 241 - Risk Monitoring Unit;

[0085] 242 - Report Generation Unit;

[0086] S1~S5, S11~S13, S21~S23, S31~S33, S41~S43, S51~S53 - Steps. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0088] In its research on unstructured environmental manipulation of objects, this invention discovered that existing motion datasets suffer from severe sensor noise and ambiguity regarding operational intent. During motion pattern analysis and annotation, relying solely on joint angle trajectories or visual features is often insufficient to determine whether an object is about to slip or become uncontrollable. A deep understanding combining the object's physical properties, environmental conditions, and the continuity of historical operations is essential for accurate annotation. Furthermore, in predicting abnormal operations such as grasping slips and collisions, non-ideal conditions like slippery surfaces and sudden mass changes often prevent traditional feature extraction methods from capturing early signs of operational instability. This poses a significant challenge to early warning systems for the safe control of moving objects.

[0089] To address this issue, this invention employs an innovative technical approach: combining the complex logical reasoning capabilities of a multimodal large-scale model with the correlation analysis capabilities of a symbolic reasoning prior knowledge base to automatically extract and pre-label the action features of the action object. Specifically, firstly, a multimodal large-scale model with strong behavioral semantic understanding capabilities is constructed to analyze potential operation patterns and instability tendencies in historical joint angle sequences. Then, a symbolic reasoning prior knowledge base is designed, which integrates in real-time a library of object geometric models, a physical property database (mass, friction coefficient, stiffness), and dynamic simulation priors for typical operation scenarios. This knowledge base provides the model with necessary physical context references, assisting the model in identifying the physical interaction logic and instability triggers behind behavioral trajectories.

[0090] In the model building and sample annotation phase, real-time physical feedback and prior object attributes are embedded into the historical action sequence of the action object through a symbolic reasoning prior knowledge base. This enhanced composite data is then input into the multimodal large model. Based on a carefully designed guidance template, the multimodal large model performs multi-dimensional extrapolation of the future operational risk level of the action object, generating a series of generative annotation results including action trajectories, contact stability scores, and instability probabilities. Subsequently, these prediction results are verified and corrected multiple times using physical simulation validation and expert experience rules from the symbolic reasoning prior knowledge base to ensure the accuracy of the model's learning objectives and the consistency in identifying unstructured disturbances.

[0091] Through this method, the present invention constructs a high-quality behavior prediction framework and training benchmark for actions of objects in unstructured environments. This invention not only improves the accuracy of predicting abnormal operations such as slippage and collisions, but also provides the prediction model with deep physical environment perception and operational intent analysis support, significantly enhancing the model's adaptive performance and generalization ability in complex and ever-changing robot operation scenarios.

[0092] like Figure 1 The diagram shown is a flowchart of a behavior prediction method provided in the first embodiment of the present invention, which includes the following steps:

[0093] S1: Construct a multi-dimensional symbolic reasoning prior knowledge base based on object geometric model library, physical property database and dynamic simulation priors of typical operation scenarios.

[0094] like Figure 2 As shown, step S1 further includes:

[0095] S11: Construct an object geometry model library, obtain a set of three-dimensional geometric models of various target objects, and use a 3D feature extractor to map the set of three-dimensional geometric models to a high-dimensional vector space to generate geometric feature vectors.

[0096] Specifically, such as Figures 3 to 6 As shown, an object geometric model library (or object behavior scene knowledge base) is constructed. First, a set of 3D geometric models of various target objects for manipulation is collected. Secondly, 3D feature extractors based on PointNet or Transformer architectures are used. (representing the feature extraction function) maps the 3D geometric model to a high-dimensional vector space, generating geometric feature vectors. ,in Vector dimension For example, 768 or 1024.

[0097] To address the operational risks arising from subtle differences in geometrically similar shapes, a cosine similarity determination mechanism is introduced to calculate the geometric vector of the object under test. Geometric feature vectors in the object geometric model library Similarity:

[0098] ;

[0099] Will satisfy ( Geometric feature terms (with a geometric similarity threshold) are stored in the vector index engine to enable multi-dimensional retrieval and matching of objects with similar shapes.

[0100] S12: Construct a physical attribute database, obtain the physical attribute set of various operation target objects, perform weighted mapping and hash sequence generation on the physical attribute set, obtain binary fingerprints through binarization and construct hash index.

[0101] In step S12, the physical attribute set is weighted and mapped and hash sequence is generated. The binary fingerprint is obtained by binarization and a hash index is constructed. Specifically, the physical attribute set is dimension normalized and the weight factor of each attribute dimension is calculated. A sequence is generated by weighted accumulation and then binarized to obtain the binary fingerprint. The binary fingerprint and the associated physical prior information are stored in the hash index table in the form of key-value pairs.

[0102] Specifically, such as Figures 3 to 6 As shown, a physical attribute database (or behavior rule base) is constructed. This involves collecting a set of known physical attributes of the target object. ,in (j=1,2,3……m) represents the physical attribute vector of the j-th object, where physical attributes include parameters such as mass, coefficient of friction, stiffness, and damping ratio. A feature hashing algorithm (e.g., SimHash algorithm) is used to analyze the physical attribute vector. To perform fingerprint processing:

[0103] ① Weighted mapping: mapping the object attribute vector Perform dimensional normalization and calculate the dimension of each attribute. Weighting factors (such as mass, coefficient of friction, etc.) ;

[0104] ② Fingerprint generation: Let the hash value of the attribute dimension be... We obtain the sequence by weighting and accumulating each digit of the fingerprint:

[0105] ;

[0106] in, For attribute dimensions The corresponding preset weight system.

[0107] ③ Binarization: Obtain an n-bit binary fingerprint (n is the preset fingerprint length). middle x>0, otherwise it is Binary fingerprints are linked to prior operational risks using key-value pairs. (in, The corresponding prior information about operational risks is stored in a hash index table. During retrieval, the Hamming distance is calculated. To measure text similarity:

[0108] ;

[0109] in, The binary fingerprint of the object under test. This represents the p-th bit of the binary fingerprint of the j-th object.

[0110] like If the physical properties of the object are highly similar to those of known objects in the behavior rule library, then the object can inherit its corresponding prior information on operational risks.

[0111] S13: Construct dynamic simulation priors for typical operation scenarios, simulate typical operation scenarios through a physics engine, and extract action-contact mapping sets; use a time-series state graph to store the operation evolution path of the action-contact mapping sets, define operation state nodes and quantify contact stability; use the action-contact mapping sets, operation state nodes and contact stability quantification results as abstract physical state information and convert them into natural language descriptors.

[0112] The operation status node includes joint angle status, contact status quantity, and operation stability score. The contact stability is quantified by the contact force distribution function, and the sum of the weights of each component is 1.

[0113] Specifically, such as Figures 3 to 6 As shown, a dynamic simulation prior (or behavior pattern library) for typical operation scenarios is constructed. Online simulation and data accumulation of typical operation scenarios are performed using real-time physics engines (such as MuJoCo and Isaac Gym), extracting the action-contact mapping set for the current operation task. The operation evolution path is stored using a timing state diagram, and operation state nodes are defined. ,in This indicates the joint angle state at time t (e.g., approach phase, contact phase, withdrawal phase). It represents the contact state quantities (including the distribution of contact points and the ratio of normal force to tangential force). This represents the operational stability score (between 0 and 1, where 1 indicates complete stability and 0 indicates instability). Contact stability is determined by the contact force distribution function. Quantification:

[0114] ;

[0115] The sum of the weights of each component These represent the contributions of normal force, tangential force, and torque, respectively. The above abstract physical state information (including the action-contact mapping set, operational state nodes, and contact stability quantification results) is then transformed into natural language descriptors. As a global physical context parameter during multimodal large-scale model inference, it provides physical semantic support for identifying the instability-inducing logic of action objects in specific operational scenarios. When constructing the behavior pattern library, a Hidden Markov Model (HMM) is used to perform state matching on the operation evolution path and extract temporal evolution rules (such as state transitions in the approach phase, contact phase, and withdrawal phase).

[0116] S2: Extract continuous behavioral data of the sample action object within a preset time window, and perform association retrieval with the symbolic reasoning prior knowledge base to obtain matching associated physical prior information. Embed the associated physical prior information into the continuous behavioral data to generate an enhanced behavioral sequence.

[0117] like Figure 7 As shown, step S2 further includes:

[0118] S21: Perform behavioral feature extraction and state segmentation on continuous behavioral data, and use bidirectional temporal coding technology to extract the behavioral patterns, contact states and force features of each behavioral node.

[0119] Specifically, such as Figures 3 to 6 As shown, behavioral feature extraction and state segmentation are performed on continuous behavioral data of the action object. Let the original behavioral sequence (i.e., continuous behavioral data) be... Each of them It includes multi-dimensional sensing data such as joint angles, angular velocities, torque feedback, and tactile signals. Bidirectional timing coding techniques (such as Bi-LSTM or Bi-Transformer) are used to extract the behavioral patterns, contact states, and force characteristics of each behavioral node.

[0120] S22: The extracted features are vector-weighted and merged to generate the physical fingerprint of the behavior sequence to be tested. Hash collision retrieval is performed in the symbolic reasoning prior knowledge base to calculate the Hamming distance. If the Hamming distance meets the preset threshold, the corresponding associated physical prior information is hit and the associated physical prior information is converted into explicit physical constraint labels.

[0121] Specifically, firstly, the SimHash value is calculated for the original behavior sequence X to generate a behavior data fingerprint. Then, the extracted features (such as behavior patterns, contact states, and forces) are vector-weighted and merged to generate a binary fingerprint of the behavior sequence to be tested. Perform hash collision retrieval in the physical attribute database of the symbolic reasoning prior knowledge base, and calculate its physical attribute fingerprint with that of known objects in the physical attribute database. Hamming distance If the following conditions are met:

[0122] ;

[0123] This involves identifying the prior operational risk of the object (i.e., associated physical prior information). This prior is then transformed into an explicit physical constraint label. (For example, "The surface friction coefficient of this object is low, and there is a risk of slipping"), which is embedded as a hard constraint in the behavioral context to be analyzed.

[0124] S23: Using explicit physical constraint labels and continuous behavioral data as input, calculate the similarity between the global behavioral semantic vector and each operational state node in the symbolic reasoning prior knowledge base. Select operational state nodes that meet the similarity conditions, and synthesize the physical state information in the selected operational state nodes with explicit physical constraint labels to generate an enhanced behavioral sequence.

[0125] Calculate the global behavioral semantic vector of the original behavioral sequence X and the operational state nodes in the dynamic simulation prior of the symbolic reasoning prior knowledge base. The similarity is calculated using a Hidden Markov Model (HMM) for state matching, calculating the similarity of contact force distributions, and selecting the operational state node with the highest matching degree. The Top-K (first K) operational state nodes with the highest similarity are selected, and text is synthesized based on a pre-defined multi-dimensional template. The synthesis template Temp is defined as:

[0126] ;

[0127] For example, physical constraint labels can be linked to operational phase states through feature concatenation. Contact stability distribution function The samples are fused to generate an enhanced composite behavior sample sequence T (i.e., an enhanced behavior sequence). This composite behavior sample sequence not only contains the original behavior trajectory, but also explicitly includes its corresponding object physical properties and dynamic environment priors, providing complete feature support for subsequent inference.

[0128] The process described above of obtaining associated physical prior information through association retrieval and embedding it into continuous behavioral data is collectively referred to as the symbolic reasoning prior embedding algorithm.

[0129] S3: Input the enhanced behavior sequence into the multimodal large model, guide the multimodal large model to perform reasoning through the prompt template containing step-by-step reasoning guidance instructions, and generate generative labeled results including operation intention evaluation, contact stability score and instability risk level.

[0130] like Figure 8 As shown, step S3 includes:

[0131] S31: Construct a prompt template, which includes character settings, task description, physical knowledge background, and the step-by-step reasoning guidance instructions.

[0132] S32: Fill the cue template with the enhanced behavior sequence to guide the multimodal large model in reasoning.

[0133] S33: The multimodal large model outputs generative annotation results, which also include inference path descriptions and predicted future action evolution probabilities.

[0134] In other words, as shown in the figure, a prompt template is constructed that includes character settings, task descriptions, physical knowledge background, and thought process guidance.

[0135] The composite behavior sample sequence T is filled into the prompt template to guide the multimodal large model to output structured labeled data containing "reasoning path, operation intention, and instability risk level".

[0136] The model outputs structured labeled data (i.e., generative labeled results). ,in For the description of the reasoning path, Classification of operation intent (e.g., stable grasp, impending slip, collision risk, normal operation). Assess the confidence level for the risk of instability. This is the predicted probability of future action evolution (such as continuing to stabilize, slipping, adjusting the grasp, etc.).

[0137] S4: Train the behavior prediction model using a labeled dataset composed of multiple generative annotation results.

[0138] like Figure 9 As shown, step S4 further includes:

[0139] S41: Vectorize and align the original behavioral features, features corresponding to associated physical prior information, and dynamic environment features corresponding to the symbolic reasoning prior knowledge base in the enhanced behavioral sequence, and use the gated fusion unit to achieve nonlinear integration of multiple features.

[0140] Specifically, the enhanced behavior sequence T generated in step S2 is vectorized and aligned. Let T contain the original behavior features. (Joint angles, angular velocities, torques), physical prior label features (Mass, coefficient of friction, stiffness) and dynamic environment characteristics (Contact state, operation phase). Nonlinear integration of multiple features is achieved using a gated fusion unit:

[0141] ;

[0142] ;

[0143] in, and The weight matrix is ​​a learnable matrix. and For learnable bias terms; It is the Sigmoid activation function. This represents element-wise multiplication. A gating mechanism automatically adjusts the contribution weights of different sources of physical knowledge in operational risk assessment.

[0144] S42: Construct a multi-layer bidirectional Transformer encoder architecture, introduce relative position encoding, and extract behavioral trajectory features through multi-layer self-attention blocks.

[0145] Specifically, to capture the bidirectional behavioral dependence and unstable evolution trend of action objects over long time, a multi-layer Bidirectional Transformer-Encoder architecture is constructed: Relative Positional Encoding is introduced, enabling the model to perceive the time interval span between different action nodes and distinguish between stable operations and transient perturbations; Deep feature extraction: The potential trajectory features of the action object's evolution from "stable operation" to "unstable state" (such as slippage, collision, loss of control) are extracted through N layers of self-attention blocks, generating a high-dimensional hidden state sequence. }

[0146] S43: Utilize generative annotation results as soft labels, and use knowledge distillation techniques to guide the fine-tuning of the parameters of the behavior prediction model.

[0147] Specifically, the generatively labeled result Y (containing the operational intent reasoning path and instability risk score) generated by the multimodal large model in step S3 is used as a soft label to guide the fine-tuning of the behavior prediction model's parameters. A joint loss function is defined. :

[0148] ;

[0149] in, The predicted output of the behavior prediction model. Hard labels that are actually labeled. The soft label distribution output by the multimodal large model. The regularization coefficient is . For model parameters, For L2 regularization terms, For cross-entropy loss targeting risk categories, To predict the KL divergence between the output and the labeled distribution of the multimodal large model, These are the distillation weighting coefficients. Through knowledge distillation technology, lightweight behavior prediction models can learn the deep physical intent inference capabilities of multimodal large models, achieving efficient and accurate prediction in real-time control scenarios.

[0150] S5: Input the real-time behavior data of the target to be predicted into the trained behavior prediction model, and output an abnormal operation risk assessment report for the target in an unstructured environment.

[0151] like Figure 10 As shown, step S5 further includes:

[0152] S51: Access the real-time behavior data of the object to be predicted, and perform a correlation retrieval of the real-time physical feedback and object attribute priors with the symbolic reasoning prior knowledge base to obtain matching associated physical prior information. Embed the associated physical prior information and map it to the feature vector space to generate the feature matrix to be tested.

[0153] Specifically, access This includes joint angle sensors, torque sensors, tactile sensors, and visual feedback. A lightweight computing engine is used to perform data cleaning and standardization. For the behavioral state of the object to be predicted, the symbolic reasoning prior embedding algorithm from step S2 is invoked in real time to map the real-time physical feedback and object attribute priors to the feature vector space, generating the feature matrix to be tested. .

[0154] S52: Input the feature matrix to be tested into the trained behavior prediction model, use the sliding window mechanism to extract continuous behavior segments, output the probability distribution of instability risk at each time step, and calculate the risk severity index. When the risk severity exceeds the preset mutation threshold, it is identified as an operational state change.

[0155] Specifically, Input the behavior prediction model trained in step S4. Use a sliding window mechanism to extract recent continuous behavior segments of the target, and utilize the behavior prediction model to output the probability distribution of instability risk at each time step. Define risk severity indicators. Used to monitor the dynamic rate of change of probability values:

[0156] ;

[0157] when When the preset mutation threshold is exceeded, it is automatically identified as "operational state change" (such as suddenly entering the pre-slip state from stable grasping), realizing early detection of the evolution of covert contact instability from the latent stage to the out-of-control stage.

[0158] S53: When the probability of instability exceeds the preset risk threshold, an abnormal operation risk assessment report is automatically generated. The abnormal operation risk assessment report includes action evolution path prediction and control strategy adjustment suggestions.

[0159] when When the preset risk threshold is exceeded, an early warning briefing is automatically generated, and an abnormal operation risk assessment report is output, which includes predictions of the motion evolution path (such as "slippage is expected in 0.3 seconds") and suggestions for adjusting the control strategy (such as "increase the gripping force" or "adjust the gripping posture").

[0160] This invention predicts whether instability events such as slippage or collision will occur in the future based on the current and historical behavior of the object and combined with prior knowledge of the object's physics, and issues early warnings or adjustment suggestions in advance.

[0161] The following are apparatus embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments remain valid in this embodiment, and will not be repeated here to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0162] like Figure 11 The diagram shown is a structural schematic of a behavior prediction device 2 provided in the second embodiment of the present invention, which adopts the behavior prediction method of the first embodiment described above and includes:

[0163] The symbolic reasoning prior knowledge base construction module 20 is used to construct a multi-dimensional symbolic reasoning prior knowledge base based on the object geometric model library, physical attribute database, and dynamic simulation priors of typical operation scenarios.

[0164] The behavior enhancement module 21 is used to extract continuous behavior data of the sample action object within a preset time window, and to perform association retrieval with the symbolic reasoning prior knowledge base to obtain matching associated physical prior information. The associated physical prior information is then embedded into the continuous behavior data to generate an enhanced behavior sequence.

[0165] The multimodal reasoning module 22 is used to input the enhanced behavior sequence into the multimodal large model, guide the multimodal large model to reason through the prompt template containing step-by-step reasoning guidance instructions, and generate generative annotation results including operation intention evaluation, contact stability score and instability risk level;

[0166] The behavior prediction model training module 23 is used to train the behavior prediction model using a labeled dataset composed of multiple generative annotation results.

[0167] Output module 24 is used to input the real-time behavior data of the target to be predicted into the trained behavior prediction model and output an abnormal operation risk assessment report for the target in an unstructured environment.

[0168] like Figure 12 As shown, the symbolic reasoning prior knowledge base construction module 20 also includes:

[0169] The first construction unit 200 is used to construct an object geometric model library, obtain a set of three-dimensional geometric models of various operation target objects, and use a 3D feature extractor to map the set of three-dimensional geometric models to a high-dimensional vector space to generate geometric feature vectors.

[0170] The second construction unit 201 is used to construct a physical attribute database, obtain a set of physical attributes of various operation target objects, perform weighted mapping and hash sequence generation on the physical attribute set, obtain binary fingerprints through binarization and construct hash indexes;

[0171] The third building unit 202 is used to build a dynamic simulation prior for typical operation scenarios. It simulates typical operation scenarios through a physics engine and extracts action-contact mapping sets. It uses a time-series state graph to store the operation evolution path of the action-contact mapping sets, defines operation state nodes and quantifies contact stability. It uses the action-contact mapping sets, the operation state nodes and the contact stability quantification results as abstract physical state information and converts them into natural language descriptors.

[0172] like Figure 13 As shown, the behavior enhancement module 21 also includes:

[0173] The first feature extraction unit 210 is used to extract behavioral features and segment states from continuous behavioral data, and to extract the behavioral patterns, contact states and force features of each behavioral node using bidirectional temporal coding technology.

[0174] The first retrieval and matching unit 211 is used to perform vector weighted merging of the extracted features to generate the physical fingerprint of the behavior sequence to be tested, perform hash collision retrieval in the symbolic reasoning prior knowledge base, calculate the Hamming distance, and if the Hamming distance meets the preset threshold, hit the corresponding associated physical prior information and convert the associated physical prior information into explicit physical constraint labels.

[0175] The fusion enhancement unit 212 is used to take explicit physical constraint labels and continuous behavior data as input, calculate the similarity between the global behavior semantic vector and each operation state node in the symbolic reasoning prior knowledge base, select operation state nodes that meet the similarity conditions, and synthesize the physical state information in the selected operation state nodes with the explicit physical constraint labels to generate the enhanced behavior sequence.

[0176] like Figure 14 As shown, the multimodal reasoning module 22 also includes:

[0177] The prompt template construction unit 220 is used to construct the prompt template, which includes character settings, task description, physical knowledge background and step-by-step reasoning guidance instructions;

[0178] The filling guidance unit 221 is used to fill the prompt template with the enhanced behavior sequence to guide the multimodal large model to perform inference;

[0179] The annotation result generation unit 222 is used to output generative annotation results from a multimodal large model. The generative annotation results also include inference path descriptions and predicted future action evolution probabilities.

[0180] like Figure 15 As shown, the behavior prediction model training module 23 also includes:

[0181] Integration unit 230 is used to vectorize and align the original behavioral features, features corresponding to associated physical prior information, and dynamic environment features corresponding to the symbolic reasoning prior knowledge base in the enhanced behavioral sequence, and to achieve nonlinear integration of multiple features using a gated fusion unit.

[0182] The second feature extraction unit 231 is used to construct a multi-layer bidirectional Transformer encoder architecture, introduce relative position encoding, and extract behavioral trajectory features through multi-layer self-attention blocks.

[0183] Adjustment unit 232 is used to use generative annotation results as soft labels to guide the fine-tuning of parameters of behavior prediction model through knowledge distillation technology.

[0184] like Figure 16 As shown, the output module 24 also includes:

[0185] The second retrieval and matching unit 240 is used to access the real-time behavior data of the object to be predicted, associate the real-time physical feedback and object attribute prior with the symbolic reasoning prior knowledge base, obtain the matching associated physical prior information, embed the associated physical prior information, map it to the feature vector space, and generate the feature matrix to be tested.

[0186] The risk monitoring unit 241 is used to input the feature matrix to be tested into the trained behavior prediction model, use the sliding window mechanism to extract continuous behavior segments, output the probability distribution of instability risk at each time step, and calculate the risk severity index. When the risk severity exceeds the preset mutation threshold, it is identified as an operational state change.

[0187] The report generation unit 242 is used to automatically generate the abnormal operation risk assessment report when the probability of instability risk exceeds a preset risk threshold. The abnormal operation risk assessment report includes action evolution path prediction and control strategy adjustment suggestions.

[0188] The behavior prediction device of this invention enhances the original behavior trajectory with knowledge through an object geometric model library, a physical attribute database, and dynamic simulation priors, generating prior embedded samples carrying complete physical context information. This effectively solves the problems of strong concealment of contact instability and difficulty in identifying operational intentions of action objects in unstructured environments. Furthermore, through a multimodal large model, logical reasoning and generative prediction are performed on the historical behavior sequence after embedding physical priors based on operation strategy prompts. This can accurately capture the instability tendency or contact risk behind the action object's operation, forming a high-confidence future behavior prediction sequence.

[0189] A third embodiment of the present invention provides an electronic device (not shown), including: a processor and a memory, wherein the memory is used to store a computer program; when the computer program is executed by the processor, it implements the steps of the behavior prediction method as described in the first embodiment above.

[0190] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the behavior prediction method as described in the first embodiment above.

[0191] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms fall within the scope of protection of the present invention.

Claims

1. A behavior prediction method, characterized in that, Includes the following steps: S1: Construct a multi-dimensional symbolic reasoning prior knowledge base based on the object geometric model library, physical property database, and dynamic simulation priors of typical operation scenarios; S2: Extract continuous behavior data of the sample action object within a preset time window, and perform association retrieval with the symbolic reasoning prior knowledge base to obtain matching associated physical prior information. Embed the associated physical prior information into the continuous behavior data to generate an enhanced behavior sequence. S3: Input the enhanced behavior sequence into the multimodal large model, guide the multimodal large model to perform reasoning through the prompt template containing step-by-step reasoning guidance instructions, and generate generative annotation results containing operation intention evaluation, contact stability score and instability risk level; S4: Train the behavior prediction model using the labeled dataset composed of multiple generative annotation results; S5: Input the real-time behavior data of the target to be predicted into the trained behavior prediction model, and output an abnormal operation risk assessment report for the target to be predicted in an unstructured environment.

2. The behavior prediction method according to claim 1, characterized in that, Step S1 further includes: S11: Construct the object geometric model library, obtain a set of three-dimensional geometric models of various operation target objects, and use a 3D feature extractor to map the set of three-dimensional geometric models to a high-dimensional vector space to generate geometric feature vectors; S12: Construct the physical attribute database, obtain the physical attribute set of various operation target objects, perform weighted mapping and hash sequence generation on the physical attribute set, obtain binary fingerprints through binarization and construct hash index; S13: Construct a dynamic simulation prior for the typical operation scenario, simulate the typical operation scenario through a physics engine, and extract the action-contact mapping set; use a time-series state graph to store the operation evolution path of the action-contact mapping set, define operation state nodes and quantify contact stability; use the action-contact mapping set, the operation state nodes and the contact stability quantification results as abstract physical state information and convert them into natural language descriptors.

3. The behavior prediction method according to claim 1, characterized in that, Step S2 further includes: S21: Perform behavioral feature extraction and state segmentation on the continuous behavioral data, and use bidirectional temporal coding technology to extract the behavioral patterns, contact states and force features of each behavioral node; S22: The extracted features are vector-weighted and merged to generate the physical fingerprint of the behavior sequence to be tested. Hash collision retrieval is performed in the symbolic reasoning prior knowledge base to calculate the Hamming distance. If the Hamming distance meets the preset threshold, the corresponding associated physical prior information is hit and the associated physical prior information is converted into explicit physical constraint labels. S23: Using the explicit physical constraint label and the continuous behavior data as input, calculate the similarity between the global behavior semantic vector and each operation state node in the symbolic reasoning prior knowledge base, select the operation state nodes that meet the similarity conditions, and synthesize the physical state information in the selected operation state nodes with the explicit physical constraint label to generate the enhanced behavior sequence.

4. The behavior prediction method according to claim 1, characterized in that, Step S3 further includes: S31: Construct the prompt template, which includes character settings, task description, physical knowledge background, and the step-by-step reasoning guidance instructions; S32: Fill the enhanced behavior sequence into the prompt template to guide the multimodal large model to perform reasoning; S33: The multimodal large model outputs the generative annotation results, which also include inference path descriptions and predicted future action evolution probabilities.

5. The behavior prediction method according to claim 1, characterized in that, Step S4 further includes: S41: The original behavioral features in the enhanced behavioral sequence, the features corresponding to the associated physical prior information, and the dynamic environment features corresponding to the symbolic reasoning prior knowledge base are vectorized and aligned, and the nonlinear integration of multiple features is achieved using a gated fusion unit; S42: Construct a multi-layer bidirectional Transformer encoder architecture, introduce relative position encoding, and extract behavioral trajectory features through multi-layer self-attention blocks; S43: Using the generative annotation results as soft labels, the parameters of the behavior prediction model are fine-tuned through knowledge distillation technology.

6. The behavior prediction method according to claim 1, characterized in that, Step S5 further includes: S51: Access the real-time behavior data of the object to be predicted, associate and retrieve the real-time physical feedback and object attribute prior with the symbolic reasoning prior knowledge base, obtain matching associated physical prior information, embed the associated physical prior information, map it to the feature vector space, and generate the feature matrix to be tested. S52: Input the feature matrix to be tested into the trained behavior prediction model, use the sliding window mechanism to extract continuous behavior segments, output the probability distribution of instability risk at each time step, and calculate the risk severity index. When the risk severity exceeds the preset mutation threshold, it is identified as an operational state change. S53: When the probability of instability exceeds the preset risk threshold, the abnormal operation risk assessment report is automatically generated. The abnormal operation risk assessment report includes action evolution path prediction and control strategy adjustment suggestions.

7. The behavior prediction method according to claim 2, characterized in that, In step S12, the physical attribute set is weighted and mapped and hash sequence is generated. After binarization, a binary fingerprint is obtained and a hash index is constructed. Specifically, the physical attribute set is dimension normalized and the weight factor of each attribute dimension is calculated. A sequence is generated by weighted accumulation and then binarized to obtain a binary fingerprint. The binary fingerprint and the associated physical prior information are stored in the hash index table in the form of key-value pairs.

8. The behavior prediction method according to claim 2, characterized in that, The operation status node includes joint angle status, contact status quantity, and operation stability score. The contact stability is quantified by the contact force distribution function, and the sum of the weights of each component is 1.

9. A behavior prediction device, characterized in that, include: The symbolic reasoning prior knowledge base construction module is used to build a multi-dimensional symbolic reasoning prior knowledge base based on the object geometric model library, physical property database, and dynamic simulation priors of typical operation scenarios. The behavior enhancement module is used to extract continuous behavior data of sample action objects within a preset time window, and to perform association retrieval of the continuous behavior data with the symbolic reasoning prior knowledge base to obtain matching associated physical prior information. The associated physical prior information is then embedded into the continuous behavior data to generate an enhanced behavior sequence. The multimodal reasoning module is used to input the enhanced behavior sequence into the multimodal large model, guide the multimodal large model to reason through a prompt template containing step-by-step reasoning guidance instructions, and generate generative annotation results including operation intention evaluation, contact stability score and instability risk level; The behavior prediction model training module is used to train the behavior prediction model using a labeled dataset composed of multiple generative annotation results. The output module is used to input the real-time behavior data of the target action object into the trained behavior prediction model and output an abnormal operation risk assessment report for the target action object in an unstructured environment.

10. An electronic device, characterized in that, include: processor; Memory, used to store computer programs; When the computer program is executed by the processor, it implements the behavior prediction method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the behavior prediction method as described in any one of claims 1 to 8.

Images