Robot action data management system and method based on AI model
By using an AI-based robot motion data management method, video sequences are collected and analyzed in real time, key event points of atomic actions are identified and marked, scene and ontology state features are integrated, and standard feature operation chains are generated. This solves the problems of insufficient motion data structuring and insufficient cross-modal fusion capabilities in traditional methods, and improves the efficiency of motion reuse.
Patent Information
- Application Number
- CN202610124677.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-29
- Publication Date
- 2026-03-03
AI Technical Summary
Traditional robot motion data management methods suffer from insufficient structuring of motion data and lack of cross-modal fusion capabilities, which leads to the need for re-adjustment and adaptation when reusing motions, increasing the time and cost of task deployment.
A robot motion data management method based on AI models is adopted. By acquiring video sequences in real time, a lightweight multimodal feature encoding network and a neural network parsing model are used to identify and mark key event points of atomic actions, integrate scene information and robot body state features, and generate standard feature operation chains.
It achieves deep interactive fusion of robot motion data, improves the structuring of motion data, establishes multi-dimensional associations between operations, scenarios and states, and reduces the debugging and adaptation costs when reusing motions.
Smart Images
Figure CN121589830A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot motion data management technology, specifically to a robot motion data management system and method based on an AI model. Background Technology
[0002] With the increasing demand for intelligent robots in fields such as industrial automation and home services, robot motion data management, as a core link in robot task execution and knowledge reuse, directly affects the robot's adaptability to multiple scenarios and the accuracy of task execution due to its degree of structuring, cross-modal correlation capabilities, and reuse efficiency.
[0003] However, traditional robot motion data management methods often face the following problems when dealing with the needs of multi-scene motion adaptation, cross-modal data integration, and motion knowledge reuse: First, the motion data is not sufficiently structured. Traditional robot motion data is mostly stored in the form of raw video sequences or simple operation step lists, without being associated with the scene context of the action execution or the robot's body state. For example, when a robot performs a grasping action, the traditional management method only records the operation steps of the action, but does not synchronously associate it with the scene at which the action occurs, the position information of the object being manipulated, and the joint angle state of the robot. This results in the need for re-adaptation and debugging when reusing the action, increasing the time cost of task deployment. Second, motion data management lacks cross-modal fusion capabilities. In traditional methods, robot operation step data, scene visual data, and body state data are usually stored as discrete data that are not integrated with features. Summary of the Invention
[0004] The purpose of this invention is to provide a robot motion data management system and method based on AI models to solve the problems raised in the prior art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a robot motion data management method based on an AI model, the method comprising the following steps: It receives robot commands, matches corresponding actions in the standard action library and drives their execution, and collects video sequences during the action execution process in real time. The robot task video is decoded and segmented into frames to generate state feature vectors at each moment. Adjacent feature vectors are analyzed to determine the action category. If a switch is detected, the corresponding moment is recorded as the segmentation node. The nodes are traversed and collected to form a set. The video is then cut into standard operation video segments. After the robot performs the task, it integrates the divided standard operation video segments into a set, processes each video segment in the set separately, takes the continuous frame sequence of the video segment and the synchronous quantization state data as input, and identifies and marks the key event points of the corresponding atomic actions through a neural network action parsing model that integrates temporal action recognition and ontology state decoding, and outputs an ordered set of operations. For the standard operation video segment corresponding to the task, determine its operation set, extract the context feature set containing scene information and robot body state, and generate the feature operation node corresponding to the atomic action through integration and encoding; For each robot task, the feature operation nodes in its corresponding standard operation video are arranged in the order of task execution time and integrated into an ordered node sequence covering all operation links of the task, and then connected to form the standard feature operation chain corresponding to the task.
[0006] The robot receives commands, matches corresponding actions in the standard action library, and drives their execution. It also captures video sequences of the action execution process in real time. Specific steps include: An AI model is deployed on the edge of the robot core board, integrating a voice acquisition sensor, a voice recognition module, and a vision sensor. The robot acquires external sound signals to be processed through the voice acquisition sensor, and the voice recognition module performs preliminary decoding of the signals. The edge AI model then performs semantic enhancement and parsing on the decoded content to extract the command operation. Based on this, the corresponding standard operation action is matched from the preset standard action library, and the robot core board drives the actuator to execute the above standard operation action. The vision sensor collects video sequences of the robot's action execution process in real time.
[0007] The robot task video is decoded and segmented into frames to generate state feature vectors at each moment. Adjacent feature vectors are analyzed to determine the action category. If a switch is detected, the corresponding moment is recorded as a segmentation node. The nodes are traversed and collected to form a set. The video is then segmented using this set as the standard operation video segment. The specific steps include: The video sequence acquired during the robot's task execution is decoded and segmented to obtain a video frame sequence {f1, f2, ..., fm, ..., fM} arranged in the acquisition time sequence, where fm represents the visual data of the m-th frame in the video frame sequence, and M represents the total number of frames in the video frame sequence; Using a lightweight multimodal feature encoding network deployed on the core board, an initial visual feature vector is extracted from the video frame at time t. This initial visual feature vector and the synchronously acquired and normalized / quantized robot body state data are used as dual inputs. The feature fusion module fuses the contextual semantics of the video frame with the quantized body state data to output the state feature vector φ at time t. t ; On the core board, the state feature vectors φ at adjacent time points are analyzed sequentially. t and φ t+1The state feature vector is input into a pre-trained lightweight action state classifier to determine the action category at the current time. When a change in the determined action category is detected from time t to time t+1, time t+1 is determined to be an action segmentation node in the video sequence, where φ t+1 This represents the state feature vector at time t+1; Traverse the entire task execution process to obtain a set of action partitioning nodes. Using the node times in this set as boundaries, cut the video sequence into n standard operation video segments. Here, n represents the number of standard operation video segments obtained after cutting with action partitioning nodes as boundaries. Each standard operation video segment corresponds to a predefined or learned atomic action of the robot, which is the smallest indivisible action unit.
[0008] After the robot performs its task, it integrates the divided standard operation video segments into a set. Each video segment within the set is processed separately. Taking the continuous frame sequence and synchronization quantization state data of the video segment as input, a neural network action parsing model that integrates temporal action recognition and ontology state decoding is used to identify and mark the key event points of the corresponding atomic actions, and output an ordered set of operations. The specific steps include: When the robot performs task t, its standard operation video sequence is divided as described above, resulting in n standard operation video segments, denoted as the set ST={s(t,1),s(t,2),...,s(t,j),...,s(t,n)}, where s(t,j) represents the j-th standard operation video segment of the robot performing task t; Each standard operation video segment within the set ST is processed individually. Taking the continuous frame sequence of the j-th standard operation video segment of the robot performing task t and its synchronized quantized state data as input, the key event points within the atomic action are identified and marked using the constructed motion parsing model, outputting an ordered operation set Ot. j ={ot j，1 ot j，2 , ..., ot j，k , ..., ot j，K}, where ot j，k This represents the k-th structured operation description tuple in the ordered set of operations, where K represents the number of structured operation description tuples in the ordered set of operations.
[0009] The action parsing model is a lightweight neural network combining temporal action recognition and ontology state decoding. This model includes a visual temporal feature extraction branch, an ontology state feature decoding branch, and a cross-modal feature fusion layer. The visual temporal branch extracts action temporal features from consecutive frames of a video segment based on 3D-CNN. The ontology state branch decodes quantized robot ontology state data through a fully connected layer. The fusion layer uses an attention mechanism to weightedly fuse the features from both branches, outputting the key event point recognition result for the atomic action. The key event point represents the instant during the execution of an atomic action that marks the start or end of a specific sub-functional phase or a qualitative change in the robot's interaction state with the environment. The training objective of the action parsing model is to locate the timestamps of key event points and determine their event types based on the input continuous frame sequence and synchronization state data.
[0010] For the standard operation video segment corresponding to the task, determine its operation set, extract the context feature set containing scene information and robot body state, and generate the feature operation node corresponding to the atomic action through integration and encoding. The specific steps include: In the standard operation video segment set ST corresponding to task t, the operation set corresponding to the j-th standard operation video segment s(t,j) is Ot. j From the start frame, key frame and end frame of s(t,j), the scene context feature information set is extracted and denoted as c(t,j). The scene context feature information set includes the key object detection box of the scene at the start / end, the scene classification label, and the robot body state snapshot before and after the start and end of the atomic action. c(t,j) represents the scene context feature information set corresponding to the j-th standard operation video segment s(t,j) when the robot performs task t. The key frame is selected based on the timestamp corresponding to the identified key event point, and the video frame within the preset time window before and after each key event point is selected as the key frame. The set of operations Ot corresponding to the j-th standard operation video segment s(t,j) j The information is integrated and encoded with the scene context feature information set c(t,j) to obtain the j-th feature operation node N(t,j) of task t; The construction method of the j-th feature operation node N(t,j) of task t satisfies: N(t,j) = Encoder(Ot) j ,c(t,j)), where Encoder() represents a neural network that maps the multi-source information sequence of the operation set and the scene context feature information set into a fixed-dimensional vector; N(t,j) represents the key operation sequence required to perform a specific atomic action in a specific scene context.
[0011] The Encoder() function employs a multimodal fusion architecture based on a hierarchical attention mechanism, including: an operation set encoding module, a scene context encoding module, and a cross-modal alignment and fusion module. The operation set encoding module uses a bidirectional long short-term memory network (Bi-LSTM) to encode the ordered operation set Ot. j The structured operations described in the text encode a sequence of tuples, where each tuple ot j，k The scene context encoding module converts the data into dense vectors through an embedding layer, inputs them into a Bi-LSTM to obtain context-aware feature representations for each time step, forming an operational feature sequence. The scene context encoding module performs feature processing in steps: first, for key object detection boxes in the scene context, RoIAlign technology is used to extract region features from the corresponding video feature map, which are then projected onto a unified feature space via a multilayer perceptron to obtain visual features; subsequently, scene classification labels are converted into semantic vectors through a word embedding layer, and robot body state snapshots are encoded into state feature vectors through a fully connected network; a gated attention mechanism is used to adaptively weight and fuse these three types of features to obtain the scene context feature vector; the cross-modal alignment and fusion module uses the scene context feature vector as the query vector and employs a cross-attention mechanism to perform cross-attention on each part of the operational feature sequence. The elements are weighted to obtain scene-enhanced operational features. Then, the aggregated representation of the operational feature sequence after self-attention pooling is used as the query. Attention weights are applied to the object, semantic, and state parts of the scene context to generate operation-aware scene features. The two types of features are concatenated to form a combined feature sequence, which is input into an L-layer Transformer encoder. In each layer, through a multi-head self-attention mechanism, information interaction and weighted aggregation can be performed between any two feature elements in the sequence. Each layer of the encoder contains layer normalization and a feedforward neural network, and residual connections are used. The feature vector of the content label bit of the output of the last Transformer encoder layer is taken, or global average pooling is performed on all output sequences, and then projected through a fully connected layer to obtain fixed-dimensional feature operation nodes.
[0012] For each robot task, all feature operation nodes in its corresponding standard operation video are arranged in chronological order according to the task execution time, and integrated into an ordered node sequence covering all operation stages of the task. This sequence is then linked together to form the standard feature operation chain corresponding to the task. Specific steps include: For each robot task, sort all the feature operation nodes that appear in the corresponding standard operation video sequence according to the time order of task execution; The sorted feature operation nodes are formed into an ordered sequence, where each node corresponds to a feature operation step of the task, and the sequence contains all the feature operation nodes required to complete the task. The ordered sequence of nodes is constructed into a standard feature operation chain for the corresponding task. This operation chain is formed by connecting each feature operation node in the order of task execution.
[0013] This AI-based robot motion data management system includes: an instruction receiving and parsing module, a multimodal motion segmentation module, an atomic motion parsing module, a feature operation node generation module, and a standard feature operation chain construction module. The instruction receiving and parsing module receives robot instructions, matches them to a standard motion library, executes them, and collects motion video sequences in real time. The multimodal motion segmentation module decodes the video, divides it into frames to generate state feature vectors, switches markers based on the motion categories of adjacent vectors, and segments the video into standard operation video segments. The atomic motion parsing module processes the continuous frames and synchronized quantized state data of each video segment, identifies and marks key event points of corresponding atomic actions using a neural network motion parsing model that integrates temporal motion recognition and ontology state decoding, and outputs an ordered set of operations. The feature operation node generation module integrates the operation set with contextual features including scene and robot state to generate feature operation nodes. The standard feature operation chain construction module arranges nodes according to the task execution sequence and connects them to form a corresponding standard feature operation chain.
[0014] Compared with the prior art, the beneficial effects of the present invention are: 1. Compared with the limitations of existing technologies where multi-source heterogeneous data such as operation sequences, scene vision, and ontology state are fragmented and lack integration, this invention achieves deep interactive fusion of the above-mentioned multiple types of data by incorporating a cross-modal fusion architecture and a neural network encoder. On this basis, feature operation nodes are sequentially connected according to the task execution sequence to form a standardized feature operation chain. 2. By integrating and encoding the set of robot atomic actions with scene context features and ontology state features, standardized feature operation nodes are generated, realizing the transformation of action data from discrete information to structured knowledge units. Unlike the existing technology where action data is stored independently and lacks correlation, this invention establishes a multi-dimensional correlation between operations, scenes and states, thereby improving the structuring degree of action data. Attached Figure Description
[0015] Figure 1 This is a flowchart illustrating the robot motion data management method based on an AI model according to the present invention. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] like Figure 1 As shown, this invention provides a technical solution: a robot motion data management method based on an AI model, which includes the following steps: It receives robot commands, matches corresponding actions in the standard action library and drives their execution, and collects video sequences during the action execution process in real time. The robot task video is decoded and segmented into frames to generate state feature vectors at each moment. Adjacent feature vectors are analyzed to determine the action category. If a switch is detected, the corresponding moment is recorded as the segmentation node. The nodes are traversed and collected to form a set. The video is then cut into standard operation video segments. After the robot performs the task, it integrates the divided standard operation video segments into a set, processes each video segment in the set separately, takes the continuous frame sequence of the video segment and the synchronous quantization state data as input, and identifies and marks the key event points of the corresponding atomic actions through a neural network action parsing model that integrates temporal action recognition and ontology state decoding, and outputs an ordered set of operations. For the standard operation video segment corresponding to the task, determine its operation set, extract the context feature set containing scene information and robot body state, and generate the feature operation node corresponding to the atomic action through integration and encoding; For each robot task, the feature operation nodes in its corresponding standard operation video are arranged in the order of task execution time and integrated into an ordered node sequence covering all operation links of the task, and then connected to form the standard feature operation chain corresponding to the task.
[0018] The robot receives commands, matches corresponding actions in the standard action library, and drives their execution. It also captures video sequences of the action execution process in real time. Specific steps include: An AI model is deployed on the edge of the robot core board, integrating a voice acquisition sensor, a voice recognition module, and a vision sensor. The robot acquires external sound signals to be processed through the voice acquisition sensor, and the voice recognition module performs preliminary decoding of the signals. The edge AI model then performs semantic enhancement and parsing on the decoded content to extract the command operation. Based on this, the corresponding standard operation action is matched from the preset standard action library, and the robot core board drives the actuator to execute the above standard operation action. The vision sensor collects video sequences of the robot's action execution process in real time.
[0019] The robot task video is decoded and segmented into frames to generate state feature vectors at each moment. Adjacent feature vectors are analyzed to determine the action category. If a switch is detected, the corresponding moment is recorded as a segmentation node. The nodes are traversed and collected to form a set. The video is then segmented using this set as the standard operation video segment. The specific steps include: The video sequence acquired during the robot's task execution is decoded and segmented to obtain a video frame sequence {f1, f2, ..., fm, ..., fM} arranged in the acquisition time sequence, where fm represents the visual data of the m-th frame in the video frame sequence, and M represents the total number of frames in the video frame sequence; Using a lightweight multimodal feature encoding network deployed on the core board, an initial visual feature vector is extracted from the video frame at time t. This initial visual feature vector and the synchronously acquired and normalized / quantized robot body state data are used as dual inputs. The feature fusion module fuses the contextual semantics of the video frame with the quantized body state data to output the state feature vector φ at time t. t ; On the core board, the state feature vectors φ at adjacent time points are analyzed sequentially. t and φ t+1 The state feature vector is input into a pre-trained lightweight action state classifier to determine the action category at the current time. When a change in the determined action category is detected from time t to time t+1, time t+1 is determined to be an action segmentation node in the video sequence, where φ t+1 This represents the state feature vector at time t+1; Traverse the entire task execution process to obtain a set of action partitioning nodes. Using the node times in this set as boundaries, cut the video sequence into n standard operation video segments. Here, n represents the number of standard operation video segments obtained after cutting with action partitioning nodes as boundaries. Each standard operation video segment corresponds to a predefined or learned atomic action of the robot, which is the smallest indivisible action unit.
[0020] After the robot performs its task, it integrates the divided standard operation video segments into a set. Each video segment within the set is processed separately. Taking the continuous frame sequence and synchronization quantization state data of the video segment as input, a neural network action parsing model that integrates temporal action recognition and ontology state decoding is used to identify and mark the key event points of the corresponding atomic actions, and output an ordered set of operations. The specific steps include: When the robot performs task t, its standard operation video sequence is divided as described above, resulting in n standard operation video segments, denoted as the set ST={s(t,1),s(t,2),...,s(t,j),...,s(t,n)}, where s(t,j) represents the j-th standard operation video segment of the robot performing task t; Each standard operation video segment within the set ST is processed individually. Taking the continuous frame sequence of the j-th standard operation video segment of the robot performing task t and its synchronized quantized state data as input, the key event points within the atomic action are identified and marked using the constructed motion parsing model, outputting an ordered operation set Ot. j ={ot j，1 ot j，2 , ..., ot j，k , ..., ot j，K}, where ot j，k This represents the k-th structured operation description tuple in the ordered set of operations, where K represents the number of structured operation description tuples in the ordered set of operations.
[0021] The action parsing model is a lightweight neural network combining temporal action recognition and ontology state decoding. This model includes a visual temporal feature extraction branch, an ontology state feature decoding branch, and a cross-modal feature fusion layer. The visual temporal branch extracts action temporal features from consecutive frames of a video segment based on 3D-CNN. The ontology state branch decodes quantized robot ontology state data through a fully connected layer. The fusion layer uses an attention mechanism to weightedly fuse the features from both branches, outputting the key event point recognition result for the atomic action. The key event point represents the instant during the execution of an atomic action that marks the start or end of a specific sub-functional phase or a qualitative change in the robot's interaction state with the environment. The training objective of the action parsing model is to locate the timestamps of key event points and determine their event types based on the input continuous frame sequence and synchronization state data.
[0022] For the standard operation video segment corresponding to the task, determine its operation set, extract the context feature set containing scene information and robot body state, and generate the feature operation node corresponding to the atomic action through integration and encoding. The specific steps include: In the standard operation video segment set ST corresponding to task t, the operation set corresponding to the j-th standard operation video segment s(t,j) is Ot. j From the start frame, key frame and end frame of s(t,j), the scene context feature information set is extracted and denoted as c(t,j). The scene context feature information set includes the key object detection box of the scene at the start / end, the scene classification label, and the robot body state snapshot before and after the start and end of the atomic action. c(t,j) represents the scene context feature information set corresponding to the j-th standard operation video segment s(t,j) when the robot performs task t. The key frame is selected based on the timestamp corresponding to the identified key event point, and the video frame within the preset time window before and after each key event point is selected as the key frame. The set of operations Ot corresponding to the j-th standard operation video segment s(t,j) jThe information is integrated and encoded with the scene context feature information set c(t,j) to obtain the j-th feature operation node N(t,j) of task t; The construction method of the j-th feature operation node N(t,j) of task t satisfies: N(t,j) = Encoder(Ot) j ,c(t,j)), where Encoder() represents a neural network that maps the multi-source information sequence of the operation set and the scene context feature information set into a fixed-dimensional vector; N(t,j) represents the key operation sequence required to perform a specific atomic action in a specific scene context.
[0023] The Encoder() function employs a multimodal fusion architecture based on a hierarchical attention mechanism, including: an operation set encoding module, a scene context encoding module, and a cross-modal alignment and fusion module. The operation set encoding module uses a bidirectional long short-term memory network (Bi-LSTM) to encode the ordered operation set Ot. j The structured operations described in the text encode a sequence of tuples, where each tuple ot j，k The scene context encoding module converts the data into dense vectors through an embedding layer, inputs them into a Bi-LSTM to obtain context-aware feature representations for each time step, forming an operational feature sequence. The scene context encoding module performs feature processing in steps: first, for key object detection boxes in the scene context, RoIAlign technology is used to extract region features from the corresponding video feature map, which are then projected onto a unified feature space via a multilayer perceptron to obtain visual features; subsequently, scene classification labels are converted into semantic vectors through a word embedding layer, and robot body state snapshots are encoded into state feature vectors through a fully connected network; a gated attention mechanism is used to adaptively weight and fuse these three types of features to obtain the scene context feature vector; the cross-modal alignment and fusion module uses the scene context feature vector as the query vector and employs a cross-attention mechanism to perform cross-attention on each part of the operational feature sequence. The elements are weighted to obtain scene-enhanced operational features. Then, the aggregated representation of the operational feature sequence after self-attention pooling is used as the query. Attention weights are applied to the object, semantic, and state parts of the scene context to generate operation-aware scene features. The two types of features are concatenated to form a combined feature sequence, which is input into an L-layer Transformer encoder. In each layer, through a multi-head self-attention mechanism, information interaction and weighted aggregation can be performed between any two feature elements in the sequence. Each layer of the encoder contains layer normalization and a feedforward neural network, and residual connections are used. The feature vector of the content label bit of the output of the last Transformer encoder layer is taken, or global average pooling is performed on all output sequences, and then projected through a fully connected layer to obtain fixed-dimensional feature operation nodes.
[0024] For each robot task, all feature operation nodes in its corresponding standard operation video are arranged in chronological order according to the task execution time, and integrated into an ordered node sequence covering all operation stages of the task. This sequence is then linked together to form the standard feature operation chain corresponding to the task. Specific steps include: For each robot task, sort all the feature operation nodes that appear in the corresponding standard operation video sequence according to the time order of task execution; The sorted feature operation nodes are formed into an ordered sequence, where each node corresponds to a feature operation step of the task, and the sequence contains all the feature operation nodes required to complete the task. The ordered sequence of nodes is constructed into a standard feature operation chain for the corresponding task. This operation chain is formed by connecting each feature operation node in the order of task execution.
[0025] In Example 1: After the user issues a voice command, the robot acquires the sound signal through the voice acquisition component, converts the audio signal into processable content through the voice decoding component, and then performs semantic enhancement analysis by the edge artificial intelligence model to extract the command operation of picking up and placing the object. Subsequently, the robot matches the corresponding operation action from the preset standard action library, drives the actuator to start the task, and at the same time, the vision sensor collects the video sequence of the entire process of action execution in real time, and synchronously captures the status data of the robot body. After the task is completed, the acquired task video is decoded and segmented to obtain continuous video frames arranged in the acquisition order. The initial visual features of each frame are extracted through a lightweight multimodal feature encoding network and then fused with the synchronized quantized ontology state data to generate state feature vectors at each time step. The feature vectors of adjacent time steps are analyzed, and a pre-trained action state classifier is used to determine the change of action category. When a category switch is detected, it is marked as a splitting node. Based on these nodes, the complete video is cut into standard operation video segments, and each video segment corresponds to an indivisible atomic action. For each standard operation video segment, its continuous frame sequence and synchronized ontology state data are used as input. Through a neural network action parsing model that integrates temporal action recognition and ontology state decoding, key event points within each atomic action are identified and marked, and an ordered set of operations is output. The specific execution steps of each atomic action are recorded. At the same time, scene context features are extracted from the start frame, key frame and end frame of each video segment, including the detection information of key objects in the scene at the start and end of the action, scene classification labels, and robot ontology state snapshots before and after the start and end of the atomic action, forming a complete set of scene context features. By using a cross-modal fusion architecture and a neural network encoder, the operation set and scene context feature set corresponding to each video segment are deeply integrated and encoded to generate feature operation nodes. These nodes fully represent the key information required to execute the corresponding atomic actions in a specific scene, realizing the association and binding of operation steps with scene and state. Then, all feature operation nodes corresponding to the same task are arranged in the order of task execution time, integrated into an ordered node sequence covering all operation links, and then connected to form the standard feature operation chain of the task, which is stored in the robot's motion database.
[0026] This AI-based robot motion data management system includes: an instruction receiving and parsing module, a multimodal motion segmentation module, an atomic motion parsing module, a feature operation node generation module, and a standard feature operation chain construction module. The instruction receiving and parsing module receives robot instructions, matches them to a standard motion library, executes them, and collects motion video sequences in real time. The multimodal motion segmentation module decodes the video, divides it into frames to generate state feature vectors, switches markers based on the motion categories of adjacent vectors, and segments the video into standard operation video segments. The atomic motion parsing module processes the continuous frames and synchronized quantized state data of each video segment, identifies and marks key event points of corresponding atomic actions using a neural network motion parsing model that integrates temporal motion recognition and ontology state decoding, and outputs an ordered set of operations. The feature operation node generation module integrates the operation set with contextual features including scene and robot state to generate feature operation nodes. The standard feature operation chain construction module arranges nodes according to the task execution sequence and connects them to form a corresponding standard feature operation chain.
[0027] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A robot motion data management method based on AI models, characterized in that: The method includes the following steps: It receives robot commands, matches corresponding actions in the standard action library and drives their execution, and collects video sequences during the action execution process in real time. The robot task video is decoded and segmented into frames to generate state feature vectors at each moment. Adjacent feature vectors are analyzed to determine the action category. If a switch is detected, the corresponding moment is recorded as the segmentation node. The nodes are traversed and collected to form a set. The video is then cut into standard operation video segments. After the robot performs the task, it integrates the divided standard operation video segments into a set, processes each video segment in the set separately, takes the continuous frame sequence of the video segment and the synchronous quantization state data as input, and identifies and marks the key event points of the corresponding atomic actions through a neural network action parsing model that integrates temporal action recognition and ontology state decoding, and outputs an ordered set of operations. For the standard operation video segment corresponding to the task, determine its operation set, extract the context feature set containing scene information and robot body state, and generate the feature operation node corresponding to the atomic action through integration and encoding; For each robot task, the feature operation nodes in its corresponding standard operation video are arranged in the order of task execution time and integrated into an ordered node sequence covering all operation links of the task, and then connected to form the standard feature operation chain corresponding to the task.
2. The robot motion data management method based on an AI model according to claim 1, characterized in that: The robot receives commands, matches corresponding actions in the standard action library, and drives their execution. It also captures video sequences of the action execution process in real time. Specific steps include: An AI model is deployed on the edge of the robot core board, integrating a voice acquisition sensor, a voice recognition module, and a vision sensor. The robot acquires external sound signals to be processed through the voice acquisition sensor, and the voice recognition module performs preliminary decoding of the signals. The edge AI model then performs semantic enhancement and parsing on the decoded content to extract the command operation. Based on this, the corresponding standard operation action is matched from the preset standard action library, and the robot core board drives the actuator to execute the above standard operation action. The vision sensor collects video sequences of the robot's action execution process in real time.
3. The robot motion data management method based on an AI model according to claim 2, characterized in that: The robot task video is decoded and segmented into frames to generate state feature vectors at each moment. Adjacent feature vectors are analyzed to determine the action category. If a switch is detected, the corresponding moment is recorded as a segmentation node. The nodes are traversed and collected to form a set. The video is then segmented using this set as the standard operation video segment. The specific steps include: The video sequence acquired during the robot's task execution is decoded and segmented to obtain a video frame sequence {f1, f2, ..., fm, ..., fM} arranged in the acquisition time sequence, where fm represents the visual data of the m-th frame in the video frame sequence, and M represents the total number of frames in the video frame sequence; Using a lightweight multimodal feature encoding network deployed on the core board, an initial visual feature vector is extracted from the video frame at time t. This initial visual feature vector and the synchronously acquired and normalized / quantized robot body state data are used as dual inputs. The feature fusion module fuses the contextual semantics of the video frame with the quantized body state data to output the state feature vector φ at time t. t ; On the core board, the state feature vectors φ at adjacent time points are analyzed sequentially. t and φ t+1 The state feature vector is input into a pre-trained lightweight action state classifier to determine the action category at the current time. When a change in the determined action category is detected from time t to time t+1, time t+1 is determined to be an action segmentation node in the video sequence, where φ t+1 This represents the state feature vector at time t+1; Traverse the entire task execution process to obtain a set of action partitioning nodes. Using the node times in this set as boundaries, cut the video sequence into n standard operation video segments. Here, n represents the number of standard operation video segments obtained after cutting with action partitioning nodes as boundaries. Each standard operation video segment corresponds to a predefined or learned atomic action of the robot, which is the smallest indivisible action unit.
4. The robot motion data management method based on an AI model according to claim 3, characterized in that: After the robot performs its task, it integrates the divided standard operation video segments into a set. Each video segment within the set is processed separately. Taking the continuous frame sequence and synchronization quantization state data of the video segment as input, a neural network action parsing model that integrates temporal action recognition and ontology state decoding is used to identify and mark the key event points of the corresponding atomic actions, and output an ordered set of operations. The specific steps include: When the robot performs task t, its standard operation video sequence is divided as described above, resulting in n standard operation video segments, denoted as the set ST={s(t,1),s(t,2),...,s(t,j),...,s(t,n)}, where s(t,j) represents the j-th standard operation video segment of the robot performing task t; Each standard operation video segment within the set ST is processed individually. Taking the continuous frame sequence of the j-th standard operation video segment of the robot performing task t and its synchronized quantized state data as input, the key event points within the atomic action are identified and marked using the constructed motion parsing model, outputting an ordered operation set Ot. j ={ot j，1 ot j，2 , ..., ot j，k , ..., ot j，K }, where ot j，k This represents the k-th structured operation description tuple in the ordered set of operations, where K represents the number of structured operation description tuples in the ordered set of operations.
5. The robot motion data management method based on an AI model according to claim 4, characterized in that: The action parsing model is a lightweight neural network combining temporal action recognition and ontology state decoding. This model includes a visual temporal feature extraction branch, an ontology state feature decoding branch, and a cross-modal feature fusion layer. The visual temporal branch extracts action temporal features from consecutive frames of a video segment based on 3D-CNN. The ontology state branch decodes quantized robot ontology state data through a fully connected layer. The fusion layer uses an attention mechanism to weightedly fuse the features from both branches, outputting the key event point recognition result for the atomic action. The key event point represents the instant during the execution of an atomic action that marks the start or end of a specific sub-functional phase or a qualitative change in the robot's interaction state with the environment. The training objective of the action parsing model is to locate the timestamps of key event points and determine their event types based on the input continuous frame sequence and synchronization state data.
6. The robot motion data management method based on an AI model according to claim 5, characterized in that: For the standard operation video segment corresponding to the task, determine its operation set, extract the context feature set containing scene information and robot body state, and generate the feature operation node corresponding to the atomic action through integration and encoding. The specific steps include: In the standard operation video segment set ST corresponding to task t, the operation set corresponding to the j-th standard operation video segment s(t,j) is Ot. j From the start frame, key frame and end frame of s(t,j), the scene context feature information set is extracted and denoted as c(t,j). The scene context feature information set includes the key object detection box of the scene at the start / end, the scene classification label, and the robot body state snapshot before and after the start and end of the atomic action. c(t,j) represents the scene context feature information set corresponding to the j-th standard operation video segment s(t,j) when the robot performs task t. The key frame is selected based on the timestamp corresponding to the identified key event point, and the video frame within the preset time window before and after each key event point is selected as the key frame. The set of operations Ot corresponding to the j-th standard operation video segment s(t,j) j The information is integrated and encoded with the scene context feature information set c(t,j) to obtain the j-th feature operation node N(t,j) of task t; The construction method of the j-th feature operation node N(t,j) of task t satisfies: N(t,j) = Encoder(Ot) j ,c(t,j)), where Encoder() represents a neural network that maps the multi-source information sequence of the operation set and the scene context feature information set into a fixed-dimensional vector; N(t,j) represents the key operation sequence required to perform a specific atomic action in a specific scene context.
7. The robot motion data management method based on an AI model according to claim 6, characterized in that: The Encoder() function employs a multimodal fusion architecture based on a hierarchical attention mechanism, including: an operation set encoding module, a scene context encoding module, and a cross-modal alignment and fusion module. The operation set encoding module uses a bidirectional long short-term memory network (Bi-LSTM) to encode the ordered operation set Ot. j The structured operations described in the text encode a sequence of tuples, where each tuple ot j，k The scene context encoding module transforms the data into dense vectors through an embedding layer, inputs them into a Bi-LSTM to obtain context-aware feature representations for each time step, forming an operational feature sequence. The scene context encoding module performs feature processing in steps: first, for key object detection boxes in the scene context, RoIAlign technology is used to extract region features from the corresponding video feature map, which are then projected onto a unified feature space via a multilayer perceptron to obtain visual features; subsequently, scene classification labels are converted into semantic vectors through a word embedding layer, and robot body state snapshots are encoded into state feature vectors through a fully connected network; a gated attention mechanism is used to adaptively weight and fuse these three types of features to obtain the scene context feature vector; the cross-modal alignment and fusion module uses the scene context feature vector as the query vector and a cross-attention mechanism to weight each element of the operational feature sequence to obtain scene enhancement. Operational features are generated by combining the operation feature sequence with the aggregated representation after self-attention pooling. Attention weights are applied to the object, semantic, and state parts of the scene context to generate operation-aware scene features. The two types of features are concatenated to form a combined feature sequence, which is then input into an L-layer Transformer encoder. In each layer, through a multi-head self-attention mechanism, any two feature elements in the sequence can interact and aggregate with weights. Each layer of the encoder contains layer normalization and a feedforward neural network, and residual connections are used. The feature vector of the output content label bit of the last Transformer encoder layer is taken, or global average pooling is performed on all output sequences. After projection through a fully connected layer, a fixed-dimensional feature operation node is obtained, where L represents the number of network stacking layers of the Transformer encoder.
8. The robot motion data management method based on an AI model according to claim 7, characterized in that: For each robot task, all feature operation nodes in its corresponding standard operation video are arranged in chronological order according to the task execution time, and integrated into an ordered node sequence covering all operation stages of the task. This sequence is then linked together to form the standard feature operation chain corresponding to the task. Specific steps include: For each robot task, sort all the feature operation nodes that appear in the corresponding standard operation video sequence according to the time order of task execution; The sorted feature operation nodes are formed into an ordered sequence, where each node corresponds to a feature operation step of the task, and the sequence contains all the feature operation nodes required to complete the task. The ordered sequence of nodes is constructed into a standard feature operation chain for the corresponding task. This operation chain is formed by connecting each feature operation node in the order of task execution.
9. A robot motion data management system based on an AI model, applied to the robot motion data management method based on an AI model as described in any one of claims 1-8, characterized in that: The system includes: an instruction receiving and parsing module, a multimodal action segmentation module, an atomic action parsing module, a feature operation node generation module, and a standard feature operation chain construction module. The instruction receiving and parsing module receives robot instructions, matches them to a standard action library, executes them, and acquires action video sequences in real time. The multimodal action segmentation module decodes the video, divides it into frames to generate state feature vectors, switches markers based on the action categories of adjacent vectors, and segments the video into standard operation video segments. The atomic action parsing module processes the continuous frames and synchronized quantized state data of each video segment, identifies and marks key event points of corresponding atomic actions using a neural network action parsing model that integrates temporal action recognition and ontology state decoding, and outputs an ordered set of operations. The feature operation node generation module integrates the operation set with contextual features including scene and robot state to generate feature operation nodes. The standard feature operation chain construction module arranges nodes according to the task execution sequence and connects them to form a corresponding standard feature operation chain.