Data construction method, device, medium and product for embodied intelligence training

Abstract

The application discloses a data construction method and device for embodied intelligent training, a medium and a product, and comprises the following steps: acquiring video data; based on the interaction relationship change between the operator and the target object in the video data and the state change of the target object, constructing the operation activity degree representing the interaction degree between the operator and the target object, identifying the triggering moment of the operation event based on the operation activity degree, determining the operation event occurrence interval, segmenting the video data, and obtaining a plurality of operation segments; for each operation segment, based on the interaction relationship change and the state change of the target object, determining whether the operation segment meets the preset effective operation condition, wherein the effective operation condition at least includes the existence of the interaction relationship and the cause of the state change of the target object, and screening to obtain the effective operation segment; performing cross-embodied structure alignment processing on the effective operation segment to construct the cross-embodied alignment training sample; and training the embodied intelligent model based on the cross-embodied alignment training sample.

Description

Technical Field

[0001] This application relates to the field of embodied intelligence, and more particularly to a data construction method, device, medium, and product for embodied intelligence training. Background Technology

[0002] With the development of artificial intelligence technology, embodied artificial intelligence has become a core key for robots to perform complex interactive tasks in the real physical world. To enable embodied artificial intelligence models to generalize their operations, massive amounts of physical interaction data are typically required. Currently, using a large number of existing human operation videos on the internet as data sources to train robots is gradually becoming an important way to obtain training data.

[0003] However, existing technologies have significant limitations when training embodied intelligence models using human video data. On the one hand, raw videos typically contain a large amount of redundant information and invalid frames, lacking precise action boundary segmentation based on interaction intent and physical contact states, resulting in extracted action segments containing a high proportion of noise. On the other hand, there are significant cross-embodied differences between human manipulators and robot end effectors and robotic arms in terms of kinematic structure, degrees of freedom, and physical form. Without targeted structured alignment, robots cannot directly understand and reproduce human actions in videos, severely restricting the deployment and application capabilities of embodied intelligence models in real-world physical environments. Summary of the Invention

[0004] One objective of this application is to provide a data construction method, device, medium, and product for embodied intelligence training, at least to solve the problem in the prior art that when training embodied intelligence models using human operation videos, it is difficult to directly construct high-quality cross-embodied alignment training samples that can be directly used for robot execution from human videos due to the lack of accurate segmentation and validity verification of video action sequences and the failure to effectively handle the morphological and structural differences between humans and robots.

[0005] To achieve the above objectives, some embodiments of this application provide the following aspects:

[0006] This application provides a data construction method for embodied intelligence training, the method comprising:

[0007] Acquire video data containing the actions performed by the operator on the target object;

[0008] Based on the changes in the interaction between the operator and the target object and the state changes of the target object in the video data, an operation activity level is constructed to characterize the degree of interaction between the operator and the target object. The triggering time of the operation event is identified based on the operation activity level, the interval of the operation event occurrence is determined according to the triggering time, and the video data is segmented to obtain multiple operation segments.

[0009] For each operation segment, based on the change in the interaction relationship and the change in the state of the target object, it is determined whether the operation segment meets the preset valid operation conditions, wherein the valid operation conditions include at least the existence of an interaction relationship and causing a change in the state of the target object, and valid operation segments are selected.

[0010] The effective operation fragments are subjected to cross-embodied structure alignment processing to construct cross-embodied alignment training samples;

[0011] The embodied intelligence model is trained based on the cross-embodied alignment training samples.

[0012] Secondly, some embodiments of this application also provide an electronic device, the electronic device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method described above.

[0013] Thirdly, some embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the steps of the method described above.

[0014] Fourthly, some embodiments of this application also provide a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the method described above.

[0015] Compared with related technologies, the solution provided in this application, by acquiring video data containing the actions of an operator on a target object, and deeply analyzing the changes in the interaction between the operator and the target object, as well as the state changes of the target object, can accurately identify the triggering time of the operation event and define the operation range. This multi-dimensional state analysis enables fine-grained atomic action segmentation of redundant video data, and, combined with preset effective operation conditions, performs secondary screening. This not only removes invalid contacts and meaningless frames, but also ensures the high quality and purity of the training samples from the source, laying a solid data foundation for subsequent model training.

[0016] Building upon this foundation, a cross-embodied structured alignment processing mechanism is further introduced. Based on relative positional relationships and target object support relationships, a structured action representation containing physical constraints such as approach direction and grasping center is constructed and converted into a sequence of robot-executable actions. This enables the embodied intelligence model to directly map complex human interaction logic, significantly improving the model's accuracy in action reproduction on real hardware and its task execution generalization ability. Attached Figure Description

[0017] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0018] Figure 1 A flowchart illustrating a data construction method for embodied intelligence training, provided as an exemplary embodiment of this disclosure;

[0019] Figure 2 An exemplary structural diagram of the electronic device provided for some embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Figure 1 An exemplary flowchart of a data construction method for embodied intelligence training provided as an exemplary embodiment of this disclosure, the method comprising:

[0022] S101. Obtain video data containing the operation behavior performed by the operator on the target object.

[0023] Specifically, continuous video data can be acquired using a first-person perspective acquisition device, a fixed camera device, or a mobile camera device. The video data may include monocular image sequences, multi-view image sequences, or image sequences containing depth information. The operating entity may include a human hand, a tool, or other interactive subject performing an operation. The target object may include goods, containers, shelves, work surfaces, or other objects to be operated on.

[0024] In one implementation, the video data can record the process of the operator performing operations such as grasping, moving, placing, adjusting, or organizing the target object, thereby reflecting the state changes of the target object during the operation. To ensure the consistency and stability of subsequent data processing, basic preprocessing operations can be performed on the acquired video data, including timestamp alignment, frame rate unification, resolution normalization, image distortion correction, and abnormal noise frame filtering.

[0025] Furthermore, during the data acquisition phase, basic indexing information can be established for the video data. For example, the acquisition time, scene category, task type, target object category, or operation batch information of the corresponding video data can be recorded, thereby providing basic data support for subsequent operation event recognition, operation segment construction, and cross-embodiment alignment processing.

[0026] S102. Based on the changes in the interaction relationship between the operator and the target object and the state changes of the target object in the video data, construct an operation activity level that represents the degree of interaction between the operator and the target object, identify the trigger time of the operation event based on the operation activity level, determine the interval of the operation event occurrence based on the trigger time, and segment the video data to obtain multiple operation segments.

[0027] Specifically, the system can detect and temporally track the operator and target object in a video sequence to obtain information on their spatial positional relationship, relative motion relationship, and interaction state changes. Combined with the target object's state change process, an operational activity level reflecting the degree of interaction between the operator and the target object is constructed. The target object's state changes can include changes in position, posture, spatial occupancy, support relationships, or relationships with environmental objects. The operational activity level characterizes the impact of the interaction between the operator and the target object on the target object's state changes. It comprehensively reflects information such as operator motion changes, target object motion changes, interaction relationship changes, and target object state changes, and is used to identify effective operational processes related to the target object's state changes.

[0028] In one implementation, the trigger time corresponding to the operation event can be identified based on the approach process, contact establishment process, contact maintenance process, and contact release process between the operator and the target object, combined with the continuity of the target object's state changes. When the operation activity exceeds a preset threshold and the target object's state changes continuously, it can be determined that the current time interval corresponds to a valid operation process, and the corresponding operation event occurrence interval can be determined based on the trigger time.

[0029] Furthermore, the continuous video data can be structurally segmented based on the intervals in which the operation events occur, to obtain multiple operation segments. These operation segments represent the operational processes that cause changes in the state of the target object, thereby enabling subsequent training data to focus more on effective operational behaviors related to changes in the target object's state.

[0030] It should be noted that the changes in interaction relationships, state changes, and operational activity can be obtained through various technical means. For example, they can be obtained based on visual detection results, temporal change characteristics, trajectory analysis results, or rule-based constraints; this embodiment does not limit this approach.

[0031] S103. For each operation segment, based on the change in the interaction relationship and the change in the state of the target object, determine whether the operation segment meets the preset valid operation conditions, wherein the valid operation conditions include at least the existence of an interaction relationship and causing a change in the state of the target object, and filter to obtain valid operation segments.

[0032] Specifically, the system can identify whether there is an interaction between the operator and the target object in each operation segment, and analyze whether the target object undergoes an operation-related state change within that segment. If an operation segment does not demonstrate effective interaction or does not cause a change in the target object's state, it can be deemed an invalid segment and discarded. If an operation segment simultaneously satisfies the conditions of an interaction and a change in the target object's state, it can be deemed a valid operation segment. This method filters out segments without practical operational significance, thereby improving the effectiveness and quality of subsequent training data.

[0033] S104. Perform cross-embodied structured alignment processing on the effective operation fragments to construct cross-embodied alignment training samples.

[0034] Specifically, the operational process within an effective operational segment can be structurally represented, unifying the processing of differences related to the specific morphology of the operational entity, thereby extracting key feature information that characterizes the operational process. For example, the operational process can be abstractly expressed based on the spatial relationship, movement trend, and operational result between the operational entity and the target object, and a unified representation can be constructed. This representation can be used for mapping between different embodied forms, enabling operational data from different entities to be aligned and utilized within a unified framework, thus forming training samples with a consistent structure.

[0035] S105. Train the embodied intelligence model based on the cross-embodied alignment training samples.

[0036] Specifically, the training samples can be input into an embodied intelligence model for training. This model can include, but is not limited to, a model structure based on joint modeling of vision, language, and action. By learning from the training samples, the model can establish a correlation between operational processes and environmental states, thereby improving its decision-making and execution capabilities in practical tasks. Various training strategies can be employed during the training process, such as batch training or iterative update methods; this embodiment does not limit these methods.

[0037] Through the above embodiments, this disclosure achieves structured segmentation of human operation videos by introducing an operation event recognition mechanism based on changes in interaction relationships and target object states. High-quality training samples are constructed through effective operation segment selection and cross-embodied alignment processing, thus avoiding the data redundancy problems caused by relying on simple temporal or semantic segmentation in traditional methods. Simultaneously, selecting effective operations helps improve the effectiveness and consistency of training data, thereby enhancing the training efficiency and generalization ability of the embodied intelligence model. Furthermore, the cross-embodied structured alignment method supports data fusion and transfer between different operating subjects, enhancing the model's adaptability under multi-scenario and multi-task conditions.

[0038] Furthermore, in one embodiment, when performing step S101 to acquire video data containing the operation behavior of the operator on the target object, the operation process of the operator in the real business scenario can be continuously acquired by a first-view acquisition device.

[0039] Specifically, operators perform tasks such as retrieving and placing goods, replenishing shelves, straightening goods, picking orders, or packing goods into baskets in scenarios such as supermarkets, retail displays, or warehouse organization. The acquisition device is used to acquire continuous video data and can simultaneously acquire voice task description information and optional auxiliary signals. The auxiliary signals may include depth information, inertial measurement information, and operator posture information. The acquisition device can be any commercially available device capable of outputting timestamp-aligned image sequences; this embodiment does not limit the specific hardware type.

[0040] To facilitate subsequent automated processing, in a preferred implementation, the acquired video data meets preset basic specifications. For example, the video frame rate is no less than 30 frames per second, and the resolution is no less than 1280×720 pixels; when depth data is present, the time synchronization error between the depth data and the RGB image does not exceed 33 milliseconds; when inertial measurement or attitude information is present, the time synchronization error between it and the video data does not exceed 20 milliseconds. For longer video data, the original video can be initially divided according to a preset time window, for example, establishing a basic segment index in units of 3 to 10 minutes, so as to facilitate subsequent finer-grained segment segmentation.

[0041] After acquiring the raw video data, basic preprocessing can be performed. Specifically, time synchronization and segment indexing can be performed first, that is, the video data, audio data, depth data and other auxiliary signals are aligned based on a unified timestamp, and index information is built for continuously acquired video data, while attaching scene or task-related identifiers, such as scene category, task type or target object category.

[0042] Furthermore, privacy protection and noise reduction operations can be performed. For example, sensitive information areas in the video can be blurred or desensitized, and segments with severe blurring, abnormal exposure, or continuous frame drops can be marked or removed. In addition, observational standardization processing can be performed on the video data, including distortion correction, resolution unification, color normalization, and lens shake compensation, thereby outputting a standardized video sequence with a uniform frame rate and resolution.

[0043] After completing the above preprocessing, preliminary semantic annotation can be performed on the video data. Specifically, by combining voice information, prior scene information, and target object recognition results, a coarse-grained task description can be generated for the video segment to represent the goal or intent of the current operation. For example, task tags related to the operation can be generated as auxiliary information for subsequent operation event recognition and data construction. The semantic annotation process can be implemented using rule-based or model-based methods, and this embodiment does not limit this approach.

[0044] Furthermore, in one embodiment, after completing video data acquisition and basic preprocessing, three basic perception results—hand, target object, and scene—can be constructed to serve as the foundational data support for subsequent operation event segmentation and cross-embodiment alignment processing. Specifically, multimodal perception analysis can be performed frame-by-frame on the video sequence to extract structured representations of the operator, target object, and their surrounding environment.

[0045] Specifically, in terms of hand detection and representation, the hand region of the operator in the video frame can be detected first, and the corresponding two-dimensional key point information can be extracted. In one implementation, the bounding boxes and two-dimensional key point positions of the left and right hands can be obtained through a hand detection model; further, the hand can be parameterized based on a three-dimensional hand pose or mesh model to obtain three-dimensional hand description information including pose parameters and shape parameters. In a preferred implementation, the hand state at each moment can be represented as:

[0046]

[0047] in,( () represents the coordinates of a two-dimensional key point. Represents depth information, and These represent the hand posture parameters and shape parameters, respectively. and These represent wrist rotation and translation information, respectively. For monocular videos lacking depth information, relative depth information can be recovered using a 3D hand model, and estimation errors can be reduced through temporal smoothing.

[0048] In the detection, segmentation, and tracking of target objects, a combination of detection and segmentation can be used for object identification and refined representation. Specifically, candidate objects in video frames can be detected based on the target description to obtain the initial region of the target object; subsequently, a pixel-level mask of the target object is generated through a segmentation model to obtain more accurate contour information; in the temporal dimension, a cross-frame tracking mechanism is used to maintain the continuous trajectory of the target object. When tracking is interrupted due to occlusion or other reasons, joint re-association can be performed based on the overlap of detection boxes, the overlap of masks, and the similarity of appearance features to recover the continuous trajectory of the target object.

[0049] Furthermore, a two-stage scheme of "open vocabulary detection + video segmentation / point tracking" is preferred. For known task words, Grounding DINO is used to detect candidate targets; for target contours, Grounded-SAM2 is used to generate pixel-level masks; for cross-frame tracking, the tracking mode of the SAM3 model is used to maintain object tracking. If a single tracker loses the target under strong occlusion, the trajectory is recovered using a joint re-association method of "detection box IoU + mask overlap + appearance embedding cosine similarity".

[0050] In terms of scene geometry and support relationship analysis, the environmental structure in the video can be analyzed to identify support areas related to the operation, such as shelf shelves, containers, or work surfaces. In one implementation, the above-mentioned support structures can be identified by combining a segmentation model with preset category information, and a scene support relationship representation can be constructed. When depth information or multi-view data is available, the geometric relationships of the scene can be further recovered to help determine the spatial state of the target object, such as whether it is in a supported state or has undergone a positional change.

[0051] To improve the stability of the above sensing results, the time-series data can be filtered. For example, for the translational trajectory of the wrist... A smooth trajectory can be obtained using Kalman filtering; simultaneously, when the displacement changes between adjacent frames... Exceeding the preset speed threshold When this happens, the corresponding frame can be marked as a perceived abnormal frame. In a preferred implementation, the velocity threshold... The speed can be set from 2.0 m / s to 2.5 m / s. Through the above processing, the continuity and stability of the hand and target object trajectories can be effectively improved, thus providing a reliable foundation for subsequent operation event recognition and data construction.

[0052] In one embodiment, the timing of the operation event triggering is determined based on the timing of the establishment, maintenance, or termination of the interaction relationship between the operator and the target object, as well as the timing of changes in the state of the target object.

[0053] Specifically, based on the aforementioned hand, target object, and scene perception results, key moments in the operation process can be identified, thereby providing a basis for subsequent video segmentation.

[0054] In one implementation, the establishment, maintenance, or termination of an interaction relationship can be identified based on the spatial relationship between the hand and the target object and its changes over time. For example, when the hand gradually approaches the target object and forms a spatial contact relationship, the interaction relationship can be determined to be established; when the hand maintains continuous contact with the target object and the target object moves or changes in state, the interaction relationship can be determined to be maintained; when the hand separates from the target object, the interaction relationship can be determined to be terminated. The determination of the above interaction relationship can be comprehensively analyzed by combining the key points of the hand, the location of the target object area, and the relative distance or overlap between the two.

[0055] Furthermore, key moments in the operation event can be determined by combining information on changes in the target object's state. These state changes can include changes in the target object's position, posture, contact with the supporting surface, or relative relationship with other objects. For example, when the target object moves from a shelf into a container, or is adjusted from an inclined state to a stable position, this can be considered a moment of state change. Detecting these state changes can help identify key nodes in the operation process.

[0056] In a preferred implementation, keyframes in the time series can be filtered by comprehensively considering changes in interaction relationships and changes in the state of the target object. When the establishment or termination of an interaction relationship is detected, or when a significant change in the state of the target object is detected, the corresponding time point can be marked as the trigger time of the operation event. Furthermore, the interval between adjacent trigger times can be used to determine the occurrence interval of the operation event and serve as the basis for subsequent video segmentation.

[0057] Furthermore, to improve the stability of trigger moment recognition, the aforementioned temporal filtering results can be combined to smooth the hand trajectory and target object trajectory, thereby reducing the impact of detection noise. In the presence of short-term occlusion or unstable detection, the trigger moment can be corrected through continuous frame consistency judgment to avoid segmentation deviations caused by single-frame misjudgments.

[0058] By using the above methods, the triggering time of operation events can be accurately identified based on the changes in the interaction between the operator and the target object, as well as the changes in the state of the target object. This enables structured segmentation of video data and provides a reliable foundation for subsequent effective operation segment selection and cross-embodiment alignment.

[0059] In one embodiment, the step of constructing an operation activity level characterizing the degree of interaction between the operator and the target object, identifying the trigger time of the operation event based on the operation activity level, determining the operation event occurrence interval based on the trigger time, and segmenting the video data includes:

[0060] Extract the motion intensity signal of the operator, the motion intensity signal of the target object, and the contact probability signal between the operator and the target object from the video data;

[0061] Based on the motion intensity signal of the operator, the motion intensity signal of the target object, and the contact probability signal, the operational activity level, which characterizes the degree of interaction between the operator and the target object, is calculated.

[0062] The peak value of the operation activity is extracted as the operation activity peak value, and the contact state switching point is determined according to the contact probability signal;

[0063] The video data is divided into multiple atomic motion segments by using the peak activity level, the contact state switching point, and the state change point of the target object as candidate segmentation boundaries.

[0064] Specifically, when extracting the motion intensity signal of the operator, the amplitude of the operator's motion can be measured based on the changes in the position of the hand key points or wrist in consecutive frames. For example, by analyzing the positional changes of the hand key points or the changes in the wrist translation trajectory in adjacent frames, a motion intensity sequence that changes over time can be obtained, thereby reflecting the degree of motion activity of the operator at each moment.

[0065] When extracting the motion intensity signal of a target object, its motion state can be characterized based on changes in its position, mask, or tracking trajectory across consecutive frames. For example, by analyzing the displacement or contour changes of the target object in an image, a motion intensity sequence can be obtained, which can be used to describe whether the target object is in a manipulated state.

[0066] When acquiring the contact probability signal between the operator and the target object, estimation can be based on the spatial relationship between them. For example, the distance relationship, overlap, and relative motion trend between key hand points and the target object area can be combined to probabilistically describe whether the two are in contact or close to contact, thus obtaining a contact probability signal that changes over time. This contact probability signal reflects the changing process of the interaction relationship between the operator and the target object.

[0067] After obtaining the above three types of signals, they can be fused to calculate the operational activity, which characterizes the degree of interaction between the operator and the target object. The operational activity comprehensively reflects the movement of the operator, the response of the target object, and changes in the interaction relationship, thereby identifying key moments in the operation process. In one implementation, the operational activity can be calculated based on the changing trends or intensity changes of the above signals, forming an activity curve that changes over time.

[0068] Furthermore, local maxima can be extracted from the operational activity curve as operational activity peaks to represent the moments when the interaction between the operator and the target object is most significant. Simultaneously, based on changes in the contact probability signal, the time points at which contact relationships are established or dissolved can be identified as contact state switching points. In addition, the time points at which significant state changes occur in the target object can be determined by combining the aforementioned detection results of target object state changes.

[0069] Based on this, the peak activity level, the contact state switching point, and the state change point of the target object can be used together as candidate segmentation boundaries to segment the original video data, thereby obtaining multiple atomic action segments. Each atomic action segment corresponds to a relatively complete basic operation process, providing a basic unit for subsequent operation segment merging, validity determination, and cross-embodiment alignment processing.

[0070] The above method enables operation event-driven segmentation based on multi-signal fusion. Compared with methods based on fixed time windows or simple semantic segmentation, it can more accurately reflect the key changes in the actual operation process, thereby improving the accuracy and consistency of the segmentation results.

[0071] Furthermore, in one embodiment, based on the initial segmentation completed according to the changes in the interaction relationship and state changes between the operator and the target object, the operational activity can be further modeled by multimodal temporal signal fusion, thereby improving the accuracy and stability of atomic action segmentation.

[0072] Specifically, multiple types of temporal signals can be extracted from each frame of the video sequence, including: the rate of change of hand keypoint trajectory, the rate of change of target object bounding box or mask, the probability of hand-object contact, changes in the gaze attention area, speech keyword boundary information, camera motion intensity, and the degree of scene semantic change. Based on these multiple types of signals, an operational activity function can be constructed. It is used to comprehensively characterize the degree of interaction between the operator and the target object.

[0073] In a preferred implementation, the operational activity function can be expressed as:

[0074]

[0075] in, Indicates the intensity of hand movements. Indicates the intensity of the target object's motion. Indicates the probability of hand-object contact. Indicates the amount of change in the gaze area. This indicates the amount of scene change after camera motion compensation. Indicates the boundary signal of speech or text keywords. to This is a weighting parameter. In a preferred implementation, it can be set... This is to enhance the impact of contact relationships and changes in the target object's state on activity calculation.

[0076] Furthermore, the intensity of the hand movement The motion intensity of the target object can be calculated comprehensively based on wrist translation speed, palm rotation angular velocity, and finger opening and closing rate of change; Calculations can be performed based on the target object's center displacement, the rate of change of the target frame area, the change in the object's orientation, and the change in height relative to the supporting surface. Using these methods, the dynamic changes of the manipulated object and the target object over time can be characterized.

[0077] After obtaining the operational activity sequence, it is possible to... A sliding window smoothing process is performed, preferably using a sliding window of 9 to 21 frames in length, to reduce the impact of noise. Subsequently, local peaks are extracted from the smoothed activity sequence as candidate segmentation boundaries to identify time points where the level of interaction significantly increases during the operation.

[0078] In terms of contact relationship modeling, in a preferred embodiment, an explicit hand-object contact detection mechanism can be used to calculate the contact probability between each hand and each target object at time (t). The contact probability can be expressed as:

[0079]

[0080] in, These represent the masks for the hand and the target object, respectively. This represents the minimum distance from the keypoints of the hand to the boundary of the target object's mask. This represents the distance attenuation coefficient, used to adjust the degree to which the minimum distance affects the probability of contact. These represent the velocity vectors of the hand and the target object, respectively. Indicates the degree of similarity in depth or the change in monocular depth. Indicates a priori contact event. This represents the Sigmoid function. In a preferred implementation, when ( If the value is greater than 0.55 and remains so for more than 3 consecutive frames, it is considered a contact establishment; when ( If the value is less than 0.35 and remains so for more than 5 consecutive frames, the contact is considered released. In the absence of depth information, the contact probability can be calculated based solely on overlap, distance, and velocity-related terms. to This represents the weighting coefficient of the corresponding feature term, used to adjust the influence of region overlap, distance attenuation, motion consistency, depth proximity, and prior contact events on the contact probability calculation results.

[0081] After completing the above activity analysis and contact state recognition, the peak activity level, contact state switching points, and target object state change points can be used as candidate segmentation boundaries to initially segment the video data, resulting in multiple atomic action segments. Further, adjacent atomic action segments can be merged based on whether the target object is continuous, whether the operation target is consistent, whether the time interval between adjacent segments is below a preset threshold, and whether the object state undergoes a clear change, thereby forming higher-level skill segments.

[0082] In a preferred embodiment, atomic fragment boundary fractions can be further defined. It is used to assist in boundary screening, and its calculation method is as follows:

[0083]

[0084] in, Represents the feature vector of a multimodal frame. This indicates a change in the state of the target object. This indicates changes in the boundaries of spoken keywords. Among them, to This represents the weight coefficient of the corresponding feature term, used to adjust the influence of multimodal feature changes, contact state changes, target object state changes, and speech boundary changes on the boundary determination result of atomic action segments. When:

[0085]

[0086] in, Represents the boundary fraction of atomic action segments The average value within a preset time window. Represents the boundary fraction of the atomic action segment The standard deviation within the time window. When the atomic action segment boundary fraction... When the change in boundary value is greater than the sum of the average value and a preset multiple of the standard deviation, the boundary value corresponding to the current moment is determined to be significant and identified as a candidate segmentation boundary. For example, when the time interval between adjacent boundaries exceeds 12 frames, the corresponding moment can be identified as the boundary of an atomic action segment. Under a video condition of 30 frames per second, this constraint can effectively suppress erroneous segmentation caused by short-term fluctuations, thereby improving the stability and consistency of the segmentation results.

[0087] Through the above implementation methods, refined operation segmentation based on multimodal signal fusion and explicit contact modeling can be achieved, providing high-quality basic data for subsequent operation segment selection and cross-embodiment alignment.

[0088] In one embodiment, the video data is segmented to obtain multiple operation segments, specifically including:

[0089] By comparing adjacent atomic action segments, the continuity characteristics of the target object and the consistency characteristics of the operation target are obtained;

[0090] Based on the continuity characteristics of the target object and the consistency characteristics of the operation target, adjacent atomic action segments are merged into skill segments;

[0091] The operation segment is constructed based on the temporal correlation, target object correlation, and state change continuity among the multiple skill segments.

[0092] Specifically, the video data can first be preliminarily segmented based on the aforementioned candidate segmentation boundaries to obtain multiple atomic action segments. These atomic action segments represent the smallest unit of operational behavior, corresponding to the time interval in which the target object's state undergoes a local change or its interaction relationship changes.

[0093] After obtaining the atomic action fragments, adjacent atomic action fragments can be compared to obtain the continuity characteristics of the target object and the consistency characteristics of the operation target, thereby determining whether the atomic action fragments need to be aggregated.

[0094] In one implementation, the continuity characteristic of the target object can be determined based on the target object's identification information, spatial location, and trajectory continuity in adjacent atomic action segments. For example, by comparing the detection results, tracking identifiers, or mask overlaps of target objects in adjacent segments, it can be determined whether they correspond to the same target object. Simultaneously, by combining the target object's trajectory continuity in the time dimension, its motion path in adjacent segments can be analyzed to determine whether the target object maintains continuous existence in multiple atomic action segments. When the target object in adjacent atomic action segments exhibits continuity in both identification and trajectory, it can be determined that it possesses the target object continuity characteristic.

[0095] In one implementation, the consistency characteristic of the operation target can be determined based on the semantic information and state change process of the operation. For example, the operation targets corresponding to adjacent atomic action segments can be compared by combining task description information, operation context information, and target object state changes. When the operation behaviors in adjacent atomic action segments point to the same operation purpose, and the target object state changes show a continuous progression, it can be determined that they possess the operation target consistency characteristic. In addition, the time interval information between adjacent atomic action segments can be combined; when the time interval is lower than a preset threshold, the judgment of operation continuity can be further enhanced.

[0096] Based on this, when adjacent atomic action segments simultaneously satisfy the characteristics of target object continuity and operation target consistency, the adjacent atomic action segments can be merged to form a skill segment. The skill segment represents a relatively complete staged operation process, which contains multiple consecutive atomic action segments and reflects the continuous state change process within a certain stage. For example, a skill segment can correspond to staged operation processes such as approaching a target object, grasping a target object, moving a target object, or placing a target object.

[0097] Furthermore, after constructing the skill segments, multiple skill segments can be aggregated at a higher level based on their temporal relationships, target object relationships, and continuous state change relationships to form operation segments. These operation segments describe the complete task execution flow, covering the entire process of the target object changing from its initial state to its target state, thus reflecting the complete operational logic and task structure. For example, an operation segment can correspond to a complete replenishment process, sorting process, or organization process.

[0098] Specifically, multiple skill segments can be analyzed for correlation based on their continuity over time and the relationship between their corresponding operational objectives. When adjacent skill segments exhibit a continuous progression in terms of operational object, operational purpose, and state change process, they can be merged into a single operational segment.

[0099] Furthermore, in one implementation, to support the automatic aggregation of atomic action fragments into skill fragments, the corresponding fragment embedding features can be extracted for each atomic action fragment. The fragment embedding features can be constructed by fusing multi-source information, such as visual features, interaction relationship statistics, motion trajectory statistics, and corresponding text summary information. Specifically, the above-mentioned multiple features can be spliced ​​or fused to obtain a unified fragment representation, which is used to characterize the operational semantics and dynamic features of atomic action fragments.

[0100] After obtaining the embedded representations of each atomic action segment, clustering methods can be used to group the atomic action segments to identify sets of segments with similar operation patterns. In a preferred implementation, density-based clustering or graph-based clustering methods can be used to perform cluster analysis on the embedded features, thereby obtaining several candidate operation pattern categories. For atomic action segments in the same video, when they are temporally adjacent, have consistent clustering results, and the time interval between adjacent segments is less than a preset threshold, they can be automatically aggregated into the same skill segment.

[0101] Furthermore, in cross-video data scenarios, atomic action segments from different videos can be uniformly embedded and clustered to identify common operation patterns across samples. By analyzing the clustering results, a set of skill patterns containing various typical staged operational behaviors can be constructed, such as grasping skills, movement skills, placement skills, or adjustment skills. This set of skill patterns can serve as a reference structure for subsequent operation segment construction, model training, or cross-embodied operation alignment, thereby improving data consistency and model generalization ability across different scenarios.

[0102] Through the above implementation methods, skill fragments and operation fragments can be constructed step by step based on atomic action fragments, thereby forming an operation representation with a hierarchical structure. This reduces the data fragmentation problem caused by overly fine segmentation and improves the structural integrity, semantic consistency, and task expression ability of training data.

[0103] In one embodiment, determining whether the operation fragment is a valid operation fragment based on the change in the interaction relationship and the change in the state of the target object includes:

[0104] Identify the contact relationship between the operating body and the target object in the operation segment, and extract the state change amount of the target object;

[0105] If the contact relationship is not detected and the state change is lower than a preset threshold, the operation segment is determined to be an invalid operation segment.

[0106] If the preset activity level condition is met and the operation segment is not determined to be invalid, the operation segment is determined to be a valid operation segment.

[0107] Specifically, when identifying the contact relationship between the operator and the target object, the determination can be based on the spatial relationship between the hand and the target object and its changes over time. In one implementation, the distance relationship between key points of the hand and the target object mask, the overlap of regions, and the relative motion trend can be combined to determine whether the operator and the target object are in contact, and to obtain the temporal distribution of the contact relationship within the operation segment. In a preferred implementation, the contact relationship can be determined based on the aforementioned contact probability signal. When the contact probability exceeds a preset threshold for several consecutive frames, a contact relationship is determined to exist; when the contact probability remains below another preset threshold, a contact relationship is determined to not have occurred.

[0108] When extracting the state change of the target object, it can be measured based on the spatial state change of the target object within the operation segment. For example, the state change can be obtained by analyzing the changes in the target object's position, orientation, or relationship with the supporting structure at the beginning and end of the segment. In one implementation, the overall state change can be comprehensively evaluated by combining the changes in the target object's center position, orientation, and contact state with the supporting surface, thereby determining whether the operation segment has caused an effective change in the target object.

[0109] Furthermore, when determining whether an operation segment is invalid, if the detection result indicates that no contact relationship has been formed between the operator and the target object, and the amount of state change of the target object within the operation segment is lower than a preset threshold, the operation segment can be determined as invalid. For example, if the operator only moves near the target object without making actual contact, or the contact time is extremely short and does not cause a significant state change in the target object, the corresponding segment can be determined as invalid, thereby avoiding the use of data without practical operational significance for subsequent training.

[0110] When determining whether an operation segment is a valid operation segment, constraints can be further imposed by incorporating operation activity information. In one implementation, when the operation activity within an operation segment reaches a preset condition, and the operation segment is not determined to be an invalid operation segment, it can be determined to be a valid operation segment. The operation activity condition can be used to ensure that the operation process has a certain dynamic change intensity, thereby avoiding misjudging noise or slight disturbances as valid operations.

[0111] Furthermore, in a preferred embodiment, the determination result of a valid operation segment can be further corrected by combining the duration of the contact relationship, the magnitude of the state change, and the trend of operational activity. For example, when the contact relationship exists only for a very short time and is not accompanied by significant state changes, its validity determination weight can be reduced; when the target object's state changes significantly and is consistent with the contact relationship in time, the credibility of its validity determination result can be enhanced. The contact relationship includes direct contact relationships or indirect interaction relationships formed through tools.

[0112] Through the above implementation methods, it is possible to determine effective operation segments based on changes in interaction relationships and changes in the state of the target object, thereby filtering out training data with practical operational significance, reducing the interference of invalid data on model training, and improving the quality and stability of subsequent cross-embodiment alignment and model training.

[0113] Furthermore, in one embodiment, after completing the preliminary determination of the validity of the operation segment, data screening and quality control processing can be further performed on the operation segment to filter out invalid or low-value data and reasonably divert abnormal data, thereby improving the overall quality of the training dataset.

[0114] Specifically, multiple rejection rules can be defined to filter operation segments. In one implementation, an empty operation rejection rule can be executed. When no contact relationship between the operator and the target object is detected in an operation segment, and the target object does not undergo a significant state change, the operation segment can be determined as an invalid segment, such as segments corresponding to browsing, waiting, or searching processes. In a preferred implementation, the proportion of active frames can be statistically analyzed based on operation activity-related signals, and defined as follows:

[0115]

[0116] in, This indicates that the hand movement intensity is met. or the intensity of the target object's movement The number of frames exceeding the preset threshold This indicates the total number of frames in the segment. When the activity ratio is below 0.2, the segment can be identified as an inactive segment and removed.

[0117] In one implementation, a low-information-content removal rule can be executed. When the content changes little across multiple consecutive frames in an operation segment, or when the operation object only undergoes slight movement without causing a change in the target object's state, the segment can be marked as a low-information-content segment. In a preferred implementation, the determination can be based on optical flow amplitude and object state changes. When the average optical flow amplitude is below 0.8 pixels / frame and the degree of target object state change is below a preset threshold, the corresponding segment can be marked as a low-information-content segment and removed from the training dataset.

[0118] In one implementation, failed and abnormal segments can be handled separately. When a segment encounters issues such as a falling target object, accidental capture, occlusion leading to inability to determine the outcome, operation interruption, or being taken over by another entity, the segment is not directly deleted. Instead, based on subsequent quality evaluation results, it is allocated to different datasets, such as a set of failed samples, a set of samples used for world model training, or a set of datasets awaiting manual review, thereby preserving its potential value in specific training tasks.

[0119] In one implementation, duplicate segment deduplication can be performed. Specifically, the visual feature representation, linguistic feature representation, and motion trajectory feature of the operation segment can be embedded, and cluster analysis can be performed on the segment based on a similarity metric. In a preferred implementation, a joint determination method of cosine similarity and dynamic time warping distance can be used, when the following condition is met: When two segments are identified as duplicates, the redundant segments can be removed to reduce homogeneous samples in the training data. This represents the similarity calculated based on visual feature embedding. This represents the similarity calculated based on text semantic embedding. This represents the distance between two trajectory sequences calculated using the dynamic time warping algorithm. Distance threshold representing trajectory similarity.

[0120] In one implementation, a long-tail retention mechanism can be introduced. For fragments that occur infrequently but have clear operational processes and well-defined changes in the target object's state, they are prioritized for retention even with a small sample size, thereby enhancing the model's learning ability for uncommon tasks. In a preferred implementation, a maximum sample retention cap and a minimum long-tail retention quantity can be set for different skill categories, thus controlling the size of popular samples while ensuring sufficient coverage of long-tail samples.

[0121] Through the aforementioned multi-dimensional data filtering and quality control mechanisms, invalid data can be effectively eliminated, the proportion of redundant samples can be reduced, and abnormal data can be made reasonable use, thereby improving the effectiveness and diversity of the training dataset and providing a high-quality data foundation for subsequent training of embodied intelligent models.

[0122] In one embodiment, the cross-body structure alignment process includes:

[0123] The grasping center and grasping orientation are determined based on the relative positional relationship between the operator and the target object. Combined with the support relationship of the target object, a structured motion representation is constructed that includes the approach direction, grasping center, grasping orientation, target object displacement, and the target placement area.

[0124] Specifically, when determining the grasping center and grasping orientation, calculations can be performed based on the relative positional relationship between the manipulator and the target object. In one implementation, the spatial distribution relationship between hand key points and the target object mask can be used to determine the main area of ​​action of the manipulator on the target object, thereby obtaining the position of the grasping center. At the same time, the orientation information during grasping can be determined by combining hand posture information and the geometric features of the target object, so as to characterize the direction in which the manipulator exerts its force on the target object.

[0125] Furthermore, when constructing a structured action representation, the key action elements during the operation process can be uniformly expressed by combining the support relationship of the target object. For example, based on the change in the support state of the target object before and after the operation, it can be determined whether it has moved from one support surface to another, and the placement target area information can be extracted accordingly. At the same time, the approach direction and displacement information of the target object can be extracted by combining the movement trajectory of the manipulator, thereby forming a structured description containing multiple key action elements.

[0126] In one implementation, the structured action representation can be used to describe the interaction process between the operator and the target object. It does not depend on the morphological parameters of the specific operator, but is expressed based on spatial relationships and state changes. Therefore, this structured action representation can be shared and mapped between different embodied forms. For example, a structured action representation containing the operator's actions on the target object can be mapped to the robot's execution space, and a corresponding executable action sequence can be generated through a subsequent action decoding process. The executable action refers to an action that satisfies the robot's kinematic constraints, collision constraints, and execution stability requirements.

[0127] Furthermore, in a preferred embodiment, the structured action representation can be standardized to ensure a consistent expression across different samples. For example, spatial coordinates can be normalized, directional information can be uniformly represented, and displacement can be scaled to improve comparability and consistency among different samples.

[0128] The above methods can achieve cross-embodied structure alignment of operation segments, transforming human operation data into a unified form of action expression, thereby providing a foundation for subsequent training data construction and model learning.

[0129] In one embodiment, constructing cross-embodied alignment training samples includes:

[0130] Establish the correspondence between the structured action representation and the robot action block;

[0131] Based on the correspondence, the structured action representation is converted into a sequence of robot-executable actions by retrieving the action block codebook and solving the inverse kinematics based on constraints.

[0132] The robot's executable action sequence is associated with the corresponding operation segment to construct the cross-embodied alignment training sample.

[0133] Specifically, when establishing the correspondence between the structured motion representation and the robot motion block, a robot motion block codebook can be pre-constructed to describe the basic motion units of the robot at different operation stages. The motion block can be represented as a motion segment with a defined function, such as approaching, grasping, moving, or placing. By analyzing information such as the approach direction, grasping center, grasping orientation, and target object displacement contained in the structured motion representation, it can be mapped to the corresponding robot motion block, thereby establishing the association between the structured motion representation and the robot motion block.

[0134] Furthermore, when converting the structured action representation into a sequence of executable actions for the robot, the action block combination that best matches the current structured action representation can be retrieved from the action block codebook based on the correspondence. In one implementation, candidate action blocks can be filtered and sorted according to the spatial constraints in the structured action representation to obtain a preliminary action sequence. Subsequently, the action sequence can be modified by solving the inverse kinematics based on the constraints to meet the physical constraints of the robot during actual execution, such as joint angle limitations, range of motion limitations, and collision constraints, thereby generating an action sequence that meets the execution requirements.

[0135] After generating the action sequence, the robot's executable action sequence can be associated with the corresponding operation segments to form cross-embodied alignment training samples. The training samples may include observation information corresponding to the operation segments and the corresponding robot action sequences, thereby establishing a mapping relationship between input and output. In one implementation, the observation information may include video frame sequences, target object state information, or task description information, while the output is a robot action sequence or action block sequence.

[0136] Through the above implementation methods, human operation data can be transformed into robot-executable actions, thereby constructing cross-embodied alignment training samples. This enables the model to learn the correspondence between operational behaviors and executed actions under a unified data representation, improving the model's applicability and generalization ability in different embodied systems.

[0137] Furthermore, in one implementation, each cross-embodied alignment training sample may include at least the following fields: task language field, observation field, hand motion field, object state field, contact map field, cross-embodied action prior field, and quality field. The task language field describes the target information of the operation task, including the target object category, initial state, and expected ending state. The observation field records visual observation information corresponding to the operation segment, including the start frame, key intermediate frames, and end frame, and optionally includes depth or point cloud information. The hand motion field describes the movement process of the manipulated body, including hand keypoint sequences, wrist pose sequences, and gripping center trajectory. The object state field describes the state changes of the target object during the operation, including changes in the detection area, segmentation mask, pose information, and relationship with the supporting structure. The contact map field represents the contact or adjacency relationships between entities during the operation. The quality field records the data quality evaluation results.

[0138] In a preferred embodiment, the cross-embodiment action prior field is used to characterize the operational intent independent of the specific manipulator's form, and is key to achieving cross-embodiment alignment. The cross-embodiment action prior may include approach direction, grasp center, grasp orientation, grasp width variation trend, target object displacement, target placement area, contact state, and operation stage information. In one implementation, the cross-embodiment action prior at each moment can be represented as:

[0139]

[0140] in, This indicates that the center position should be captured. Indicates the direction of the grab, Indicates the capture width. Indicates the displacement of the target object. Indicates the orientation of the placement. Indicates the contact status. Indicates the operation stage label. The possible values ​​are one or more of "approach, pre-grasp, grasp, transport, place, release, refine", where "approach" represents the approach phase, "pre-grasp" represents the pre-grab phase, "grasp" represents the grab phase, "transport" represents the transport phase, "place" represents the placement phase, "release" represents the release phase, and "refine" represents the refinement phase.

[0141] Furthermore, in one implementation, the hand reference coordinate system, the gripping axis between the thumb and index finger, the palm normal, and the wrist posture information can be estimated based on the key hand point information to obtain the posture expression of the manipulator in space. Combined with the relative positional relationship between the hand and the target object, candidate grasping centers and grasping directions are determined. Based on this, the size, shape, support relationship of the target object, and obstacle information in the surrounding environment can be combined to generate a corresponding cross-embodiment action prior sequence in the task space, used to describe the complete operation process from approach, grasping, transporting to placement.

[0142] In one implementation, a small amount of real robot operation data can be used to establish the correspondence between the cross-embodied action prior and the robot action block. For example, the operation intention representation and the robot action block can be aligned and trained using a contrastive learning method, or candidate action sequences can be obtained through time series matching and action block retrieval, and the actions can be corrected by combining constraint-based inverse kinematics solution, thereby generating a robot action prior that satisfies the execution constraints.

[0143] In a preferred embodiment, establishing the correspondence may include two stages: action block codebook retrieval and constraint-based inverse kinematics solution. Specifically, an action block codebook may first be constructed based on the robot's actual trajectory.

[0144]

[0145] Subsequently, one can learn mapping functions. This enables matching between embodied action priors and action blocks. In one implementation, contrastive learning can be used to train this mapping, and its loss function can be expressed as:

[0146]

[0147] in and These are the human prior and the robot action block projection matrices, respectively. The temperature parameter is used. After obtaining the initial robot motion sequence through motion block retrieval, it is then continuously corrected using constrained inverse kinematics. Its objective function can be expressed as:

[0148]

[0149] in, , , , as well as This represents the weighting coefficient of the corresponding loss term, used to adjust the degree of influence of end-effector position error, end-effector posture error, motion continuity constraint, collision constraint, and joint limit constraint on the inverse kinematics solution results; Indicates the robot's joint state The following are the positive kinematic results; Indicates the position of the target endpoint; Indicates the robot's end-effector posture; Indicates the target's end attitude; This represents the posture difference measurement function; Indicates the collision constraint penalty term; This indicates a penalty item for joint limit constraints.

[0150] Preferably, when the end position error is less than Attitude error less than An action is considered executable if the collision penalty is 0 and the joint velocity does not exceed the limit. If these conditions are not met, the executable score of the sample is reduced or it is transferred to the world model training bucket.

[0151] Finally, the generated robot-executable action sequences can be associated with the corresponding operation segments to construct cross-embodied alignment training samples.

[0152] Through the above implementation methods, while preserving the task process and interactive semantic information in human operation videos, action expressions can be abstracted into structured representations independent of specific embodied forms. This enables cross-embodied data alignment and reuse, reduces reliance on strict human-machine pairing data, and improves the flexibility of data construction and the generalization ability of model training.

[0153] In one embodiment, before training the embodied intelligence model based on the cross-embodied alignment training samples, the method further includes:

[0154] Quality assessment of cross-embodied alignment training samples was conducted, and tiered screening was performed based on the assessment results;

[0155] The embodied intelligence model is trained using the selected training samples;

[0156] New candidate samples are generated based on the trained model, and the quality of the candidate samples is evaluated and then fed back into the training dataset.

[0157] Specifically, in one implementation, multi-dimensional quality indicators can be calculated for each cross-embodied alignment training sample to comprehensively evaluate the sample quality. These quality indicators may include semantic consistency indicators, operational effectiveness indicators, executability indicators, and temporal stability indicators. The semantic consistency indicator assesses the degree of matching between the task language field and the observed field; the operational effectiveness indicator assesses whether the operational fragment reflects clear state changes and interaction processes; the executability indicator assesses whether the robot action sequence obtained from the cross-embodied action prior mapping satisfies execution constraints; and the temporal stability indicator assesses the continuity and consistency of the operation process in the time dimension. In a preferred implementation, a comprehensive quality score can be generated based on the above multi-dimensional indicators, and the cross-embodied alignment training samples can be classified accordingly.

[0158] Furthermore, the training samples can be filtered based on the grading results. In one implementation, high-quality samples can be prioritized for policy model training, medium-quality samples can be used for auxiliary training or data augmentation, and low-quality samples can be discarded or transferred to a dataset awaiting review. This approach allows for the construction of training datasets with different quality levels, thereby improving the stability and convergence efficiency of model training.

[0159] After sample selection, the embodied intelligence model can be subjected to hybrid training based on the selected training samples. In one implementation, the hybrid training can combine data from different sources, such as cross-embodied aligned training samples containing actions performed by an operator on a target object, and data from actual robot actions, thereby improving the model's adaptability to different data distributions. During training, different weights can be assigned to different samples based on their quality and source to achieve more rational data utilization.

[0160] After model training is complete, new candidate samples can be generated based on the trained model. In one implementation, the embodied intelligence model can be used to predict a given task, thereby generating a corresponding action sequence or state change process, which can then be used as candidate samples. Furthermore, a quality assessment process can be performed on the candidate samples, and samples that meet preset conditions can be selected based on the assessment results and fed back into the training dataset, thereby continuously expanding the scale of the training data and optimizing the data distribution.

[0161] Through the above implementation methods, by introducing a multi-dimensional quality assessment and hierarchical screening mechanism before training, and combining hybrid training and sample re-feedback strategies, collaborative optimization of data and model is achieved. On the one hand, by finely screening cross-embodied alignment training samples, the effectiveness and consistency of training data can be significantly improved. On the other hand, by introducing model-generated candidate samples and performing quality control before re-feeding them back into the training dataset, the data scale can be continuously expanded and the data distribution optimized, thereby improving the adaptability and generalization ability of the embodied intelligence model in complex task scenarios. In addition, this method does not rely on a large amount of strictly aligned human-machine demonstration data, reducing data collection costs and improving the flexibility and scalability of data construction.

[0162] Furthermore, in one embodiment, before training the embodied intelligence model based on the cross-embodied alignment training samples, quality assessment and hierarchical screening can be performed on the cross-embodied alignment training samples to construct a high-quality training dataset, and on this basis, iterative optimization of data generation and model training can be achieved.

[0163] Specifically, in one implementation, multi-dimensional quality metrics can be calculated for each cross-embodied alignment training sample to comprehensively evaluate the sample quality. These quality metrics may include perceptual quality score, semantic quality score, operational quality score, executability quality score, and modeling value score, where each sub-score can be normalized to an interval. Among them, the perceived quality score The quality score is used to reflect the quality of the data acquisition process, such as image clarity, occlusion level, key point tracking stability, and target object tracking stability; the semantic quality score is used to measure the consistency between task instructions and video content; the operation quality score is used to evaluate the rationality of contact relationships and state changes during operation; the executability quality score is used to evaluate whether the robot action sequence obtained from cross-embodied action prior mapping satisfies execution constraints; and the modeling value score is used to measure the potential contribution of samples to model training.

[0164] In a preferred embodiment, the perceived quality score Calculated based on metrics such as sharpness, occlusion rate, keypoint tracking stability, object tracking stability, and depth confidence.

[0165] In a preferred embodiment, a semantic consistency evaluation method can be used to assess semantic quality. The calculation can be expressed in the following form:

[0166]

[0167] in This is the original instruction text. The description generated by the visual language model from the keyframes. Keyframe visual features; It can be given by the cosine similarity of the text encoder. It can be calculated using CLIP, SigLIP, or SigLIP2. For fetching tasks, the preferred method can be selected. and If any of them is below the threshold, it will be downweighted or discarded.

[0168] In a preferred embodiment, the operation quality can be scored based on abnormal situations during the operation process. An evaluation can be expressed in the following form:

[0169]

[0170] in, Represents the counting of causeless motion. This indicates the trajectory jump count. Indicates idling count, This indicates a discrepancy between the contact relationship and the state change. If the target object experiences an instantaneous displacement exceeding 20% ​​of its frame width, and no change in the contact or support relationship between the hand and the object is detected within the preceding and following 5 frames, then it is counted as one causeless motion.

[0171] In a preferred embodiment, executability quality can be scored based on action execution constraints. An evaluation can be expressed in the following form:

[0172]

[0173] in, Indicates the accessibility score of the workspace. Indicates the degree of matching of the crawling scale. Indicates the score for the continuity of movement. This indicates the collision and safety score. If... If the sample is not selected, it will not be included in the policy training data set.

[0174] In a preferred embodiment, the modeling value is... The value of a fragment can be estimated based on its contribution to the training objective. For example, long-tailed objects, rare tasks, key failure samples, and complex contact samples can be given higher weights.

[0175] Furthermore, a comprehensive quality score can be calculated based on the above sub-scorings, and its form can be expressed as:

[0176]

[0177] in, to Each represents a sub-rating. to This indicates the corresponding weight.

[0178] In a preferred embodiment, different weights may be used for different stages. For the VLA action training set, priority is given to retaining... , , A higher sample size; for the world model training set, appropriate retention is possible. Samples with high accuracy but with failed tasks; for the data synthesis seed set, samples with complete scenes, clear state changes, and strong interpretability of actions are prioritized for retention. In a preferred embodiment, it can be set... This is to highlight the feasibility of actions and the consistency of causal contact.

[0179] In one implementation, cross-embodied alignment training samples can be categorized and screened based on the comprehensive quality score, and divided into different datasets. For example, when the comprehensive quality score and the executability quality score meet preset conditions, the samples can be assigned to the direct policy training dataset; when the comprehensive quality score is at a moderate level and the operational quality score meets the requirements, the samples can be assigned to the world model training dataset; when the comprehensive quality score is low or key information is missing, the samples can be assigned to the dataset awaiting review or further processing.

[0180] Furthermore, after completing the hierarchical screening of the cross-embodied alignment training samples, the embodied intelligence model can be subjected to hybrid training based on the screened training samples. Specifically, the model can be jointly trained by combining samples from different datasets, where high-quality samples are used for policy model training and medium-quality samples are used for auxiliary training or data augmentation, thereby improving the model's adaptability to different data distributions.

[0181] After model training is complete, new candidate samples can be generated based on the trained embodied intelligence model. Specifically, the embodied intelligence model can be used to reason about a given task, generating corresponding action sequences or state change processes, which can then be used as candidate samples. Subsequently, the candidate samples can be quality evaluated, and candidate samples that meet preset conditions can be fed back into the training dataset for subsequent model training.

[0182] Through the above implementation methods, multi-dimensional quality evaluation and refined screening of cross-embodied alignment training samples can be achieved. Combined with hybrid training and sample return mechanisms, data and model synergistic optimization can be realized, improving the utilization efficiency of training data and the generalization ability and stability of embodied intelligent models in complex task scenarios.

[0183] Furthermore, cross-embodied alignment training samples can be binned based on the comprehensive quality score, so that samples of different quality levels can serve different types of model training tasks.

[0184] Specifically, in one implementation, the cross-embodied alignment training samples can be divided into at least three datasets: a direct policy training dataset, a world model training dataset, and a dataset awaiting review or reprocessing. The direct policy training dataset is used to train or fine-tune the embodied intelligent model, requiring samples to have high action feasibility and strong semantic alignment capabilities. The world model training dataset allows the inclusion of failure segments or counterfactual segments, used to learn the evolution of environmental states, error consequences, and recovery strategies. The dataset awaiting review or reprocessing stores samples of uncertain quality but with potential value for subsequent automatic relabeling, completion labeling, or manual review.

[0185] In a preferred embodiment, the binning threshold can be set based on the overall quality score and each sub-score. For example, when the overall quality score... And the executability quality score When, the samples can be divided into the direct policy training data set; when And operational quality score When, the samples can be divided into the world model training dataset; when If a sample has missing key fields, the sample can be classified into a data set that is pending review or further processing.

[0186] Furthermore, for samples that contain failure processes but have high informational value, supplementary judgments can be made based on the observability of their state changes. In one implementation, when the observability of a sample's state is greater than a preset threshold (e.g., 0.7), it can be allowed to bypass the comprehensive quality score threshold and be directly assigned to the world model training dataset, in order to retain anomalous or failed samples that are of significant value to model learning.

[0187] The aforementioned bucketing mechanism avoids the coarse-grained processing of simply keeping or discarding samples in traditional data processing methods. It allows samples from the same data source to be allocated to different training paths based on their quality and characteristics, thereby improving data utilization efficiency and enhancing the learning and generalization capabilities of embodied intelligent models in complex scenarios.

[0188] In one embodiment, training the embodied intelligence model based on the cross-embodied alignment training samples further includes:

[0189] The world model is trained based on the cross-embodied alignment training samples, and synthetic training samples are generated using the world model.

[0190] The quality of the synthetic training samples is evaluated, and synthetic training samples that meet the preset conditions are returned to the training dataset.

[0191] The embodied intelligence model is iteratively trained based on the aforementioned training dataset;

[0192] In this training dataset, samples from different sources are dynamically matched and sampled based on the training phase, and weighted based on sample quality.

[0193] Specifically, in one implementation, the world model can be trained based on the cross-embodied alignment training samples. The world model learns the dynamic evolution of environmental states as operational behaviors change. Its inputs may include observation information, task instructions, object state descriptions, and cross-embodied action priors, and its output may include future state changes or corresponding video sequence representations. By training on selected high-quality samples, the world model can learn the mapping relationship between state changes and actions during real-world operations.

[0194] After the world model is trained, it can be used to generate synthetic training samples. In one implementation, initial states, task instructions, and action conditions can be input into the world model to generate corresponding future state sequences or video data, thereby obtaining new candidate samples. The synthetic training samples can cover different perspectives, different scene layouts, and different initial states, thus expanding the diversity of training data.

[0195] Furthermore, a quality assessment process can be performed on the synthesized training samples. In one implementation, the synthesized samples can be evaluated based on the aforementioned multi-dimensional quality assessment mechanism, and samples that meet preset conditions can be selected based on the evaluation results. For example, when a synthesized sample meets preset standards in terms of semantic consistency, operational rationality, and executability, it can be determined as a valid sample.

[0196] After completing the quality assessment, the synthetic training samples that meet the preset conditions can be fed back into the training dataset, forming an updated training dataset together with the original cross-embodied alignment training samples. Subsequently, the embodied intelligence model can be iteratively trained based on the updated training dataset, thereby improving model performance as new samples are continuously introduced.

[0197] In a preferred embodiment, the samples from different sources in the training dataset can be dynamically proportioned based on the training phase. For example, in the early stage of training, the sampling ratio of human operation samples and samples with strong semantic information can be increased to enhance the model's ability to understand task semantics; in the later stage of training, the sampling ratio of robot execution samples and fine operation samples can be increased to improve the model's ability to control execution details.

[0198] Furthermore, in one implementation, a weighted sampling strategy can be set for different samples based on sample quality. Specifically, sampling weights can be assigned to samples according to their overall quality score, so that high-quality samples have a higher probability of being selected during training, while low-quality samples have a correspondingly lower frequency of participation in training. By combining dynamic matching sampling with quality-weighted sampling, the stability and convergence efficiency of the training process can be improved while ensuring data diversity.

[0199] Through the above implementation methods, a data generation and feedback mechanism based on the world model can be realized. Combined with dynamic sampling strategies and quality weighting strategies, the training data and model capabilities can be continuously optimized, thereby improving the generalization and execution capabilities of the embodied intelligent model in complex task scenarios.

[0200] Furthermore, in one implementation, a course-based hybrid training strategy can be adopted for the vision-language-action model in the embodied intelligence model. In the early stages of training, the sampling ratio of high-level semantic segments in cross-embodied aligned training samples and real robot samples can be increased, enabling the model to prioritize learning task understanding, state change modeling, and operational intent prediction capabilities. In the later stages of training, the sampling ratio of samples containing real contact behaviors, fine operation processes, and termination control information can be increased, thereby gradually enhancing the model's control over grasping timing, placement accuracy, and action termination conditions.

[0201] Regarding the model structure, the embodied intelligence model can adopt a unified architecture including a visual encoding module, a language encoding module, and an action decoding module to achieve multimodal information fusion modeling; the world model can adopt a state prediction model based on action conditions to learn the state evolution rules during the operation process. The above model structure is a preferred implementation method, used to illustrate the feasible implementation path of this embodiment, and does not constitute a limitation on the model type.

[0202] Furthermore, in one embodiment, cross-embodied alignment training samples can be organized in a unified format to construct an input-output data structure suitable for model training. The model input may include the current observation frame sequence, task instruction information, target object state description, and a summary of historical operation segments; the model output may include a robot action block sequence, the next key state, a termination marker, and corresponding confidence information. The summary of historical operation segments can be compressed based on keyframe information, contact relationship representation, and cross-embodied action priors, thereby enhancing the model's ability to utilize long-term task context.

[0203] Furthermore, in a preferred embodiment, a multi-task joint loss function can be used to train the embodied intelligence model, which can be expressed as follows:

[0204]

[0205] in, Indicates the loss from action prediction. This represents the state prediction loss. Indicates the termination marker for predicted loss. This indicates the monitoring loss during the contact phase. , as well as This represents the weight coefficient of the corresponding loss term, used to adjust the influence of state prediction, termination marker prediction, and contact phase supervision on the model training results. Through multi-task joint optimization, the model's capabilities in action generation, state understanding, and contact relationship modeling can be improved simultaneously.

[0206] For training the world model, in one implementation, high-quality samples that have undergone segmentation, filtering, and cross-embodiment alignment can be preferentially used for training, enabling the world model to learn the contact relationships, spatial layout, and multi-step state transition rules between the target object and the environment. In a preferred embodiment, the input of the world model may include the starting keyframe, task instruction representation, object state representation, cross-embodiment action prior representation, and contact relationship representation, and its output may be the state changes at several future time steps or the corresponding observation representation. Its training loss can be expressed as:

[0207]

[0208] in, Indicates the losses incurred during reconstruction. Indicates perceived loss. This indicates a loss of consistency in actions. This indicates a loss of consistency in contact relationships. This represents the time-series smoothing loss. , , as well as This represents the weight coefficient of the corresponding loss term, used to adjust the degree of influence of perceptual consistency, action consistency, contact relationship consistency, and temporal smoothness constraints on the world model training results.

[0209] Furthermore, after the world model training is completed, the fine-tuned world model can be used to generate various forms of synthetic training samples to enhance the scene coverage and state change diversity of the training data.

[0210] In one implementation, the world model can be used to perform at least one data augmentation operation, including: view augmentation, scene augmentation, state augmentation, failure recovery augmentation, and action candidate evaluation.

[0211] Specifically, the viewpoint augmentation is used to generate operation processes under different observation viewpoints while maintaining consistency in operation semantics and state change processes, such as generating observation sequences that are closer to the robot's wrist view, robot's head view, or fixed top-down view; the scene augmentation is used to change the appearance of the target object, scene layout, lighting conditions, or background environment to improve the model's generalization ability to different scenes; the state augmentation is used to generate operation processes under different initial states, different target object arrangements, and different target object combinations; and the failure recovery augmentation is used to generate training samples that include abnormal processes such as grasping deviation, placement offset, target object occlusion, or operation mismatch, as well as corresponding recovery processes.

[0212] Furthermore, in one implementation, the world model can be used to predict the forward states of action candidates generated by the embodied intelligence model. Specifically, the action candidates output by the embodied intelligence model can be input into the world model to predict future state changes, and it can be determined whether the predicted future states are consistent with the target task requirements, thereby selecting high-value training samples that meet the task objectives.

[0213] Preferably, for candidate samples generated by the world model, action executability assessment or reverse action decoding verification can be further performed to determine whether the candidate sample can be mapped to an executable action of the robot. Only when the candidate sample satisfies the action execution constraints and passes the quality evaluation threshold is it allowed to be fed back into the training dataset for subsequent model training.

[0214] The above methods can form a closed-loop data optimization mechanism that includes data acquisition, video segmentation, cross-embodied alignment, quality evaluation, model hybrid training, sample generation, re-quality evaluation, and training data feedback, thereby achieving continuous iterative optimization of training data and model capabilities.

[0215] Furthermore, in a preferred embodiment, for candidate samples generated by the world model, action executability assessment and quality evaluation can be performed. Only samples that meet the execution constraints and pass the quality threshold screening are fed back into the training dataset, thereby forming an updated training dataset together with the original cross-embodied alignment training samples, forming a closed-loop optimization process of data construction and model training.

[0216] In one implementation, a dynamic matching sampling strategy can be used for training samples from different sources. Assume that training samples aligned across embodied models, real robot samples, and world model-generated samples are used in the training step. The sampling probabilities are respectively , and Then the following condition is met:

[0217]

[0218] In a preferred embodiment, the ratio of cross-embodied alignment training samples to robot real samples can be increased in the early stage of training to enhance task semantic learning; the ratio of synthetic samples can be gradually increased in the middle stage of training to improve scene coverage; and the ratio of robot real samples can be further increased in the later stage of training to enhance execution accuracy and stability.

[0219] Furthermore, in one implementation, a weighted sampling weight can be set based on the overall quality score of the sample. Let the sample... The quality score is Then its sampling weight can be expressed as:

[0220]

[0221] in, This represents the temperature coefficient. In a preferred embodiment, an additional weight coefficient can be introduced for failed recovery class samples to increase their sampling probability in world model training, while reducing their sampling weight in direct policy training.

[0222] Through the above implementation methods, collaborative training can be achieved across embodied alignment training samples, robot real samples, and world model generated samples. Training efficiency can be improved through dynamic ratio sampling and quality weighting mechanisms. At the same time, combined with the world model's data generation and feedback mechanism, continuous optimization of training data and model capabilities can be achieved.

[0223] The following example, using a supermarket beverage replenishment task, illustrates the method described in this application. The operator wears a first-person perspective recording device and performs beverage replenishment operations in a real supermarket environment, retrieving multiple bottles of beverage from a turnover box and placing them on designated shelf positions. The collected raw video data is approximately 6 minutes long, including multiple continuous operation steps such as walking to the target shelf, observing shelf space, grabbing beverages, moving to the target shelf, placing the beverages and adjusting their orientation, and leaving the replenishment area.

[0224] Specifically, the original video data is first subjected to basic preprocessing and coarse-grained task annotation, including time synchronization, image normalization, and task recognition based on visual and semantic information. Subsequently, the video data is segmented based on changes in the interaction between the operator and the target object, as well as changes in the target object's state. Specifically, operational activity can be constructed by combining changes in hand-object contact, beverage bottle pose, and shelf space status to characterize the degree of interaction between the operator and the target object. The trigger time of the operational event is identified based on this operational activity, and the video data is segmented into multiple operational segments accordingly. In one implementation, the operational segments can be further merged and filtered to form operational segments with clear semantics, such as "approaching the target shelf," "grabbing a beverage from a turnover box," "moving the beverage to the target shelf," "placing and correcting the label orientation," and "checking the spacing and fine-tuning," etc.

[0225] After obtaining the operation segment, its validity can be determined based on changes in interaction relationships and changes in the target object's state, and segments that do not meet preset valid operation conditions can be removed. For example, segments that only contain actions such as walking, waiting, or line-of-sight searching that do not cause changes in the target object's state can be determined as invalid operation segments and removed, thereby retaining valid operation segments that are directly related to changes in the target object's state.

[0226] Based on this, cross-embodied structured alignment processing can be performed on the effective operation segments to construct cross-embodied alignment training samples. Specifically, key hand points, segmentation masks and pose change information of the target object, shelf shelf positions, and contact relationships between the operator and the target object can be extracted during the operation process to form a multimodal structured representation. Simultaneously, the human operation process can be mapped to a cross-embodied action prior representation; for example, the grasping and placing process can be represented as an operation sequence including stages such as approach, gripping, lifting, translation, placement, and release. In one implementation, a small amount of real robot execution data can be further utilized to establish the correspondence between the cross-embodied action prior and robot action blocks, thereby generating a sequence of robot-executable actions and constructing cross-embodied alignment training samples.

[0227] Furthermore, multidimensional quality assessments can be performed on the cross-embodied alignment training samples, and hierarchical screening can be conducted based on the assessment results. In one implementation, samples with high quality scores and good executability can be assigned to the direct policy training dataset; samples containing slight operational biases but with recovery processes can be simultaneously assigned to the world model training dataset for learning abnormal states and recovery strategies.

[0228] After sample selection, the embodied intelligence model can be trained using the cross-embodied alignment training samples and a small amount of real robot data, and then combined with the world model for data generation and augmentation. Specifically, the world model can be used to generate synthetic samples under different shelf heights, beverage appearances, and empty space distributions, and these synthetic samples can be evaluated for feasibility and quality. Synthetic samples that meet preset conditions can be fed back into the training dataset for subsequent model training.

[0229] Through the above embodiments, this disclosure first achieves the goal of constructing high-quality embodied training data at low cost by utilizing scalable human operation videos and combining data processing mechanisms driven by interaction relationships and state changes. This eliminates the need to rely on expensive dedicated acquisition equipment or large-scale robot teleoperation data, significantly reducing the data acquisition threshold.

[0230] Secondly, by performing hierarchical segmentation, filtering of empty operation segments, deduplication of duplicate samples, and splitting of failed segments on long-term human videos, the data entering the training process is more focused on effective segments with clear operational semantics, contact relationships, and state changes. This effectively avoids invalid data interfering with model training and improves the effectiveness and structure of the training data.

[0231] Furthermore, by constructing cross-embodied action priors oriented towards the task space, the human operation process is abstracted into an operation intention representation independent of the specific executor. By combining a small amount of real robot data to establish a mapping relationship, an effective conversion from human operation to robot-executable actions is achieved, thereby bridging the embodied differences between humans and robots without relying on strict human-machine pairing data.

[0232] Finally, by using cross-embodied alignment training samples simultaneously for training both the embodied intelligence model and the world model, the model can not only learn action generation strategies, but also environmental state evolution and anomaly recovery mechanisms, thereby improving the model's generalization ability and execution stability in complex task scenarios.

[0233] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0234] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as cellular phones, smartphones, wearable devices, and other similar computing devices.

[0235] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 2 An exemplary structural diagram of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations. The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0236] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103 and output device 1104 may be connected by a bus or other means, as shown in the figure, which is connected by a bus.

[0237] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display, a light-emitting diode display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0238] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device (e.g., a cathode ray tube or LCD monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0239] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0240] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0241] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0242] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0243] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile optical discs or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0244] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0245] In the above embodiments, all or part of the implementation can be achieved through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the above steps or functions. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0246] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0247] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0248] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural.

[0249] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily made by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims, and the above embodiments should be regarded as exemplary and non-limiting.

Claims

1. A data construction method for embodied intelligence training, characterized in that, The method includes: Acquire video data containing the actions performed by the operator on the target object; Based on the changes in the interaction between the operator and the target object and the state changes of the target object in the video data, an operation activity level is constructed to characterize the degree of interaction between the operator and the target object. The triggering time of the operation event is identified based on the operation activity level, the interval of the operation event occurrence is determined according to the triggering time, and the video data is segmented to obtain multiple operation segments. For each operation segment, based on the change in the interaction relationship and the change in the state of the target object, it is determined whether the operation segment meets the preset valid operation conditions, wherein the valid operation conditions include at least the existence of an interaction relationship and causing a change in the state of the target object, and valid operation segments are selected. The effective operation fragments are subjected to cross-embodied structure alignment processing to construct cross-embodied alignment training samples; The embodied intelligence model is trained based on the cross-embodied alignment training samples.

2. The method as described in claim 1, characterized in that, The timing of the operation event is determined based on the establishment, maintenance, or termination of the interaction relationship between the operator and the target object, as well as the moment when the state of the target object changes.

3. The method as described in claim 1, characterized in that, The step of identifying the trigger time of the operation event based on the operation activity, determining the operation event occurrence interval based on the trigger time, and segmenting the video data specifically includes: Extract the motion intensity signal of the operator, the motion intensity signal of the target object, and the contact probability signal between the operator and the target object from the video data; Based on the motion intensity signal of the operator, the motion intensity signal of the target object, and the contact probability signal, the operational activity level, which characterizes the degree of interaction between the operator and the target object, is calculated. The peak value of the operation activity is extracted as the operation activity peak value, and the contact state switching point is determined according to the contact probability signal; The video data is divided into multiple atomic motion segments by using the peak activity level, the contact state switching point, and the state change point of the target object as candidate segmentation boundaries.

4. The method as described in claim 3, characterized in that, The video data is segmented to obtain multiple operation segments, specifically including: By comparing adjacent atomic action segments, the continuity characteristics of the target object and the consistency characteristics of the operation target are obtained; Based on the continuity characteristics of the target object and the consistency characteristics of the operation target, adjacent atomic action segments are merged into skill segments; The operation segment is constructed based on the temporal correlation, target object correlation, and state change continuity among the multiple skill segments.

5. The method as described in claim 1, characterized in that, The step of determining whether the operation segment is a valid operation segment based on the changes in the interaction relationship and the state changes of the target object includes: Identify the contact relationship between the operating body and the target object in the operation segment, and extract the state change amount of the target object; If the contact relationship is not detected and the state change is lower than a preset threshold, the operation segment is determined to be an invalid operation segment. If the preset activity level condition is met and the operation segment is not determined to be invalid, the operation segment is determined to be a valid operation segment.

6. The method as described in claim 1, characterized in that, The cross-body structured alignment process includes: The grasping center and grasping orientation are determined based on the relative positional relationship between the operator and the target object. Combined with the support relationship of the target object, a structured motion representation is constructed that includes the approach direction, grasping center, grasping orientation, target object displacement, and the target placement area.

7. The method as described in claim 6, characterized in that, The construction of cross-embodied alignment training samples includes: Establish the correspondence between the structured action representation and the robot action block; Based on the correspondence, the structured action representation is converted into a sequence of robot-executable actions by retrieving the action block codebook and solving the inverse kinematics based on constraints. The robot's executable action sequence is associated with the corresponding operation segment to construct the cross-embodied alignment training sample.

8. The method as described in claim 1, characterized in that, Before training the embodied intelligence model based on the cross-embodied alignment training samples, the method further includes: Quality assessment of cross-embodied alignment training samples was conducted, and tiered screening was performed based on the assessment results; The embodied intelligence model is trained using the selected training samples; New candidate samples are generated based on the trained model, and the quality of the candidate samples is evaluated and then fed back into the training dataset.

9. The method as described in claim 1, characterized in that, The training of the embodied intelligence model based on the cross-embodied alignment training samples also includes: The world model is trained based on the cross-embodied alignment training samples, and synthetic training samples are generated using the world model. The quality of the synthetic training samples is evaluated, and synthetic training samples that meet the preset conditions are returned to the training dataset. The embodied intelligence model is iteratively trained based on the aforementioned training dataset; In this training dataset, samples from different sources are dynamically matched and sampled based on the training phase, and weighted based on sample quality.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method as described in any one of claims 1 to 9.

11. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

Images