Method and device for constructing image prediction samples based on multi-source real interaction data

Abstract

The application provides a method and device for constructing image prediction samples based on multi-source real interaction data, and relates to the technical field of artificial intelligence. The method for constructing image prediction samples based on multi-source real interaction data comprises the following steps: acquiring multi-source real interaction data; extracting candidate prediction samples from each piece of interaction data of the multi-source real interaction data; in an initial state image, a recognizable visual feature of a target object is presented, and the target object is not associated with an interaction action indicated by a target interaction instruction; in a target state image, the recognizable visual feature of the target object is presented, and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is an object acted on by the interaction action; and target prediction samples are obtained based on the candidate prediction samples. The application can obtain logically valid image prediction evaluation samples from multi-source data composed of robot operation data and first perspective interaction data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for constructing image prediction samples based on multi-source real interactive data. Background Technology

[0002] With the rapid development of image generation models and world models, evaluating their ability to predict subsequent visual states based on current visual states and verbal instructions has become a critical requirement. Current mainstream image editing datasets are mostly synthesized from static images and text editing instructions; the target images often only reflect superficial modifications, making it difficult to verify the model's understanding of state transitions caused by real physical actions. Meanwhile, real robot operation data or first-person perspective interaction data are typically stored in the form of long videos, trajectories, or clips. These multi-source datasets exhibit significant differences in frame rate, timestamp accuracy, instruction granularity, shooting angle, and annotation fields.

[0003] In related technologies, randomly extracting consecutive frames as state pairs from real robot operation data or first-person perspective interaction data cannot reflect the true causal time sequence. Furthermore, relying solely on pixel-level differences to filter data cannot accurately determine the effective state transitions of the target object.

[0004] Therefore, how to obtain logically valid image prediction and evaluation samples from multi-source data consisting of robot operation data and first-person perspective interaction data is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a method and apparatus for constructing image prediction samples based on multi-source real interaction data, in order to solve the above-mentioned defects in the prior art, and to obtain logically valid image prediction evaluation samples from multi-source data composed of robot operation data and first-person perspective interaction data.

[0006] This application provides a method for constructing image prediction samples based on multi-source real interaction data, including the following steps.

[0007] Acquire multi-source real-world interaction data; extract candidate prediction samples from each interaction data point of the multi-source real-world interaction data; wherein, the candidate prediction samples include a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image; in the initial state image, identifiable visual features of the target object are presented, and the target object is not associated with the interaction action indicated by the target interaction instruction; in the target state image, identifiable visual features of the target object are presented, and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is the object acted upon by the interaction action; based on the candidate prediction samples, obtain target prediction samples, and use the target prediction samples to train or evaluate the target task execution capability of the embodied intelligent agent.

[0008] According to the method provided in this application for constructing image prediction samples based on multi-source real interaction data, the interaction data includes multiple frames of images carrying timestamps and interaction instruction information; the step of extracting candidate prediction samples from each interaction data of the multi-source real interaction data includes: for each interaction data, performing the following operations to obtain the candidate prediction sample corresponding to the interaction data: determining an action pre-window and a result stabilization window based on the temporal information of the multi-frame images and the interaction state of the interactive subject and the target object in each frame image; wherein, the action pre-window is a time window in which the interactive subject has not yet executed the interaction action; the result stabilization window is a time window in which the execution result of the interaction action is displayed; the temporal information of the multi-frame images is determined by the timestamps carried by each frame image; selecting a first image in the action pre-window from the multi-frame images as the initial state image; selecting a second image in the result stabilization window from the multi-frame images as the target state image; and performing structured processing on the interaction instruction information to obtain the target interaction instruction.

[0009] According to the method for constructing image prediction samples based on multi-source real interaction data provided in this application, the action pre-window is determined in the following manner: based on the temporal information of the multi-frame images and the interaction state of the interactive subject and the target object in each frame image, the start time of the interactive action is determined based on the detected interaction temporal clues; wherein, the interaction temporal clues include at least one of the following: the start information of the action, the contact trigger information between the interactive execution component and the target object, the target object state change information, and the multimodal event recognition information for the interactive execution component; wherein, the interactive execution component includes a human hand, an end effector of an embodied intelligent agent, or a tool controlled by a human or the embodied intelligent agent; the time interval in the time interval corresponding to the multi-frame images that is before the start time of the interactive action is determined as the action pre-window.

[0010] According to the method for constructing image prediction samples based on multi-source real-world interactive data provided in this application, the stable window of the result is determined in the following manner: The moment when the change in the interaction result caused by the interaction action first exceeds a preset change threshold, starting from the start time of the interaction action, is taken as the moment when the interaction result is displayed. After the moment when the interaction result is displayed, it is determined whether the interaction result remains stable in a preset number of consecutive image frames. If so, the continuous time interval corresponding to the preset number of consecutive image frames is determined as the result stabilization window.

[0011] According to the method for constructing image prediction samples based on multi-source real-world interaction data provided in this application, the step of obtaining a target prediction sample based on the candidate prediction samples includes: The candidate prediction samples are subjected to at least one type of screening; based on the results of the at least one type of screening, a target prediction sample is determined from the candidate prediction samples; wherein, the at least one type of screening includes temporal validity screening, interaction state change significance screening, and real interaction constraint screening; the temporal validity screening is used to verify whether the temporal information between the initial state image and the target state image in the candidate prediction samples is reasonable; the interaction state change significance screening is used to eliminate candidate prediction samples whose state changes of the target object are not significant; the real interaction constraint screening is used to evaluate whether the change of the target state image relative to the initial state image conforms to the realism constraint.

[0012] According to the method for constructing image prediction samples based on multi-source real interaction data provided in this application, the candidate prediction samples are determined to pass the temporal legality screening in the following way: for each candidate prediction sample containing an initial state image and a target state image, if the acquisition time of the initial state image is earlier than the start time of the interaction action, and the acquisition time of the target state image is not earlier than the display time of the interaction result, then the candidate prediction sample is determined to pass the temporal legality screening.

[0013] According to the method for constructing image prediction samples based on multi-source real interaction data provided in this application, the following steps are taken to determine whether the candidate prediction sample passes the interaction state change significance screening: For the initial state image and the target state image in each candidate prediction sample, the following operations are performed: In the initial state image, a first object region where the target object is located and a first interaction region where both the target object and the interaction execution component exist simultaneously are determined; In the target state image, a second object region where the target object is located and a second interaction region where both the target object and the interaction execution component exist simultaneously are determined; A first interaction state change amount is determined; wherein, the first interaction state change amount is determined based on at least one of a first perceptual difference amount between the first interaction region and the second interaction region, a first attribute change amount of the target object, and a first relationship change amount of the target object; A second interaction state change amount is determined; wherein, the second interaction state change amount is determined based on at least one of a second perceptual difference amount between the first object region and the second object region, a second attribute change amount of the target object, and a second relationship change amount of the target object; Based on the first interaction state change amount and / or the second interaction state change amount, it is determined whether the candidate prediction sample passes the interaction state change significance screening.

[0014] According to the method for constructing image prediction samples based on multi-source real interaction data provided in this application, the step of determining whether the candidate prediction sample passes the interaction state change significance screening based on the first interaction state change amount and / or the second interaction state change amount includes: if the global pixel change amount between the initial state image and the target state image is greater than a preset global threshold and satisfies a preset judgment condition, then it is determined that the candidate prediction sample has not passed the interaction state change significance screening; wherein, the preset judgment condition includes at least one of the following: the first perceptual difference amount and / or the second perceptual difference amount does not exceed a preset perceptual difference threshold; the first attribute change amount and / or the second attribute change amount does not exceed a preset attribute change threshold; the first relationship change amount and / or the second relationship change amount does not exceed a preset relationship change threshold.

[0015] According to the method for constructing image prediction samples based on multi-source real interaction data provided in this application, the step of determining whether the candidate prediction sample passes the interaction state change significance screening based on the first interaction state change amount and / or the second interaction state change amount includes: determining a first weighted result of the first perceptual difference amount, the first attribute change amount, and the first relationship change amount; determining a second weighted result of the second perceptual difference amount, the second attribute change amount, and the second relationship change amount; and determining whether the candidate prediction sample passes the interaction state change significance screening based on the comparison result of the first weighted result and a first change significance threshold, and / or the comparison result of the second weighted result and a second change significance threshold.

[0016] According to the method for constructing image prediction samples based on multi-source real interaction data provided in this application, the following steps are taken to determine whether a candidate prediction sample passes the real interaction constraint screening: For each candidate prediction sample, multiple review models are used to evaluate the real interaction constraints from at least one dimension, and the real interaction constraint scores of each review model for the candidate prediction sample in the at least one dimension are obtained; For each candidate prediction sample, based on its real interaction constraint scores in the at least one dimension, it is determined whether the candidate prediction sample passes the real interaction constraint screening; wherein, the real interaction constraint scores in the at least one dimension include at least one of: action execution compliance score, scene consistency score, and physical rationality score; wherein: the action execution compliance score is used to characterize whether the target state image presents the expected result corresponding to the target interaction command; the scene consistency score is used to characterize whether the interaction subject, the background of the image, and the shooting scene corresponding to the image have undergone unreasonable changes; the physical rationality score is used to characterize whether the state change of the target object conforms to the physical constraint conditions.

[0017] According to the method for constructing image prediction samples based on multi-source real interaction data provided in this application, the step of determining whether a candidate prediction sample passes the real interaction constraint screening based on its real interaction constraint score in at least one dimension for each candidate prediction sample includes: for each candidate prediction sample, performing a weighted average aggregation of its real interaction constraint scores from multiple review models in each dimension to obtain a dimension aggregation score for the candidate prediction sample in that dimension; for each candidate prediction sample, obtaining a dimension divergence degree in each dimension based on the dispersion of its real interaction constraint scores in each dimension; for each candidate prediction sample, obtaining a comprehensive quality score based on the weighted sum of its corresponding dimension aggregation scores; for each candidate prediction sample, obtaining a review divergence degree based on the weighted sum of its corresponding dimension divergence degrees; if the comprehensive quality score reaches a quality threshold, and the dimension aggregation scores corresponding to each dimension all reach their respective dimension thresholds, and the review divergence degree does not exceed the divergence threshold, then the candidate prediction sample is determined to pass the real interaction constraint screening.

[0018] This application also provides an apparatus for constructing image prediction samples based on multi-source real interaction data, comprising the following modules: The first acquisition module is used to acquire multi-source real interaction data; the extraction module is used to extract candidate prediction samples from each interaction data of the multi-source real interaction data; wherein, the candidate prediction sample includes a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image; in the initial state image, the identifiable visual features of the target object are presented and the target object is not associated with the interaction action indicated by the target interaction instruction; in the target state image, the identifiable visual features of the target object are presented and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is the object to which the interaction action is applied; the second acquisition module is used to obtain a target prediction sample based on the candidate prediction sample, so as to use the target prediction sample to train or evaluate the target task execution capability of the embodied intelligent agent.

[0019] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method for constructing image prediction samples based on multi-source real interactive data as described above.

[0020] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing image prediction samples based on multi-source real interactive data as described above.

[0021] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method for constructing image prediction samples based on multi-source real interactive data as described above.

[0022] This application provides a method and apparatus for constructing image prediction samples based on multi-source real interaction data. The method acquires multi-source real interaction data and extracts candidate prediction samples from each interaction data point within the multi-source real interaction data. Each candidate prediction sample includes a target interaction command and an initial state image and a target state image associated with that command. The initial state image presents identifiable visual features of the target object but has not yet executed an interaction action, while the target state image presents identifiable visual features of the target object and has completed an interaction action. Through unified access to multi-source data and precise anchoring of the interaction time sequence, a clear causal relationship is established between the extracted initial state image, the target interaction command, and the target state image. This completely avoids the problem of arbitrarily extracting frames from long interaction sequences, where the initial frame already contains the action result or the target frame has not yet presented a stable result. This provides a temporal foundation for subsequently generating high-quality prediction samples. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the method for constructing image prediction samples based on multi-source real-world interactive data provided in this application.

[0025] Figure 2 This is a flowchart illustrating the method for obtaining target prediction samples based on candidate prediction samples provided in this application.

[0026] Figure 3 This is a schematic diagram of the candidate prediction sample construction process provided in this application.

[0027] Figure 4 This is a schematic diagram illustrating the process of hierarchical screening of candidate prediction samples provided in this application.

[0028] Figure 5 This is a schematic diagram of the fusion evaluation process using the review model provided in this application.

[0029] Figure 6 This is a schematic diagram of the training sample collection for candidate prediction samples based on the coverage balancing strategy provided in this application.

[0030] Figure 7 This is a schematic diagram of the device provided in this application for constructing image prediction samples based on multi-source real interactive data.

[0031] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] The following is combined with Figures 1-8 This application describes a method for constructing image prediction samples based on multi-source real-world interactive data.

[0034] Figure 1 This is a flowchart illustrating the method for constructing image prediction samples based on multi-source real-world interaction data provided in this application. Figure 1 As shown, the method includes the following: Step 101: Obtain real interaction data from multiple sources.

[0035] Multi-source real-world interaction data is data obtained from different data collection subjects and from different perspectives, recording the real interaction process between the interaction subject and the interaction object.

[0036] In practice, various forms of interactive data can be collected from different acquisition environments to obtain multi-source real-world interactive data, without being limited by the description in this specification. For example, multi-source real-world interactive data can be collected in the real physical world. Alternatively, multi-source real-world interactive data can be collected in a virtual simulation environment or a mixed reality environment.

[0037] For example, such as Figure 3 As shown, robot interaction data can be obtained from the onboard camera of a real embodied intelligent robot, first-person perspective interaction sequences of users performing daily housework can be obtained from wearable first-person perspective acquisition devices, human-object interaction data in human-machine collaborative operation scenarios can be obtained from a dedicated human-object interaction acquisition workstation, and global perspective interaction sequences can be obtained from fixed monitoring cameras deployed in the corners of the scene.

[0038] In the specific implementation process, each piece of interaction data in the multi-source real-world interaction data includes at least one of the following: action annotation, task description, timestamp, or trajectory-level metadata. A unified field mapping rule is established for interaction data from all different sources, converting the original frame time, trajectory identifier, camera identifier, action label, object category, and original interaction instructions from different data sources into a unified intermediate format, while retaining the original source identifier and data storage path. This ensures that the format of all data is compatible and traceable in subsequent processing, resulting in multi-source real-world interaction data.

[0039] Step 102: Extract candidate prediction samples from each interaction data point of the multi-source real interaction data.

[0040] Interactive data refers to various forms of data that characterize the interaction process between the interactive subject and the target object. In specific implementation, interactive data can include temporally continuous interactive sequence data, such as continuous video streams or time-series action logs; interactive data can also include temporally non-continuous discrete interactive data, such as keyframe combinations, multimodal discrete recordings, etc.

[0041] In some embodiments, the candidate prediction sample includes a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image.

[0042] An initial state image is an image frame that clearly reflects the initial state of the target object before the interactive action indicated by the target interaction command occurs. In the initial state image, the identifiable visual features of the target object are presented, and the target object is not associated with the interactive action indicated by the target interaction command.

[0043] The target object is the object on which the interactive action is performed. The interactive object can include, but is not limited to, physical entities, virtual objects, or optically projected objects. For example, the target object could be an embodied robot picking up a glass of water in a kitchen environment. Another example is a virtual apple grasped by an embodied agent in a simulation engine. Yet another example is a virtual keyboard projected onto a table by a projector and tapped by an embodied agent.

[0044] The target interaction command can carry one or more target objects at the same time.

[0045] As an example only, let's take the target interaction instruction "Put the plate of shrimp into the microwave oven" as an example. The interaction action is to put it in, the plate of shrimp is the object of this interaction action, and the microwave oven is the target container of this interaction action. Both are target objects.

[0046] A target state image is an image frame that clearly reflects the final state of a target object after an interactive action has been completed. The target state image presents the identifiable visual features of the target object, and the target object is associated with the interactive action indicated by the target interaction command.

[0047] Distinguishing visual features are information contained in an initial state image or a target state image that characterizes the existence state of a target object. Distinguishing visual features can be manifested as the overall presentation of the target object or the presentation of key local features; even if the target object is partially occluded, as long as the image presents distinctive local visual information about the target object, such as specific color areas of the object, unique texture patterns, specific geometric edges, key points, or the outlines of functional components, it is considered to present distinguishing visual features.

[0048] For example, in the candidate prediction samples that execute the "open drawer" instruction, the initial state image shows a closed drawer; the target state image shows the same drawer in an open state.

[0049] The interaction data includes multiple frames of images with timestamps and interaction command information.

[0050] Target interaction instructions are standardized instructions used to describe interaction actions, obtained by structuring the interaction instruction information carried by the interaction data.

[0051] Interaction instruction information is the raw data carried by the interaction data to describe the interaction task or action. Interaction instruction information includes at least one of the following: natural language instructions, action tags, task descriptions, or trajectory-level metadata. Natural language instructions refer to statements describing the interaction goal in unstructured text form, such as "put the cup on the table." Action tags are labels indicating the type, start, and end times of key actions during the interaction; for example, in a video of a robot grasping an object, the action tags could be the grasping action and its corresponding frame range. Task descriptions are general descriptions of the interaction task goal, such as "pour water." Trajectory-level metadata refers to additional information related to the motion trajectory of the interaction subject, such as the three-dimensional coordinates, velocity, and acceleration of the end effector.

[0052] In the specific implementation process, the interactive instruction information can be structured in the following way: identify at least one of the following: interactive subject, interactive action, target object, spatial relationship between interactive subject and target object, and state change of target object, and generate standardized instructions with a unified grammatical structure. For example, the original natural language instruction "Please bring me that red cup" can be standardized into a structured target interactive instruction: <action: grab>, <object: red cup>.

[0053] In some embodiments, for each piece of interaction data, the following operations are performed to obtain candidate prediction samples corresponding to the interaction data: determining the pre-action window and the result stabilization window based on the temporal information of multiple frames of images and the interaction state of the interactive subject and the target object in each frame of images; selecting the first image in the pre-action window from the multiple frames of images as the initial state image; selecting the second image in the result stabilization window from the multiple frames of images as the target state image; and performing structured processing on the interaction instruction information to obtain the target interaction instruction.

[0054] An interactive subject is an entity that can actively perceive the state of the environment, generate interactive intentions, and perform corresponding actions during the interaction process in order to exchange information and provide physical feedback with other entities. Interactive subjects may include humans or embodied intelligent agents.

[0055] The pre-action window is the time window during which the interactive subject has not yet performed an interactive action. For example, for the target interactive instruction of "pouring water from the kettle into the cup", the pre-action window is the time interval during which the robotic arm or human hand has not yet started the tilting action of the kettle. During this interval, the kettle always remains upright and no new water is injected into the cup.

[0056] The result stability window is a time window that displays the execution result of the interactive action. For example, in the water pouring scenario above, the result stability window is a continuous time interval during which the liquid level in the cup remains stable and no longer changes after the water flow stops pouring. Within this interval, the final result of the interactive action is clear and identifiable and there will be no blurry state due to dynamic transition.

[0057] The timing information of multi-frame images is determined by the timestamps carried by each frame, and can be calibrated according to the order of acquisition of each image frame.

[0058] The first image is the image frame that ultimately serves as the initial state image in the candidate prediction samples. This image frame needs to meet the following requirements: the target object is clearly visible within the action pre-window, and it is completely unrelated to the interaction action corresponding to the target interaction command. In the specific implementation process, invalid images that have already shown the execution result corresponding to the target interaction command in the action pre-window or where the target object is occluded are excluded. Then, the first image that meets the requirements is selected from the multiple frames contained in the action pre-window according to the priority of the highest image clarity and the complete unoccluded visibility of the target object.

[0059] The second image is the image frame that ultimately serves as the target state image in the candidate prediction samples. This image frame needs to meet the following requirements: it must be located within the result stability window after the interaction result is displayed, the target object must be clearly visible, and the interaction action corresponding to the target interaction command must have been completed. In the specific implementation process, the rate of state change between consecutive images within the result stability window is calculated frame by frame. Images with a rate of state change less than a preset stability threshold and whose execution results corresponding to the target interaction command are clearly identifiable are selected. The second image that meets the requirements is chosen from these images. The second image does not contain dynamic transition frames or motion-blurred frames during the action execution process and can stably present the final state after the interaction action has been completed.

[0060] In the specific implementation process, the pre-action window is determined in the following ways: Based on the temporal information of multiple frames and the interaction state of the interactive subject and the target object in each frame, the start time of the interactive action is determined based on the detected interaction temporal clues; the time interval in the time interval corresponding to the multiple frames that is before the start time of the interactive action is determined as the action pre-window.

[0061] Interaction timing cues refer to visual, tactile, or control signals that mark the formal initiation of an interactive action. Interaction timing cues include at least one of the following: initiation information of the action, contact trigger information between the interactive execution component and the target object, state change information of the target object, and multimodal event recognition information for the interactive execution component.

[0062] For example, the "start moving" command issued by the robot controller, the sudden change in the force sensor reading at the moment the robotic gripper makes contact with the object, or the visual detection of a human hand reaching towards the target object can all serve as clues to the timing of the interaction.

[0063] Interactive execution components can include a human hand, an end effector of an embodied intelligent agent, or a tool controlled by a human or embodied intelligent agent. This tool can include, but is not limited to, handheld tools such as screwdrivers and chopsticks; and wearable tools such as exoskeletons and haptic gloves.

[0064] In the specific implementation process, the result stability window is determined in the following ways: The moment when the change in the interaction result caused by the interaction action first exceeds a preset change threshold is determined as the moment when the interaction result is displayed. After the moment when the interaction result is displayed, it is determined whether the interaction result remains stable in a preset number of consecutive image frames. If so, the continuous time interval corresponding to the preset number of consecutive image frames is determined as the result stabilization window.

[0065] In the specific implementation process, the moment when the change in the interaction result caused by the interaction action first exceeds the preset change threshold can be determined through the following multiple dimensions: Object attribute dimension: Detects whether the appearance, shape, and other attributes of the target object have changed in relation to the interactive action.

[0066] Location dimension: Detects whether the spatial position of the target object changes in relation to the interactive action.

[0067] Container Relationship Dimension: Detects whether the containment relationship between the target object and the container changes in relation to the interaction action.

[0068] Open / Closed State Dimension: Detect whether the open / closed state of target objects such as doors, lids, and switches changes in relation to interactive actions.

[0069] Contact Relationship Dimension: Detects the contact state between the end effector of the hand or robotic arm and the target object, or whether the contact relationship of the target object changes in relation to the interactive action.

[0070] Scene state dimension: Detects whether the state of a local scene changes under the influence of interactive actions.

[0071] If a change in the interaction result caused by an interaction action is detected to exceed a preset change threshold for the first time in at least one of the above dimensions, that moment can be determined as the moment when the interaction result is displayed.

[0072] As an example, taking the target interaction command "put the plate containing shrimp into the microwave oven" as an example, after the robotic arm starts moving the plate, the moment when the plate is detected to be completely inside the microwave oven cavity for the first time is determined as the moment when the interaction result is displayed; subsequently, when the position of the plate no longer changes and the relative container relationship between the plate and the microwave oven remains in the state of internal containment in multiple consecutive frames of images, this continuous time interval is regarded as the result stability window.

[0073] As an example only, for the target interaction instruction "put the water cup into the storage box", the moment when the change in the interaction result caused by the interaction action first exceeds a preset change threshold can be determined in the following way: Location dimension: The spatial coordinates of the water cup are detected to be displaced, and the displacement trajectory points to the inside of the storage box. When the displacement distance exceeds a preset threshold, the time when the interaction result is displayed is determined.

[0074] Container relationship dimension: When the water cup changes from being outside the storage box to being inside the storage box, the interaction result is displayed when the state change continues to exceed a preset threshold.

[0075] Contact relationship dimension: The system detects that the contact state between the robotic arm end effector and the water cup changes from "grabbing" to "releasing". When the release action is completed, the system determines when the interaction result is displayed.

[0076] Step 103: Based on the candidate prediction samples, obtain the target prediction samples.

[0077] Target prediction samples are used to train or evaluate the target task performance capabilities of embodied agents.

[0078] A target task is a series of integrated operations involving perception, planning, decision-making, and action control that an embodied intelligent agent needs to perform autonomously or semi-autonomously in a physical environment in order to achieve a specific target state or complete a specific interaction intention.

[0079] As an example only, the target task could be a housekeeping task in a home service scenario, a parts delivery task in an industrial production scenario, or a medicine delivery task in a medical care scenario.

[0080] The target prediction samples can not only be used to train or evaluate the state prediction ability and physical law compliance of the world model carried by the embodied agent in real interaction scenarios, but also as benchmark data for training or evaluating the task planning and interaction execution ability of the embodied agent.

[0081] In practice, target prediction samples can be obtained from candidate prediction samples in various ways, without being limited by the description in this specification.

[0082] For an example of obtaining the target prediction sample based on the candidate prediction sample, please refer to the following. Figure 2 The relevant content will not be repeated here.

[0083] Figure 2 This is a flowchart illustrating the method for obtaining target prediction samples based on candidate prediction samples provided in this application. Figure 2 As shown, the method includes the following: Step 201: Perform at least one type of screening on the candidate prediction samples.

[0084] At least one type of filtering includes time sequence legality filtering, interaction state change significance filtering, and real interaction constraint filtering.

[0085] Temporal validity screening is used to verify whether the temporal information of the initial state image and the target state image in the candidate prediction sample is reasonable.

[0086] In some embodiments, the candidate prediction sample can be determined to pass the temporal legality screening in the following way: for each candidate prediction sample containing an initial state image and a target state image, if the acquisition time of the initial state image is earlier than the start time of the interaction action, and the acquisition time of the target state image is not earlier than the time when the interaction result is displayed, then the candidate prediction sample is determined to pass the temporal legality screening.

[0087] As an example only, for the target interactive instruction "put the plate containing shrimp into the microwave oven", we verify whether the acquisition time of the initial state image is earlier than the start time of the placement action, and whether the acquisition time of the target state image is not earlier than the time when the plate is completely inside the microwave oven cavity. This effectively avoids the time sequence reversal of the execution result corresponding to the target interactive instruction in the initial state image and the target state image not showing a stable execution result.

[0088] In the specific implementation process, the candidate prediction samples are first subjected to basic validity screening, and four verification tasks are completed in sequence: checking the decodability of image files in the candidate prediction samples, verifying the integrity of metadata fields associated with the candidate prediction samples and the non-emptiness of interactive instructions, evaluating the image clarity and exposure, ensuring that the image quality meets the preset standards, detecting the visibility of the target object in the image, and removing samples where the target object is severely occluded.

[0089] For all candidate prediction samples that pass the basic validity screening, a temporal validity screening is performed. For the initial state image and target state image contained in each candidate prediction sample, the logical relationship between their acquisition time and the interaction event time is verified. The candidate prediction sample is determined to pass the temporal validity screening only if all three conditions are met: the acquisition time of the initial state image is earlier than the start time of the interaction action, the acquisition time of the target state image is not earlier than the time when the interaction result is displayed, and the initial state image and target state image originate from the same interaction trajectory or are confirmed to belong to the same scene.

[0090] After temporal validity screening, near-duplicate screening can be performed. This involves calculating the similarity between candidate prediction samples using methods such as perceptual hashing, visual embedding features, and interactive semantic embedding features, removing near-duplicate samples that are completely identical to or highly similar to existing candidate prediction samples in the candidate prediction sample library. Finally, the initial screening process is complete: all candidate prediction samples that fail any of the aforementioned screening steps are directly eliminated; samples with only missing information but potential for repair are individually marked and sent to the corresponding backflow queue for further targeted processing.

[0091] The interaction state change significance screening is used to eliminate candidate prediction samples whose state changes of the target object are not significant.

[0092] For example, if the initial state image and target state image of a candidate prediction sample only contain global pixel changes caused by camera shake, scene lighting changes, or the passing of irrelevant people, and the target object itself does not produce a state change corresponding to the target interaction command, the candidate prediction sample will be directly eliminated by the interaction state change significance filter to avoid misjudging irrelevant visual disturbances as valid execution results.

[0093] Before filtering the significance of interactive state changes, for interactive data collected from motion cameras, it is necessary to estimate geometric transformation using static background features and align the initial state image and the target state image.

[0094] In some embodiments, the filtering of the significance of interaction state changes includes: For each candidate prediction sample, in both the initial state image and the target state image, the following operations are performed: In the initial state image, a first object region where the target object is located, and a first interaction region where both the target object and the interaction execution component exist simultaneously are determined; In the target state image, a second object region where the target object is located, and a second interaction region where both the target object and the interaction execution component exist simultaneously are determined; A first interaction state change amount is determined; wherein the first interaction state change amount is determined based on at least one of a first perceptual difference amount within the first interaction region and the second interaction region, a first attribute change amount of the target object, and a first relationship change amount of the target object; A second interaction state change amount is determined; wherein the second interaction state change amount is determined based on at least one of a second perceptual difference amount within the first object region and the second object region, a second attribute change amount of the target object, and a second relationship change amount of the target object; Based on the first interaction state change amount and / or the second interaction state change amount, it is determined whether the candidate prediction sample passes the interaction state change significance screening.

[0095] The first perceptual difference measure is used to quantify the degree of visual appearance change between the first interaction area and the second interaction area, and can be obtained by calculating the pixel-level difference between the first interaction area and the second interaction area.

[0096] The second perceptual difference measure is used to quantify the degree of visual appearance change between the first object region and the second object region, and can be obtained by calculating the pixel-level difference between the first object region and the second object region.

[0097] The change in the first attribute of the target object is the change in the target object's inherent physical or semantic attributes, such as shape, color, state, temperature, etc., between the first and second interaction areas.

[0098] The change in the second attribute of the target object is the change in the target object's inherent physical or semantic attributes, such as shape, color, state, temperature, etc., between the first object region and the second object region.

[0099] The first relationship change of the target objects is the change in the spatial positional relationship, contact relationship or logical subordination relationship between the target objects or between the target objects and the environment between the first interaction area and the second interaction area.

[0100] The second relational change of the target object is the change in the spatial positional relationship, contact relationship or logical subordination relationship between the target objects or between the target object and the environment between the first object region and the second object region.

[0101] In practice, various methods can be used to determine whether a candidate prediction sample passes the screening based on the significance of the interaction state change, based on the change amount of the first interaction state and / or the change amount of the second interaction state, without being limited by the description in this specification.

[0102] In some embodiments, if the global pixel change between the initial state image and the target state image is greater than a preset global threshold and a preset judgment condition is met, then the candidate prediction sample is determined to have failed the interaction state change significance screening; wherein, the preset judgment condition includes at least one of the following: the first perceptual difference and / or the second perceptual difference does not exceed a preset perceptual difference threshold; the first attribute change and / or the second attribute change does not exceed a preset attribute change threshold; the first relationship change and / or the second relationship change does not exceed a preset relationship change threshold.

[0103] In the candidate prediction samples, the global pixel change between the initial state image and the target state image is greater than the preset global threshold, indicating that the global picture of the target state image has changed significantly relative to the initial state image. However, the perceptual effect, self-attribute, and interrelationship of the target objects in the interaction area have not changed significantly as expected. Therefore, the change of the target state image relative to the initial state image is caused by changes that are completely unrelated to the target interaction command, such as camera pose shift and background scene flow in the shooting scene. By using the global pixel change and preset judgment conditions, the interference of such pseudo-valid samples can be accurately eliminated.

[0104] In some embodiments, a first weighted result of a first perceived difference, a first attribute change, and a first relationship change is determined; a second weighted result of a second perceived difference, a second attribute change, and a second relationship change is determined; and based on a comparison of the first weighted result with a first change significance threshold, and / or a comparison of the second weighted result with a second change significance threshold, it is determined whether the candidate prediction sample passes the interaction state change significance screening.

[0105] Realistic interaction constraint screening is used to evaluate whether the changes in the target state image relative to the initial state image meet realistic constraints.

[0106] Realism constraints are the conditions under which interactive actions can be implemented in the real physical world and the results of the interaction conform to physical laws and common sense.

[0107] In some embodiments, it is determined whether a candidate prediction sample passes the real interaction constraint screening by: for each candidate prediction sample, using multiple review models to evaluate the real interaction constraints from at least one dimension, and obtaining the real interaction constraint scores of each review model for the candidate prediction sample in at least one dimension; for each candidate prediction sample, it is determined whether the candidate prediction sample passes the real interaction constraint screening based on its real interaction constraint scores in at least one dimension, wherein the real interaction constraint scores in at least one dimension may include at least one of the following scores: action execution compliance score, scene consistency score, and physical rationality score.

[0108] In the specific implementation process, a visual language model can be used as a review model.

[0109] Action execution compliance score is a quantitative score used to characterize whether the target state image presents the expected result corresponding to the target interaction instruction. For example, for the target interaction instruction "put the plate containing shrimp into the microwave oven", the action execution compliance score is used to determine whether the target state image actually presents the expected interaction result of the plate being completely placed inside the microwave oven cavity.

[0110] Scene consistency scoring is a quantitative assessment used to characterize whether the interactive subject, the background of the image, and the corresponding shooting scene of the image have undergone unreasonable changes. Scene consistency scoring verifies whether the background content, interactive execution component attributes, and core features of the shooting perspective corresponding to non-target objects in the initial state image and the target state image are consistent, thus avoiding abnormal samples where the original interactive scene in the target state image has been completely replaced by other unrelated scenes.

[0111] Physical rationality scoring is a quantitative assessment used to characterize whether the state changes of a target object conform to physical constraints. Physical constraints include real-world rules covering dimensions such as the object's own properties, spatial relationships between target objects, interaction states, and the physical feasibility of interactive actions. Specifically, they encompass common sense concepts such as gravity, support, occlusion, accessibility, and object functionality to ensure that the interaction process corresponding to the generated sample fully conforms to the operating logic of the real world.

[0112] In the specific implementation process, for each candidate prediction sample, the weighted average of its real interaction constraint scores from multiple review models in each dimension is aggregated to obtain the dimensional aggregate score of the candidate prediction sample in each dimension; for each candidate prediction sample, the dimensional divergence degree in each dimension is obtained based on the dispersion of its real interaction constraint scores in each dimension; for each candidate prediction sample, the comprehensive quality score is obtained based on the weighted sum of its corresponding dimensional aggregate scores; for each candidate prediction sample, the review divergence degree is obtained based on the weighted sum of its corresponding dimensional divergence degrees; if the comprehensive quality score reaches the quality threshold, and the dimensional aggregate scores corresponding to each dimension all reach the corresponding dimensional threshold, and the review divergence degree does not exceed the divergence threshold, the candidate prediction sample is determined to have passed the real interaction constraint screening.

[0113] In the specific implementation process, such as Figure 5 As shown, candidate prediction samples containing the initial state image, target interaction instructions, and target state image can be input into M multimodal review models: review model 1, review model 2, review model 3… review model M. For each dimension… ,Will The actual interaction constraint scores output by each review model and reasons Perform weighted average aggregation.

[0114] Specifically, based on the model reliability weights Calculate the aggregated score of the dimensions : ; By using weighted variance or range, and based on the dispersion of the true interaction constraint scores given by multiple review models in each dimension, the dimensionality divergence of the candidate prediction samples in each dimension is calculated. .

[0115] Then, for each candidate prediction sample, a weighted sum of the aggregated scores for each corresponding dimension is used to obtain the overall quality score of the candidate prediction sample. : ; The weights for aggregating scores for the k-th dimension.

[0116] For each candidate prediction sample, the review disagreement is obtained by weighting the disagreement of each dimension. : ; The weight of the divergence in the k-th dimension.

[0117] If the above calculation results simultaneously meet the following conditions, then the candidate predicted samples are determined to pass the screening of real interaction constraints: Overall quality score The quality threshold has been reached; Dimensional aggregated scores for each dimension All reached the corresponding dimensional threshold; Review Disagreement Not exceeding the divergence threshold.

[0118] For the overall quality score Samples located on the boundary between the pass threshold and the rejection threshold, or the reasons for different review models. Conflicting samples are sent to the manual review queue.

[0119] Step 202: Based on the results of at least one type of screening, determine the target prediction sample from the qualified prediction sample.

[0120] A qualified prediction sample refers to a candidate prediction sample that has passed at least one of the following screening criteria: temporal legality screening, interaction state change significance screening, and real interaction constraint screening.

[0121] In practice, target prediction samples can be determined from qualified prediction samples based on the priority of the screening results, the quality score, or the coverage requirements of specific scenarios. For example, the qualified prediction samples with the highest overall quality score and the lowest review disagreement can be selected as target prediction samples to ensure the high quality and consistency of the training data.

[0122] In the specific implementation process, such as Figure 6 As shown, a multi-dimensional coverage vector is established, which covers data source, scene type such as kitchen, living room, workbench, etc., action category such as picking, placing, pouring, opening, closing, heating, etc., target object attribute such as rigid, flexible, liquid, etc., viewpoint type such as robot viewpoint, first person viewpoint, etc., and state change type such as position change, state change, container relationship change, etc.

[0123] Based on this coverage vector, all qualified predicted samples are binned and classified, and an equalization strategy is implemented: for high-frequency combination bins with a sample number exceeding the preset upper limit, a quality-priority downsampling operation is performed to retain high-quality samples with higher comprehensive scores; for low-frequency combination bins with a sample number below the preset lower limit, their retention priority is increased while meeting basic quality requirements, thereby achieving coverage balance for different types of samples.

[0124] In the subsequent process of dividing the target prediction samples into training, validation and test sets, they are isolated by trajectory, scene or object instance, and cross-set near duplication checks are performed based on visual near duplication features and interactive semantic features to strictly avoid sample leakage issues. All final output target prediction samples are accompanied by complete and traceable quality metadata. Recorded fields include: sample_id (unique identifier), source_id (source identifier), trajectory_id (interaction trajectory identifier), camera_id (camera identifier), initial_frame (initial state image path), target_frame (target state image path), instruction_raw and instruction_normalized (raw and normalized interaction instructions), interaction start time t_a, interaction result display time t_r, result stability window stable_interval, interaction_region, saliency_score, action_score, scene_score, physics_score, judge_disagreement, review_status, filter_reason (filter reason), split (set partition), and version identifier. By recording the filter version and scoring reasons, the filtering process can be replayed after the review model is updated.

[0125] In the embodiments provided in this application, the problems of class distribution imbalance and sample leakage that occur after merging multi-source interactive samples are solved by coverage balancing and cross-set sample isolation. The final target prediction samples have high effectiveness, consistency and scene coverage.

[0126] The method for constructing image prediction samples based on multi-source real interactive data provided in this application will be further described below with reference to specific embodiments.

[0127] Example 1 Taking the robot interaction event of a kettle pouring water into a cup as an example, firstly, based on the interaction timing clues, the moment when the robotic arm begins to tilt the kettle is determined as the start time of the interaction action, and the moment when the water flows into the cup and the liquid level in the cup is clearly visible for the first time is determined as the moment when the interaction result is displayed.

[0128] From the pre-action window located before the start of the interaction, select an image frame showing the kettle upright and no new water added to the cup as the initial state image; and from the result stabilization window located after the interaction result is displayed, select an image frame showing a stable liquid surface as the target state image.

[0129] In the stage of selecting the target prediction sample from the candidate prediction sample, the following judgment is made on each candidate prediction sample: Interactive state change significance screening: If the target state image only shows the change in camera viewpoint and the water level in the cup does not change, the interactive state change significance is insufficient and the sample does not pass the screening.

[0130] Realistic Interaction Constraint Screening: Scene Consistency Score: If the kettle, cup, or workbench in the target state image is replaced with objects from different scenes, the scene consistency is deemed unqualified; Physical Reasonableness Score: If the target state image presents phenomena that do not conform to physical laws, such as water suspending in the air or penetrating the cup wall, the physical reasonableness is deemed unqualified.

[0131] Only when the candidate prediction sample meets the requirements of action execution compliance, scene consistency, and physical rationality will it be determined to pass the real interaction constraint screening and enter the qualified sample pool.

[0132] Example 2 Taking the target interactive command "Place the plate containing shrimp into the microwave oven" as an example, the selected initial state image should show that the plate is outside the microwave oven and the key object is identifiable, while the target state image should show that the same plate is inside the microwave oven cavity.

[0133] In the process of filtering real-world interactive constraints, not only were the positional changes of the plate between the initial and target state images compared, but further verification was also conducted: Changes in relationships between objects: Has the relationship between the plate and the microwave oven container correctly changed from external to internal?

[0134] Scene consistency: Does the microwave oven and its surrounding environment remain consistent before and after the interaction?

[0135] Physical rationality: Is the size of the plate reasonable in relation to the space of the microwave oven cavity, and are there any physically impossible situations such as overlapping?

[0136] Figure 4 This is a schematic diagram illustrating the process of hierarchical screening of candidate prediction samples provided in this application.

[0137] like Figure 4 The candidate triples shown correspond to candidate prediction samples extracted from each interaction data in multi-source real interaction data. They are composed of the target interaction command, the initial state image, and the target state image. The initial state image shows the identifiable visual features of the target object but has not yet performed the corresponding interaction action. The target state image shows the identifiable visual features of the target object but has completed the corresponding interaction action.

[0138] Perform basic quality and timing checks on candidate prediction samples at the first layer.

[0139] The verification dimensions include sample format integrity, image clarity, temporal order validity, and absence of near-duplicate samples.

[0140] The second layer performs interaction saliency verification, checking whether there are observable state changes corresponding to the target interaction command within the interaction area where the target object and the interaction execution end coexist. Specifically, it calculates the perceptual difference, attribute change, and relationship change between the interaction area and the target object area, and removes invalid samples that are only affected by changes in the image due to camera shake, lighting changes, etc., but for which the target object has not undergone a valid change in interaction state.

[0141] The third layer performs authenticity and physical constraint verification, checking whether the main subject of the scene remains consistent and whether the action results conform to physical constraints and common sense. Specifically, a multi-review model outputs realistic interaction constraint scores in dimensions such as scene consistency score and physical rationality score, excluding abnormal samples that violate the physical laws of the real world, such as background / subject abrupt changes, object clipping, and levitation.

[0142] In the fourth layer, quality confidence and distribution balance checks are performed to verify multi-review consistency, low uncertainty, and category coverage constraints. Specifically, through reviewer disagreement calculation and relevant rules for sample distribution balance, it ensures high consistency of multi-model scores and low judgment uncertainty for single samples, while also constraining the sample coverage of different action and scenario categories from a global perspective to avoid class imbalance in the dataset.

[0143] The abnormal backflow handling mechanism ensures that candidate prediction samples that fail the screening at any of the above layers will not be directly and permanently discarded. Instead, they will be selectively removed or moved to the boundary sample pool for manual review.

[0144] For candidate prediction samples that pass all four layers of screening consecutively, they are stored as qualified samples and the screening evidence is saved. The final output target prediction sample will simultaneously retain the screening metadata of the entire chain, including the verification score of each level, the degree of review disagreement, source identification and other information, so as to realize the quality traceability of the entire sample life cycle and ultimately ensure the high purity and high reliability of the dataset stored in the database.

[0145] The apparatus for constructing image prediction samples based on multi-source real interaction data provided in this application is described below. The apparatus for constructing image prediction samples based on multi-source real interaction data described below can be referred to in correspondence with the method for constructing image prediction samples based on multi-source real interaction data described above.

[0146] Figure 7 This is a schematic diagram of the device provided in this application for constructing image prediction samples based on multi-source real-world interactive data. Figure 7As shown, the apparatus 700 for constructing image prediction samples based on multi-source real-world interaction data includes the following modules: The first acquisition module 710 is used to acquire real interactive data from multiple sources.

[0147] The extraction module 720 is used to extract candidate prediction samples from each piece of interaction data in the multi-source real interaction data; wherein, the candidate prediction sample includes a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image; in the initial state image, identifiable visual features of the target object are presented, and the target object is not associated with the interaction action indicated by the target interaction instruction; in the target state image, identifiable visual features of the target object are presented, and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is the object to which the interaction action is performed.

[0148] The second acquisition module 730 is used to obtain the target prediction sample based on the candidate prediction sample.

[0149] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a method for constructing image prediction samples based on multi-source real interaction data. This method includes: acquiring multi-source real interaction data; extracting candidate prediction samples from each interaction data point of the multi-source real interaction data; wherein the candidate prediction samples include a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image; in the initial state image, identifiable visual features of a target object are presented, and the target object is not associated with the interaction action indicated by the target interaction instruction; in the target state image, identifiable visual features of the target object are presented, and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is the object acted upon by the interaction action; and based on the candidate prediction samples, a target prediction sample is obtained.

[0150] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method provided by the above methods for constructing image prediction samples based on multi-source real interaction data. The method includes: acquiring multi-source real interaction data; extracting candidate prediction samples from each interaction data of the multi-source real interaction data; wherein the candidate prediction samples include a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image; in the initial state image, identifiable visual features of a target object are presented, and the target object is not associated with the interaction action indicated by the target interaction instruction; in the target state image, identifiable visual features of the target object are presented, and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is the object acted upon by the interaction action; and a target prediction sample is obtained based on the candidate prediction samples.

[0152] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for constructing image prediction samples based on multi-source real interaction data provided by the methods described above. The method includes: acquiring multi-source real interaction data; extracting candidate prediction samples from each interaction data point of the multi-source real interaction data; wherein the candidate prediction samples include a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image; the initial state image presents identifiable visual features of a target object and the target object is not associated with the interaction action indicated by the target interaction instruction; the target state image presents identifiable visual features of the target object and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is the object acted upon by the interaction action; and obtaining a target prediction sample based on the candidate prediction samples.

[0153] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0154] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for constructing image prediction samples based on multi-source real-world interaction data, characterized in that, include: Acquire real-world interaction data from multiple sources; From each interaction data point of the multi-source real interaction data, candidate prediction samples are extracted; wherein, the candidate prediction sample includes a target interaction command, an initial state image associated with the target interaction command, and a target state image; in the initial state image, identifiable visual features of the target object are presented, and the target object is not associated with the interaction action indicated by the target interaction command; in the target state image, identifiable visual features of the target object are presented, and the target object is associated with the interaction action indicated by the target interaction command; the target object is the object on which the interaction action is performed. Based on the candidate prediction samples, target prediction samples are obtained, and the target prediction samples are used to train or evaluate the target task execution capabilities of the embodied intelligent agent.

2. The method for constructing image prediction samples based on multi-source real-world interactive data according to claim 1, characterized in that, The interactive data includes multiple frames of images with timestamps and interactive command information; The step of extracting candidate prediction samples from each interaction data point of the multi-source real interaction data includes: For each piece of interaction data, perform the following operations to obtain the candidate prediction samples corresponding to the interaction data: Based on the temporal information of the multi-frame images and the interaction state between the interactive subject and the target object in each frame image, a pre-action window and a result-stabilized window are determined; wherein, the pre-action window is a time window in which the interactive subject has not yet performed the interactive action; the result-stabilized window is a time window in which the execution result of the interactive action is displayed; the temporal information of the multi-frame images is determined by the timestamp carried by each frame image; The first image within the window before the action is selected from the multiple frames of images and used as the initial state image; The second image within the result stabilization window is selected from the multi-frame images and used as the target state image; The interactive instruction information is processed in a structured manner to obtain the target interactive instruction.

3. The method for constructing image prediction samples based on multi-source real-world interactive data according to claim 2, characterized in that, The pre-action window is determined in the following manner: Based on the temporal information of the multi-frame images and the interaction state between the interactive subject and the target object in each frame image, the start time of the interactive action is determined based on the detected interaction timing clues; wherein, the interaction timing clues include at least one of the following: the start information of the action, the contact trigger information between the interactive execution component and the target object, the target object state change information, and the multimodal event recognition information for the interactive execution component; wherein, the interactive execution component includes a human hand, an end effector of an embodied intelligent agent, or a tool controlled by a human or the embodied intelligent agent; The time interval corresponding to the multi-frame images that is before the start time of the interactive action is determined as the action pre-window.

4. The method for constructing image prediction samples based on multi-source real-world interactive data according to claim 3, characterized in that, The stability window of the result is determined in the following way: The moment when the change in the interaction result caused by the interaction action first exceeds a preset change threshold, starting from the start time of the interaction action, is taken as the moment when the interaction result is displayed. After the interaction result is displayed, determine whether the interaction result remains stable for a consecutive preset number of image frames; If so, the continuous time interval corresponding to the preset number of consecutive image frames is determined as the result stabilization window.

5. The method for constructing image prediction samples based on multi-source real-world interactive data according to any one of claims 3 to 4, characterized in that, The process of obtaining the target prediction sample based on the candidate prediction sample includes: At least one type of screening is performed on the candidate prediction samples; Based on the results of the at least one type of screening, a target prediction sample is determined from the candidate prediction samples; The at least one type of screening includes temporal validity screening, interaction state change significance screening, and real interaction constraint screening; the temporal validity screening is used to verify whether the temporal information of the initial state image and the target state image in the candidate prediction samples is reasonable; the interaction state change significance screening is used to remove candidate prediction samples whose state changes of the target object are not significant; the real interaction constraint screening is used to evaluate whether the change of the target state image relative to the initial state image meets the realism constraint.

6. The method for constructing image prediction samples based on multi-source real-world interactive data according to claim 5, characterized in that, The candidate predicted samples are determined to pass the temporal validity screening in the following manner: For each candidate prediction sample containing an initial state image and a target state image, if the acquisition time of the initial state image is earlier than the start time of the interaction action, and the acquisition time of the target state image is not earlier than the display time of the interaction result, then the candidate prediction sample is determined to pass the temporal legality screening.

7. The method for constructing image prediction samples based on multi-source real-world interactive data according to any one of claims 5 to 6, characterized in that, Whether the candidate prediction sample passes the interaction state change significance screening is determined by the following method: For each candidate prediction sample, perform the following operations on the initial state image and the target state image: In the initial state image, a first object region where the target object is located and a first interaction region where both the target object and the interaction execution component exist are determined; In the target state image, a second object region where the target object is located, and a second interaction region where both the target object and the interaction execution component exist simultaneously are determined; Determine the change in the first interactive state; Wherein, the first interaction state change amount is determined based on at least one of the first perceptual difference amount between the first interaction area and the second interaction area, the first attribute change amount of the target object, and the first relationship change amount between the target objects; Determine the second interaction state change amount; wherein the second interaction state change amount is determined based on at least one of the following: a second perceptual difference amount between the first object region and the second object region, a second attribute change amount of the target object, and a second relationship change amount of the target object; Based on the first interaction state change amount and / or the second interaction state change amount, determine whether the candidate prediction sample passes the interaction state change significance screening.

8. The method for constructing image prediction samples based on multi-source real-world interactive data according to claim 7, characterized in that, The step of determining whether the candidate prediction sample passes the interaction state change significance screening based on the first interaction state change amount and / or the second interaction state change amount includes: If the global pixel change between the initial state image and the target state image is greater than a preset global threshold and meets the preset judgment condition, then it is determined that the candidate prediction sample has not passed the interaction state change significance screening. The preset determination conditions include at least one of the following: the first perceived difference amount and / or the second perceived difference amount does not exceed a preset perceived difference threshold; the first attribute change amount and / or the second attribute change amount does not exceed a preset attribute change threshold; the first relationship change amount and / or the second relationship change amount does not exceed a preset relationship change threshold.

9. The method for constructing image prediction samples based on multi-source real-world interactive data according to claim 7, characterized in that, The step of determining whether the candidate prediction sample passes the interaction state change significance screening based on the first interaction state change amount and / or the second interaction state change amount includes: Determine the first weighted result of the first perceived difference, the first attribute change, and the first relationship change; Determine a second weighted result of the second perceived difference, the second attribute change, and the second relationship change; Based on the comparison result of the first weighted result and the first change significance threshold, and / or the comparison result of the second weighted result and the second change significance threshold, it is determined whether the candidate prediction sample passes the interaction state change significance screening.

10. The method for constructing image prediction samples based on multi-source real-world interactive data according to any one of claims 6 to 9, characterized in that, Whether a candidate predicted sample passes the screening of the real interaction constraints is determined by the following method: For each candidate prediction sample, multiple review models are used to evaluate the real interaction constraints from at least one dimension, and the real interaction constraint scores of each review model for the candidate prediction sample in the at least one dimension are obtained. For each candidate prediction sample, determine whether the candidate prediction sample passes the real interaction constraint screening based on its real interaction constraint score in at least one dimension. The at least one dimension of the real interaction constraint score includes at least one of the following: action execution compliance score, scene consistency score, and physical rationality score; in: The action execution compliance score is a quantitative score used to characterize whether the target state image presents the expected result corresponding to the target interaction instruction; The scene consistency score is a quantitative score used to characterize whether the interactive subject, the background of the image, and the shooting scene corresponding to the image have undergone unreasonable changes. The physical rationality score is a quantitative score used to characterize whether the state changes of the target object conform to physical constraints.

11. The method for constructing image prediction samples based on multi-source real-world interactive data according to claim 10, characterized in that, The step of determining whether a candidate prediction sample passes the real interaction constraint screening based on its real interaction constraint score in at least one dimension for each candidate prediction sample includes: For each candidate prediction sample, the weighted average of its real interaction constraint scores from the multiple review models in each dimension is aggregated to obtain the dimensional aggregate score of the candidate prediction sample in that dimension. For each candidate prediction sample, the dimensional divergence degree of the candidate prediction sample in each dimension is obtained based on the dispersion of its true interaction constraint score in each dimension. For each candidate prediction sample, the overall quality score of the candidate prediction sample is obtained by weighting the aggregated scores of its corresponding dimensions. For each candidate prediction sample, the review disagreement degree of the candidate prediction sample is obtained by weighting the disagreement degree of each corresponding dimension. If the overall quality score reaches the quality threshold, and the aggregated scores of each dimension all reach the corresponding dimension threshold, and the review disagreement does not exceed the disagreement threshold, then the candidate prediction sample is determined to have passed the screening of the real interaction constraints.

12. An apparatus for constructing image prediction samples based on multi-source real-world interactive data, characterized in that, include: The first acquisition module is used to acquire real interaction data from multiple sources; An extraction module is used to extract candidate prediction samples from each piece of interaction data in the multi-source real interaction data; wherein, the candidate prediction sample includes a target interaction instruction, an initial state image associated with the target interaction instruction, and a target state image; in the initial state image, identifiable visual features of the target object are presented, and the target object is not associated with the interaction action indicated by the target interaction instruction; in the target state image, identifiable visual features of the target object are presented, and the target object is associated with the interaction action indicated by the target interaction instruction; the target object is the object on which the interaction action is performed. The second acquisition module is used to obtain a target prediction sample based on the candidate prediction sample, so as to use the target prediction sample to train or evaluate the target task execution capability of the embodied intelligent agent.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method for constructing image prediction samples based on multi-source real interactive data as described in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for constructing image prediction samples based on multi-source real interactive data as described in any one of claims 1 to 11.

Images