Alignment methods, devices, and electronic equipment for multimodal sensing data
By generating physical causal event anchors and optimizing the causal delay field, multimodal sensing data is mapped to a common physical time axis, solving the misalignment problem in heterogeneous data synchronization of multimodal sensing systems and achieving high-precision and robust data alignment.
Patent Information
- Application Number
- CN202611508677.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-24
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal sensing systems suffer from high costs and difficulty in coping with dynamic interferences such as network jitter, sensor packet loss, or clock drift during heterogeneous data synchronization, leading to misalignment of data across different modalities on the time axis.
By detecting changes in physical state caused by robot actions or interactions with objects, physical causal event anchors are generated, event anchor bundles are established, and the causal delay field is optimized within a sliding time window. The data from each modality of perception are mapped to a common physical time axis to generate multimodal decision frames.
It achieves high-precision, robust, and adaptive dynamic alignment of heterogeneous modalities, reduces alignment costs, and improves the accuracy and reliability of data alignment.
Smart Images

Figure CN123115691A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of robot control technology, and more specifically, to a method, apparatus, and electronic device for aligning multimodal sensing data. Background Technology
[0002] With the rapid development of embodied intelligence technology, existing multimodal perception systems in the field of robot control face a severe challenge in synchronizing heterogeneous data during practical deployment. Different sensors (such as RGB cameras, depth cameras, haptic arrays, and torque sensors) typically have independent hardware clocks and different sampling frequencies. For example, a camera's frame rate may be only 30Hz, and there may be exposure delay; while a torque sensor's sampling rate may be as high as 1kHz. Traditional solutions usually rely on hardware-level synchronization (such as Precision Time Protocol (PTP) or hardware trigger signals) to align the timestamps of each modality. However, in practical applications, hardware synchronization is not only costly but also difficult to cope with dynamic interference such as network jitter, sensor packet loss, or clock drift, resulting in misalignment of data across modalities on the timeline. Summary of the Invention
[0003] This disclosure provides a method, apparatus, and electronic device for aligning multimodal sensing data, which solves the technical problem that existing time alignment methods for multimodal sensing data are not only costly, but also still have misalignment on the time axis after alignment.
[0004] According to one aspect of the present disclosure, a method for aligning multimodal sensing data is provided. The method includes: acquiring multimodal sensing data; detecting identifiable responses caused by changes in the same physical state in each modal sensing data; and determining the set of identifiable responses as physical causal event anchors; wherein the physical state change is a state transition event caused by robot action or robot-object interaction; generating event anchor bundles based on the physical causal event anchors; wherein the event anchor bundles include multiple event anchors; jointly optimizing the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor bundles within a sliding time window; wherein the causal delay field is used to map the local time of each modal sensing data to a common physical time axis; and generating a multimodal decision frame based on the optimized causal delay field, mapping the local time of each modal sensing data to the common physical time axis, and according to the action phase boundary preservation rule; wherein the multimodal decision frame includes at least the multimodal sensing data after physical causal alignment.
[0005] Optionally, the physical causal event anchor includes event type, modal local time, host reception time, object identifier, action identifier, and confidence level; the event type of the physical causal event anchor includes at least one of action start, mechanism start, initial contact, stable contact, slip, collision, object displacement, and release.
[0006] Optionally, generating an event anchor bundle based on physical causal event anchors includes: filtering candidate event anchors from the generated event anchors based on object identifiers and action identifiers; and forming an event anchor bundle by combining at least two cross-modal candidate events whose time intervals satisfy preset conditions and have causal relationships based on predefined causal reachability relationships in a directed acyclic graph of action causality.
[0007] Optionally, the event anchor bundle includes at least one robot control mode event anchor and one external environment mode event anchor.
[0008] Optionally, the functional expression for the causal time-delay field is:
[0009] in, For modality The mapping function, For the modal local time, Let be the spline basis function matrix. is the spline control point vector.
[0010] Optionally, the above method further includes: applying a derivative constraint to the function of the causal time-delay field to maintain time monotonicity; the derivative constraint is:
[0011] in, and For the preset positive constant, Let be the time derivative of the function of the causal time-delay field.
[0012] Optionally, the objective function for jointly optimizing the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor bundle includes at least: a data alignment term, used to force the local time of different modalities within the same event anchor bundle to converge to the same physical time after being mapped to the common physical time axis; a causal penalty term, used to make the optimization result conform to the physical causality law; a smoothing constraint term, used to limit the second-order rate of change of the causal delay field and prevent the causal delay field from exhibiting severe jitter; and a reception time prior term, used to take the reception time of the sensed data as a weak prior.
[0013] Optionally, a multimodal decision frame is generated according to the action phase boundary preservation rule, including: when the target physical moment falls within the protection width of the action phase boundary, the observations before the boundary and the observations after the boundary are aggregated respectively to generate two independent candidate decision frames, and one of them is selected as the final output based on the causal consequence events that have occurred.
[0014] Optionally, the multimodal decision frame includes modal features, common physical time, action phase label, modality missing mask, and aligned confidence intervals generated based on causal delay field covariance.
[0015] Optionally, the above method further includes: when the alignment confidence of the key modality is lower than a first threshold or the default amount of the action causal directed acyclic graph exceeds a second threshold, controlling the robot to scale its speed, maintain its position, or backtrack along the most recently verified free space segment.
[0016] According to another aspect of the present disclosure, an alignment device for multimodal sensing data is provided. The device includes: a first processing module, configured to acquire multimodal sensing data, detect identifiable responses caused by changes in the same physical state in each modal sensing data, and determine the set of identifiable responses as physical causal event anchors; wherein the physical state change is a state transition event caused by robot action or robot-object interaction; a second processing module, configured to generate event anchor bundles based on the physical causal event anchors; wherein the event anchor bundles include multiple event anchors; an optimization module, configured to jointly optimize the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor bundles within a sliding time window; wherein the causal delay field is used to map the local time of each modal sensing data to a common physical time axis; and a third processing module, configured to map the local time of each modal sensing data to a common physical time axis based on the optimized causal delay field, and generate a multimodal decision frame according to the action phase boundary preservation rule; wherein the multimodal decision frame includes at least the multimodal sensing data after physical causal alignment.
[0017] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the above-described method.
[0018] According to another aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described method.
[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described method.
[0020] The beneficial effects of the technical solution provided in this disclosure are as follows: The technical solution proposed in this disclosure does not rely on equal timestamps of data packets as a sufficient condition. Instead, it utilizes cross-modal state transitions caused by robot actions in the physical world to establish event anchors, estimates the causal delay field from each modality to the common physical time axis online, and generates boundary-preserving multimodal decision frames based on action risk. Compared to timestamp pairing relying on the same clock, this disclosure uses the same physical change as the alignment basis for modal sensing data. The event anchors can reposition late-arriving data using physical consequences, achieving high-precision, robust, and adaptive dynamic alignment of heterogeneous modalities. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.
[0022] Figure 1 A schematic diagram of the architecture of a modal dynamic alignment system based on physical causal event anchors provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating a method for aligning multimodal sensing data provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of an alignment device for multimodal sensing data provided in an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation
[0023] The embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions of the embodiments of this disclosure.
[0024] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this disclosure mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element are connected through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term, for example, “A and / or B” or “A, B” indicates implementation as “A,” or implementation as “B,” or implementation as “A and B.”
[0025] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.
[0026] Existing time alignment methods for multimodal sensing data are not only costly, but also result in misalignment of the aligned modal data along the time axis.
[0027] The method, apparatus, and electronic device for aligning multimodal sensing data disclosed herein are intended to solve at least one of the above-mentioned technical problems in the prior art.
[0028] The following description of several exemplary embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.
[0029] Figure 1 This is a schematic diagram of the architecture of a modal dynamic alignment system based on physical causal event anchors provided in an embodiment of this disclosure, as shown below. Figure 1 As shown, the system includes: a multimodal acquisition layer, an event extraction layer, a cause-effect graph and time delay field estimation layer, a decision frame generation layer, and a secure execution layer.
[0030] The multimodal acquisition layer is used to receive RGB / RGB-D images, tactile arrays, six-dimensional torque, microphones, joint encoders, and controller motion logs; each data entry retains the sensor's local time, reception time, motion identifier, and object identifier.
[0031] The event extraction layer operates independently within its respective modality. The vision end detects abrupt changes in object pose or optical flow, the tactile end detects changes in contact area and pressure gradient, the auditory end detects the impact envelope, the torque end detects changes in the end wrench, and the encoder end detects changes in speed or acceleration.
[0032] Cause-effect graphs and time-delay field estimation layers are used to connect these events into event anchors.
[0033] The decision frame generation layer is used to read the reliable features of each modality according to the common physical time and generate multimodal decision frames.
[0034] The safety execution layer is used to determine whether the robot should continue, slow down, maintain, or retreat based on confidence level, action risk, and causal default amount.
[0035] In some embodiments, a set of hardware may employ a 30 Hz RGB-D camera, a 1 kHz haptic array, a 48 kHz microphone, a 500 Hz six-dimensional torque sensor, and 1 kHz joint servo feedback. The algorithm may be deployed on the non-real-time coprocessing core of the robot's real-time controller, providing 20 Hz alignment decision frames to the Vision-Language-Action (VLA) strategy.
[0036] This disclosure provides a method for aligning multimodal sensing data, such as... Figure 2 As shown, the method includes the following steps: Step S201: Acquire multimodal perception data, detect identifiable responses caused by the same physical state change in each modal perception data, and determine the set of identifiable responses as physical causal event anchors; wherein, physical state change is a state transition event caused by robot action or robot interaction with an object.
[0037] In embodiments of this disclosure, a physical causal event anchor is a set of identifiable responses generated by the same physical state change in different modes.
[0038] A change in physical state refers to a real event that occurs in the physical world. For example, patting a table with your hand. The change in physical state is manifested as the hand changing from motion to rest, and the surface of the table changing from rest to vibration.
[0039] In this disclosure, "modal" refers to different sensing methods or sensor types. The same physical event can be recorded in different forms by different sensors. For example: Visual modality: The camera captures a hand slapping a table.
[0040] Auditory modality: The microphone recorded a "pop" sound.
[0041] Tactile / force modality: A force sensor on the table detects a sudden increase in pressure.
[0042] Kinematic mode: The accelerometer detected vibrations on the table surface.
[0043] According to some optional embodiments of this disclosure, a physical causal event anchor includes an event type, a modal local time, a host receiving time, an object identifier, an action identifier, and a confidence level; the event type of the physical causal event anchor includes at least one of action start, mechanism initiation, initial contact, stable contact, slippage, collision, object displacement, and release.
[0044] In the embodiments of this disclosure, the first An event anchor is represented by the following expression:
[0045] in, For the first An event anchor; The event type indicates the category of physical interaction to which the event anchor belongs; Modal local time refers to the original timestamp of the event recorded by the sensor or acquisition terminal under the local hardware clock. The host reception time refers to the system time at which the central processing unit (host) actually receives the event data packet; Modal identifiers are used to distinguish the sensor type from which the data originates; For object identification, used to identify the physical entity or environmental object associated with the event (such as: target object, obstacle, work surface, etc.); This is an action identifier used to associate the robot's low-level action command or high-level intent label that triggered the event. Confidence level represents the reliability probability value or weighting coefficient of the event detection result. .
[0046] In some embodiments, event types include at least action initiation, mechanism activation, initial contact, stable contact, slippage, collision, object displacement, and release. Event anchors must reflect the causal coupling between robot actions and environmental feedback. The system considers an alignment anchor point valid only when both the internal action and the external response are detected simultaneously and satisfy the partial order constraints in the action causal graph. This fundamentally guarantees the physical authenticity of multimodal perception alignment.
[0047] Step S202: Generate an event anchor bundle based on the physical causal event anchor; wherein, the event anchor bundle includes multiple event anchors.
[0048] In some embodiments, an event anchor bundle refers to a set of event anchors formed by at least two modes transitioning to the same state within a preset physical time neighborhood.
[0049] In some embodiments, the event anchor bundle is not a single synchronous pulse; for example, the gripper closing action can form a four-modal bundle of motor current rise, fingertip pressure appearance, collision sound, and target visual velocity reduction. Even if the microphone data is delayed by 80ms due to operating system buffering, it can still be mapped to the same physical contact moment as long as its event type and amplitude pattern in the causal graph are consistent.
[0050] Step S203: Within the sliding time window, jointly optimize the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor bundle; wherein, the causal delay field is used to map the local time of the sensing data of each modality to the common physical time axis.
[0051] According to some optional embodiments of this disclosure, the causal delay field is a piecewise monotonic function that maps modal local time to physical time.
[0052] Step S204: Based on the optimized causal delay field, the local time of each modal sensing data is mapped to the common physical time axis, and a multimodal decision frame is generated according to the action phase boundary preservation rule; wherein, the multimodal decision frame includes at least the multimodal sensing data after physical causal alignment.
[0053] The above technical solution utilizes the cross-modal state transitions caused by robot actions in the physical world to establish event anchors, online estimates the causal time delay fields from each modality to the common physical time axis, and generates boundary-preserving multimodal decision frames based on action risks. Compared to pairing based on timestamps from the same clock, this disclosure uses the same physical change as the alignment basis for modal sensing data. Event anchors can reposition late-arriving data using physical consequences, achieving high-precision, robust, and adaptive dynamic alignment of heterogeneous modalities.
[0054] According to some optional embodiments of this disclosure, step S202 generates an event anchor bundle based on physical causal event anchors, which is achieved by: filtering candidate event anchors from the generated event anchors based on object identifiers and action identifiers; and forming an event anchor bundle by combining at least two cross-modal candidate events whose time intervals meet preset conditions and have causal relationships based on predefined causal reachability relationships in a directed acyclic graph of action causality.
[0055] According to some optional embodiments of this disclosure, candidate bundles are first filtered by action identifier and object identifier, then by event type based on reachable paths in the action causal graph, and finally by a relaxed time threshold. Maximum weight matching is performed within the bundle. The bundle weights simultaneously consider event type compatibility, feature amplitude similarity, current time delay field prediction residuals, and modal health. Candidate bundling refers to the process of aggregating and binding candidate events from different modalities based on whether they point to the same physical state transition, forming an event anchor bundle.
[0056] In this embodiment, firstly through and The system confines a massive number of events to specific action and object interaction contexts. For example, when performing the action of grasping a water cup, the system only focuses on events related to the cup and the grasp, directly filtering out irrelevant visual or force noise in the background. Then, a motion causal graph is introduced as prior knowledge. Only event types that conform to physically causally reachable paths (e.g., initial contact can only occur after the mechanism is activated, rather than jumping directly to release) are included in the candidate pool. This logically excludes out-of-order events that violate physical common sense. Only after passing the first two levels of filtering is a maximum-weighted match performed in the time dimension. The relaxed time threshold here is crucial; it acknowledges the differences in physical response and transmission delay between different sensors, allowing the search for the best match within a reasonable time window, rather than requiring absolute synchronization.
[0057] In some embodiments, if an event may correspond to multiple physical changes, multiple hypotheses are retained and disambiguated in a subsequent window based on causal residuals, without premature hard binding.
[0058] Using the above technical solution, the system does not rely on all sensors sharing hardware triggering or PTP; it only requires local time to be monotonic and the consequences of actions to be observable. Compared to pairing by reception time or most recent timestamp, event anchoring can relocate late-arriving data using physical consequences, reducing the probability of stitching different action phases into a single frame.
[0059] In some embodiments, for each type of skill template (e.g., grasping, placing, tightening screws), a directed acyclic graph of action causality is pre-built. The expression of the directed acyclic graph of action causality is:
[0060] in, This represents a directed acyclic graph representing the cause and effect of actions. The set of nodes includes commands (control instructions issued by the upper-level strategy), mechanism states (kinematic / dynamic responses of the robot body), contact states (physical interaction states between the robot and the environment), and environmental results (the final changes caused by the action to the physical world). These four types of nodes correspond to the complete causal chain of robot actions from intention to physical realization. Let be the set of edges, representing the causal transition relationships between states.
[0061] The expression for the edges of a causal directed acyclic graph is:
[0062] in, Represents the edges of a causal directed acyclic graph; Indicates the parent event, This indicates a child event, meaning the parent event should occur before the child event. For example: The mechanism starts before the initial contact occurs; the robot must move first before it can come into contact with the object. Initial contact occurs before object displacement; you must make contact with the object before you can move it. This represents the lower bound of the time difference, the minimum physical time interval between the parent and child events; This represents the upper bound of the time difference, the maximum allowed time interval between the parent and child events.
[0063] The physical time difference between the parent and child events should be within the interval. Within. This interval can be initialized using equipment specifications, no-load identification, and historical task statistics.
[0064] In some embodiments, a causality margin is calculated for each anchor bundle at runtime, and the expression for the causality margin is: , in, For causal margin; Represents an edge in a causal directed acyclic graph. Indicates the lower bound of the time difference; Indicates the actual occurrence time of the sub-event; Indicates the actual time when the parent event occurred; This indicates the upper bound of the time difference.
[0065] This formula is essentially a range out-of-bounds detector. It determines the actual time difference. Does it fall within the legal range? Inside.
[0066] Scenario 1: Within the normal range ( ≤ ≤ ) at this time, - ≤0, and - ≤0, =0 The causal sequence is normal and requires no intervention.
[0067] Scenario 2: The time difference is too small ( < (The incident happened too early) at this time, - >0, and - <0, >0 A child event occurring too early after a parent event violates the lower bound of physical laws. For example, a sudden change in torque after a command is issued could be due to sensor noise or a false trigger.
[0068] The larger the number, the earlier the violation occurred, and the more serious the violation.
[0069] Scenario 3: Excessive time difference ( > (The incident happened too late) at this time, - <0, and - >0, >0 A child event occurs too late after its parent event, exceeding a reasonable waiting time. For example, if a grab command is issued but contact is detected a long time later, it could be due to object position shift or actuator malfunction.
[0070] The larger the number, the later the violation, and the more serious the violation.
[0071] When an event anchor causes multiple high-confidence causal edges to appear simultaneously If the value is greater than 0, the system will determine that the reliability of the anchor bundle is questionable, and will reduce the matching weight or split the anchor bundle.
[0072] When the same edge exhibits a long-term unidirectional offset (e.g.) Slightly larger If so, then exponential forgetting update is used. However, the direction of the edge remains unchanged.
[0073] The causal directed acyclic graph proposed in this embodiment specifies partial order relationships such as command precedes mechanism response, mechanism approach precedes contact, contact precedes impact sound and force consequences, so that dynamic time distortion will not produce spurious matches that violate physical causality.
[0074] In some embodiments, the acoustic impact and torque rise that occur simultaneously after contact do not have a strict sequential relationship. The causal graph sets them as common child nodes of the contact-establishing node, rather than creating an edge between them. This design avoids the spurious causal constraints introduced by forcibly ordering parallel events.
[0075] According to some optional embodiments of this disclosure, the event anchor bundle includes at least a robot control mode event anchor and an external environment mode event anchor.
[0076] In some embodiments, an effective event anchor bundle cannot contain only single-viewpoint events; it must simultaneously include both robot-interactive state changes and interactions with the external environment to serve as a reliable anchor point for cross-modal time alignment. Control modal events reflect the robot's internal state changes, originating from internal sensors or controller commands. External environment modal events reflect the robot's interactions with the external environment or target objects, originating from external sensors.
[0077] According to some optional embodiments of this disclosure, the functional expression of the causal time delay field is:
[0078] in, For modality The mapping function, For the modal local time, Let be the spline basis function matrix. is the spline control point vector.
[0079] In this embodiment, Used to convert modes local time Mapped to a common physical time axis; local time of the modality refers to modality The original timestamp of the sensor sampling; It is composed of basis functions of B-splines or Hermite splines, and is time-dependent. The function; It is modal The optimization variables control the shape and amplitude of the time delay field.
[0080] This disclosure provides a possible implementation method that applies a derivative constraint to the function of the causal time-delay field to maintain time monotonicity; the derivative constraint is:
[0081] in, and For the preset positive constant, Let be the time derivative of the function of the causal time-delay field.
[0082] In some optional embodiments of this disclosure, derivative constraints are imposed on the function of the causal time-delay field, for example, This ensures that the local time order is not reversed (i.e., >0), while limiting the magnitude of time dilation / compression within a reasonable range. If <0 means time travel, which is physically impossible; if Too large or too small a value means that the time delay field has undergone an unreasonable and drastic distortion.
[0083] The technical solution proposed in this disclosure, or the segmented monotonic delay field, can describe the slowly varying delay caused by operating system scheduling, network queuing, and internal filtering of sensors; the action-causal directed acyclic graph can incorporate the physical response interval into the constraints, avoiding misjudging the actual contact propagation delay as clock deviation.
[0084] In some optional embodiments of this disclosure, the objective function for jointly optimizing the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor bundle in step S203 includes at least: a data alignment term, used to force the local time of different modalities within the same event anchor bundle to converge to the same physical time after being mapped to the common physical time axis; a causal penalty term, used to make the optimization result conform to the physical causality law; a smoothing constraint term, used to limit the second-order rate of change of the causal delay field and prevent the causal delay field from experiencing severe jitter; and a reception time prior term, used to take the reception time of the sensed data as a weak prior.
[0085] In this embodiment, the function parameters of the causal delay field to be optimized refer to those mentioned above. , i.e., the spline control point vector. The objective function for joint optimization is expressed as: + + +
[0086] in, The objective function is the joint optimization objective, which is composed of a weighted sum of each sub-item. The smaller the overall sum, the higher the alignment quality. ( ) represents a robust kernel function (such as the Huber kernel) used to suppress outliers; For modality Mapping function; Modal local time; The estimated time of the k-th event anchor on the common physical time axis; For the first The measurement noise variance of each observation is used for weighting; This is the causal penalty weighting coefficient; For causal margin; For smoothing weighting coefficients; It is a second-order difference operator (or a second-order derivative operator); The spline control point vector; To receive prior weight coefficients; For the first The actual timestamp of each observation received at the receiving end; For modality Fixed system offsets (such as the average latency of the driver cache).
[0087] For data alignment, observations from different modalities within the same event anchor bundle are forced to converge to the same physical moment after being mapped to a common physical time axis. .
[0088] As a causal penalty term, the action-causal graph mentioned above is embedded as a soft constraint into the optimization objective. When the actual time sequence violates the expected range of the causal graph, a penalty is generated, forcing the optimization result to conform to the physical causality law.
[0089] This is a smoothing constraint term used to limit the second-order rate of change of the time delay field and prevent drastic fluctuations in the time delay field. Physically, the transmission delay of a sensor cannot change abruptly in a very short time; this term reflects the prior physical plausibility.
[0090] As a priori term for the reception time, the reception time of the data packet is used as a weak prior. Although the reception time is not equal to the actual time when the physical event occurs (because it includes transmission delay, queuing delay, etc.), it provides a rough upper bound reference to prevent the optimization results from deviating too far.
[0091] According to some optional embodiments of this disclosure, step S204 generates a multimodal decision frame according to the action phase boundary preservation rule, which is achieved by the following method: when the target physical moment falls within the protection width of the action phase boundary, the observation number before the boundary and the observation data after the boundary are aggregated respectively to generate two independent candidate decision frames, and one of them is selected as the final output according to the causal consequence events that have occurred.
[0092] In traditional time alignment, systems typically employ simple interpolation or nearest neighbor matching.
[0093] Assume the robot touches the table at t=1.0s.
[0094] The visual frame is at t=0.98s (no contact).
[0095] Torque data at t=1.02s (already available).
[0096] Traditional algorithms might forcibly stitch these two types of data frames together into a single decision frame, leading to artifacts where the upper-level policy perceives contact but the visual contact hasn't occurred, or the visual contact has occurred but the force hasn't arrived. This stitching of data frames can cause the upper-level policy to output incorrect actions (e.g., continuing to apply force when contact has already occurred).
[0097] In embodiments of this disclosure, interpolation across physical boundaries is prohibited. Even if the data is discontinuous, false data that violates physical laws cannot be created.
[0098] In embodiments of this disclosure, a restricted area, i.e., a protection width, is defined before and after the action phase boundary (such as switching from free motion to contact). When the target physical time Falling into protection width At that time, the system generates two candidate representations: one before the boundary and one after the boundary. The action-cause graph is used to determine the currently occurring consequence event (such as whether there is a force mutation). If a contact consequence has occurred, the system forces the selection of the representation after the boundary, prohibiting the averaging of the visual frame before contact and the force data after contact into the same token, thus ensuring physical realism.
[0099] In some embodiments, if data from a non-critical sensor (such as a microphone) is lost, it is directly marked as missing, rather than being filled in with data from the previous time step (i.e., long-range interpolation). Filling in the missing data with predicted features is only allowed when the world model's prediction variance is below a preset threshold, and the prediction source is then labeled to the upper-layer policy.
[0100] According to some optional embodiments of this disclosure, the multimodal decision frame includes modal features, common physical time, action phase label, modality missing mask, and aligned confidence intervals generated based on causal delay field covariance.
[0101] In some embodiments, the final output decision frame is not just a bunch of numbers, but a smart package with metadata, including: Modal features, i.e., aligned multimodal data; Common physical time serves as a unified time reference. Action phase labels, for example, clearly indicate whether the current phase is approaching, contacting, or locking; Modality missing mask is used to notify upper-layer policies which sensor data is valid; Aligned confidence intervals generated based on the covariance of causal delay fields are used to inform the upper-layer policy of the confidence level of the decision frame.
[0102] This disclosure provides a possible implementation method in which, when the alignment confidence of a key modality is lower than a first threshold or the default amount of the action-causal directed acyclic graph exceeds a second threshold, the robot's speed is controlled to scale, position is maintained, or it backs away along the most recently verified free space segment.
[0103] In this embodiment, if the real-time confidence level of a key modality (such as force perception in a contact task) is lower than a safety threshold, it indicates that the modality may have failed or been interfered with. If the breach amount (i.e., causal margin) of the acyclic graph exceeds a preset threshold, it indicates that the actual timing deviates significantly from the expected causal graph, possibly due to unmodeled external disturbances or sensor failures. These two conditions correspond to two types of anomalies: unreliable perception and physical inconsistency, respectively, covering the main sources of safety risks in embodied tasks.
[0104] If the confidence level of the critical modality is lower than the safety threshold (i.e., the first threshold) or the causal default exceeds the preset threshold (i.e., the second threshold), the robot's speed is controlled to scale, position is maintained, or it regresses along the most recently verified free space segment; after recovery, a new event anchor bundle must be formed and the phase reconfirmed before the original action can continue.
[0105] This mechanism ensures that alignment errors have clear control consequences, preventing multimodal fusion models from outputting high-confidence actions even under unknown misalignments.
[0106] This disclosure provides an alignment device for multimodal sensing data, such as... Figure 3 As shown, the device 30 may include: The first processing module 301 is used to acquire multimodal perception data, detect identifiable responses caused by the same physical state change in each modal perception data, and determine the set of identifiable responses as physical causal event anchors; wherein, the physical state change is a state transition event caused by robot action or robot interaction with an object.
[0107] The second processing module 302 is used to generate an event anchor bundle based on the physical causal event anchor; wherein the event anchor bundle includes multiple event anchors.
[0108] The optimization module 303 is used to jointly optimize the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor within the sliding time window; wherein, the causal delay field is used to map the local time of the sensing data of each modality to the common physical time axis.
[0109] The third processing module 304 is used to map the local time of each modal sensing data to a common physical time axis based on the optimized causal delay field, and generate a multimodal decision frame according to the action phase boundary preservation rule; wherein, the multimodal decision frame includes at least the multimodal sensing data after physical causal alignment.
[0110] The apparatus of this disclosure embodiment can execute the method provided in this disclosure embodiment, and its implementation principle is similar, and it has corresponding technical effects. The actions performed by each module in the apparatus of each embodiment of this disclosure correspond to the steps in the method of each embodiment of this disclosure. For a detailed functional description of each module of the apparatus, please refer to the description in the corresponding method shown above, and it will not be repeated here.
[0111] This disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method provided in any optional embodiment of this disclosure. Compared with the prior art, it can achieve the technical effects of high precision, strong robustness, and adaptive dynamic alignment of heterogeneous modalities.
[0112] In one alternative embodiment, an electronic device is provided, such as Figure 4 As shown, Figure 4 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure.
[0113] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure herein. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0114] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0115] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.
[0116] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.
[0117] Electronic devices include, but are not limited to, robots.
[0118] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.
[0119] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.
[0120] It should be understood that although arrows indicate various operation steps in the flowcharts of the embodiments of this disclosure, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of the embodiments of this disclosure, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured as required, and the embodiments of this disclosure do not limit this.
[0121] The above description is only an optional implementation method for some implementation scenarios of this disclosure. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this disclosure without departing from the technical concept of this disclosure also fall within the protection scope of the embodiments of this disclosure.
Claims
1. A method for aligning multimodal sensing data, characterized in that, include: Acquire multimodal sensing data, detect identifiable responses caused by the same physical state change in each modal sensing data, and determine the set of identifiable responses as physical causal event anchors; wherein, the physical state change is a state transition event caused by robot action or robot interaction with an object; An event anchor bundle is generated based on the physical causal event anchor; wherein the event anchor bundle includes a plurality of the event anchors; Within a sliding time window, the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor are jointly optimized; wherein, the causal delay field is used to map the local time of each modality sensing data to the common physical time axis; Based on the optimized causal delay field, the local time of each modal sensing data is mapped to the common physical time axis, and a multimodal decision frame is generated according to the action phase boundary preservation rule; wherein, the multimodal decision frame includes at least the multimodal sensing data after physical causal alignment.
2. The method according to claim 1, characterized in that, The physical causal event anchor includes event type, modal local time, host reception time, object identifier, action identifier, and confidence level; The event types of the physical causal event anchor include at least one of the following: action start, mechanism start, initial contact, stable contact, slip, collision, object displacement, and release.
3. The method according to claim 2, characterized in that, The generation of event anchor bundles based on the physical causal event anchors includes: Candidate event anchors are filtered from the generated event anchors based on the object identifier and the action identifier; Based on the causal reachability relationships in a predefined directed acyclic graph of action causality, at least two cross-modal candidate events with time intervals that meet preset conditions and have causal correlation are grouped into the event anchor bundle.
4. The method according to claim 3, characterized in that, The event anchor bundle includes at least one robot control mode event anchor and one external environment mode event anchor.
5. The method according to claim 1, characterized in that, The functional expression for the causal time delay field is: in, For modality The mapping function, The local time of the mode. Let be the spline basis function matrix. is the spline control point vector.
6. The method according to claim 5, characterized in that, The method further includes: A derivative constraint is applied to the function of the causal time-delay field to maintain time monotonicity; the derivative constraint is: in, and For the preset positive constant, Let be the time derivative of the function of the causal time-delay field.
7. The method according to claim 1, characterized in that, The objective function for jointly optimizing the causal time delay fields of each mode and the common physical moment corresponding to the event anchor bundle includes at least the following: The data alignment term is used to force the local time of different modes within the same event anchor bundle to converge to the same physical time after being mapped to a common physical time axis; Causal penalty term, used to ensure that the optimization result conforms to the physical causality law; A smoothing constraint term is used to limit the second-order rate of change of the causal time delay field to prevent the causal time delay field from experiencing severe jitter. The reception time prior is used to treat the reception time of the sensed data as a weak prior.
8. The method according to claim 1, characterized in that, The generation of multimodal decision frames according to the action phase boundary preservation rule includes: When the target physical moment falls within the protection width of the action phase boundary, the observations before the boundary and the observations after the boundary are aggregated to generate two independent candidate decision frames, and one of them is selected as the final output based on the causal consequence events that have occurred.
9. The method according to claim 1 or 8, characterized in that, The multimodal decision frame includes modal features, common physical time, action phase label, modality missing mask, and aligned confidence intervals generated based on causal delay field covariance.
10. The method according to claim 1, characterized in that, The method further includes: When the alignment confidence of a key modality is lower than a first threshold or the default amount of the causal directed acyclic graph of actions exceeds a second threshold, the robot's speed is controlled to scale, position is maintained, or it regresses along the most recently verified free space segment.
11. A device for aligning multimodal sensing data, characterized in that, include: The first processing module is used to acquire multimodal perception data, detect identifiable responses caused by the same physical state change in each modal perception data, and determine the set of identifiable responses as physical causal event anchors; wherein, the physical state change is a state transition event caused by robot action or robot interaction with an object; The second processing module is used to generate an event anchor bundle based on the physical causal event anchor; wherein the event anchor bundle includes a plurality of the event anchors; An optimization module is used to jointly optimize the function parameters of the causal delay field of each modality and the common physical time corresponding to the event anchor within a sliding time window; wherein, the causal delay field is used to map the local time of the sensing data of each modality to the common physical time axis; The third processing module is used to map the local time of each modal sensing data to the common physical time axis based on the optimized causal delay field, and generate a multimodal decision frame according to the action phase boundary preservation rule; wherein, the multimodal decision frame includes at least the multimodal sensing data after physical causal alignment.
12. An electronic device, comprising: A memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the method of any one of claims 1 to 10.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 10.
14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 10.
