Object physical property estimation method, device and medium based on machine vision observation

Abstract

The application discloses an object physical property estimation method and device based on machine vision observation and a medium, and relates to the field of embodied intelligent technology.The method comprises the following steps: acquiring a video frame sequence of an object moving from a first physical state to a second physical state; performing object segmentation and motion tracking on the video frame sequence to obtain an object segmentation mask and an object motion trajectory; and inputting the object segmentation mask and the object motion trajectory into a physical inverse model to obtain a physical property prediction value of the object; wherein the physical inverse model is obtained by training based on visual data of object interaction and corresponding physical property true values.The method extracts an object segmentation mask and an object motion trajectory from a video frame sequence, and uses a physical inverse model to infer the internal physical property of an object from visual motion appearance, thereby realizing effective estimation of key physical properties of an object under non-contact conditions.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method, device and medium for estimating the physical properties of objects based on machine vision observation. Background Technology

[0002] In the field of embodied intelligence, when robots perform tasks such as grasping and pushing, they not only need to identify the type and pose of objects, but also need to know the physical properties of the objects, such as mass, surface friction coefficient, and moment of inertia, in order to rationally plan the maneuvering force and motion trajectory. However, these physical properties are not intuitively visible visual features; they are implicit in the mechanical interaction between the object and its environment. In motion planning and control driven by physical models, the lack of physical parameters of the object will directly lead to the mechanical model's inability to accurately predict the motion state after contact, making it difficult for the robot to achieve safe and dexterous operations.

[0003] In current technologies, the primary method for acquiring the physical properties of objects still relies on physical contact. For example, robots repeatedly and tentatively grasp objects with grippers, gradually increasing the gripping force until the object is stably lifted, thereby inferring its mass and surface friction characteristics; or they use mechanical sensors integrated into the end effector for direct measurement. However, these contact methods require a complete physical test cycle for each operation, which is inefficient and poses a risk of damage to fragile or easily deformable objects. In open scenarios such as service robots and field operations, robots often deal with one-time, unknown objects, making pre-contact testing impractical.

[0004] In the field of machine vision, existing research has attempted to estimate the physical properties of objects from images. However, current methods typically rely on prior identification of the object's material or category. They require pre-establishing a mapping database between material types and physical property values, and cannot provide effective property estimations when encountering unfamiliar objects not included in the database. Furthermore, obtaining accurate physical property annotation data for real-world objects is extremely difficult. Measuring the mass, moment of inertia, and coefficient of friction of a large number of objects individually requires significant manpower and specialized equipment, resulting in a severe scarcity of training data in real-world scenarios. These limitations mean that robots generally lack reliable non-contact physical perception capabilities when facing unknown objects, becoming a key bottleneck restricting the autonomous operation of embodied intelligent systems in the real world. Summary of the Invention

[0005] This invention provides a method, device, and medium for estimating the physical properties of objects based on machine vision observation. The technical problem it aims to solve is: how to provide a solution that can effectively estimate the key physical properties of objects under non-contact conditions.

[0006] In a first aspect, the present invention provides a method for estimating the physical properties of an object based on machine vision observation, comprising: Acquire a video frame sequence of an object moving from a first physical state to a second physical state; The video frame sequence is subjected to object segmentation and motion tracking to obtain the object segmentation mask and the object motion trajectory; The object segmentation mask and the object's motion trajectory are input into the physical inverse model to obtain the predicted physical properties of the object. The physical inverse model is trained based on visual data of object interactions and the corresponding ground truth values ​​of physical attributes.

[0007] Optionally, the physical inverse model is trained in the following manner: In a physical simulation environment, the physical properties of virtual objects are randomly set to simulate the interactive motion of the virtual objects and generate a simulation video frame sequence and the corresponding true values ​​of physical properties. An initial neural network model is trained using the simulated video frame sequence to obtain a pre-trained physical inverse model; Obtain the labeled video frame sequence of object interactions in a real scene and the corresponding physical attribute labels, and fine-tune the pre-trained physical inverse model based on the labeled video frame sequence to obtain the physical inverse model.

[0008] Optionally, the randomization setting of the physical properties of the virtual object includes: S21, define the corresponding probability distribution type and distribution parameters for mass, friction coefficient and moment of inertia respectively, the distribution parameters are determined based on the statistical prior of the properties of real-world objects; S22, Based on the probability distribution type and distribution parameters, independently sample to obtain random values ​​of mass, friction coefficient, and moment of inertia; S23, The sampled random values ​​of mass, friction coefficient, and moment of inertia are used as a set of attribute combinations and assigned to a target virtual object in the physical simulation environment; S24. Repeat steps S22-S23 to generate multiple target virtual objects with different attribute combinations, and record the random values ​​of mass, friction coefficient and moment of inertia assigned to each target virtual object as the true values ​​of the physical attributes corresponding to each target virtual object.

[0009] Optionally, the step of fine-tuning the pre-trained physical inverse model based on the labeled video frame sequence to obtain the physical inverse model includes: Extract the object segmentation mask and object motion trajectory from the labeled video frame sequence; The extracted object segmentation mask and object motion trajectory are input into the pre-trained physical inverse model to obtain the predicted physical properties; Calculate the error between the predicted physical property and the labeled physical property value; Based on the error, only the parameters of the network used for regression prediction in the pre-trained physical inverse model are adjusted, while the parameters of the network used for spatiotemporal feature extraction remain unchanged, to obtain the physical inverse model.

[0010] Optionally, the step of performing object segmentation and motion tracking on the video frame sequence to obtain an object segmentation mask and an object motion trajectory includes: Instance segmentation is performed on each frame of the video frame sequence to extract the region where the interacting object is located, and an object segmentation mask sequence is generated. The centroid position of the object in each frame of the image is determined based on the object segmentation mask sequence. An initial object trajectory is generated based on the centroid position; The initial object trajectory is normalized to obtain the object's trajectory.

[0011] Optionally, generating the initial object trajectory based on the centroid position includes: Detect at least one feature point among corner points and edge points within the object region of the object segmentation mask sequence; Optical flow tracking is performed on the feature points between adjacent frames to obtain the motion trajectory of the feature points; A random sampling consensus algorithm is used to fit the motion trajectory of the feature points to a rigid body motion model, and outlier feature points that do not conform to the laws of rigid body motion are removed to obtain the motion trajectory of the center of mass of the object. The motion trajectory of the center of mass is then used as the initial motion trajectory of the object.

[0012] Optionally, the normalization process for the initial object trajectory to obtain the object trajectory includes: Scaling each mask in the object segmentation mask sequence to a uniform fixed size; A local coordinate system is established with the centroid position of the object in the first frame of the video frame sequence as the origin, and the coordinate axes of the local coordinate system are aligned with the direction of the image coordinate system. The coordinates of each point in the initial object's trajectory are transformed to the local coordinate system to obtain the normalized trajectory. Calculate the average area of ​​the object in the object segmentation mask sequence, divide the coordinate values ​​of each point in the normalized motion trajectory by the square root of the average area to obtain the scale-normalized motion trajectory, and use the scale-normalized motion trajectory as the object motion trajectory.

[0013] Optionally, the physical inverse model includes a spatiotemporal feature extraction network and a regression network, wherein the spatiotemporal feature extraction network is a convolutional neural network with three-dimensional convolutional kernels; The step of inputting the object segmentation mask and the object's motion trajectory into the physical inverse model to obtain the predicted physical properties of the object includes: The object segmentation mask and the object motion trajectory are concatenated in the channel dimension to generate a multi-channel spatiotemporal input tensor; The multi-channel spatiotemporal input tensor is input into the spatiotemporal feature extraction network, and the three-dimensional convolution kernel is used to perform convolution in both the spatial and temporal dimensions to obtain a spatiotemporal feature map. The spatiotemporal feature map is input into the regression network to obtain the predicted physical properties of the object.

[0014] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0015] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0016] This invention provides a method, device, and medium for estimating the physical properties of objects based on machine vision observation. It involves acquiring a video frame sequence of an object moving from a first physical state to a second physical state, performing object segmentation and motion tracking on the video frame sequence to obtain an object segmentation mask and an object motion trajectory. These are then input into a physical inverse model to obtain predicted values ​​for the object's physical properties. The object segmentation mask and the object motion trajectory describe the object's behavioral characteristics during interaction from the spatial appearance dimension and the temporal motion dimension, respectively, providing complete visual cues for physical property inference. Furthermore, the physical inverse model is trained based on visual data of object interaction and corresponding ground truth values ​​of physical properties. By learning the inverse mapping relationship from visual motion appearance to intrinsic physical properties, it enables the inference of object physical properties using only visual observation data. Therefore, the entire scheme achieves effective estimation of key physical properties of objects without relying on any physical contact. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for estimating the physical properties of an object based on machine vision observation, provided in an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0021] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0022] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0023] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0024] Please see Figure 1 This invention provides a method for estimating the physical properties of objects based on machine vision observation. This method is applied to scenarios where embodied intelligent robots perform non-contact perception of manipulated objects. In the field of embodied intelligence, before performing physical interaction operations such as grasping and pushing, robots not only need to identify the object's category and spatial pose, but also need to know the object's intrinsic physical properties such as mass, surface friction coefficient, and moment of inertia in advance. This provides crucial prior information for precise grasping force control and safe motion planning. However, these physical properties are not surface optical features that can be directly observed through visual sensors; they are implicit in the dynamic response process of the object's mechanical interaction with its environment. Existing methods for acquiring physical properties mainly rely on physical contact, such as repeatedly and tentatively grasping the object with grippers to gradually increase the gripping force until it is lifted, or directly measuring it by integrating multi-dimensional force / torque sensors. These methods are inefficient and pose a risk of damage to fragile or easily deformable items. Machine vision refers to the technical field of acquiring image or video data through visual sensors such as cameras and using computer algorithms to simulate the human visual system to process and understand the data, thereby endowing machines with environmental perception capabilities.

[0025] Therefore, the technical solution of this embodiment enables robots to "see" the physical properties of objects through machine vision. Its core concept lies in "inferring" properties rather than directly "measuring" them. By observing the complete interactive process of an object moving from one physical state to another in a video, such as the sliding process of being pushed from rest and then slowed down by friction before coming to rest again, two complementary visual cues—the object's appearance mask and its motion trajectory—are extracted. Then, using a physical inverse model trained with a large amount of data, the underlying physical properties driving this motion are inferred from the appearance of these visual movements. Specifically, the method includes the following steps: S1, acquire the video frame sequence of the object moving from the first physical state to the second physical state.

[0026] In specific implementation, the system acquires a sequence of video frames showing the object moving from a first physical state to a second physical state. The first physical state refers to a definable stable mechanical state of the object before it is subjected to interactive forces, such as a stationary state or a state of uniform motion along a straight line; this invention does not specifically limit this state. The second physical state refers to a new stable mechanical state that the object eventually reaches after escaping the interactive forces and undergoing self-adjustment, such as a state of returning to a stationary state due to frictional energy dissipation, or a state of motion at a new constant speed; this invention does not specifically limit this state either. Furthermore, this video frame sequence is a set of image frames continuously acquired by a vision sensor (e.g., an RGB camera or a depth camera) mounted on the robot and arranged in chronological order. It completely records the continuous spatiotemporal process of the target object's change in motion state under interactive forces.

[0027] S2, Perform object segmentation and motion tracking on the video frame sequence to obtain the object segmentation mask and the object motion trajectory.

[0028] In specific implementation, the acquired video frame sequence is subjected to object segmentation and motion tracking to obtain an object segmentation mask and object motion trajectory. Further, object segmentation refers to a pixel-level image processing procedure that separates a specific object of interest from the background and other objects in a video frame, with the aim of filtering out irrelevant visual information. The object segmentation mask is the output of the object segmentation step, specifically represented as a binary image sequence of the same size as the original image frames. In each frame, pixels belonging to the target object are labeled with specific foreground values, while all other pixels not belonging to the object are labeled with background values, thereby accurately outlining the independent spatial contour and geometric deformation of the object at each moment.

[0029] Furthermore, object motion tracking refers to the process of continuously estimating and recording the positional changes of a target object within a continuous sequence of video frames. The object trajectory is the output of the motion tracking step; it describes the path of the object's position change as the video frame timeline progresses, typically represented as a sequence of pixel coordinates of the object's centroid in each frame. Further, the segmentation and tracking operations refine the original redundant pixel information into two highly abstract and complementary modalities: the object segmentation mask reflects the object's geometry and scale information, while the object trajectory quantifies the object's kinematic response to forces during interaction.

[0030] In some preferred embodiments, the step of performing object segmentation and motion tracking on the video frame sequence to obtain an object segmentation mask and an object motion trajectory includes: performing instance segmentation on each frame of the video frame sequence, extracting the region where the interacting object is located, and generating an object segmentation mask sequence; determining the centroid position of the object in each frame of the image based on the object segmentation mask sequence; generating an initial object motion trajectory based on the centroid position; and normalizing the initial object motion trajectory to obtain the object motion trajectory.

[0031] In practice, accurately extracting the appearance mask and motion trajectory of the target object from the original video frame sequence is the data foundation for the entire physical attribute estimation process. If the edges of the segmentation mask are rough or contain background noise, or if the extracted motion trajectory has jitter and deviations, these errors in the front-end processing will be amplified layer by layer by the physical inverse model during the reverse inference process, directly affecting the accuracy and reliability of the final physical attribute prediction. Therefore, this embodiment adopts a method combining instance segmentation and frame-by-frame centroid localization, unifying pixel-level appearance extraction and motion information acquisition into a fine processing chain, providing high-quality basic data for the physical inverse model. The specific implementation is as follows: First, instance segmentation is performed on each frame of the acquired video frame sequence to extract the precise region where the interacting object is located, thereby generating a sequence of object segmentation masks arranged in chronological order. Instance segmentation is a more refined task in computer vision than semantic segmentation; it requires not only distinguishing which class of an object a pixel belongs to, but also different individual instances of the same class of object. Furthermore, this step can be performed using a deep learning-based instance segmentation network model (such as Mask R-CNN). For each frame of the video frame sequence, this model can automatically detect all potential object instances in the image and generate a pixel-level binary segmentation mask for each instance. Further, this step focuses only on the target object that is interacting, thus selecting its corresponding mask. Arranging the masks corresponding to all video frames in chronological order constitutes the object segmentation mask sequence. Furthermore, each mask in this sequence accurately labels the region occupied by the target object in the image space at the corresponding time, cleanly eliminating interference from the background and other irrelevant objects, providing pure spatial geometric information for the physical inverse model.

[0032] Furthermore, based on the generated object segmentation mask sequence, the centroid position of the object within each frame of the sequence is precisely determined. The image centroid is the weighted average position of pixel locations in an image, weighted by their pixel values. For a binary object segmentation mask, it represents the geometric center of the object region. By calculating the zeroth moment (i.e., the sum of all pixel values ​​within the mask region, representing the object area) and the first moment (representing the weighted sum of pixel coordinates) of the mask image, and dividing the first moment by the zeroth moment, the centroid pixel coordinates of the object in that frame can be obtained. This calculation is performed on all frames in the mask sequence to obtain a temporally ordered sequence of centroid positions.

[0033] Furthermore, based on the determined centroid position of each frame, an initial object motion trajectory is generated. The centroid pixel coordinates calculated for each frame are then connected sequentially according to their temporal order in the video, forming a continuous path within the two-dimensional image plane that describes the evolution of the object's spatial position over time. This path is the initial object motion trajectory. Moreover, this initial trajectory is the most direct and concise mathematical expression of the object's overall translational motion, directly reflecting the object's dynamic response to the net external force.

[0034] Furthermore, the generated initial object trajectory undergoes a series of normalization processes to obtain the final object trajectory used as input to the model. The main purpose of the normalization process is to eliminate the influence of external factors unrelated to the object's intrinsic physical properties, such as shooting angle, distance between the camera and the object, and image resolution, on the trajectory data. This ensures that the processed trajectory consistently and stably reflects the object's true relative motion pattern, regardless of the shooting conditions. Ultimately, the normalized trajectory serves as the final object trajectory provided to the physical inverse model.

[0035] This embodiment performs pixel-level segmentation of video frame sequences frame by frame using an instance segmentation network. Under complex and dynamic background conditions, it can accurately locate and separate the spatial region of the target object, generating a sharp and clean object segmentation mask sequence. This effectively eliminates the influence of background interference and potential occlusions, ensuring the purity of the input appearance information. Furthermore, based on this, a high-precision centroid position of the object is calculated using image moments and concatenated into an initial motion trajectory, achieving robust extraction of the object's overall translational information at sub-pixel accuracy. Further, by normalizing the initial trajectory, the influence of irrelevant changes in shooting conditions and image scale on the trajectory data is eliminated. This allows the trajectory data input to the physical inverse model to focus on the core motion features reflecting the object's essential physical properties, laying a solid and reliable data foundation for subsequent high-precision physical property inference.

[0036] In some preferred embodiments, generating the initial object motion trajectory based on the centroid position includes: detecting at least one feature point among corner points and edge points within the object region of the object segmentation mask sequence; performing optical flow tracking on the feature points between adjacent frames to obtain the feature point motion trajectory; using a random sampling consensus algorithm to fit the feature point motion trajectory to a rigid body motion model, eliminating outlier feature points that do not conform to the rigid body motion law, obtaining the centroid motion trajectory of the object, and using the centroid motion trajectory as the initial object motion trajectory.

[0037] In practice, the trajectory is generated directly by calculating and concatenating the centroids of the mask for each frame. This method is simple and effective when dealing with pure translational motion. However, when the object rotates significantly or there is local occlusion, the trajectory relying solely on the centroid loses valuable rotational motion information. How the object rotates is precisely the key physical clue for inferring its moment of inertia and friction distribution. Furthermore, local occlusion or segmentation noise may also cause occasional deviations in the centroid calculation of some frames. Therefore, this embodiment introduces a method based on dense feature point optical flow tracking and robust rigid body motion model fitting to generate initial object motion trajectories with richer physical information and stronger anti-interference capabilities. The specific implementation is as follows: First, within the object region defined by the generated object segmentation mask sequence, at least one feature point is detected, either a corner point or an edge point. Feature points are pixels in an image that possess distinct local structure and are easily re-identified and located in consecutive frames. Corner points refer to points where two or more edges intersect, or points with drastic local brightness changes; edge points are points located on the boundaries of an object's contour. These feature points act like "virtual markers" attached to the object's surface and contour, and their collective motion can comprehensively reflect the object's overall translation and rotation around its center of mass. Corner points and edge points are chosen because they possess stable mathematical properties that allow for robust tracking in the image.

[0038] Furthermore, for the detected feature points, optical flow tracking is performed between adjacent frames of the video frame sequence to obtain the motion trajectory of each feature point. Optical flow tracking is an algorithm based on the assumption of image brightness constancy. It iteratively optimizes and finds the most likely corresponding position of a feature point in the previous frame in the current frame, thereby obtaining the velocity vector of that point. Further, for each successfully detected feature point in each frame, optical flow tracking is repeatedly performed frame by frame, thus establishing a pixel coordinate sequence describing the cross-frame motion of each feature point, i.e., the feature point's motion trajectory. This set of trajectories constitutes a dense, holographic description of the motion of each point on the object's surface, containing both translational and rotational information.

[0039] Furthermore, the Random Sample Consensus (RANSAC) algorithm is used to fit a rigid body motion model with outlier removal to the motion trajectories of all the feature points obtained in the previous step, ultimately obtaining the motion trajectory of the object's center of mass, which is then used as the initial object motion trajectory. The core idea of ​​the RANSAC algorithm lies in iteration: each time, the minimum number of samples (i.e., the smallest subset) required to solve the model are randomly selected from the dataset to fit a rigid body motion model, and then the consistency (number of interior points) of all other samples in the dataset with the model is counted. Further, after a large number of iterations, the model that can contain the most interior point samples is the optimal model, while samples that deviate too much from this optimal model are identified as outliers caused by segmentation errors, local deformation, or tracking drift, and are removed. Furthermore, the rigid body motion model assumes that the distance between any two points on the object remains constant during its motion, and its overall motion can be completely decomposed into a unique rotational motion and translational motion, which conforms to the physical characteristics of most man-made objects. By using all the interior feature points and their trajectories retained after RANSAC filtering, the rotation and translation parameters of the object in each frame can be accurately solved by the least squares method, thereby calculating a more accurate trajectory of the object's center of mass.

[0040] This embodiment expands the description of object motion from a single centroid point to a dense set of feature points attached to the object's surface through feature point detection and optical flow tracking. This comprehensively and synchronously captures the translational and rotational behaviors of the object during interaction, providing crucial and rich motion information for subsequent inference of physical properties related to rotational inertia. Furthermore, based on this, it innovatively employs the Random Sample Consensus Algorithm (RANSAC) for rigid body motion model fitting. Leveraging the prior knowledge that "the motion of the main body of an object is essentially rigid body motion," it can automatically identify and accurately remove outlier feature points that do not conform to the overall motion law due to local deformation, occlusion, or tracking errors. This further enhances the physical robustness of the final generated centroid motion trajectory, enabling reliable and stable object motion trajectories even under non-ideal conditions with significant occlusion or complex image noise, significantly improving the accuracy and stability of the entire system's physical property estimation in complex real-world scenarios.

[0041] In some preferred embodiments, the normalization process of the initial object motion trajectory to obtain the object motion trajectory includes: scaling each mask in the object segmentation mask sequence to a uniform fixed size; establishing a local coordinate system with the centroid position of the object in the first frame of the video frame sequence as the origin, wherein the coordinate axes of the local coordinate system are aligned with the direction of the image coordinate system; transforming the coordinates of each point in the initial object motion trajectory to the local coordinate system to obtain the normalized motion trajectory; calculating the average area of ​​the object in the object segmentation mask sequence, dividing the coordinate values ​​of each point in the normalized motion trajectory by the square root of the average area to obtain the scale-normalized motion trajectory, and using the scale-normalized motion trajectory as the object motion trajectory.

[0042] In practice, the image size of an object and the pixel scale occupied by its motion trajectory change significantly and non-linearly with the distance between the camera and the object. The closer the camera is to the object, the larger the image area of ​​the object, and the greater the pixel displacement corresponding to the same physical distance moved on the image plane. This scale effect, unrelated to physical properties, can seriously mislead the inference of the physical inverse model. If this raw pixel scale information, which is strongly correlated with shooting conditions, is directly input into the physical inverse model, the model is likely to incorrectly attribute this visual difference caused by shooting distance to differences in the object's own physical properties (such as mass or coefficient of friction), leading to inconsistent estimation results for the same object at different distances, severely impairing the model's cross-scene generalization ability. Therefore, this embodiment eliminates the influence of shooting distance and viewpoint on motion representation from a mathematical perspective by establishing the object's own coordinate system and introducing an area scale factor. The specific implementation is as follows: First, each mask in the previously generated object segmentation mask sequence is scaled to a uniform, fixed size using image scaling operations. For example, the largest bounding rectangle of the object in the mask can be scaled to a standard size of 128×128 pixels or 256×256 pixels. During scaling, the original aspect ratio of the object region is maintained to ensure that the object's shape information is not distorted. This step allows all objects to be processed on a uniform scale, eliminating differences in object image size caused by camera distance and achieving spatial scale appearance normalization.

[0043] Furthermore, a local coordinate system is established, fixed to the object itself, with the centroid of the object in the first frame of the video frame sequence as the origin. The X and Y axes of this local coordinate system are consistent with the coordinate axes of the original image coordinate system to ensure that the directions are not confused. Furthermore, since the origin of this local coordinate system is chosen at the object's initial centroid, its coordinate axes move with the object. Therefore, the trajectory described in this coordinate system reflects the displacement change of the object relative to its initial starting position at different times, rather than the absolute position change in the image. This step eliminates the influence of the object's initial absolute position in the image on the trajectory representation, achieving position-independent motion description.

[0044] Furthermore, the coordinates of each sampling point in the previously generated initial object motion trajectory are transformed one by one from the original global image coordinate system to the newly established local coordinate system to obtain a normalized motion trajectory. Specifically, the transformation method is as follows: the original X and Y coordinates of the centroid in the first frame image are subtracted from the original X and Y coordinates of the centroid in the trajectory at each moment, and the resulting difference coordinates are the displacement sequence of the object relative to the initial position.

[0045] Further, the average area of ​​the object within all masks in the object segmentation mask sequence is calculated. Since the imaged area of ​​an object is inversely proportional to the square of the distance from the camera to the object, and the pixel displacement of the object in the image is inversely proportional to the camera distance, dividing the X and Y coordinate values ​​of each point in the normalized motion trajectory by the square root of the average area mathematically approximates the scaling effect caused by the camera distance, achieving scale normalization. Further, the trajectory obtained after this step is called the scale-normalized motion trajectory, and it is used as the final object motion trajectory for the physical inverse model. Furthermore, the motion amplitude of this scale-normalized trajectory no longer depends on the shooting distance, but mainly reflects the motion pattern of the object at the real physical scale, enabling the model to accurately perceive key dynamic parameters such as acceleration.

[0046] This embodiment successfully migrates the motion trajectory from the volatile absolute coordinate system of the image to the relative reference system of the object itself by establishing a local coordinate system with the initial centroid of the object as the origin and performing coordinate transformation, completely eliminating the representation differences introduced by different initial positions of the object in the image. Furthermore, through scale normalization processing based on the average area of ​​the object's imaging, the pixel amplitude of the motion trajectory is correlated and corrected with the imaging scale of the object, mathematically offsetting the nonlinear effect of changes in camera shooting distance on the trajectory amplitude. Moreover, the object's motion trajectory after these two normalization processes can purely and consistently express the object's motion pattern and amplitude in a unified standardized space. This allows the downstream inverse physics model to focus entirely on deep reasoning of physical properties from the motion pattern itself, without being interfered with by irrelevant factors related to shooting conditions. This fundamentally and significantly improves the model's cross-scene generalization ability and the consistency of multiple estimation results under different shooting distances and resolutions.

[0047] S3, input the object segmentation mask and the object motion trajectory into the physical inverse model to obtain the predicted physical attribute values ​​of the object; wherein, the physical inverse model is trained based on the visual data of object interaction and the corresponding ground truth values ​​of physical attributes.

[0048] In practice, the obtained object segmentation mask and the object's motion trajectory are input into the physical inverse model to obtain the predicted values ​​of the object's physical properties. The physical inverse model is a pre-trained deep neural network model; the "inverse" in its name refers to its relationship to the forward solution process in classical mechanics. In classical mechanics, given the complete physical properties of an object (mass, coefficient of friction, moment of inertia) and initial motion conditions, the future motion trajectory of the object can be completely simulated through physical laws and numerical calculations—a forward process "from cause (property) to effect (motion)." Furthermore, the physical inverse model does the exact opposite: it observes the complete visual motion phenomenon that an object has already undergone (i.e., the "effect") and infers what intrinsic physical properties the object must possess (i.e., the "cause") to produce such motion. This unique reasoning ability of "inferring cause from effect" is the core technological innovation of this solution in achieving physical property perception under non-contact conditions. Furthermore, the predicted physical property value includes at least one of the following properties: the mass of the object, used to quantify the magnitude of the object's inertia, i.e., the object's ability to resist changes in its translational motion state; the coefficient of friction of the object, a dimensionless value used to characterize the frictional force characteristics between the object and the contact surface that hinder their relative sliding tendency; and the moment of inertia of the object, used to quantify the object's ability to resist changes in its rotational motion state.

[0049] Furthermore, the physical inverse model is trained based on visual data of object interactions and the corresponding ground truth values ​​of physical properties. Visual data of object interactions refers to video data recording the mechanical interaction process of objects. Ground truth values ​​of physical properties refer to the numerical labels of physical properties as standard answers, obtained through precision instrument measurement or direct acquisition from a simulation environment. By using a large amount of visual data containing different objects and different interaction methods as input, and their corresponding ground truth values ​​as supervision signals, the neural network model can be iteratively trained to learn a complex nonlinear mapping function from visual appearance to physical properties.

[0050] Furthermore, this embodiment acquires a complete video frame sequence of an object's motion between two physical states, and extracts the object segmentation mask and object trajectory as input to the physical inverse model. This enables the estimation of key physical properties such as object mass and friction coefficient under non-contact conditions based solely on machine vision observation. The object segmentation mask and object trajectory, from two complementary dimensions of spatial appearance and temporal motion, jointly characterize the object's behavioral features in mechanical interaction, providing complete visual cues for the "effect-to-cause" reasoning of the physical inverse model. Furthermore, by learning this complex inverse mapping relationship, the physical inverse model allows the robot to obtain prior physical information about the object without any physical contact before grasping, providing a crucial perceptual foundation for the robot's safe and dexterous manipulation of unknown objects. This effectively overcomes the shortcomings of traditional contact-based measurement schemes, which are characterized by low efficiency and high risk when dealing with fragile, irregular, or unknown objects.

[0051] In some preferred embodiments, the physical inverse model is trained in the following manner: in a physical simulation environment, the physical properties of virtual objects are randomly set, the interactive motion of the virtual objects is simulated, and a simulation video frame sequence and corresponding physical property truth values ​​are generated; an initial neural network model is trained using the simulation video frame sequence to obtain a pre-trained physical inverse model; an annotated video frame sequence of object interactions in a real scene and corresponding physical property annotation values ​​are obtained, and the pre-trained physical inverse model is fine-tuned based on the annotated video frame sequence to obtain the physical inverse model.

[0052] In practice, the inference performance of the physical inverse model is highly dependent on its training process. However, this solution faces a fundamental data dilemma: obtaining accurate true values ​​of the physical properties of objects in motion in the real world is an extremely difficult and costly task. Precisely measuring the mass, friction coefficient, and moment of inertia of each object individually requires significant time and specialized precision measurement equipment, resulting in a severe scarcity of high-quality training data in real-world scenarios, which cannot support the effective training of large-scale deep neural networks. To address this training data bottleneck, this embodiment constructs a training strategy of "simulation pre-training plus real-world fine-tuning." The core of this strategy lies in utilizing the physical simulation environment to generate massive amounts of precisely labeled simulation data at extremely low cost, helping the model learn most of the general physical laws; then, only a very small amount of high-value real-world labeled data is needed to perform lightweight adaptive adjustments to the model, enabling it to transfer excellently to real-world scenarios and bridging the data distribution differences between the simulation and real domains. Specifically, the specific steps for obtaining the physical inverse model are as follows: First, in a physical simulation environment, the physical properties of virtual objects are randomly set, and the interactive motion of these virtual objects is simulated to generate a sequence of simulated video frames and corresponding true values ​​of their physical properties. The physical simulation environment refers to a virtual three-dimensional space constructed using computer software platforms (such as PyBullet, MuJoCo, etc.). The built-in physics engine of such platforms can accurately simulate and calculate the motion trajectory and collision response of virtual objects under force conditions based on Newton's equations of motion. Furthermore, in this environment, a large number of virtual objects with different shapes, sizes, and textures can be randomly generated, and a complete set of physical property values, including but not limited to mass, coefficient of friction, and moment of inertia, can be independently and randomly set for each virtual object. Further, by simulating typical interactive scenarios such as pushing, collision, and free fall, these virtual objects are driven from a first physical state to a second physical state, and a sequence of simulated video frames depicting this complete process is recorded simultaneously. Furthermore, because the physical attribute values ​​are directly set when creating virtual objects, the simulation engine can simultaneously output and record these physical attribute values ​​as true values, forming zero-noise true values ​​of physical attributes that correspond strictly one-to-one with each segment of the simulation video frame sequence, serving as the gold standard label for subsequent model training. The scale of simulation data generation is virtually unlimited, theoretically covering an arbitrarily wide physical attribute space.

[0053] Furthermore, an initial neural network model is trained using the generated sequence of simulated video frames to obtain a pre-trained physical inverse model. During training, the object segmentation mask and object motion trajectory are extracted from each segment of the simulated video frame sequence as model input, and the model outputs a predicted physical attribute. The predicted physical attribute is numerically compared with the corresponding ground truth physical attribute, the error between the two is calculated, and the weight parameters inside the model are iteratively updated using the gradient backpropagation algorithm with the goal of minimizing this error. Furthermore, after sufficient training on simulation data covering a large number of scenes, the model can learn the general basic ability to infer physical attributes from visual motion appearances in the simulation environment, thus forming a pre-trained physical inverse model.

[0054] Furthermore, a sequence of annotated video frames depicting object interactions in a few real-world scenarios, along with their corresponding physical attribute annotations, is acquired. Based on these annotated video frame sequences, the pre-trained physical inverse model is fine-tuned to obtain the final deployable physical inverse model. The annotated video frame sequences from real-world scenarios are obtained through actual physical experiments. During filming, high-precision electronic scales, mechanical testing machines, and other specialized equipment are used to precisely calibrate the target object's mass, friction coefficient, and other attributes, thereby obtaining the physical attribute annotations. Furthermore, this high-quality, real-world annotated data, which is extremely costly to acquire, is used to perform the final stage of supervised training on the pre-trained physical inverse model. This allows the model to adapt to lighting conditions, background textures, and shooting noise that differ between the real-world and simulation environments, completing knowledge transfer and adaptation from the simulation domain to the real domain. The fine-tuned model is the final physical inverse model used for actual inference.

[0055] This embodiment fundamentally solves the dilemma of the extreme scarcity of real physical attribute labeled data, which makes it difficult for models to train effectively, by constructing a two-stage training framework of "simulation pre-training + real fine-tuning". The physical simulation environment can generate simulated video frame sequences and their corresponding zero-noise physical attribute ground truth values ​​at zero cost and in unlimited quantities, providing the model with a large-scale and high-quality basic training supervision signal, enabling it to fully learn general physical laws. Furthermore, based on this, only a very small amount of real scene labeled data obtained through precise measurement needs to be introduced for lightweight fine-tuning, which can enable the model to effectively overcome the data distribution differences between simulation and reality and quickly adapt to the complex characteristics of the real world. Furthermore, this training strategy significantly reduces the dependence on expensive and time-consuming real physical measurements, significantly shortens the cycle from data preparation to model deployment, and makes high-precision visual physical attribute estimation schemes economical and feasible in practical applications, paving the way for the large-scale deployment of embodied intelligent systems in variable physical environments.

[0056] In some preferred embodiments, the randomization of the physical properties of the virtual object includes: S21, defining corresponding probability distribution types and distribution parameters for mass, friction coefficient, and moment of inertia, wherein the distribution parameters are determined based on prior statistical data of real-world object properties; S22, independently sampling random values ​​for mass, friction coefficient, and moment of inertia according to the probability distribution types and distribution parameters; S23, assigning the sampled random values ​​for mass, friction coefficient, and moment of inertia as a set of attribute combinations to a target virtual object in the physical simulation environment; S24, repeating steps S22-S23 to generate multiple target virtual objects with different attribute combinations, and recording the random values ​​for mass, friction coefficient, and moment of inertia assigned to each target virtual object as the true values ​​of the physical properties corresponding to each target virtual object.

[0057] In practical implementation, when generating training data using a physical simulation environment, an effective training sample not only needs accurate true values ​​of physical properties, but also needs to ensure that the simulated object property combinations can statistically reasonably cover and closely approximate the common distribution space of object properties in the real world. If no prior knowledge and completely uniform random sampling is used, it may generate a large number of property combinations that are almost impossible to exist in physical reality and violate common sense, such as objects with "huge mass but a friction coefficient close to zero." Training on these invalid samples will mislead the model's learning of physical laws and may even lead to the learning of incorrect property associations. Therefore, this embodiment solves this simulation data quality problem by introducing a probability distribution sampling mechanism based on statistical prior knowledge of real-world properties, improving the effectiveness and realism of the training data. The specific implementation method is as follows: First, for the three core physical quantities—mass, coefficient of friction, and moment of inertia—we define their corresponding probability distribution types and distribution parameters. These parameters are determined based on statistical priors of the properties of real-world objects. Specifically, for common object categories likely encountered in the target application scenario, such as industrial parts, standard packaging boxes, and everyday household items, statistical information on their physical properties can be collected in advance by consulting materials science databases or conducting sampling experiments. Further, based on the collected statistical data, a log-normal distribution can be determined for mass, with its mean and variance obtained statistically from the logarithmic values ​​of real object masses, as mass distributions are typically skewed towards positive numbers and have a long tail. Further, a beta distribution can be determined for the coefficient of friction, with its distribution parameters determined based on the typical range and central tendency of the friction coefficient of real contact surfaces between 0 and 1. Further, a derived distribution related to the mass and shape distributions can be determined for the moment of inertia, with its distribution parameters calculated based on the statistical laws governing mass distribution and geometry. This step ensures that the sampling range is constrained within reasonable limits of real-world physics.

[0058] Furthermore, based on the probability distribution type and distribution parameters defined for each physical quantity in the previous step, independent random sampling is performed to obtain a random value for mass, a random value for friction coefficient, and a random value for moment of inertia. Furthermore, the sampling process for each attribute is mathematically independent, with each randomly drawing a sample value from its own dedicated probability distribution. Moreover, the advantage of this independent sampling is that it can effectively decouple the artificially strong correlations between attributes, allowing the model to encounter more diverse attribute combinations during training than observed in the real world. At the same time, because the distribution parameters are set based on statistical priors from the real world, the sampling results still generally conform to the frequency patterns of object attribute values ​​in reality, avoiding the generation of physically absurd combinations.

[0059] Furthermore, the random values ​​of mass, friction coefficient, and moment of inertia obtained from this sampling are treated as a complete and mutually matching attribute combination and assigned to a newly created target virtual object in the physical simulation environment. Specifically, in the object attribute setting interface of the simulation engine, the mass parameter of the virtual object is set to the sampled random value of mass, the friction coefficient between it and the contact surface is set to the sampled random value of friction coefficient, and its moment of inertia is set to the sampled random value of moment of inertia, thereby completing the configuration of the physical attributes of the object.

[0060] Furthermore, steps S22 and S23 are repeated, i.e., the processes of "independent sampling" and "assigning virtual objects" are cyclically executed to generate a large number of target virtual objects with different attribute combinations. Simultaneously, the random values ​​of mass, friction coefficient, and moment of inertia assigned to each target virtual object are recorded and saved as unique true values ​​of the physical attributes corresponding to that target virtual object. These recorded precise numerical labels will serve as unique and correct supervision signals in the subsequent model training phase, guiding the model to learn the mapping from vision to attributes.

[0061] It should be noted that the independent sampling of each physical parameter in this embodiment is not unconstrained random generation, but rather strictly based on preset physical constraints. Specifically, the sampling of mass, friction coefficient, and moment of inertia is limited to reasonable values ​​within the statistical distribution range of similar object properties in the real physical world. Furthermore, during the numerical simulation process, the physics simulation engine verifies the interactive motion of each attribute combination according to Newton's laws of motion, eliminating abnormal samples that are physically inconsistent or clearly violate the conservation of energy and momentum. This ensures that the simulation data ultimately used for model training are all physically realizable and mechanically reasonable valid samples.

[0062] This embodiment introduces a probability distribution sampling mechanism based on statistical priors of real-world object attributes, replacing uniform random sampling without prior knowledge. This makes the virtual objects generated in the physical simulation environment more consistent with the statistical distribution of objects in the real physical world in terms of attribute combinations. This effectively avoids generating a large number of physically unrealistic and invalid sample combinations, thereby significantly improving the overall quality and effectiveness of the simulation training data and ensuring that the model focuses its limited training capacity on learning valuable physical laws. Furthermore, each physical attribute adopts an independent sampling strategy, maximizing the diversity of attribute combinations while maintaining the prior constraints of real distribution. This helps the model access a wider range of attribute joint spaces during training, thus significantly enhancing the model's inference robustness and generalization ability when facing unknown objects with rare attribute combinations.

[0063] In some preferred embodiments, the step of fine-tuning the pre-trained physical inverse model based on the labeled video frame sequence to obtain the physical inverse model includes: extracting object segmentation masks and object motion trajectories from the labeled video frame sequence; inputting the extracted object segmentation masks and object motion trajectories into the pre-trained physical inverse model to obtain predicted physical attributes; calculating the error between the predicted physical attributes and the labeled physical attribute values; and based on the error, adjusting only the parameters of the network portion used for regression prediction in the pre-trained physical inverse model while keeping the parameters of the network portion used for spatiotemporal feature extraction unchanged to obtain the physical inverse model.

[0064] In practical implementation, a core technical challenge in the fine-tuning stage of the transfer from the simulation domain to the real domain lies in how to adapt the model to the data distribution of the real world with only a very small number of real labeled samples, without catastrophically destroying the general visual spatiotemporal representation capabilities that the model has already learned on a large amount of simulation data. If all network parameters of the pre-trained physical inverse model are fine-tuned directly, due to the extremely small number of real labeled samples, the model is very likely to "overfit" to these few samples. That is, the model "memorizes" the specific appearance and motion patterns of these few samples, but loses the ability to generalize and infer a wider range of unseen objects. To overcome this problem, this embodiment adopts a parameter freezing fine-tuning strategy, the specific steps of which are as follows: First, object segmentation masks and object motion trajectories are extracted from the acquired annotated video frame sequence. The data processing pipeline for this extraction process is kept completely consistent with the simulation pre-training phase to ensure that the feature distribution and format of the input data remain unchanged.

[0065] Furthermore, the extracted object segmentation mask and object motion trajectory are input into the pre-trained physical inverse model for forward propagation calculation to obtain a predicted physical property.

[0066] Furthermore, the error between the predicted physical attribute and the actually collected labeled physical attribute is calculated. This error is calculated using a preset loss function, such as the commonly used mean squared error loss function, to quantify the degree of deviation between the model's current predicted value and the standard answer.

[0067] Further, and most crucially, backpropagation is performed based on the calculated error. However, when updating the model parameters, only the parameters of the network specifically used for regressing and predicting physical values ​​in the pre-trained physical inverse model are adjusted, while the parameters of the network specifically used for extracting spatiotemporal features from visual and motion data remain completely unchanged. The underlying technical principle is that the network responsible for spatiotemporal feature extraction learns how to understand the "general motion pattern of objects"—a fundamental visual intelligence—from pixels and trajectories. This ability is highly consistent and reusable in both simulation and real-world environments because physical laws are universal. The network responsible for regression prediction maps the abstract "motion pattern" to specific "physical parameter values." This mapping needs to be specifically adapted to potential systemic biases between the simulation and real domains. Therefore, by freezing the feature network and fine-tuning only the regression network, valuable generalized visual spatiotemporal feature knowledge accumulated during the pre-training phase can be perfectly preserved while achieving domain adaptation, resulting in a highly generalizable physical inverse model.

[0068] It should be further explained that, in the physical inverse model constructed in this application, the network used for spatiotemporal feature extraction specifically refers to a convolutional neural network with three-dimensional convolutional kernels. It is responsible for receiving a multi-channel spatiotemporal input tensor composed of object segmentation masks and object motion trajectories. By performing sliding convolution operations simultaneously in the spatial dimension (image height and width) and the temporal dimension (video frame sequence), it extracts a deep spatiotemporal feature map describing how changes in the appearance and motion of an object evolve together over time. The network used for regression prediction specifically refers to a regression network composed of stacked fully connected layers. It is responsible for taking the aforementioned extracted high-dimensional abstract spatiotemporal feature map as input, and through stepwise dimensionality reduction and linear / nonlinear transformations, finally mapping and outputting specific predicted values ​​of the object's physical properties (such as mass, coefficient of friction, and moment of inertia). The strategy of "freezing the former and updating only the latter" adopted in the fine-tuning stage is technically reasonable because: the spatiotemporal feature extraction network learns a general visual motion pattern understanding ability, which is highly reusable in both simulation and real environments due to the universality of physical laws. On the other hand, the regression network needs to be specifically calibrated for the difference in domain distribution between simulation and real data. Therefore, adjusting only the parameters of the regression network can effectively avoid the risk of overfitting due to insufficient real labeled samples while adapting to real scenes, and fully retain the valuable general knowledge accumulated in the pre-training stage.

[0069] This embodiment employs a parameter-freezing fine-tuning strategy. When adjusting the model on a very small amount of real-world labeled data, only the network components directly related to physical parameter regression are updated, while the spatiotemporal feature extraction component responsible for general visual understanding remains stable. Furthermore, this strategy effectively transfers and adapts from simulation to the real-world data domain while preventing catastrophic overfitting due to insufficient fine-tuning samples, maximizing the preservation of the powerful general visual spatiotemporal feature extraction capabilities learned by the pre-trained model on rich simulation data. Moreover, the resulting physical inverse model, even when faced with object shapes, categories, and motion patterns never seen during training, can still output accurate and robust physical property predictions based on stable, invariant feature understanding and specifically adapted regression mappings, significantly improving the model's reliability and generalization performance in real-world, open, and uncertain scenarios.

[0070] In some preferred embodiments, the physical inverse model includes a spatiotemporal feature extraction network and a regression network, wherein the spatiotemporal feature extraction network is a convolutional neural network with three-dimensional convolutional kernels; the step of inputting the object segmentation mask and the object motion trajectory into the physical inverse model to obtain the predicted physical attribute values ​​of the object includes: concatenating the object segmentation mask and the object motion trajectory in the channel dimension to generate a multi-channel spatiotemporal input tensor; inputting the multi-channel spatiotemporal input tensor into the spatiotemporal feature extraction network, and simultaneously convolving it in the spatial and temporal dimensions using the three-dimensional convolutional kernels to obtain a spatiotemporal feature map; and inputting the spatiotemporal feature map into the regression network to obtain the predicted physical attribute values ​​of the object.

[0071] In practice, object segmentation masks belong to the spatial appearance modality, describing the static shape and region of an object; object motion trajectories belong to the temporal motion modality, describing the dynamic process of an object's position changing over time. These two heterogeneous data contain different physical cues and have a natural spatiotemporal coupling relationship. For example, the magnitude of the friction coefficient directly affects the deceleration of an object, which visually manifests as the curvature and nonlinearity of the motion trajectory, while the object's appearance mask may only change when deformation or rotation occurs. If the two modalities are processed separately using independent network branches and then fused at a higher level, it will be difficult to capture the fine-grained, deep coupling relationship between appearance changes and motion changes in the spatiotemporal dimension from the bottom layer, like listening to only the melody and beat separately without perceiving the overall harmony of the music. Therefore, this embodiment designs a spatiotemporal feature extraction network containing three-dimensional convolutional kernels and concatenates the two modalities at the input layer, achieving a tight fusion of appearance and motion information from the bottom layer to capture more discriminative spatiotemporal joint features. The implementation is as follows: The physical inverse model structurally comprises a spatiotemporal feature extraction network and a regression network. The spatiotemporal feature extraction network is specifically defined as a convolutional neural network with a three-dimensional convolutional kernel. Unlike two-dimensional convolutional kernels, which can only slide along the spatial dimensions (height and width of the image) to extract single-frame spatial features, the three-dimensional convolutional kernel possesses three dimensions: height, width, and temporal depth. This allows it to perform sliding convolution operations along the temporal dimension of the input data, thereby directly and simultaneously learning local, short-term spatiotemporal patterns describing how an object's appearance and motion jointly evolve over time, such as "the appearance expands while the motion trajectory begins to decelerate."

[0072] Those skilled in the art will understand that the three-dimensional convolutional kernel is a primitive operation constituting the convolution operation in the spatiotemporal feature extraction network. In actual network implementation, by setting multiple three-dimensional convolutional kernels to form output feature maps along the feature channel dimension, and by stacking multiple layers of three-dimensional convolutional kernels with different receptive field scales, a complete spatiotemporal feature extraction network capable of capturing multi-scale spatiotemporal dependencies can be constructed. The specific number of layers, kernel size, and number of channels of the three-dimensional convolutional kernel are conventional network parameter designs that can be performed by those skilled in the art based on the complexity of the actual task and computational constraints.

[0073] Further, the model inference process begins. First, the object segmentation mask and the object motion trajectory, which serve as input data, are concatenated along the channel dimension in the data preprocessing layer, thereby generating a multi-channel spatiotemporal input tensor that composites information. The object segmentation mask sequence itself is a three-dimensional data block, whose dimensions can be represented by time length, image height, and image width. The object motion trajectory can be encoded into a motion feature map that is fully aligned with the spatiotemporal dimensions of the mask in various ways, such as generating a two-dimensional Gaussian heatmap based on the trajectory points, or generating a multi-channel image containing velocity and acceleration vectors. By concatenating these mask channels representing appearance information and trajectory channels representing motion information along the channel dimension, a multi-channel spatiotemporal input tensor that contains both frame-by-frame detailed appearance and embedded continuous motion information is formed. This early concatenation ensures that subsequent three-dimensional convolutional kernels can simultaneously access the original low-level information from both modalities, thereby learning the inherent spatiotemporal relationship between them.

[0074] Furthermore, the multi-channel spatiotemporal input tensor generated by the concatenation is fed into the spatiotemporal feature extraction network. The three-dimensional convolutional kernels in this network begin to slide and perform convolution operations simultaneously along the spatial (height and width) and temporal dimensions of the input tensor. The spatial convolution handles the appearance features such as shape, contour, and texture within each frame of the image; the temporal convolution captures the continuous changes in these appearance features and the corresponding motion signals over time, such as the deformation process of an object, the increase or decrease in motion speed, and changes in trajectory curvature. More importantly, the three-dimensional convolution can directly learn the causal relationship and complex spatiotemporal synchronization between shape changes and motion changes, such as how the asymmetry of an object's shape causes its rotation trajectory to exhibit a specific pattern of precession. Through layer-by-layer abstraction, the network finally outputs a highly condensed spatiotemporal feature map, which contains all the key spatiotemporal information extracted from the input data.

[0075] Further, the obtained spatiotemporal feature map is input into the regression network. The regression network typically consists of a series of fully connected layers stacked together. Its task is to map the high-dimensional, abstract spatiotemporal feature map into a low-dimensional, concrete physical property numerical space through progressive dimensionality reduction and linear / nonlinear transformations. Those skilled in the art can determine the number of three-dimensional convolutional layers, the kernel size of each layer, and the number of channels based on the feature granularity required for the actual task and computational resource constraints. This is a conventional design in the field and does not constitute a limitation of the present invention. The final output layer of the regression network directly provides the predicted physical property values ​​of the object, for example, a vector containing three floating-point numbers, representing the predicted mass, coefficient of friction, and moment of inertia, respectively.

[0076] This embodiment constructs a multi-channel spatiotemporal input tensor containing complete appearance and motion cues with strict spatiotemporal alignment by concatenating the object segmentation mask and object motion trajectory in the channel dimension at the initial stage of data input. This achieves thorough fusion of multimodal information from the data source. Furthermore, a convolutional neural network with three-dimensional convolutional kernels is employed as the spatiotemporal feature extraction network. This network can perform convolution operations simultaneously in the spatial and temporal dimensions, enabling it to directly learn deep, fine-grained spatiotemporal joint feature patterns between changes in object appearance and motion. Moreover, compared to the common approach of modeling appearance and motion separately and then fusing them, this end-to-end spatiotemporal joint learning method can more effectively capture the deep coupling relationship and causal logic between physical properties such as friction and deceleration behavior, and rotational inertia and changes in rotational attitude, and multimodal visual phenomena. This ensures that the physical inverse model can automatically extract the most crucial complex spatiotemporal cues for physical reasoning from the original multimodal visual observations, ultimately outputting more accurate and robust physical property predictions.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0079] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0080] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute a method for estimating the physical properties of an object based on machine vision observations.

[0081] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0082] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a method for estimating the physical properties of an object based on machine vision observation.

[0083] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0084] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of an object physical property estimation method based on machine vision observation provided in any of the above method embodiments.

[0085] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0086] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0087] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of a machine vision-based object physical property estimation method provided in any of the above-described method embodiments.

[0088] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0089] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0090] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0091] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0093] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0094] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0095] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for estimating the physical properties of objects based on machine vision observation, characterized in that, include: Acquire a video frame sequence of an object moving from a first physical state to a second physical state; The video frame sequence is subjected to object segmentation and motion tracking to obtain the object segmentation mask and the object motion trajectory; The object segmentation mask and the object's motion trajectory are input into the physical inverse model to obtain the predicted physical properties of the object. The physical inverse model is trained based on visual data of object interactions and the corresponding ground truth values ​​of physical attributes.

2. The method for estimating the physical properties of objects based on machine vision observation according to claim 1, characterized in that, The physical inverse model is trained in the following way: In a physical simulation environment, the physical properties of virtual objects are randomly set to simulate the interactive motion of the virtual objects and generate a simulation video frame sequence and the corresponding true values ​​of physical properties. An initial neural network model is trained using the simulated video frame sequence to obtain a pre-trained physical inverse model; Obtain the labeled video frame sequence of object interactions in a real scene and the corresponding physical attribute labels, and fine-tune the pre-trained physical inverse model based on the labeled video frame sequence to obtain the physical inverse model.

3. The method for estimating the physical properties of objects based on machine vision observation according to claim 2, characterized in that, The randomization of the physical properties of virtual objects includes: S21, define the corresponding probability distribution type and distribution parameters for mass, friction coefficient and moment of inertia respectively, the distribution parameters are determined based on the statistical prior of the properties of real-world objects; S22, Based on the probability distribution type and distribution parameters, independently sample to obtain random values ​​of mass, friction coefficient, and moment of inertia; S23, The sampled random values ​​of mass, friction coefficient, and moment of inertia are used as a set of attribute combinations and assigned to a target virtual object in the physical simulation environment; S24. Repeat steps S22-S23 to generate multiple target virtual objects with different attribute combinations, and record the random values ​​of mass, friction coefficient and moment of inertia assigned to each target virtual object as the true values ​​of the physical attributes corresponding to each target virtual object.

4. The method for estimating the physical properties of objects based on machine vision observation according to claim 2, characterized in that, The step of fine-tuning the pre-trained physical inverse model based on the labeled video frame sequence to obtain the physical inverse model includes: Extract the object segmentation mask and object motion trajectory from the annotated video frame sequence; The extracted object segmentation mask and object motion trajectory are input into the pre-trained physical inverse model to obtain the predicted physical properties; Calculate the error between the predicted physical property and the labeled physical property value; Based on the error, only the parameters of the network used for regression prediction in the pre-trained physical inverse model are adjusted, while the parameters of the network used for spatiotemporal feature extraction remain unchanged, to obtain the physical inverse model.

5. The method for estimating the physical properties of objects based on machine vision observation according to claim 1, characterized in that, The step of performing object segmentation and motion tracking on the video frame sequence to obtain the object segmentation mask and object motion trajectory includes: Instance segmentation is performed on each frame of the video frame sequence to extract the region where the interacting object is located, and an object segmentation mask sequence is generated. The centroid position of the object in each frame of the image is determined based on the object segmentation mask sequence. An initial object trajectory is generated based on the centroid position; The initial object trajectory is normalized to obtain the object's trajectory.

6. The method for estimating the physical properties of objects based on machine vision observation according to claim 5, characterized in that, The generation of the initial object trajectory based on the centroid position includes: Detect at least one feature point among corner points and edge points within the object region of the object segmentation mask sequence; Optical flow tracking is performed on the feature points between adjacent frames to obtain the motion trajectory of the feature points; A random sampling consensus algorithm is used to fit the motion trajectory of the feature points to a rigid body motion model, and outlier feature points that do not conform to the laws of rigid body motion are removed to obtain the motion trajectory of the center of mass of the object. The motion trajectory of the center of mass is then used as the initial motion trajectory of the object.

7. The method for estimating the physical properties of objects based on machine vision observation according to claim 5, characterized in that, The normalization process for the initial object trajectory to obtain the object trajectory includes: Scaling each mask in the object segmentation mask sequence to a uniform fixed size; A local coordinate system is established with the centroid position of the object in the first frame of the video frame sequence as the origin, and the coordinate axes of the local coordinate system are aligned with the direction of the image coordinate system. The coordinates of each point in the initial object's trajectory are transformed to the local coordinate system to obtain the normalized trajectory. Calculate the average area of ​​the object in the object segmentation mask sequence, divide the coordinate values ​​of each point in the normalized motion trajectory by the square root of the average area to obtain the scale-normalized motion trajectory, and use the scale-normalized motion trajectory as the object motion trajectory.

8. The method for estimating the physical properties of objects based on machine vision observation according to claim 1, characterized in that, The physical inverse model includes a spatiotemporal feature extraction network and a regression network, wherein the spatiotemporal feature extraction network is a convolutional neural network with three-dimensional convolutional kernels; The step of inputting the object segmentation mask and the object's motion trajectory into the physical inverse model to obtain the predicted physical properties of the object includes: The object segmentation mask and the object motion trajectory are concatenated in the channel dimension to generate a multi-channel spatiotemporal input tensor; The multi-channel spatiotemporal input tensor is input into the spatiotemporal feature extraction network, and the three-dimensional convolution kernel is used to perform convolution in both the spatial and temporal dimensions to obtain a spatiotemporal feature map. The spatiotemporal feature map is input into the regression network to obtain the predicted physical properties of the object.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images