Method and system for three-dimensional positioning of physical objects based on video data

Abstract

The application discloses a physical object three-dimensional positioning method and system based on video data, and relates to the technical field of three-dimensional positioning. The method is used to solve the problems of low spatial positioning accuracy, unstable pose estimation and poor time sequence continuity of objects in a video scene. First, the input video stream is analyzed to generate an object segmentation mask and extract feature points, camera motion parameters are calculated through inter-frame matching, and a scene sparse three-dimensional point cloud is reconstructed. According to the segmentation mask and the point cloud, a candidate three-dimensional bounding box is generated, the optimal bounding box is selected by combining the geometric matching degree and the feature similarity evaluation, and the preliminary three-dimensional positioning result of the object is obtained. Then, the positioning result is fused with the historical frame motion data to construct a spatio-temporal constraint optimization model, and the six-degree-of-freedom pose of the object is solved. Finally, a neural radiance field representation is established based on the pose, the pose and neural parameters are optimized through the combination of differentiable rendering and back propagation, the continuous and accurate update of the three-dimensional position of the object is realized, and the positioning stability and robustness in a dynamic scene are improved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional positioning technology, specifically to a method and system for three-dimensional positioning of physical objects based on video data. Background Technology

[0002] In terms of social context, with the rapid development of applications such as intelligent manufacturing, autonomous driving, augmented reality, and digital twins, numerous tasks in the real world require high-precision 3D localization and attitude estimation of physical objects in video data to achieve realistic perception and virtual-real fusion of the spatial environment. For example, industrial robots need to identify and locate workpieces through video data to perform grasping operations; autonomous driving systems rely on vehicle cameras to perform 3D modeling and dynamic tracking of surrounding obstacles; and augmented reality devices need to accurately obtain the position of objects based on scene videos to achieve virtual-real overlay display. Video data has continuity and rich spatiotemporal information, making it a primary information source for constructing 3D spatial understanding. However, how to accurately recover the spatial position and attitude of objects from it remains a key challenge in the field of visual computing.

[0003] In existing technologies, commonly used video-based 3D localization methods mostly rely on structured light, depth cameras, or multi-view geometry reconstruction. Structured light and depth camera methods are significantly affected by ambient lighting, reflection characteristics, and viewpoint occlusion, making it difficult to maintain stable accuracy in complex scenes. Multi-view geometry-based 3D reconstruction methods typically assume objects are stationary, generating sparse point clouds through feature point matching. However, when there is object movement, lighting changes, or dynamic occlusion, the reconstruction results are prone to distortion and mismatches. Furthermore, traditional methods often separate object detection from 3D localization, lacking a joint optimization mechanism. This leads to error accumulation between pose estimation results and image observation, making it difficult to achieve high-precision, temporally consistent 3D localization of multiple objects in dynamic scenes. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for three-dimensional positioning of physical objects based on video data, thus solving the problems mentioned in the background.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for 3D localization of physical objects based on video data, comprising the following steps: S1. Extracting continuous frame images from the input video stream, generating a segmentation mask for each object through a segmentation network, extracting feature points from the continuous frame images and establishing inter-frame feature point matching relationships, calculating camera motion parameters based on the matching relationships, and constructing a sparse 3D point cloud of the scene; S2. Generating multiple candidate 3D bounding boxes for each object based on the segmentation mask and the sparse point cloud, projecting each 3D bounding box onto a 2D image plane, calculating the geometric matching degree between the projected contour and the segmentation mask, and the similarity between image features within the projected area and object template features, and performing a comprehensive evaluation, selecting the optimal 3D bounding box as the initial 3D localization method for the object. S3. The preliminary 3D localization results of the object are fused with the motion data of historical frames to establish the motion trajectory model of the object between consecutive frames. At the same time, a spatial constraint network based on the relative position and contact relationship between objects is constructed. By minimizing the smoothness constraint of the motion trajectory and the consistency constraint of the spatial relationship, the six-degree-of-freedom pose of the object is obtained. S4. Based on the six-degree-of-freedom pose of the object, a neural radiation field representation is established for each object. The object image and segmentation mask are synthesized from the current viewpoint through differentiable rendering. The photometric loss and geometric loss of the synthesized result and the actual observation data are calculated. The neural representation parameters and object pose parameters are jointly optimized through backpropagation. The optimized object pose parameters are fed back to the motion trajectory model for state updates in subsequent frames.

[0006] Furthermore, the specific process of extracting consecutive frame images from the input video stream, generating a segmentation mask for each object through a segmentation network, extracting feature points from consecutive frame images, and establishing inter-frame feature point matching relationships is as follows: acquire consecutive multi-frame images with adjacent timestamps from the video stream, parse each frame image through a segmentation network to obtain object segmentation masks, extract multi-scale feature points from each frame image, and establish correspondences between feature points of adjacent frames through a matching algorithm; through dual verification of geometric constraints and descriptor similarity, filter the initially matched feature point pairs, remove mismatched points that do not meet spatial continuity constraints, and perform positional registration between the verified feature point matching relationships and the corresponding segmentation masks.

[0007] Furthermore, the specific process of calculating camera motion parameters and constructing a sparse 3D point cloud of the scene based on the matching relationship is as follows: the essential matrix between adjacent frames is calculated based on the verified feature point matching relationship, the relative pose parameters of the camera are decomposed, the initial coordinates of the matching feature points in 3D space are calculated through triangulation, and the initial point cloud structure of the scene is established; the camera pose and point cloud coordinates are jointly optimized by combining the principle of minimizing reprojection error, a sparse 3D point cloud is generated, and its spatial correspondence with the object segmentation region is established.

[0008] Furthermore, based on the segmentation mask and sparse point cloud, multiple candidate 3D bounding boxes are generated for each object. The specific process of projecting each 3D bounding box onto the 2D image plane is as follows: Based on the pixel distribution of the object segmentation mask and the spatial distribution of the point cloud, the 3D spatial range of the object is determined. Within this range, candidate 3D bounding boxes of different sizes and orientations are generated through multi-resolution sampling. Based on the camera intrinsic parameters and pose parameters of the current frame, the candidate bounding boxes are projected onto the image plane. The geometric fit between the projected contour and the object segmentation mask is calculated, and candidate bounding boxes that meet the geometric constraints are selected. The retained candidate bounding boxes are grouped and clustered, maintaining the differences in size and orientation of the bounding boxes.

[0009] Further, the geometric matching degree between the projected contour and the segmentation mask, and the similarity between image features within the projected region and object template features are calculated and comprehensively evaluated to select the optimal 3D bounding box as the preliminary 3D localization result of the object. The specific process is as follows: The intersection-union ratio (IUR) of the projected contour of each retained candidate bounding box and the object segmentation mask is calculated as the geometric matching degree. The image features within the projected region and the object template features are extracted and their similarity is calculated. Through an adaptive weight allocation mechanism, the fusion ratio of geometric matching degree and feature similarity is dynamically adjusted according to the image quality and occlusion degree of the current frame. A comprehensive score is calculated for each candidate bounding box, and non-maximum suppression is performed to eliminate redundant bounding boxes. Geometric consistency is verified by combining the spatial distribution of sparse point clouds. Through multiple rounds of iterative optimization, the 3D bounding box with the highest comprehensive score and that meets the spatial constraints is selected as the preliminary 3D localization result of the object.

[0010] Furthermore, the initial 3D localization results of the object are fused with the motion data of historical frames to establish the motion trajectory model of the object between consecutive frames. The construction logic is as follows: extract the 3D position coordinates and attitude angles of each object in the current frame, and read the pose and velocity information of the corresponding object from the historical frames. Calculate the pose change of the object between adjacent frames, and derive the duct velocity vector and angular velocity vector to describe the instantaneous motion state of the object in 3D space. Record the motion state sequence of the object in time order to construct a pose time series set between consecutive frames. During the update process, smooth abnormal velocity or direction changes to maintain the continuity of the trajectory in the time dimension, and output the object motion trajectory data after time fusion and continuity correction.

[0011] Furthermore, a spatial constraint network based on the relative positions and contact relationships between objects is constructed. The specific process of solving for the six-DOF pose of the object by minimizing the smoothness constraint of the motion trajectory and the consistency constraint of the spatial relationship is as follows: Based on the object motion trajectory data, the contact, support, and adjacency relationships between objects in the scene are identified, and geometric constraint expressions are established for each relationship. Among them, the support relationship is constrained by maintaining consistency in the normal direction, and the contact relationship is constrained by minimizing the distance between the contact points. A joint optimization objective function is constructed, in which the trajectory smoothness constraint is expressed as a second-order continuous term of the pose change between adjacent frames, and the spatial relationship constraint is expressed as a relative pose error term between objects. Iterative optimization is performed in the three-dimensional rigid body motion space, and the weights are dynamically adjusted according to the constraint residuals to output the six-DOF pose of the object that satisfies the continuity of motion and the consistency of spatial constraints.

[0012] Furthermore, based on the six-DOF pose of the object, a neural radiation field representation is established for each object. The object image and segmentation mask are synthesized from the current viewpoint through differentiable rendering. The specific process of calculating the photometric loss and geometric loss between the synthesized result and the actual observation data is as follows: Based on the six-DOF pose of the object, a neural radiation field representation centered on the object is constructed, and the volume density and color distribution are modeled simultaneously through a multi-layer network; through a differentiable rendering process, ray sampling and volume rendering are performed from the current camera viewpoint to generate a synthesized image and segmentation mask, and the difference between the synthesized result and the real observation image at the brightness and structure levels is calculated to obtain the photometric loss; the difference between the contour and overlap of the synthesized mask and the real segmentation mask is calculated to obtain the geometric loss, and a regularization term based on normal consistency is introduced to maintain the smoothness of the reconstructed surface.

[0013] Furthermore, the neural representation parameters and object pose parameters are jointly optimized through backpropagation, and the optimized object pose parameters are fed back to the motion trajectory model for state updates in subsequent frames. The specific process is as follows: An end-to-end differentiable optimization framework is established, with photometric loss and geometric loss as objective functions. The neural radiation field parameters and object pose parameters are updated simultaneously through gradient backpropagation. Lie algebra representation is used in pose updates to ensure the numerical stability and geometric consistency of rotation calculations. The optimized six-DOF object pose is written into the motion trajectory model to update the latest object state information, providing prior information for localization in the next frame.

[0014] A video-based physical object 3D localization system includes the following modules: a scene parsing and 3D reconstruction module, used to extract continuous frame images from the input video stream, generate a segmentation mask for each object through a segmentation network, extract feature points from continuous frame images and establish inter-frame feature point matching relationships, calculate camera motion parameters based on the matching relationships, and construct a sparse 3D point cloud of the scene; an object localization module, used to generate multiple candidate 3D bounding boxes for each object based on the segmentation mask and sparse point cloud, project each 3D bounding box onto a 2D image plane, calculate the geometric matching degree between the projected contour and the segmentation mask, and the similarity between image features within the projected area and object template features, and perform comprehensive evaluation to select the optimal 3D bounding box as the preliminary 3D localization result of the object; and a spatiotemporal constraint optimization module. The first module fuses the initial 3D localization results of an object with motion data from historical frames to establish a motion trajectory model of the object across consecutive frames. Simultaneously, it constructs a spatial constraint network based on the relative positions and contact relationships between objects. By minimizing the smoothness constraints of the motion trajectory and the consistency constraints of the spatial relationships, it solves for the object's six-DOF pose. The second module, neural representation and pose optimization, establishes a neural radiation field representation for each object based on its six-DOF pose. It synthesizes the object image and segmentation mask from the current viewpoint using differentiable rendering, calculates the photometric and geometric losses between the synthesized result and the actual observation data, jointly optimizes the neural representation parameters and object pose parameters through backpropagation, and feeds the optimized object pose parameters back to the motion trajectory model for state updates in subsequent frames.

[0015] The present invention has the following beneficial effects:

[0016] (1) A physical object 3D localization method based on video data achieves collaborative analysis of object regions and spatial structures by introducing a segmentation network and feature point matching in consecutive frames. This method completes camera motion parameter estimation and sparse 3D point cloud reconstruction while extracting object segmentation masks, making object detection and scene geometry modeling mutually constrained, significantly improving the accuracy and stability of 3D structure recovery in dynamic video scenes. By fusing segmentation masks and sparse point clouds to generate candidate 3D bounding boxes, and combining a joint evaluation mechanism of geometric matching degree and feature similarity, the optimal bounding box can be robustly selected under the conditions of occlusion or illumination changes, achieving accurate preliminary localization of objects.

[0017] (2) The physical object 3D localization system based on video data, by fusing the preliminary localization results with historical motion data, constructs a motion trajectory model and a spatial constraint network, which can maintain the continuity and spatial consistency of pose inference in multi-object dynamic interaction scenarios. Furthermore, by utilizing neural radiation field representation and differentiable rendering mechanism, the neural representation parameters and pose parameters of the object are jointly optimized at the photometric and geometric levels, enabling the system to adaptively correct localization errors and continuously update motion states, thereby obtaining high-precision, temporally consistent object six-degree-of-freedom pose estimation results, improving the overall accuracy and robustness of 3D localization.

[0018] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0019] Figure 1 This is a flowchart of the physical object three-dimensional positioning method based on video data according to the present invention.

[0020] Figure 2 This is a flowchart of the physical object three-dimensional positioning system based on video data according to the present invention. Detailed Implementation

[0021] This application's embodiments address the problems of difficulty in accurately restoring the spatial position of objects in dynamic video scenes, the susceptibility of pose estimation to occlusion and changes in lighting, and the lack of temporal consistency in object motion trajectories through a physical object 3D localization method and system based on video data.

[0022] The overall concept of the solution in this application embodiment is as follows:

[0023] By analyzing the input video stream frame by frame, a segmentation network is first used to generate an object mask and combine it with feature point matching to construct a sparse 3D point cloud, thereby obtaining the spatial structure information of the scene. On this basis, candidate 3D bounding boxes are generated by fusing the segmentation mask and the sparse point cloud, and the preliminary 3D localization result of the object is determined by a comprehensive evaluation of geometric matching and feature similarity. Subsequently, the preliminary pose of the object is jointly modeled with the motion information of historical frames, and the precise six-degree-of-freedom pose of the object is obtained by optimizing the trajectory smoothing constraint and spatial relationship consistency. Finally, a differentiable representation of the object is established based on the neural radiation field, and the object pose and neural parameters are optimized by backpropagation of rendering reprojection error, so as to achieve adaptive correction and continuous updating of the localization result, thereby achieving high-precision 3D localization and motion perception of multiple objects in complex dynamic environments.

[0024] Please see Figure 1This invention provides a technical solution: a method for 3D localization of physical objects based on video data, comprising the following steps: S1. Extracting continuous frame images from the input video stream, generating a segmentation mask for each object through a segmentation network, extracting feature points from the continuous frame images and establishing inter-frame feature point matching relationships, calculating camera motion parameters based on the matching relationships, and constructing a sparse 3D point cloud of the scene; S2. Generating multiple candidate 3D bounding boxes for each object based on the segmentation mask and the sparse point cloud, projecting each 3D bounding box onto a 2D image plane, calculating the geometric matching degree between the projected contour and the segmentation mask, and the similarity between image features within the projected area and object template features, and performing a comprehensive evaluation, selecting the optimal 3D bounding box as the preliminary 3D localization result of the object. S3. The preliminary 3D localization results of the object are fused with the motion data of historical frames to establish the motion trajectory model of the object between consecutive frames. At the same time, a spatial constraint network based on the relative position and contact relationship between objects is constructed. By minimizing the smoothness constraint of the motion trajectory and the consistency constraint of the spatial relationship, the six-degree-of-freedom pose of the object is obtained. S4. Based on the six-degree-of-freedom pose of the object, a neural radiation field representation is established for each object. The object image and segmentation mask are synthesized from the current viewpoint through differentiable rendering. The photometric loss and geometric loss of the synthesized result and the actual observation data are calculated. The neural representation parameters and object pose parameters are jointly optimized through backpropagation. The optimized object pose parameters are fed back to the motion trajectory model for state updates in subsequent frames.

[0025] In this implementation, S1: Images are extracted frame by frame from the input video stream. First, a segmentation network generates pixel-level segmentation masks for each object in each frame, thus obtaining the spatial region information of the object on the image plane. Simultaneously, local feature points are extracted and matched between frames. Based on these matching relationships, the relative motion between cameras is estimated, and the geometric structure of the scene is restored, thereby constructing a sparse 3D point cloud of the scene. This step provides two key foundational pieces of information: first, pixel-level object regions (segmentation masks); and second, a sparse 3D structure reflecting the scene geometry (point cloud) and camera pose. This information lays the foundation for subsequent geometry- and appearance-based 3D localization and couples detection (semantic) with geometric restoration, reducing the propagation of errors from independent modules. Segmentation network: A pixel-level classification deep neural network used to classify images into foreground targets and background or multi-class target regions. Feature points: Pixels in an image with stable local structures (e.g., corner points or extreme values ​​of local descriptors), facilitating the establishment of correspondences between different frames. Inter-frame feature point matching: Matching the same physical points in the current frame with those in adjacent frames for calculating relative motion and 3D reconstruction. Camera motion parameter estimation: Camera extrinsic parameters (translation and rotation) are solved using matching points, typically through geometric methods such as the essential matrix / fundamental matrix. Sparse 3D point cloud: A set of 3D coordinates obtained by triangulating feature points from multiple frames, representing the structural skeleton of the scene (non-dense reconstruction), facilitating rapid geometric verification and localization. S2: Using the segmentation mask and sparse point cloud obtained in S1, multiple candidate bounding boxes (candidate solutions of different sizes, orientations, or positions) are generated for each target in 3D space. Each 3D bounding box is projected back onto the 2D image plane, and (A) the geometric matching degree between the projected contour and the segmentation mask, and (B) the similarity between image features within the projected region and object template features are calculated respectively. The best 3D bounding box is selected as the preliminary 3D localization result of the object based on the comprehensive score of the two. This step performs explicit verification in the local search space, which can still robustly provide an initial pose guess even in the presence of occlusion, incomplete sparse point cloud, or reconstruction errors, providing a reliable starting point for subsequent trajectory fusion and refinement. 3D Bounding Box: The smallest (or candidate) rectangular box described by length, width, height, and orientation in 3D space, used to approximate the spatial occupancy of an object. Projected Contour Geometric Matching: After projecting the 3D bounding box onto the image, its geometric measure (overlap, boundary consistency, etc.) is compared with the contour of the segmentation mask to evaluate whether the candidate box is geometrically consistent with the observation. Image Feature and Object Template Feature Similarity: The appearance similarity between the observed region and a pre-built or online object template is calculated based on local descriptors or depth features to supplement texture and color information that may be overlooked by geometry alone. Candidate Generation Strategies: These can include estimating size priors based on sparse point cloud projection, generating multi-scale candidates from historical frame scale changes, or generating orientation candidates using learned shape priors. These strategies demonstrate creative design for handling uncertainty.S3: The preliminary 3D localization obtained in S2 is fused with the motion information of historical frames to establish a cross-frame motion trajectory model for each object. Simultaneously, a spatial constraint network based on the relative positions and contact relationships between objects is introduced to impose spatial consistency constraints (e.g., non-overlapping, contact relationship preservation, relative position priors, etc.) on the multi-object scene. By jointly minimizing the trajectory smoothness term (encouraging motion continuity and velocity / acceleration smoothness) and the spatial relationship consistency term (maintaining physical coherence between objects), the six-degree-of-freedom pose (position + rotation) of each object is solved in the spatiotemporal domain. This step extends single-frame estimation to a temporally consistent optimal solution, significantly reducing instantaneous noise, jumps caused by occlusion, and pose anomalies caused by local mismatches. Motion trajectory model: A mathematical model describing the displacement and attitude of an object over time, typically including state variables (pose, velocity) and constraints (smoothness, dynamic priors). Smoothness constraint: Penalizes the second-order or first-order difference of pose over time to suppress unreasonable instantaneous jumps. Spatial Constraint Network: A set of constraints in graph / matrix form, where nodes represent object poses and edges represent spatial relationships between objects (e.g., contact, relative orientation, or minimum distance), used to ensure global spatial consistency during joint optimization. Six-DOF Pose: The position (x, y, z) and orientation of an object in 3D space (usually represented by rotation matrices or quaternions). S4: Initializing with the six-DOF pose obtained in S3, a neural radiation field representation or other differentiable implicit neural representation is established for each object; the RGB image and segmentation mask of the object are synthesized from the current camera's viewpoint using a differentiable renderer, and the photometric and geometric losses (e.g., depth or contour differences) between the synthesized result and the actual observation (image and mask) are calculated. Backpropagation simultaneously updates the neural representation parameters and object pose parameters, allowing the neural representation to better fit the observation and correct pose errors; after optimization, the improved pose is fed back to the motion trajectory model for state updates in subsequent frames. This step achieves fine-grained pose refinement through rendering-based visual supervision, thereby obtaining more accurate six-DOF estimates under complex appearance and occlusion conditions. Neural Radiation Field: A technique that uses neural networks to implicitly represent volumetric emissivity and color, enabling the synthesis of high-quality views from arbitrary perspectives based on the network. Differentiable Rendering: The rendering process is differentiable with respect to parameters (geometry / pose / material / neural representation, etc.), allowing rendering errors to be backpropagated to the parameters for optimization. Photometric Loss: Compares the pixel-level errors between the rendered image and the observed image to measure appearance consistency. Geometric Loss: Measures the difference in geometric structure between the rendered result and the observation. Joint Optimization and Pose Feedback Mechanism: Simultaneously uses neural representation parameters and pose parameters as optimization variables, and writes the optimized pose back to the trajectory model, forming an online closed-loop correction.

[0026] Specifically, the process of extracting consecutive frame images from the input video stream, generating a segmentation mask for each object through a segmentation network, extracting feature points from consecutive frame images, and establishing inter-frame feature point matching relationships is as follows: Multiple consecutive frame images with adjacent timestamps are acquired from the video stream; each frame image is parsed through a segmentation network to obtain the object segmentation mask; multi-scale feature points of each frame image are extracted, and a matching algorithm is used to establish the correspondence between feature points of adjacent frames; through dual verification of geometric constraints and descriptor similarity, the initially matched feature point pairs are screened, and mismatched points that do not conform to spatial continuity constraints are removed; the verified feature point matching relationships are then matched with the corresponding segmentation masks.

[0027] In this implementation scheme, joint analysis of the appearance region and geometric features of objects in a video is achieved through segmentation and feature point matching of consecutive frame images. Specifically, the segmentation network extracts the pixel regions of each object to be located from each frame image, limiting subsequent feature point matching to the object mask range, thereby reducing interference from background features. Feature point extraction employs a multi-scale detection operator to identify salient structural points at different image resolutions to enhance the robustness of matching. To improve the accuracy of feature point matching, the initially matched points undergo dual verification using geometric constraints and descriptor similarity. Let the k-th frame image be... The image of the (k+1)th frame is Extract their feature point sets respectively and A preliminary set of correspondences is established through a matching algorithm. The geometric constraint model is used to screen matching pairs for spatial consistency, and its geometric consistency is represented by the epipolar constraint equation: ;in: It is the basic matrix, reflecting the joint relationship between the camera's internal parameters and relative attitude; The matrix is ​​a column vector of feature points in normalized image coordinates; the matrix is ​​estimated using the RANSAC random sampling consensus algorithm. The matching points that do not meet the epipolar geometric constraints are then removed. After initial screening based on geometric constraints, further verification is performed using descriptor similarity. The feature descriptors are represented using local directional gradients or depthwise convolutional features, and the matching similarity is defined as: ;in: Similarity between feature point pairs; These are the descriptor vectors of corresponding points in the two frames; The similarity decay threshold is determined as follows: During the system training phase, the statistical descriptor difference distribution of the validation video set is used to select the percentile value that optimizes both matching precision and recall as the threshold. Only matching pairs that simultaneously satisfy geometric consistency and similarity greater than the threshold are retained, forming the final matching set M. Subsequently, the validated matching points are spatially registered according to object masks to ensure that the matching points fall within the same object region, providing a reliable observation basis for camera motion calculation. This process enables subsequent pose estimation and 3D reconstruction to be performed under object-level semantic constraints, thereby improving the robustness of scene parsing.

[0028] Specifically, the process of calculating camera motion parameters and constructing a sparse 3D point cloud of the scene based on the matching relationship is as follows: the essential matrix between adjacent frames is calculated based on the verified feature point matching relationship, the relative pose parameters of the camera are decomposed, the initial coordinates of the matching feature points in 3D space are calculated through triangulation, and the initial point cloud structure of the scene is established; the camera pose and point cloud coordinates are jointly optimized by combining the principle of minimizing reprojection error, a sparse 3D point cloud is generated, and its spatial correspondence with the object segmentation region is established.

[0029] In this implementation scheme, based on the verified feature point matching relationships, the camera's motion parameters are estimated and a sparse point cloud model of the scene is established to achieve the reconstruction from two-dimensional video frames to three-dimensional spatial structures. Specifically, the essential matrix between adjacent frames is first calculated based on the matching point set M. And obtain the relative rotation matrix and translation vector of the camera through its decomposition. Let From the formula: ;in: Each camera in frame and frame The intrinsic parameter matrix at that time; The aforementioned fundamental matrix. Through the essential matrix... Singular value decomposition yields the rotation matrix. With translation vector This indicates that the camera is from a frame. to frame The pose changes. Then, for each pair of matched feature points... The coordinates in three-dimensional space are recovered using the principle of triangulation. ;in: The coordinates of the reconstructed three-dimensional points in space; This is the projection function from a 3D point to image coordinates; and These are the camera extrinsic matrices for the two frames. The 3D position of each feature point is obtained by minimizing the dual-view reprojection error, thus constructing the initial sparse point cloud of the scene. To further improve the global consistency between point cloud and camera pose, a joint optimization strategy for reprojection error is adopted to jointly adjust the camera pose and point cloud coordinates. The optimization objective can be expressed as: ;in: For the first The weight coefficients of each feature point are used to suppress the influence of mismatches or noise points; weight coefficients The method for determining the weight is as follows: the weight is calculated based on the weighted product of the matching confidence of the feature point descriptor and the local gradient intensity of the image, and the normalized result is taken as the weight. This is the optimized 3D point cloud set. Through the above optimization, a sparse 3D point cloud with consistent structure and camera trajectory can be obtained, and its spatial correspondence with the object segmentation region can be established. This process ensures that subsequent 3D localization steps can be performed with a high-precision camera. This is performed under sparse geometric constraints, thus providing a reliable foundation for object-level 3D bounding box generation and neural radiation field optimization.

[0030] Specifically, based on the segmentation mask and sparse point cloud, multiple candidate 3D bounding boxes are generated for each object. The specific process of projecting each 3D bounding box onto the 2D image plane is as follows: Based on the pixel distribution of the object segmentation mask and the spatial distribution of the point cloud, the 3D spatial range of the object is determined. Within this range, candidate 3D bounding boxes of different sizes and orientations are generated through multi-resolution sampling. Based on the camera intrinsic parameters and pose parameters of the current frame, the candidate bounding boxes are projected onto the image plane. The geometric fit between the projected contour and the object segmentation mask is calculated, and candidate bounding boxes that meet the geometric constraints are selected. The retained candidate bounding boxes are grouped and clustered, maintaining the differences in size and orientation of the bounding boxes.

[0031] In this implementation scheme, based on the segmentation mask and sparse point cloud, multiple candidate 3D bounding boxes are generated for each object. The specific process of projecting each 3D bounding box onto the 2D image plane is as follows: The pixel distribution of the object segmentation mask and its corresponding sparse point cloud spatial coordinates are obtained. The 3D spatial extent of the object is determined by statistically analyzing the boundary points of the point cloud along the three coordinate axes. , , Within this spatial range, candidate 3D bounding boxes of different sizes, center positions, and orientations are generated through multi-resolution sampling. Each bounding box can be represented as... ,in The coordinate vector of the bounding box center; is the three-dimensional dimension vector of the bounding box; Let this be the rotation matrix representing the orientation. Based on the camera's intrinsic parameter matrix for the current frame. With pose parameters , the 3D vertices of the candidate bounding box Projecting onto a two-dimensional image plane, the projection process can be represented as follows: ;in, The coordinates of the projected two-dimensional image; Let be the rotation matrix of the camera; Let be the translation vector. Then, the geometric fit between the projected contour and the object segmentation mask is calculated, defined as: ;in, For the enclosure Projected outline area; Divide the object into mask regions; Represents the pixel area of ​​the region. Geometric fit. This reflects the degree of overlap between the bounding box projection and the segmentation mask in terms of shape and position. A geometric threshold is set. ,when At that time, the bounding box is considered to match the shape of the object. Threshold The 95% confidence interval of the matching error distribution in the training set can be used to balance recall and precision. A similarity-based clustering operation is performed on the set of bounding boxes that meet the geometric constraints, grouping bounding boxes with significant differences in size and orientation into different groups to avoid redundancy in subsequent scoring calculations. Through the above projection and filtering process, a diverse set of candidate 3D bounding boxes can be generated for each object while ensuring spatial geometric consistency, providing initial constraints for subsequent localization optimization.

[0032] Specifically, the process of calculating the geometric matching degree between the projected contour and the segmentation mask, and the similarity between the image features within the projected region and the object template features, and then comprehensively evaluating and selecting the optimal 3D bounding box as the preliminary 3D localization result of the object, is as follows: The intersection-union ratio (IUR) of the projected contours of each retained candidate bounding box and the object segmentation mask is calculated as the geometric matching degree; image features within the projected region and object template features are extracted and their similarity is calculated; through an adaptive weight allocation mechanism, the fusion ratio of geometric matching degree and feature similarity is dynamically adjusted according to the image quality and occlusion degree of the current frame; a comprehensive score is calculated for each candidate bounding box, and non-maximum suppression is performed to eliminate redundant bounding boxes; geometric consistency is verified by combining the spatial distribution of sparse point clouds; through multiple rounds of iterative optimization, the 3D bounding box with the highest comprehensive score and satisfying the spatial constraints is selected as the preliminary 3D localization result of the object.

[0033] In this implementation, for each retained candidate bounding box Calculate the intersection-union ratio (IUU) of its projected contour and the segmentation mask. As a geometric matching degree; simultaneously, image feature vectors are extracted within the bounding box projection region. and the object template feature vector Perform cosine similarity calculation: ;in, For feature similarity; and These are the feature vectors of the candidate region and the template, respectively; "." represents the vector dot product. Let represent the vector norm. To comprehensively consider the matching effect from both geometric and semantic perspectives, a comprehensive scoring function is defined: ;in, The overall score for candidate bounding boxes; These are adaptive weighting coefficients, whose values ​​are determined based on the image quality parameters of the current frame. And degree of obstruction Dynamic adjustment can be determined through a normalization function: Among them, image quality The degree of occlusion can be estimated by weighting brightness uniformity and texture sharpness. It can be calculated from the edge integrity of the segmentation mask. The comprehensive score of each candidate bounding box is then calculated. Next, non-maximum suppression (NMS) is performed to remove bounding boxes with high spatial overlap but low scores. Then, considering the spatial distribution of the sparse point cloud, geometric consistency verification is performed on the remaining bounding boxes, eliminating those that do not conform to the point cloud density distribution. Finally, through multiple rounds of iterative optimization, the bounding box with the highest overall score that satisfies the spatial constraints is selected. As a preliminary 3D localization result for the object, by combining geometric shape, image features, and a dynamic weighted fusion mechanism, robust estimation of the object's 3D position is achieved under different scene lighting, occlusion, and pose change conditions, providing highly reliable initial input for subsequent fine localization and motion tracking.

[0034] Specifically, the logic for constructing a motion trajectory model of an object across consecutive frames by fusing the preliminary 3D localization results of the object with motion data from historical frames is as follows: extract the 3D position coordinates and attitude angles of each object in the current frame, and read the pose and velocity information of the corresponding object from historical frames. Calculate the pose change of the object between adjacent frames, and derive duct velocity vectors and angular velocity vectors to describe the instantaneous motion state of the object in 3D space. Record the motion state sequence of the object in chronological order, construct a pose time series set between consecutive frames, and smooth abnormal velocity or direction changes during the update process to maintain the continuity of the trajectory in the time dimension. Output the object motion trajectory data after time-series fusion and continuity correction.

[0035] In this implementation scheme, firstly, the three-dimensional position coordinates and attitude angles of each object are extracted from the current frame, denoted as... ,in Represents the three-dimensional position of the center point of an object. This represents its Euler angle pose around the three coordinate axes. The pose and velocity information of the corresponding object are read from historical frames. Calculate the pose change between adjacent frames: Therefore, the duct velocity vector and angular velocity vector are derived: ;in, The time interval between adjacent frames; It is the linear velocity vector; This is the angular velocity vector, used to describe the instantaneous motion state of the object in three-dimensional space. Next, the sequence of the object's motion states is recorded in chronological order. Construct a set of pose time series between consecutive frames. During the update process, for frames with abrupt changes in velocity or orientation, calculate the rate of change of motion state: ;like If it is, then it is determined to be an abnormal motion state, where The threshold for velocity variation can be determined by statistically analyzing the standard deviation of the motion sequence. Outliers are corrected using Kalman filtering or an exponentially weighted smoothing function. ;in, This is a smoothing coefficient, whose value is dynamically adjusted according to scene complexity to ensure the trajectory remains continuous and stable in the time dimension. The final output object trajectory data consists of both position and pose sequences, forming a 3D motion trajectory model that has undergone temporal fusion and continuity correction. By fusing historical frame motion information with the current frame pose, temporal consistency of the motion state is achieved, avoiding positioning drift caused by instantaneous occlusion or detection jitter, and making the trajectory curve continuously differentiable in the time dimension.

[0036] Specifically, a spatial constraint network based on the relative positions and contact relationships between objects is constructed. The process of solving for the six-DOF pose of the object by minimizing the smoothness constraint of the motion trajectory and the consistency constraint of the spatial relationship is as follows: Based on the motion trajectory data of the object, the contact, support, and adjacency relationships between objects in the scene are identified, and geometric constraint expressions are established for each relationship. Among them, the support relationship is constrained by maintaining consistency in the normal direction, and the contact relationship is constrained by minimizing the distance between the contact points. A joint optimization objective function is constructed, in which the trajectory smoothness constraint is expressed as a second-order continuous term of the pose change between adjacent frames, and the spatial relationship constraint is expressed as a relative pose error term between objects. Iterative optimization is performed in the three-dimensional rigid body motion space, and the weights are dynamically adjusted according to the constraint residuals to output the six-DOF pose of the object that satisfies the continuity of motion and the consistency of spatial constraints.

[0037] In this implementation scheme, based on the aforementioned object motion trajectory data, the spatial relationships between objects in the scene are identified, including contact, support, and adjacency relationships. For support relationships, the constraint condition is that the normal direction of the support surface is consistent, and the constraint is expressed as follows: ;in, These are the normal vectors of the supporting object and the planar supporting surface, respectively. For contact relationships, the constraint is minimizing the distance between the contact points: ;in, Let be the three-dimensional coordinates of the contact point between the two objects. Construct a joint optimization objective function: ;in, The second-order continuous term represents the pose change and is used for smoothness constraints; These are the weighting coefficients for spatial smoothing, support consistency, and contact constraints, respectively. The weights are determined as follows: dynamically adjusted based on the mean residual of each constraint during the optimization process, using a normalized form: ;in, For the first The mean residual of the constraint is used; the larger the residual, the lower the weight, thus improving optimization stability. Iterative optimization using the Gauss-Newton method is performed in a three-dimensional rigid body motion space to solve for the six-degree-of-freedom pose parameters of the object. The weight coefficients are dynamically updated based on the constraint residuals, ultimately outputting an optimized pose result that satisfies the requirements of trajectory smoothness and spatial consistency. By constructing a spatiotemporal joint constraint optimization model, the physical interaction relationships between objects and trajectory continuity are uniformly incorporated into the optimization objective, effectively suppressing error propagation between independently moving objects and ensuring the physical rationality and global stability of the generalized result.

[0038] Specifically, based on the six-DOF pose of the object, a neural radiation field representation is established for each object. The object image and segmentation mask are synthesized from the current viewpoint through differentiable rendering. The specific process of calculating the photometric loss and geometric loss between the synthesized result and the actual observation data is as follows: Based on the six-DOF pose of the object, a neural radiation field representation centered on the object is constructed, and the volume density and color distribution are modeled simultaneously through a multi-layer network; through a differentiable rendering process, ray sampling and volume rendering are performed from the current camera viewpoint to generate a synthesized image and segmentation mask, and the difference between the synthesized result and the real observation image at the brightness and structure levels is calculated to obtain the photometric loss; the difference between the contour and overlap of the synthesized mask and the real segmentation mask is calculated to obtain the geometric loss, and a regularization term based on normal consistency is introduced to maintain the smoothness of the reconstructed surface.

[0039] In this implementation scheme, firstly, based on the object's six-degree-of-freedom pose... Establish a function representing the neural radiation field in the coordinate system of the object's center: ;in, The three-dimensional coordinates of the ray sampling point; The line-of-sight vector; This represents the volume density value of a point; This represents the corresponding color vector; These are the learnable parameters of the neural network, including the weights and biases of the multilayer perceptron. During differentiable rendering, a ray is projected for each pixel in the current camera view. ,in For the camera optical center, These are depth sampling parameters. Pixel color is calculated using integral form: ;in, This is the transmittance function, used to describe the attenuation of light within a volumetric space; This represents the sampling interval of the ray within the object's boundary. The calculated rendering result is compared to the actual observed image. Photometric loss: ;in, This represents the total number of pixels sampled. To synthesize pixel colors; The pixel color corresponds to the real image. To maintain geometric consistency, a synthetic segmentation mask is calculated. With the real mask Contour overlap differences: And a regularization term based on normal consistency is introduced: ;in, , The sampling points represent the real surface and the reconstructed surface, respectively. The normal vector at that location; This represents the number of sampling points along the normal direction. The total loss function is obtained by combining these values: ;in, , , These are the weighting coefficients. The weighting method employs a dynamic loss balancing mechanism: ;in, For the first The moving average of the loss terms is used to automatically adjust the optimization priority of different loss terms. Through pose-based neural radiation field modeling and a differentiable rendering process, high-precision reconstruction of object color distribution and spatial geometry is achieved. At the same time, the combination of photometric and geometric constraints effectively suppresses rendering artifacts and improves the detail fidelity and spatial consistency of 3D reconstruction.

[0040] Specifically, the process of jointly optimizing neural representation parameters and object pose parameters through backpropagation, and feeding the optimized object pose parameters back to the motion trajectory model for state updates in subsequent frames is as follows: An end-to-end differentiable optimization framework is established, with photometric loss and geometric loss as objective functions. Neural radiation field parameters and object pose parameters are updated simultaneously through gradient backpropagation. Lie algebra representation is used in pose updates to ensure numerical stability and geometric consistency of rotation calculations. The optimized six-DOF object pose is written into the motion trajectory model to update the latest object state information, providing prior information for localization in the next frame.

[0041] In this implementation scheme, an end-to-end differentiable optimization framework is established, with the total loss... The objective function is to simultaneously update the neural radiation field parameters through gradient backpropagation. With object pose parameters In pose optimization, to ensure the stability and geometric consistency of rotation calculations, Lie algebraic forms are used to represent rotation components. The rotation matrix is ​​then... Represented as an exponential mapping: ;in, The rotation Lie algebra representation is used; pose optimization is achieved through small update rules. ;in, For rotation increments; For translation increments, a joint update is performed using an adaptive learning rate optimizer (such as Adam): ;in, These are the learning rates for the neural network parameters and pose parameters, respectively, which can be automatically adjusted based on the gradient variance. Once the optimization converges, the updated pose parameters will be... The updated pose is written into the motion trajectory model to replace the predicted state of the corresponding object in the current frame. This updated pose serves as the motion prior for the next frame, providing high-precision initial conditions for temporal prediction between consecutive frames. Through backpropagation, the neural radiation field and pose parameters are jointly optimized, allowing rendering errors to be directly fed back into the spatial geometric representation and motion state estimation. This forms a 3D localization system based on closed-loop optimization of observation data, achieving high-precision adaptive updating and dynamic reconstruction at the object level.

[0042] Please see Figure 2A video-based physical object 3D localization system includes the following modules: a scene parsing and 3D reconstruction module, used to extract continuous frame images from the input video stream, generate a segmentation mask for each object through a segmentation network, extract feature points from continuous frame images and establish inter-frame feature point matching relationships, calculate camera motion parameters based on the matching relationships, and construct a sparse 3D point cloud of the scene; an object localization module, used to generate multiple candidate 3D bounding boxes for each object based on the segmentation mask and sparse point cloud, project each 3D bounding box onto a 2D image plane, calculate the geometric matching degree between the projected contour and the segmentation mask, and the similarity between image features within the projected area and object template features, and perform comprehensive evaluation to select the optimal 3D bounding box as the preliminary 3D localization result of the object; and a spatiotemporal constraint optimization module. The first module fuses the initial 3D localization results of the object with motion data from historical frames to establish a motion trajectory model of the object across consecutive frames. It also constructs a spatial constraint network based on the relative positions and contact relationships between objects. By minimizing the smoothness constraints of the motion trajectory and the consistency constraints of the spatial relationships, it solves for the object's six-DOF pose. The second module, neural representation and pose optimization, establishes a neural radiation field representation for each object based on its six-DOF pose. It synthesizes the object image and segmentation mask from the current viewpoint using differentiable rendering, calculates the photometric and geometric losses between the synthesized result and the actual observation data, jointly optimizes the neural representation parameters and object pose parameters through backpropagation, and feeds the optimized object pose parameters back to the motion trajectory model for state updates in subsequent frames.

[0043] In this implementation scheme, the scene parsing and 3D reconstruction module is used to complete the entire process from the original video stream to the initial reconstruction of the 3D scene. This module identifies target regions in video frames through a segmentation network, extracts multi-scale feature points and establishes inter-frame matching relationships, uses the matching information to estimate camera motion parameters and reconstruct a sparse 3D point cloud of the scene, thereby providing a unified spatial reference for subsequent object-level localization. The object localization module is used to perform preliminary localization of each object in the reconstructed 3D space. This module combines segmentation masks and sparse point clouds to generate candidate 3D bounding boxes. Through a joint evaluation mechanism of geometric matching and feature similarity, it selects the bounding box that best matches the observation data from multiple candidates, achieving a preliminary estimate of the object's position and scale. The spatiotemporal constraint optimization module is used to maintain the continuity of object motion in the time dimension and maintain reasonable relationships between objects in the spatial dimension. This module integrates motion data from historical frames to construct a temporal trajectory model. At the same time, it combines the support, contact and relative position constraints between objects to establish a joint optimization model, performs spatiotemporal consistency correction on the preliminary localization results, and solves for the precise six-degree-of-freedom pose of the object. The neural representation and pose optimization module further refines and optimizes the 3D representation and pose estimation of objects. Based on the optimized pose, this module establishes a neural radiation field representation for each object, synthesizes a view through differentiable rendering, compares it with real observations, calculates photometric and geometric losses, and optimizes the neural representation and pose parameters in an end-to-end manner. The optimization results are then fed back to the temporal model to achieve dynamic state updates across frames. Overall, the four modules form a closed-loop process from video parsing, scene reconstruction, object localization, spatiotemporal constraints to neural optimization, enabling accurate, continuous, and structurally consistent 3D localization of multiple objects in dynamic scenes.

[0044] In summary, this application has at least the following effects:

[0045] A method and system for 3D localization of physical objects based on video data achieves high-precision 3D localization and pose estimation of objects in dynamic video scenes through joint modeling of segmentation networks, sparse point cloud reconstruction, spatial constraint optimization, and neural radiation fields. This method introduces a spatiotemporal fusion mechanism between consecutive frames, effectively suppressing localization errors caused by occlusion, lighting variations, and camera shake, thus improving the stability and consistency of pose estimation. Differentiable rendering and end-to-end optimization further enhance the system's adaptability and reconstruction accuracy in complex environments, demonstrating high versatility and engineering application value.

[0046] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0047] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0051] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for three-dimensional positioning of a physical object based on video data, characterized in that, Includes the following steps: S1. Extract continuous frame images from the input video stream, generate a segmentation mask for each object through a segmentation network, extract feature points from the continuous frame images and establish inter-frame feature point matching relationships, calculate camera motion parameters based on the matching relationships and construct a sparse 3D point cloud of the scene. S2. Based on the segmentation mask and sparse point cloud, generate multiple candidate 3D bounding boxes for each object, project each 3D bounding box onto the 2D image plane, calculate the geometric matching degree between the projected contour and the segmentation mask and the similarity between the image features within the projected area and the object template features, and perform a comprehensive evaluation to select the optimal 3D bounding box as the preliminary 3D localization result of the object. S3. The preliminary three-dimensional positioning results of the object are fused with the motion data of historical frames to establish the motion trajectory model of the object between consecutive frames. At the same time, a spatial constraint network based on the relative position and contact relationship between objects is constructed. By minimizing the smoothness constraint of the motion trajectory and the consistency constraint of the spatial relationship, the six-degree-of-freedom pose of the object is obtained. S4. Based on the six-degree-of-freedom pose of the object, establish a neural radiation field representation for each object. Synthesize the object image and segmentation mask from the current viewpoint through differentiable rendering. Calculate the photometric loss and geometric loss between the synthesized result and the actual observation data. Jointly optimize the neural representation parameters and object pose parameters through backpropagation. Feed the optimized object pose parameters back to the motion trajectory model for state updates in subsequent frames.

2. The method of claim 1, wherein: The specific process of extracting consecutive frame images from the input video stream, generating a segmentation mask for each object through a segmentation network, extracting feature points from consecutive frame images, and establishing inter-frame feature point matching relationships is as follows: The system acquires multiple consecutive frames of images with adjacent timestamps from the video stream, parses each frame image through a segmentation network to obtain an object segmentation mask, extracts multi-scale feature points from each frame image, and establishes the correspondence between feature points of adjacent frames through a matching algorithm. By using both geometric constraints and descriptor similarity for dual verification, the initially matched feature point pairs are screened, and mismatched points that do not conform to the spatial continuity constraints are removed. The verified feature point matching relationships are then matched with the corresponding segmentation masks.

3. The method of claim 2, wherein: The specific process of calculating camera motion parameters and constructing a sparse 3D point cloud of the scene based on the matching relationship is as follows: The essential matrix between adjacent frames is calculated based on the verified feature point matching relationship. The relative pose parameters of the camera are decomposed and the initial coordinates of the matching feature points in three-dimensional space are calculated through triangulation to establish the initial point cloud structure of the scene. By combining the principle of minimizing reprojection error, the camera pose and point cloud coordinates are jointly optimized to generate sparse 3D point clouds and establish their spatial correspondence with the object segmentation region.

4. The method of claim 1, wherein: Based on the segmentation mask and sparse point cloud, multiple candidate 3D bounding boxes are generated for each object. The specific process of projecting each 3D bounding box onto the 2D image plane is as follows: Based on the pixel distribution of the object segmentation mask and the spatial distribution of the sparse 3D point cloud, the 3D spatial range of the object is determined, and within this range, candidate 3D bounding boxes of different sizes and orientations are generated through multi-resolution sampling. Based on the camera intrinsic parameters and pose parameters of the current frame, candidate bounding boxes are projected onto the image plane, the geometric fit between the projected contour and the object segmentation mask is calculated, and candidate bounding boxes that meet the geometric constraints are selected. The retained candidate bounding boxes are grouped and clustered, preserving the differences in size and orientation of the bounding boxes.

5. The method for three-dimensional localization of physical objects based on video data according to claim 4, characterized in that: The specific process of calculating the geometric matching degree between the projected contour and the segmentation mask, and the similarity between the image features within the projected region and the object template features, and then comprehensively evaluating and selecting the optimal 3D bounding box as the preliminary 3D localization result of the object, is as follows: The intersection-union ratio (IUU) of the projection contours of each retained candidate bounding box with the object segmentation mask is calculated as the geometric matching degree. The image features within the projection area are extracted and the object template features are used to calculate the similarity. Through an adaptive weight allocation mechanism, the fusion ratio of geometric matching degree and feature similarity is dynamically adjusted according to the image quality and occlusion degree of the current frame. A comprehensive score is calculated for each candidate bounding box and non-maximum suppression is performed to eliminate redundant bounding boxes. Geometric consistency is verified by combining the spatial distribution of sparse point cloud. Through multiple rounds of iterative optimization, the 3D bounding box with the highest comprehensive score and that meets the spatial constraints is selected as the preliminary 3D localization result of the object.

6. The method for three-dimensional localization of physical objects based on video data according to claim 1, characterized in that: The logic for constructing a motion trajectory model of an object across consecutive frames by fusing the preliminary 3D localization results with motion data from historical frames is as follows: Extract the three-dimensional position coordinates and attitude angles of each object in the current frame, and read the pose and velocity information of the corresponding object from the historical frames. Calculate the pose change of the object between adjacent frames, and derive the duct velocity vector and angular velocity vector to describe the instantaneous motion state of the object in three-dimensional space. Record the motion state sequence of an object in chronological order, construct a set of pose time series between consecutive frames, and smooth abnormal speed or direction changes during the update process to maintain the continuity of the trajectory in the time dimension. Output the object motion trajectory data after time fusion and continuity correction.

7. The method for three-dimensional localization of physical objects based on video data according to claim 1, characterized in that: Simultaneously, a spatial constraint network based on the relative positions and contact relationships between objects is constructed. By minimizing the smoothness constraints of the motion trajectory and the consistency constraints of the spatial relationships, the specific process of solving for the six-degree-of-freedom pose of the object is as follows: Based on the object motion trajectory data, the contact, support and adjacency relationships between objects in the scene are identified, and geometric constraint expressions are established for each relationship. Among them, the support relationship is constrained by keeping the normal direction consistent, and the contact relationship is constrained by minimizing the distance between the contact points. A joint optimization objective function is constructed, in which the trajectory smoothness constraint is expressed as a second-order continuous term of pose change between adjacent frames, and the spatial relationship constraint is expressed as a relative pose error term between objects. Iterative optimization is performed in a three-dimensional rigid body motion space. The weights are dynamically adjusted according to the constraint residuals, and the output is a six-degree-of-freedom pose of the object that satisfies both motion continuity and spatial constraint consistency.

8. The method for three-dimensional localization of physical objects based on video data according to claim 1, characterized in that: Based on the six-degree-of-freedom pose of the objects, a neural radiation field representation is established for each object. Object images and segmentation masks are synthesized from the current viewpoint using differentiable rendering. The specific process for calculating the photometric and geometric losses of the synthesized results compared to the actual observation data is as follows: Based on the six-degree-of-freedom pose of an object, a neural radiation field representation centered on the object is constructed, and volume density and color distribution are modeled simultaneously through a multi-layer network. Through a differentiable rendering process, ray sampling and volumetric rendering are performed from the current camera perspective to generate a composite image and segmentation mask. The difference between the composite result and the real observed image in terms of brightness and structure is calculated to obtain the photometric loss. The geometric loss is obtained by calculating the contour and overlap differences between the synthetic mask and the real segmentation mask, and a regularization term based on normal consistency is introduced to maintain the smoothness of the reconstructed surface.

9. The method for three-dimensional localization of physical objects based on video data according to claim 8, characterized in that: The specific process of jointly optimizing neural representation parameters and object pose parameters through backpropagation, and then feeding the optimized object pose parameters back to the motion trajectory model for state updates in subsequent frames is as follows: An end-to-end differentiable optimization framework is established, with photometric loss and geometric loss as objective functions. The neural radiation field parameters and object pose parameters are updated simultaneously through gradient backpropagation. Lie algebra representation is used in pose update to ensure the numerical stability and geometric consistency of rotation calculation; The optimized six-DOF pose of the object is written into the motion trajectory model to update the latest state information of the object and provide prior information for localization in the next frame.

10. A physical object three-dimensional positioning system based on video data, applied to the physical object three-dimensional positioning method based on video data as described in any one of claims 1-9, characterized in that, Includes the following modules: The scene parsing and 3D reconstruction module is used to extract continuous frame images from the input video stream, generate a segmentation mask for each object through a segmentation network, extract feature points from continuous frame images and establish inter-frame feature point matching relationships, calculate camera motion parameters based on the matching relationships, and construct a sparse 3D point cloud of the scene. The object localization module generates multiple candidate 3D bounding boxes for each object based on the segmentation mask and sparse point cloud. Each 3D bounding box is projected onto the 2D image plane. The geometric matching degree between the projected contour and the segmentation mask and the similarity between the image features in the projected area and the object template features are calculated and comprehensively evaluated. The optimal 3D bounding box is selected as the preliminary 3D localization result of the object. The spatiotemporal constraint optimization module is used to fuse the preliminary 3D positioning results of the object with the motion data of historical frames to establish the motion trajectory model of the object between consecutive frames. At the same time, it constructs a spatial constraint network based on the relative position and contact relationship between objects. By minimizing the smoothness constraint of the motion trajectory and the consistency constraint of the spatial relationship, the six-degree-of-freedom pose of the object is obtained. The neural representation and pose optimization module is used to establish a neural radiation field representation for each object based on its six-degree-of-freedom pose. It synthesizes the object image and segmentation mask from the current viewpoint through differentiable rendering, calculates the photometric loss and geometric loss between the synthesized result and the actual observation data, jointly optimizes the neural representation parameters and object pose parameters through backpropagation, and feeds the optimized object pose parameters back to the motion trajectory model for state updates in subsequent frames.

Images