Dynamic space relation analysis method based on three-dimensional scene reconstruction

Abstract

The embodiment of the invention provides a dynamic space relation analysis method based on three-dimensional scene reconstruction, and belongs to the technical field of three-dimensional computer vision. The method comprises the following steps: constructing an initial scene model formed by a plurality of explicit three-dimensional geometric primitives, and constructing a spatial index data structure for organizing the primitives according to geometric and semantic attributes of the primitives; in a real-time operation stage, in response to the obtained new sensor data, locally updating the primitive subset corresponding to the currently moving object in the scene model, and synchronously updating the spatial index data structure; and receiving a query request containing semantic description, efficiently executing the request on the updated index, and generating an analysis result by firstly utilizing semantic positioning and then utilizing geometric attributes to determine a spatial relationship. According to the method, high-fidelity representation, real-time local updating and high-performance semantic query are combined, so that the system can accurately analyze a dynamic three-dimensional scene at extremely high frequency, and the autonomous navigation and safe obstacle avoidance capabilities of an unmanned aerial vehicle and a robot are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional computer vision, in particular to a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction. BACKGROUND

[0002] In the forefront fields such as special scene operation, robot technology, and self-driving car, it is a core prerequisite to endow a machine (especially an unmanned aerial vehicle or a ground robot performing an autonomous inspection task) with accurate and real-time perception and understanding ability of dynamic three-dimensional environment, so as to realize advanced autonomous navigation and obstacle avoidance functions.

[0003] However, the current simultaneous localization and mapping (SLAM) technology for realizing autonomous navigation has significant limitations. The core design goal of the SLAM system is to improve the positioning robustness of the robot itself. In order to achieve this positioning accuracy, the mainstream SLAM tends to regard dynamic elements in the scene (such as other moving robots or cables slightly swinging in the wind) as abnormal values or noise interfering with positioning, and tends to exclude or filter them from the final map.

[0004] This positioning-first strategy is far from enough for the autonomous inspection task that requires high-precision obstacle avoidance. For example, an unmanned aerial vehicle flying between complex structures such as towers or outdoor frames must first perceive and avoid all objects in the environment, especially those difficult-to-model lines in real time. If the system actively ignores these key and possibly moving obstacles for a clean map, its navigation decision will be based on an incomplete and distorted world model, which has a high collision risk. The existing SLAM system can answer the question "where is the unmanned aerial vehicle in the static world?", but cannot answer the more advanced question "what is the precise three-dimensional distance between the unmanned aerial vehicle and the swinging cable?" which is crucial for obstacle avoidance. At the same time, other related technologies are also difficult to make up for this defect. High-fidelity three-dimensional reconstruction technologies such as photogrammetry or neural radiance field (NeRF) can generate high-precision static models, but they require a large amount of offline processing time and cannot meet the fast response required for real-time obstacle avoidance. Existing real-time collision detection systems such as game physics engines are fast, but they rely on simplified geometric primitives such as bounding boxes or spheres for approximate calculation, and such low-fidelity representations cannot accurately describe the complex or fine structures commonly found in autonomous inspection scenes.

[0005] The present application inventors found that the above-mentioned prior art solutions have obvious defects when serving autonomous navigation and obstacle avoidance tasks in the process of implementing the present application. SUMMARY

[0006] The embodiment of the application aims to provide a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction, which can reconstruct a three-dimensional scene dynamically with high fidelity and perform efficient spatial query to realize real-time and accurate obstacle avoidance.

[0007] To achieve the above-mentioned purpose, the embodiment of the application provides a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction, which comprises the following steps: obtaining a sensor data stream representing a dynamic three-dimensional scene, and constructing an initial scene model for representing the dynamic three-dimensional scene based on the sensor data stream, wherein the scene model is composed of a plurality of explicit three-dimensional geometric primitives, each of which has geometric attributes and semantic attributes; constructing a corresponding spatial index data structure based on the initial scene model, wherein the spatial index data structure organizes the primitives constituting the scene model according to the geometric attributes and the semantic attributes; in response to the obtained new sensor data, performing local update on a subset of primitives in the scene model corresponding to objects currently in motion, so that the scene model is consistent with the current state of the dynamic three-dimensional scene; in response to the update of the scene model, synchronously updating the spatial index data structure; receiving a query request for a spatial relationship or a spatial attribute related to a target object in the three-dimensional dynamic scene, wherein the query request further comprises a semantic description of the target object; executing the query request on the current spatial index data structure, wherein a corresponding subset of primitives is located according to the semantic description, and the spatial relationship or the spatial attribute is determined based on the geometric attributes of the primitives in the located subset of primitives to generate an analysis result.

[0008] Optionally, the three-dimensional geometric primitives are three-dimensional Gaussian primitives, and the definition parameters of each three-dimensional Gaussian primitive include position, covariance matrix, opacity and color.

[0009] Optionally, the local update on the subset of primitives in the scene model corresponding to the objects currently in motion comprises: identifying the subset of primitives corresponding to the objects currently in motion based on the new sensor data; predicting the state of the subset of primitives at the next time based on a motion model to obtain a predicted state of the subset of primitives; using the predicted state as an initial estimate, and combining the new sensor data to correct the subset of primitives.

[0010] Optionally, the identification of the subset of primitives corresponding to the objects currently in motion based on the new sensor data comprises: identify a set of candidate primitives corresponding to known movable object classes from the new sensor data using real-time semantic segmentation; perform geometry-based change detection on the set of candidate primitives by comparing the new sensor data with sensor data from a previous time instance to finally determine a set of primitives corresponding to the object currently undergoing motion from the set of candidate primitives.

[0011] Optionally, if the object currently undergoing motion is a rigid object, the motion model is a rigid kinematic model, and the kinematic model is used to predict part or all of the position, pose, linear velocity and angular velocity of the rigid object at a next time instance; if the object currently undergoing motion is a non-rigid object, the motion model is a deformation model based on optical flow vectors, the deformation model calculates a dense optical flow field corresponding to the set of primitives, and combines the depth information of the primitives to obtain a three-dimensional vector field, and predicts the state of each primitive in the set of primitives at a next time instance based on the three-dimensional vector field, the state including independent scale transformation and direction transformation.

[0012] Optionally, using the predicted state as an initial estimate, and combining the new sensor data, the set of primitives is corrected, including: integrate the predicted state of the set of primitives into a scene model, and render a predicted image based on the integrated scene model; calculate the loss between the predicted image and a real image obtained based on the new sensor data in a pixel region corresponding to the set of primitives; update the geometric properties of the primitives in the set of primitives in the scene model by backpropagating the loss.

[0013] Optionally, the spatial index data structure adopts a tree data structure, wherein each internal node of the tree data structure stores aggregated semantic information, and the aggregated semantic information summarizes the semantic properties of the primitives contained by all child nodes of the internal node, so as to prune tree branches irrelevant to the semantic description of the object when performing a query.

[0014] Optionally, constructing the spatial index data structure includes a recursive partitioning process on the three-dimensional space represented by each node, wherein for each node, the partitioning axis of the node is selected according to the following manner: calculate the average principal direction of the covariance matrix of all three-dimensional Gaussian primitives contained in the node; select the coordinate axis most perpendicular to the average principal direction as the partitioning axis.

[0015] Optionally, the spatial index data structure adopting a tree data structure is updated based on a lazy rebalancing strategy, including: In the tree, the primitives whose positions have changed are logically deleted, and the updated primitives are inserted as new nodes; In the insertion process, only when the balance factor of a node on the path affected by the update exceeds a preset threshold, a local subtree rooted at the node whose balance factor exceeds the preset threshold is triggered to be rebuilt, wherein the balance factor is used to measure the balance state of the subtree corresponding to the node.

[0016] Optionally, the update operation of the spatial index data structure contains a timestamp; and the spatial index data structure further supports a time sequence query for retrieving primitives satisfying a preset spatial relationship within a specified time period.

[0017] Through the above technical solution, the application provides a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction. By using explicit three-dimensional geometric primitives to construct a high-fidelity scene model, and combining a real-time updating mechanism for locally updating a primitive subset corresponding to an object currently in motion, it is ensured that the model is always consistent with the current state of the dynamic physical world; by synchronously updating a spatial index data structure for organizing primitives according to spatial and semantic attributes, spatial relationship queries are realized. According to the method, spatial relationship query requests containing high-level semantic descriptions can be efficiently executed, the relevant primitive subset can be quickly located using semantic descriptions, accurate spatial relationship analysis can be performed based on geometric attributes, and finally analysis results are generated. According to the method, a machine system can quickly and real-timely reason complex spatial relationships, and the efficiency and safety of its execution of autonomous navigation and dynamic obstacle avoidance and the like are improved.

[0018] Other features and advantages of the embodiments of the application will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are included to provide a further understanding of the embodiments of the application, and constitute a part of the specification, and are used together with the following detailed description to explain the embodiments of the application, but do not constitute a limitation of the embodiments of the application. In the drawings: Figure 1 is a flowchart of a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction according to an embodiment of the application; Figure 2 is a schematic diagram of the overall system architecture according to an exemplary embodiment of the application; Figure 3 is a detailed flowchart of a prediction and correction Gaussian flow updating mechanism according to an exemplary embodiment of the application; Figure 4 is a schematic diagram of a semantic time sequence tree data structure according to an exemplary embodiment of the application; Figure 5 is a structural schematic diagram of a spatial logic rules engine according to an example embodiment of the present application; Figure 6 is a step schematic diagram of an end-to-end processing example according to an example embodiment of the present application; Figure 7 is a structural schematic diagram of an electronic device according to at least one embodiment of the present application. DETAILED DESCRIPTION

[0020] The example embodiments will now be described in detail with reference to the drawings. Like reference numerals indicate corresponding parts throughout the drawings. The following detailed description implements the example embodiments and is not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the appended claims.

[0021] Embodiments of the present application can be applied to a computer system / server, which can be operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with computer system / server include, but are not limited to: personal computers, servers, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputers, mainframe computers, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0022] The computer system / server can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, that perform particular tasks or implement particular abstract data types. Computer system / server can operate in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0023] The prior art (such as SLAM) actively eliminates dynamic objects to ensure positioning accuracy, which cannot be used for accurate obstacle avoidance and spatial relationship analysis; and high-fidelity technology (such as NeRF) cannot realize real-time online processing due to huge amount of calculation. The present application proposes a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction. The present application uses an explicit three-dimensional geometric primitive (such as a three-dimensional Gaussian primitive) as the bottom representation of the scene, which guarantees the high-fidelity reconstruction quality and makes the geometric properties (such as position and shape) directly accessible and high-speed queryable, thereby avoiding the defect that the implicit representation (NeRF) is difficult to query. The present application also proposes a prediction-correction Gaussian flow mechanism for online model updating mechanism, which only performs efficient local update on the subset of primitives corresponding to the objects currently moving in the scene, thereby realizing real-time, online self-adaptation of high-fidelity explicit models. The present application also proposes a specially designed spatial index data structure. The index structure integrates the geometric properties and semantic properties of the primitives, and through the division strategy and incremental update mechanism of the anisotropic shape of the perception primitives, it can support high-throughput spatial relationship query. The present application is helpful to realize real-time and accurate understanding and reasoning of the dynamic three-dimensional world.

[0024] Reference Figure 1 which shows a flowchart of a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction according to an embodiment of the present application. The method can be performed by a computing device deployed in a robot, unmanned aerial vehicle, autonomous vehicle or fixed monitoring station. As shown in the figure, the method comprises the following steps S101-S106.

[0025] Step S101, acquiring a sensor data stream representing a dynamic three-dimensional scene, and constructing an initial scene model for representing the dynamic three-dimensional scene based on the sensor data stream, wherein the scene model is composed of a plurality of explicit three-dimensional geometric primitives, each of which has geometric properties and semantic properties.

[0026] The dynamic three-dimensional scene refers to a three-dimensional physical space containing static background (such as tower, ground, fixed machine) and dynamic objects (such as moving robot, unmanned aerial vehicle, staff or wind-swinging cable). The sensor data stream refers to a continuous data sequence from one or more sensors for perceiving the three-dimensional scene. In some embodiments, the sensor can be an RGB-D camera, which can provide color images and depth maps simultaneously. In some embodiments, the sensor can also be a LiDAR, which can directly obtain high-precision three-dimensional point clouds. In other embodiments, the sensor can also be an event camera, which can asynchronously report pixel brightness changes at a microsecond level, which is very suitable for capturing high-speed motion.

[0027] The system uses the first few frames of sensor data to globally reconstruct the scene and build an initial scene model within the first few seconds of startup. In some embodiments, techniques such as Structure-from-Motion (SfM) can be used to build the initial scene model.

[0028] The scene model is composed of a number of explicit three-dimensional geometric primitives. “Explicit” is a concept opposite to “implicit” (e.g. NeRF), according to which each geometric element in the scene has well-defined properties that can be directly accessed and queried by the system, thus avoiding the expensive ray marching or root finding operations that must be performed by implicit representation methods to determine object surfaces.

[0029] In the scene model, each primitive has geometric properties and semantic properties. Geometric properties refer to parameters that describe the physical existence of the primitive, such as its position, rotation pose, shape, and size in three-dimensional space. Semantic properties refer to labels that describe the business meaning of the primitive, such as whether it belongs to a robot arm, a cable, or the ground, etc.

[0030] In some embodiments, the explicit three-dimensional geometric primitives are three-dimensional Gaussian primitives, and the definition parameters of each three-dimensional Gaussian primitive include position, covariance matrix, opacity, and color.

[0031] According to the present embodiment, the present application uses 4D Gaussian Splatting (4D-GS) technology as the underlying representation method for the scene. Unlike traditional meshes or point clouds, 4D-GS uses a set of (e.g. millions of) three-dimensional Gaussian ellipsoids to model the scene. It has the advantages of real-time rendering performance (e.g. over 80 FPS on an RTX 3090 GPU) and direct querying. The geometric properties of the primitives can be used as the definition parameters of their 3D Gaussian ellipsoids, including: position (defining the center point), covariance matrix (defining its shape, size, and direction), opacity (defining its transparency), and color (defining its appearance).

[0032] In step S102, based on the initial scene model, a corresponding spatial index data structure is constructed, which organizes the primitives of the scene model according to the geometric properties and semantic properties.

[0033] The initial scene model may contain millions of unordered 3D Gaussian primitives, making it inefficient to directly execute query requests on this set (e.g., "calculate the nearest distance between A and B"). Therefore, according to this embodiment, a corresponding efficient spatial index data structure can be constructed. The spatial index data structure proposed in this embodiment can spatially partition primitives according to their geometric attributes (such as position and shape) to accelerate spatial proximity search, and can also organize primitives according to their semantic attributes (such as labels) to accelerate semantic queries.

[0034] Step S103: In response to the acquired new sensor data, locally update the subset of primitives in the scene model corresponding to the currently moving object, so that the scene model is consistent with the current state of the dynamic 3D scene.

[0035] After initialization, the system enters the real-time online operation phase, beginning to process new data from the sensors in real time. This embodiment employs a local update strategy: in each frame, the system first identifies currently moving objects in the scene (e.g., a swinging robotic arm) and strictly limits computational resources to updating only a subset of primitives representing these moving objects; primitives corresponding to the static background are not included in the computation. This localization strategy makes the model update speed more than an order of magnitude faster than global optimization, thereby achieving truly real-time, online adaptive scene modeling.

[0036] In some implementations, the local update may include: identifying a subset of primitives corresponding to the currently moving object based on the new sensor data; predicting the state of the primitive subset at the next moment based on the motion model to obtain the predicted state of the primitive subset; and using the predicted state as an initial estimate and combining it with the new sensor data to correct the primitive subset.

[0037] According to this implementation method, firstly, it is identified which parts of the scene are moving, then a motion model is used to predict where they will move to in the next moment, and finally, new sensor data is used to correct the prediction results to ensure the real-time accuracy of the scene model.

[0038] In some implementations, identifying the primitive subset corresponding to the currently moving object (also referred to as the dynamic object in this invention) may include: using real-time semantic segmentation to identify a set of candidate primitive subsets corresponding to known movable object categories from the new sensor data; and performing geometry-based change detection on the candidate primitive subsets by comparing the new sensor data with the sensor data at previous times, so as to finally determine the primitive subset corresponding to the currently moving object from the candidate primitive subsets.

[0039] The present embodiment proposes a two-stage recognition strategy to achieve a balance between speed and accuracy. First, the system can use a pre-trained real-time semantic segmentation model (for example, a scheme inspired by the efficient version of SAM used in DyGS-SLAM and prompted by YOLO-World) to quickly find all movable object categories (such as "people", "robot arms") from the new sensor image frame and mark them as a candidate primitive subset. For all candidates screened out, further confirmation can be made through geometric change detection to determine whether they are moving in the current frame. This can be achieved by comparing new sensor data with data at previous time, such as calculating the dense optical flow field between consecutive RGB frames, or comparing consecutive depth maps to identify areas that have undergone significant displacement. Objects that meet both the semantic movable and geometric moving conditions will be finally recognized as targets that need to be updated at the current time.

[0040] In some embodiments, if the object currently in motion is a rigid object (i.e. an object that does not deform, such as a robot link), the motion model is a rigid kinematics model (such as a state estimation filter of extended Kalman filter EKF), which is used to predict part or all of the position, pose, linear velocity and angular velocity of the rigid object at the next time, for example, the extended Kalman filter EKF can track a high-dimensional state vector (such as a 13-dimensional vector containing 3D position, 3D direction (quaternion), 3D linear velocity and 3D angular velocity), and based on a simple constant velocity physical model, quickly predict the overall position and pose of the object at the next time.

[0041] In some embodiments, if the object currently in motion is a non-rigid object (i.e. a deformable object, such as a cable swinging in the wind or a worker's clothes), the motion model is a deformation model based on optical flow vectors, which calculates the dense optical flow field corresponding to the primitive subset, and combines the depth information of the primitive (such as from an RGB-D sensor or a laser radar) to obtain a three-dimensional vector field, and based on the three-dimensional vector field, predicts the state of each primitive in the primitive subset at the next time, the state includes independent scale transformation and direction transformation, thereby modeling the deformation of the object such as stretching and bending.

[0042] In some embodiments, the correction of the primitive subset includes: integrating the predicted state of the primitive subset into the scene model, and rendering a predicted image based on the integrated scene model; calculating the loss between the predicted image and the real image obtained based on the new sensor data in the pixel region corresponding to the primitive subset; updating the geometric properties of the primitives in the primitive subset in the scene model by backpropagating the loss.

[0043] According to this implementation, the system can integrate the predicted state of the currently predicted primitive subset (i.e., the new position of the dynamic part) with the static background part that remains unchanged in the scene model to form a temporary, complete predicted scene model, and then render it. Next, rendering can be performed only in very small pixel areas affected by moving objects. Inside, calculate the predicted image ( ) and real sensor images ( The error between the predicted and ground truth images within a specific pixel region can be represented by a weighted combination of photometric L1 loss and differentiable structural similarity (D-SSIM) loss. For example, the error between the predicted and ground truth images within a specific pixel region can be calculated using the following loss function: , in, Indicates the predicted image at the pixel level. The value at that location, Represents the real image in pixels The value at that location, L1 norm represents the photometric loss, which is the absolute value of the difference between the rendered pixel value and the actual pixel value. It is used to measure the color fidelity of the predicted image. This represents the differentiable structural similarity loss, used to compare the similarity between the structure, contrast, and brightness of local regions in an image, in order to measure the structural fidelity of the predicted image. and These are weighting coefficients, which are preset hyperparameters.

[0044] Then backpropagation is performed to correct the error. During this process, the gradient flow is strictly limited. Only the geometric attribute parameters (such as position, rotation, and scaling) of primitives belonging to the identified primitive subset (i.e., dynamic objects) are updated. Static background primitives, which represent the vast majority of elements in the scene, do not participate in the calculation during this process, thus greatly saving computational resources.

[0045] To model dynamic scenes (i.e., 4D), traditional 4D Gaussian splashing methods rely on offline training of a deformation field (such as an MLP) to predict the changes of Gaussian primitives over time. However, this method cannot meet the requirements for real-time online updates. The online update mechanism proposed in this invention directly updates the geometric properties of dynamic primitives in the scene model in real time by predicting and correcting the Gaussian flow, without relying on the deformation field.

[0046] Step S104: In response to the update of the scene model, the spatial index data structure is updated synchronously.

[0047] After the primitive attributes of the dynamic objects are updated, the positions and shapes of the objects in the three-dimensional space are changed, and the spatial index data structure is correspondingly updated to enable the internal organization structure to accurately reflect the latest state of the scene.

[0048] In some embodiments, the spatial index data structure adopts a tree data structure, wherein each internal node of the tree data structure stores aggregated semantic information that summarizes semantic attributes of primitives contained in all child nodes of the internal node, to perform pruning on tree branches irrelevant to semantic descriptions of the objects when performing a query.

[0049] According to the present embodiment, each non-leaf internal node in the tree stores aggregated semantic information (for example, a bitmask or a set) that records all semantic IDs (such as worker_head and robot_link_5) present in all branches under the internal node. When performing a semantic query (for example, a query distance(robot, human)), the system can immediately prune all entire tree branches that do not contain robot or human semantic information, thereby significantly reducing the search space and improving the query efficiency.

[0050] In some embodiments, constructing the spatial index data structure includes performing a recursive partitioning process on the three-dimensional space represented by each node, wherein for each node, a partitioning axis of the node is selected according to the following manner: calculating an average principal direction of covariance matrices of all three-dimensional Gaussian primitives contained in the node; and selecting a coordinate axis most perpendicular to the average principal direction as the partitioning axis.

[0051] Unlike a standard K-d tree that generally performs partitioning at the median of data points, according to the present embodiment, when recursively partitioning the three-dimensional space, for any node, the average principal direction (that is, the average of the longest direction of the Gaussian ellipsoid) of covariance matrices of all Gaussian primitives in the node can be calculated by, for example, principal component analysis (PCA), and then a coordinate axis (such as X, Y, or Z) most perpendicular to the average principal direction can be selected as the partitioning axis, thereby effectively splitting the elongated Gaussian ellipsoid in space and ensuring that the partitioning plane can more closely wrap the primitive to achieve more efficient pruning in subsequent queries.

[0052] In some embodiments, the spatial index data structure employing a tree data structure is updated based on a lazy rebalancing strategy, including: in the tree, logically deleting a cell whose location has changed, and inserting the updated cell as a new node; only when a balance factor of a node on the path affected by the update exceeds a preset threshold, triggering a reconstruction of a local subtree rooted at the node whose balance factor exceeds the preset threshold, wherein the balance factor is used to measure the balance state of the left and right subtrees of the corresponding node.

[0053] According to the present embodiment, the incremental update efficiency of the tree data structure can be significantly improved. When the state of a Gaussian cell is updated, the system can first mark its old location as "deleted" (logically deleted) in the tree, and then insert the updated cell as a new node. During the insertion process, the balance factor of the node on the path affected by the update can be checked. In the tree data structure, the path affected by the update refers to the set of all nodes traversed by the algorithm from the root node to the target leaf node when performing the above-mentioned logical deletion and new node insertion operations. The balance factor is an indicator used to measure the balance state of the node subtree, for example, it can be the height difference or the node number ratio of the left and right subtrees. Only when the balance factor of a certain node becomes excessively unbalanced (i.e., exceeds a preset threshold) due to this insertion, a local subtree reconstruction is triggered, i.e., only the subtree rooted at the unbalanced node is reconstructed, thereby avoiding the reconstruction operation in most cases and ensuring the real-time performance of the update step.

[0054] In some embodiments, the update operation of the spatial index data structure contains a timestamp; and the spatial index data structure further supports a time-series query for retrieving cells satisfying a preset spatial relationship within a specified time period.

[0055] According to the present embodiment, each insertion is time-stamped when performing the update operation, so that the index structure can support time-series queries and efficiently answer complex spatio-temporal queries such as "find all objects that have entered the dangerous area X in the past 2 seconds".

[0056] Step S105, receiving a query request for a spatial relationship or a spatial attribute related to a target object in a three-dimensional dynamic scene, the query request further including a semantic description of the target object.

[0057] The query request is an instruction from an external system (e.g., a robot control program, a drone navigation system, or a security monitoring application) to request a specific spatial relationship / spatial attribute analysis on the current three-dimensional scene state. The target object refers to one or more specific entities in the scene that are of interest to the user or the external system, such as a specific robot arm, a worker's head, a cable, and / or a designated virtual safety area.

[0058] Spatial relationship can refer to the relative state of a target object with respect to other objects or a reference system. Spatial relationship can include topological relationship, such as querying whether different target objects intersect or contain, can also include directional relationship, such as querying whether one object is above another object, and can also include metric relationship, such as querying the minimum distance between two objects. Spatial attribute can refer to the state of a target object or its association with a specific region, for example, a query request can require counting the number of objects of a specific type within a specified region, etc.

[0059] The query request includes semantic description of the target object. Semantic description refers to the descriptive label used by the user or external system to specify the target object when initiating the query. For example, in a security monitoring application, the query request can specify the target objects 'worker_head' and 'robot_arm' through semantic description semantic_tag='worker_head' and semantic_tag='robot_arm', without needing to care about the underlying specific, hundreds of thousands of three-dimensional Gaussian primitives that make up the target objects.

[0060] Step S106, executing the query request on the current spatial index data structure, wherein the corresponding primitive subset is located according to the semantic description, and the spatial relationship or spatial attribute is determined based on the geometric properties of the primitives in the located primitive subset to generate the analysis result.

[0061] According to the present embodiment, the query execution can be divided into two stages. The first stage can be referred to as coarse screening, that is, locating the corresponding primitive subset according to the semantic description. The embedded semantic function of the data structure can be used to quickly traverse the tree, and a large number of irrelevant tree branches can be quickly removed through semantic pruning, so as to efficiently locate all primitive subsets containing the target semantic description (such as 'worker_head' and 'robot_arm').

[0062] The second stage can be referred to as fine screening, that is, determining the spatial relationship / spatial attribute based on the geometric properties of the primitives in the located primitive subset. For the small-scale candidate primitive subset returned by the coarse screening stage, the system can call the computational geometry algorithm to perform parallel and accurate geometric tests on the GPU. For example, for distance queries, the system can use an extension of the Gilbert-Johnson-Keerthi (GJK) algorithm to accurately calculate the minimum distance between two Gaussian ellipsoids (convex bodies); for intersection queries, the GJK algorithm can be used to detect whether the intersection exists.

[0063] Finally, the system can generate an analysis result, which can be a Boolean value, a distance value, or an object count value, etc., and output it to the control system or monitoring system for triggering subsequent actions or decisions.

[0064] Figure 2 A schematic diagram of the overall system architecture according to an exemplary embodiment of the present invention is shown. The system is capable of efficiently converting raw sensor data streams from data sources (such as one or more RGB-D cameras or LiDAR) into spatial knowledge that can be queried in a high-level manner by external systems (such as robot controllers or security monitoring systems) in real time.

[0065] like Figure 2 As shown, the system includes an input and preprocessing module, a core processing loop module, and a query and execution module. Data is received by the input module and sent to the preprocessing module. This preprocessing module may include a real-time instance segmentation algorithm to initially identify potential dynamic objects in the scene (such as drones and workers) and assign them preliminary semantic labels.

[0066] The processed data frames are fed into the system's core processing loop. This loop consists of a tightly coupled representation and update module and core data structures. The representation and update module is responsible for maintaining a high-fidelity geometric representation of the scene and executing a predictive-corrected Gaussian flow mechanism. Once this module efficiently updates the primitive properties representing dynamic objects based on new data, it immediately synchronizes these updates to the core data structures, such as a spatial index data structure employing a tree-like structure. This tight loop synchronization ensures that the spatial index always remains consistent with the latest state of the scene.

[0067] The query and execution modules respond to high-level spatial rules from external systems. These rules are parsed and compiled by the spatial logic rule engine in the query module to generate an efficient query plan. This plan is then sent to the execution module for efficient execution on the spatial index data structure. The execution process can be divided into two stages: coarse screening and fine screening. The query results obtained after execution (e.g., Boolean values ​​or distance values) are returned to the external system by the output module to trigger decision-making.

[0068] Figure 3 A detailed flowchart of a prediction and correction Gaussian flow update mechanism according to an exemplary embodiment of the present invention is shown. This mechanism is... Figure 2 The specific implementation of the representation and update module addresses the challenge of the standard 4D-GS method's inability to update in real-time online. As shown in the figure, when a new sensor frame (such as RGB-D data) arrives, dynamic object recognition is performed first. By combining semantic and geometric paths—that is, using a semantic segmentation model to identify known moving objects and using optical flow or depth change detection to identify motion—the system accurately identifies the currently moving object and its corresponding set of primitives.

[0069] Subsequently, a prediction step is entered. In this step, the system can employ different motion models according to the object properties. For the recognized rigid objects, a kinematic model (e.g. an extended Kalman filter) can be employed to predict the kinematic state of the object. For the non-rigid objects, a morphing model based on the optical flow vectors can be employed to predict the independent morphing of each primitive by the computed dense optical flow field. The two paths jointly generate the predicted state of the dynamic primitives at the next time instance.

[0070] A correction step is then entered. After entering the correction step, the system can perform a localization correction optimization step. The present invention reuses the spatial index based on the spatial index data structure, which first utilizes the index to efficiently query the screen space pixel region covered by the primitives affected by the predicted transformation, and then computes the photometric loss between the rendered image and the real image within the local region. By back-propagating the local loss, the system can only update the parameters of those affected dynamic Gaussian primitives, and finally generate the updated dynamic scene model.

[0071] Figure 4 A schematic diagram of a semantic temporal tree data structure according to an example embodiment of the present invention is shown. The data structure is a concrete implementation of the core data structure in Figure 2 The root node and all internal nodes (e.g. child node 1, child node 2) of the tree store aggregated semantic information, for example, the root node stores the semantic labels {'worker_head', 'table', 'robot_arm'} contained by all the primitives under it. This design enables efficient semantic pruning during query. For example, a query for only 'robot_arm' can immediately prune the entire "child node 1" branch, as its aggregated semantics does not contain the label. Meanwhile, the tree employs a Gaussian shape-aware partitioning strategy (i.e. partitioning based on principal component analysis, PCA), which partitions by selecting the coordinate axis (e.g. Y or Z) most perpendicular to the average principal direction of the Gaussian primitives within the node, thus achieving tighter packing and more efficient spatial pruning than the standard K-d tree. Figure 4 The content of a leaf node D is shown in detail in the lower left corner of FIG. 4, which contains two Gaussian primitives (ellipsoids) with explicit semantic properties.

[0072] Figure 5 A structural schematic diagram of a spatial logic rule engine according to an example embodiment of the present invention is shown. The engine is a concrete implementation of the core data structure in Figure 2The specific implementation of the spatial query module provides a powerful and easy-to-use high-level query interface for users. As shown in the figure, the external system (such as a safety controller) inputs human-readable spatial logic language (SLL) rules (for example, VERIFYdistance(A, B) > 1.5 43). These rules are first processed by the SLL parser in the engine, which converts them from a text string into an abstract syntax tree (AST) that can be understood by a computer. Subsequently, the AST is handed over to the query planner of the engine. After receiving the AST, the query planner compiles it into an optimized, executable query plan. This plan is broken down into two stages: first, a coarse screening plan that performs a quick narrowing of the search range is executed to traverse the spatial index data structure; then, a fine screening plan that performs precise geometric tests (for example, using the GJK algorithm) is executed. The two-stage executable query plan is submitted to the execution module and is executed at high speed on the index data.

[0073] Reference Figure 6 It takes the human-robot collaborative safety monitoring in a typical smart factory as an example to demonstrate the end-to-end processing flow of the dynamic spatial relationship analysis method based on three-dimensional scene reconstruction proposed according to the application.

[0074] The scenario is that a worker collaborates with a large six-axis robot arm at an assembly workstation. The entire work area is monitored by multiple RGB-D sensors, and the RGB-D video streams thereof serve as real-time inputs of the system.

[0075] In the initialization phase of system startup, the system can use the first few frames of data to construct an initial high-fidelity scene model containing static workbenches, backgrounds, and dynamic workers and robots, and simultaneously construct an initial spatial index data structure. In this process, the head of the worker and each link of the robot arm are automatically identified and given corresponding semantic labels (also known as semantic attributes), such as 'worker_head' and 'robot_arm'.

[0076] Phase one of the flow: real-time perception and update As Figure 6 As shown on the right, once the initialization is completed, the system enters a continuous real-time perception and update cycle.

[0077] When the worker and the robot both start to move, the sensors continuously input new RGB-D frames at a high frequency (for example, 30 Hz).

[0078] In each frame, the prediction and correction Gaussian flow mechanism is started, and the worker and the robot are identified as the objects that are currently moving through semantic and geometric change detection.

[0079] This mechanism can update the Gaussian cells representing workers and robots in real-time and locally, generating a scene model that is consistent with the current state of the physical world.

[0080] In the meantime, these real-time updates can be immediately used to incrementally update the spatial index data structure, ensuring that the index is always synchronized with the model.

[0081] Phase 2: Continuous querying and evaluation As Figure 6 In parallel with the perception update loop, the spatial logic rules engine can continuously execute query requests at very high frequency (e.g., over 1000 Hz, much higher than the sensor frame rate), as shown on the left.

[0082] In this example, a safety engineer can predefine a safety rule in the engine, e.g., 'distance(worker_head, robot_arm) > 0.8m', which means that the minimum distance between the worker's head and any part of the robot's arm must always be greater than 0.8 meters.

[0083] The engine will continuously execute the query plan compiled from this rule on the spatial index data structure.

[0084] Phase 3: Decision and execution As Figure 6 As shown at the bottom, each execution of the query engine generates an analysis result and triggers the corresponding decision.

[0085] Case 1 (safe): Suppose at time point T, the system computes the minimum distance between the worker's head and the robot's arm to be 0.95 meters by executing the query on the index. Since 0.95m is greater than the safety distance 0.8m, the query result is TRUE. At this time, the distance evaluation module determines that the state is safe. The system does not need to trigger any action, and the robot continues normal operation.

[0086] Case 2 (dangerous): Suppose at time point T+0.1 seconds, the robot's arm moves according to the trajectory and gets closer to the worker. The system computes the minimum distance to be 0.75 meters. Since 0.75m is less than the safety distance 0.8m, the query result becomes FALSE. At this time, the distance evaluation module determines that the state is dangerous.

[0087] This dangerous state can immediately generate a trigger signal and be sent to the external robot safety controller. After receiving the signal, the controller immediately triggers the emergency stop program.

[0088] In this way, the application can stop the robot's movement at the moment before the collision occurs, and the end-to-end delay of the entire system from capturing the dangerous posture by the sensor to receiving the stop signal by the robot controller can be controlled at a very low level (e.g., less than 15 milliseconds) to achieve high-precision, millisecond-level safety monitoring.

[0089] In summary, the application provides a dynamic spatial relationship analysis method based on three-dimensional scene reconstruction, which solves the problem that traditional high-fidelity models are difficult to perform real-time spatial queries by using high-fidelity explicit three-dimensional geometric primitives (such as three-dimensional Gaussian primitives). The application uses a prediction and correction mechanism to perform efficient local updates only on objects that are currently moving, to achieve real-time online adaptation of the model, and also uses a spatial index data structure that deeply fuses spatial and semantic attributes to support high-throughput complex queries. According to the application, spatial relationships in dynamic scenes can be accurately analyzed at high frequency, providing strong perception capabilities for the autonomous navigation and safe interaction of unmanned aerial vehicles and robots.

[0090] Figure 7 The electronic device provided for at least one embodiment of the application includes a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to implement the dynamic spatial relationship analysis method based on three-dimensional scene reconstruction according to any embodiment or implementation manner of the application when executing the computer instructions.

[0091] At least one embodiment of the application also provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the dynamic spatial relationship analysis method based on three-dimensional scene reconstruction according to any embodiment or implementation manner of the application.

[0092] It should be emphasized that the description of specific embodiments in the specification and the flowcharts shown in the drawings are intended to clearly set forth the technical solutions of the application, and not to limit the protection scope of the application to the specific structures, sequences or combinations described. Those skilled in the art can understand that the various technical features described in the embodiments can be recombined or split, and the execution order of the steps can also be adjusted without departing from the core spirit and principles of the application. In some cases, parallel or multi-task processing is also possible. Therefore, any modification, equivalent replacement and improvement made according to the technical essence of the application to the above embodiments shall be included within the protection scope of the application.

Claims

1. A dynamic spatial relationship analysis method based on 3D scene reconstruction, characterized in that, The method includes: Acquire sensor data streams representing dynamic 3D scenes, and construct an initial scene model representing the dynamic 3D scene based on the sensor data streams. The scene model consists of multiple explicit 3D geometric primitives, each of which has geometric and semantic attributes. Based on the initial scene model, a corresponding spatial index data structure is constructed. The spatial index data structure organizes the various primitives that constitute the scene model according to geometric and semantic attributes. In response to new sensor data, the subset of primitives in the scene model corresponding to the currently moving objects is locally updated to keep the scene model consistent with the current state of the dynamic 3D scene. In response to updates to the scene model, the spatial index data structure is updated synchronously. Receive a query request for spatial relationships or spatial attributes related to a target object in a three-dimensional dynamic scene, the query request also including a semantic description of the target object; The query request is executed on the current spatial index data structure, wherein the corresponding primitive subset is located according to the semantic description, and the spatial relationship or spatial attribute is determined based on the geometric attributes of the primitives in the located primitive subset, so as to generate the analysis result.

2. The method according to claim 1, characterized in that, The three-dimensional geometric primitives are three-dimensional Gaussian primitives, and the definition parameters of each three-dimensional Gaussian primitive include position, covariance matrix, opacity, and color.

3. The method according to claim 1, characterized in that, Local updates are performed on the subset of primitives in the scene model corresponding to the currently moving object, including: Based on the new sensor data, a subset of primitives corresponding to the currently moving object is identified; Based on the motion model, the state of the primitive subset at the next moment is predicted, and the predicted state of the primitive subset is obtained. The predicted state is used as an initial estimate, and the primitive subset is corrected by combining the new sensor data.

4. The method according to claim 3, characterized in that, Based on the new sensor data, a subset of primitives corresponding to the currently moving object is identified, including: Using real-time semantic segmentation, a subset of candidate primitives corresponding to known movable object categories is identified from the new sensor data; By comparing the new sensor data with the sensor data at previous times, geometry-based change detection is performed on the candidate primitive subset to ultimately determine the primitive subset corresponding to the currently moving object from the candidate primitive subset.

5. The method according to claim 3, characterized in that: If the object currently in motion is a rigid object, the motion model is a rigid kinematic model, which is used to predict some or all of the rigid object's position, attitude, linear velocity, and angular velocity at the next moment. If the object currently in motion is a non-rigid object, the motion model is a deformation model based on optical flow vectors. The deformation model calculates the dense optical flow field corresponding to the primitive subset, and combines the depth information of the primitives to obtain a three-dimensional vector field. Based on the three-dimensional vector field, it predicts the state of each primitive in the primitive subset at the next moment. The state includes independent scale transformation and orientation transformation.

6. The method according to claim 3, characterized in that, Using the predicted state as an initial estimate and combining it with the new sensor data, the primitive subset is corrected, including: The predicted states of the primitive subset are integrated into the scene model, and the predicted image is rendered based on the integrated scene model. Within the pixel region corresponding to the primitive subset, calculate the loss between the predicted image and the real image obtained based on the new sensor data; The geometric properties of primitives in the primitive subset of the scene model are updated by backpropagating the loss.

7. The method according to claim 1, characterized in that, The spatial index data structure adopts a tree-like data structure, wherein each internal node of the tree-like data structure stores aggregated semantic information. The aggregated semantic information summarizes the semantic attributes of the primitives contained in all child nodes of the internal node, so as to prune tree branches that are not related to the semantic description of the object when executing a query.

8. The method according to claim 7, characterized in that, Constructing the spatial index data structure includes a recursive partitioning process of the three-dimensional space represented by each node, wherein, for each node, the partitioning axis is selected according to the following method: Calculate the average principal direction of the covariance matrix of all three-dimensional Gaussian elements contained within the node; The coordinate axis most perpendicular to the average principal direction is selected as the dividing axis.

9. The method according to claim 7, characterized in that, The spatial index data structure, which employs a tree-like data structure, is updated based on the following lazy rebalancing strategy: In the tree, primitives whose positions have changed are logically deleted, and the updated primitives are inserted as new nodes; During the insertion process, reconstruction of the local subtree rooted at the node whose balance factor exceeds the preset threshold is triggered only when the balance factor of the node on the path affected by the update exceeds the preset threshold. The balance factor is used to measure the balance state of the subtree of the corresponding node.

10. The method according to claim 7, characterized in that, The update operation of the spatial index data structure includes a timestamp; and the spatial index data structure further supports time-series queries, which are used to retrieve primitives that satisfy a preset spatial relationship within a specified time period.

Images