A multi-person collision detection method and system for VR large-space immersive tour

Abstract

The present invention discloses a multi-person collision detection method and system for VR large-space immersive tours, comprising: obtaining RGB images and depth images through VR devices, and generating instance segmentation images based on the RGB images. Then, feature extraction is performed in combination with the instance segmentation image and the depth image to obtain static and dynamic optical flow images and 3D feature points. The initial camera posture is estimated using the static optical flow image and the static 3D feature points, while the motion residual is calculated based on the dynamic 3D feature points, and the initial object motion is optimized in combination with the dynamic optical flow image. Then, a factor graph is constructed based on the initial camera posture, object motion and 3D feature points, and the posture and motion trajectory of the dynamic object are optimized through residual calculation. Finally, multi-person collision detection is performed based on the optimized posture and trajectory to achieve high-precision and high-real-time collision warning, significantly improving the safety and immersion of the VR experience.

Description

Technical Field

[0001] The present invention belongs to the technical fields of artificial intelligence, computer vision, and computer graphics, and in particular relates to a multi-person collision detection method and system for VR large-space immersive tours. Background Art

[0002] Large-scale immersive tours typically involve groups of three to five people conducting group VR exploration within a specific location-based entertainment (LBE) venue. However, during the tour, participants may encounter various collision issues, such as collisions between people, collisions with walls, or collisions with props. These issues not only pose safety risks but also compromise the quality of the VR experience. This is especially true when there are large crowds of visitors, with multiple VR exploration teams simultaneously operating within the same LBE venue, further increasing the risk of collisions.

[0003] In order to solve the above problems and ensure that every participant can enjoy the VR experience safely and happily, it is particularly important to monitor and process the posture and motion trajectory of all dynamic objects in the venue in real time. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a multi-person collision detection method and system for VR large-space immersive tours to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above objectives, the present invention provides a multi-person collision detection method for VR large-space immersive tours, comprising:

[0006] Obtain RGB images and depth images through VR devices, and obtain instance segmentation images based on the RGB images;

[0007] Perform feature extraction based on the instance segmentation image and the depth image to obtain a static optical flow image, static 3D feature points, a dynamic optical flow image, and dynamic 3D feature points;

[0008] Obtaining an initial camera pose based on the static optical flow image and the static 3D feature points; calculating motion residuals between consecutive frames on each dynamic object based on the dynamic 3D feature points; and obtaining an initial object motion based on the dynamic optical flow image, the dynamic 3D feature points, and the motion residuals;

[0009] constructing a factor graph based on the initial camera pose, the initial object motion, static 3D feature points, and dynamic 3D feature points; performing a residual operation based on the factor graph, and obtaining the pose and motion trajectory of each dynamic object based on the residual operation result, the initial camera pose, and the initial object motion;

[0010] Multi-person collision detection is performed based on the posture and motion trajectory of each dynamic object.

[0011] Optionally, the process of extracting features based on the instance segmentation image and the depth image includes:

[0012] Selecting a feature detection algorithm based on the environment map, performing sparse feature point detection on the instance segmentation image based on the feature detection algorithm, screening the feature points using an adaptive non-maximum suppression algorithm, and then using an optical flow tracking algorithm to match and track the screened feature points between consecutive frames to generate a static optical flow image, and reversely projecting the 2D feature points on the static optical flow image into 3D feature points to obtain static 3D feature points;

[0013] An instance segmentation mask image is used to distinguish dynamic objects from static backgrounds, unique instance labels are assigned to dynamic objects, dense feature points are detected and screened, and a dynamic optical flow image is generated using an optical flow tracking algorithm. Feature points are then reversely projected to obtain dynamic 3D feature points.

[0014] Optionally, the process of obtaining an initial camera pose based on the static optical flow image and the static 3D feature points includes:

[0015] Construct the correspondence between the 2D static feature points of the current time step and the static 3D feature points of the previous time step, perform a preliminary estimate of the camera pose by minimizing the reprojection error through the PnP algorithm, and verify the estimation result; combine the static optical flow information with the verified estimation result, and obtain the initial camera pose through nonlinear least squares optimization.

[0016] Optionally, the motion residual between consecutive frames for each dynamic object The calculation is as follows:

[0017]

[0018] Where, represents the position coordinates of the 3D feature point at time step τ in the world coordinate system, represents the object motion at time steps τ-1 to τ.

[0019] Optionally, the process of obtaining the initial object motion based on the dynamic optical flow image, the dynamic 3D feature points, and the motion residual includes:

[0020] The correspondence between the 2D dynamic feature points of the current time step and the dynamic 3D feature points of the previous time step is constructed, and a preliminary estimate of the dynamic object motion is made by minimizing the reprojection error through the PnP algorithm, and the estimation result is verified; the dynamic optical flow information is combined with the verified estimation result, and the preliminary estimated dynamic object motion is optimized through nonlinear least squares. The optimized dynamic object motion is secondary optimized to obtain the initial object motion by minimizing the weighted square sum of the motion residuals between consecutive frames on each dynamic object.

[0021] Optionally, the process of constructing a factor graph includes:

[0022] The camera pose, object motion, static 3D feature points and dynamic 3D feature points are used as nodes, and the optical flow information and reprojection error are used as edge constraints to construct a factor graph.

[0023] Optionally, the residual operation includes 3D point measurement residual, camera motion residual, 3D point position residual, and object motion smoothness residual.

[0024] Optionally, an objective function is constructed based on the residual operation result, and the posture and motion trajectory of each dynamic object are obtained based on the objective function; the objective function is as follows:

[0025]

[0026] in, represents the residual of the initial camera pose, represents the camera motion residual, Represents the 3D point measurement residual, represents the 3D point position residual under object motion, Represents the residual error of object motion smoothness; ∑0, ∑ 3D,R , Respectively represent the covariance matrix of the corresponding residual; ρ h represents the Huber function, represents the set of static 3D points at time step τ, represents the set of dynamic 3D points at time step τ, represents the set of dynamic 3D points on the j-th object at time step τ, represents the set of objects observed at time step τ.

[0027] Optionally, it is characterized in that it also includes: the current VR device performs contour rendering according to the posture and motion trajectory of each dynamic object, and after cross-verification of data transmitted by multiple VR devices, the precise posture and motion trajectory of all moving objects in the scene are transmitted back to the current VR device to update the contour of the participant.

[0028] The present invention also provides a multi-person collision detection system for VR large-space immersive tours, comprising:

[0029] A VR device is used to acquire an RGB image and a depth image, and obtain an instance segmentation image based on the RGB image;

[0030] A feature extraction module is used to extract features based on the instance segmentation image and the depth image to obtain a static optical flow image, static 3D feature points, a dynamic optical flow image and dynamic 3D feature points;

[0031] a pose and motion estimation module, configured to obtain an initial camera pose based on the static optical flow image and the static 3D feature points; calculate motion residuals between consecutive frames on each dynamic object based on the dynamic 3D feature points; and obtain initial object motion based on the dynamic optical flow image, the dynamic 3D feature points, and the motion residuals;

[0032] A factor graph construction module, configured to construct a factor graph based on the initial camera pose, the initial object motion, the static 3D feature points, and the dynamic 3D feature points;

[0033] an optimization module, configured to perform a residual operation based on the factor graph, and obtain a posture and motion trajectory of each dynamic object based on a residual operation result, the initial camera posture, and the initial object motion;

[0034] The collision detection module is used to perform multi-person collision detection based on the posture and motion trajectory of each dynamic object.

[0035] Compared with the prior art, the present invention has the following advantages and technical effects:

[0036] (1) This invention directly parameterizes the motion and posture of each object through a novel residual calculation and optimization formula, ensuring that the constraints of rigid body kinematics are maintained. This formulation enables the system to more accurately estimate the motion of objects, especially in complex dynamic environments.

[0037] (2) The present invention uses multiple joint optimization algorithms to jointly optimize camera pose, object motion, and environment map, ensuring the consistency and accuracy of system state. This joint optimization strategy enables the system to better handle complex scenes in dynamic environments.

[0038] (3) The present invention adopts sliding window optimization, dynamic object and static background parallel optimization strategy, which can significantly reduce the amount of calculation and improve the real-time performance of the system; the modular design enables the system to flexibly integrate different modules and optimization algorithms to adapt to different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0040] Figure 1 A schematic diagram of a method according to an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of feature extraction process according to an embodiment of the present invention;

[0042] Figure 3 Schematic diagram of the algorithm optimization process of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Example 1

[0046] like Figure 1-3 As shown, this embodiment provides a multi-person collision detection method for VR large-space immersive tour, including:

[0047] Obtain RGB images and depth images through VR devices, and obtain instance segmentation images based on the RGB images;

[0048] Perform feature extraction based on the instance segmentation image and the depth image to obtain a static optical flow image, static 3D feature points, a dynamic optical flow image, and dynamic 3D feature points;

[0049] Specifically, the process of extracting features based on the instance segmentation image and the depth image includes:

[0050] Selecting a feature detection algorithm based on the environment map, performing sparse feature point detection on the instance segmentation image based on the feature detection algorithm, screening the feature points using an adaptive non-maximum suppression algorithm, and then using an optical flow tracking algorithm to match and track the screened feature points between consecutive frames to generate a static optical flow image, and reversely projecting the 2D feature points on the static optical flow image into 3D feature points to obtain static 3D feature points;

[0051] An instance segmentation mask image is used to distinguish dynamic objects from static backgrounds, unique instance labels are assigned to dynamic objects, dense feature points are detected and screened, and a dynamic optical flow image is generated using an optical flow tracking algorithm. Feature points are then reversely projected to obtain dynamic 3D feature points.

[0052] If the VR device has an RGB-D camera or a stereo camera, use this image to capture the RGB image directly. and depth map If the VR device does not have an RGB-D camera or a stereo camera, it can also be connected to an external device to enable the VR device to capture depth maps. Note that the superscript τ is the time step.

[0053] If the VR device does not have an RGB-D camera or a stereo camera, and it is difficult or impossible to connect an external device, first convert the RGB image Input into a pre-trained "depth detection model" to estimate the depth map Common pre-trained "deep detection models" are:

[0054] Monodepth2, which predicts depth maps from monocular images;

[0055] Depth estimation with a Deep Neural Network, which uses a deep neural network to estimate depth from monocular images;

[0056] RAFT-Depth, a depth estimation method based on the RAFT optical flow algorithm;

[0057] StereoNet, which estimates depth maps from stereo image pairs.

[0058] Convert RGB image Input into a pre-trained "instance segmentation model" and obtain the instance segmentation image through "mask operation" Common pre-trained "instance segmentation models" are:

[0059] Mask R-CNN: A popular instance segmentation network that performs simultaneous object detection and segmentation.

[0060] DeepLab: A network for semantic segmentation that can also be used for instance segmentation.

[0061] YOLO series: A real-time target detection network that can be used for instance segmentation when combined with a segmentation module.

[0062] Panoptic-DeepLab: A network for pan-visual segmentation that can handle both semantic and instance segmentation.

[0063] The superscript τ in represents the time step, i.e., time-step, which can be understood as the time interval of each frame, and each time step has a corresponding object posture.

[0064] The output of the "feature extraction" stage is a "static optical flow image" Dynamic Optical Flow Image Static feature points and dynamic feature points D 2D,τ , where the subscript 2D represents the two-dimensional feature points (i.e., the feature points on the image), and the subscript v represents the time step.

[0065] RGB images Mainly used for feature detection and tracking; depth map Represents the depth information of each pixel, used to convert 2D feature points into 3D points; instance segmentation image The instance label of the object to which each pixel in the image belongs is annotated, where background pixels are marked as 0 and dynamic object pixels are marked as 1...n object , where n object is the number of objects in the image; optical flow image or Represents the motion between pixels and is used to assist feature tracking and camera pose estimation.

[0066] Static Feature Tracking

[0067] “Sparse feature point detection”: At each time step τ, a feature detection algorithm is used to detect Detecting sparse static feature points in

[0068] For indoor environments, the Shi-Tomasi corner detection algorithm is used. The Shi-Tomasi corner detection algorithm is a method for identifying corners in an image. Its primary purpose is to provide a more stable and reliable corner selection criterion. The core of the Shi-Tomasi algorithm lies in examining the two eigenvalues ​​of the autocorrelation matrix (structure tensor) at each pixel location. For each pixel, the corresponding autocorrelation matrix is ​​calculated (usually by taking the sum of the squares of the grayscale changes within a window). The two eigenvalues ​​λ1 and λ2 of this matrix are then found. These two eigenvalues ​​represent the intensity of the grayscale changes around that pixel, with the larger eigenvalue corresponding to the direction of the most significant change, and the smaller eigenvalue corresponding to the less significant direction. The Shi-Tomasi algorithm selects the minimum of the two eigenvalues ​​as the evaluation criterion, namely R = min(λ1,λ2). Only when R exceeds a certain threshold is the corresponding pixel considered a corner.

[0069] For outdoor environments, the ORB feature detection algorithm (Oriented FAST and Rotated BRIEF) is used. ORB uses the FAST (Features from Accelerated Segment Test) algorithm to find feature points. The FAST algorithm determines whether a pixel is a corner by checking whether a circle of pixels surrounding it forms a local extremum (i.e., the number of consecutive segments that are brighter or darker than the center pixel by a certain threshold). This method is computationally efficient, but the original FAST corner points lack directional information, which is detrimental to achieving rotation invariance in feature matching. To compensate for the lack of rotation invariance in FAST corner points, ORB calculates the orientation of each corner point after detecting it. This is achieved by measuring the orientation between the grayscale centroid within the corner point's neighborhood and the corner point's location. The grayscale centroid is the center of gravity of the pixel intensity distribution within the corner point's neighborhood. By comparing the positional relationship between the centroid and the corner point, a direction vector is obtained, which gives the ORB feature a certain degree of rotation invariance. After obtaining directional key points, ORB uses a modified version of the BRIEF (Binary Robust Independent Elementary Features) descriptor to describe these key points. The standard BRIEF descriptor generates a binary string as a descriptor by comparing the brightness of a pair of randomly selected pixels. However, the original BRIEF is not rotation invariant. Therefore, ORB uses a method called rBRIEF, which takes into account the direction of the key points when constructing the descriptor, thereby enhancing the descriptor's robustness to rotation changes. Finally, the feature descriptors generated by ORB are used to match feature points. Since ORB descriptors are binary, Hamming distance can be used for efficient matching operations, which is computationally very efficient and suitable for real-time applications.

[0070] "Sparse feature point screening": To improve tracking robustness, the Adaptive Non-Maximal Suppression (ANMS) algorithm is used to screen the detected feature points, retaining the most informative ones while ensuring that the feature points are evenly distributed in the image. If the number of tracked feature points falls below the set "first threshold" η1 (in the experiment, η1 = 1000), the system will re-detect new feature points to supplement them.

[0071] "First optical flow tracking algorithm": Through the "first optical flow tracking algorithm", such as Lucas-Kanade algorithm, KLTTracking (Kanade-Lucas-Tomasi Tracking) algorithm, etc., feature points are tracked between consecutive frames. Matching and tracking are performed to generate correspondences between feature points, and finally a "static optical flow image" is generated.

[0072] "Reverse Projection": reversely project 2D feature points into 3D feature points.

[0073]

[0074] in, Represents the 2D pixel measurement coordinates on the image plane. It is a two-dimensional vector containing the row and column coordinates of the pixel. d represents the depth information, that is, the distance from the camera to the observation point. Represents a 3D measurement point in local coordinates, which is a 3D vector. represents the camera pose at time step τ, which is a 4×4 homogeneous transformation matrix. Represents a 3D point in the world coordinate system, which is a three-dimensional vector. -1 (·) represents the camera’s inverse projection function, which converts 2D pixel measurements and depth information into 3D points.

[0075] It will Convert to homogeneous coordinate form.

[0076] Obtain 2D image data and depth information from sensors such as RGB-D cameras or stereo cameras, and then use the above formula to recover the position of 3D points in the local sensor coordinate system from the 2D pixel measurements and depth information.

[0077] For example: suppose there is an RGB-D camera that captures an image at time step τ and has a 2D pixel measurement And the corresponding depth information d = 5 meters, the camera intrinsic parameter matrix is ​​τ:

[0078] First, use the inverse projection function to measure the 2D pixels And the depth information d is converted into 3D points, namely:

[0079]

[0080] Where f is the focal length of the camera.

[0081] Then, the 3D point Convert to homogeneous coordinate form:

[0082] Finally, use the camera pose The 3D point Convert to local sensor coordinates:

[0083]

[0084] Among them, R τ is the rotation matrix, t τ is the translation vector, and is a 3D point The coordinates in the world coordinate system.

[0085] Dynamic Feature Tracking

[0086] “Dense feature point detection”: For dynamic objects, the system needs to track the dense feature points D on each object. 2t,D . Using instance segmentation mask images To distinguish dynamic objects from static backgrounds and assign a unique instance label to each dynamic object. If the instance segmentation network does not provide temporally consistent instance labels, the system uses a tracking algorithm (such as ByteTrack) to ensure that the instance labels remain consistent between consecutive frames.

[0087] "Dense Feature Point Filtering": Similar to "Sparse Feature Point Filtering," to ensure robust tracking, the ANMS algorithm is used to filter detected feature points to retain the most informative ones and distribute them evenly across each dynamic object. If the number of feature points on a dynamic object is less than the set "second threshold" η2 (in the experiment, η2 = 800), the system will re-detect new feature points to supplement them.

[0088] "Second optical flow tracking algorithm": For each dynamic object, use the "second optical flow tracking algorithm", such as the Farneback algorithm, RAFT, PWC-Net, etc., to find the correspondence between feature points in consecutive frames and finally generate a "dynamic optical flow image".

[0089] The principle of “dynamic feature point reverse projection” is the same as the “feature point reverse projection” described above.

[0090] Obtaining an initial camera pose based on the static optical flow image and the static 3D feature points; calculating motion residuals between consecutive frames on each dynamic object based on the dynamic 3D feature points; and obtaining an initial object motion based on the dynamic optical flow image, the dynamic 3D feature points, and the motion residuals;

[0091] Furthermore, the process of obtaining an initial camera pose based on the static optical flow image and the static 3D feature points includes:

[0092] Construct the correspondence between the 2D static feature points of the current time step and the static 3D feature points of the previous time step, perform a preliminary estimate of the camera pose by minimizing the reprojection error through the PnP algorithm, and verify the estimation result; combine the static optical flow information with the verified estimation result, and obtain the initial camera pose through nonlinear least squares optimization.

[0093] Furthermore, the motion residual between consecutive frames on each dynamic object The calculation is as follows:

[0094]

[0095] Where, represents the position coordinates of the 3D feature point at time step τ in the world coordinate system, represents the object motion at time steps τ-1 to τ.

[0096] Furthermore, the process of obtaining the initial object motion based on the dynamic optical flow image, the dynamic 3D feature points and the motion residual includes:

[0097] The correspondence between the 2D dynamic feature points of the current time step and the dynamic 3D feature points of the previous time step is constructed, and a preliminary estimate of the dynamic object motion is made by minimizing the reprojection error through the PnP algorithm, and the estimation result is verified; the dynamic optical flow information is combined with the verified estimation result, and the preliminary estimated dynamic object motion is optimized through nonlinear least squares. The optimized dynamic object motion is secondary optimized to obtain the initial object motion by minimizing the weighted square sum of the motion residuals between consecutive frames on each dynamic object.

[0098] Input and output: The output parameters of the "feature extraction" stage are used as the input parameters of this stage, including static optical flow images Dynamic optical flow image Static 3D feature points Dynamic 3D feature point D 3D,τ ; The output of this stage is the camera trajectory Object pose and sports 3D point position Right now

[0099]

[0100] Where D represents the time step, Represents the set of all time steps, and the superscript W represents the world coordinate system. Represents the camera pose in the world coordinate system at a time step of τ. Represents the motion set of all dynamic objects from time step τ-1 to τ in the world coordinate system. It represents the set of poses of all objects at all time steps when the time step is τ in the world coordinate system. Represents the set of all points at time step τ in the world coordinate system, including dynamic points and static points.

[0101] "Camera pose estimation" + "Optical flow joint optimization": responsible for estimating the camera pose at each time step τ

[0102] Establish the correspondence between 2D points and 3D points: use the 2D feature points in the current time step τ and the 3D feature points in the previous time step τ-1 Establish a corresponding relationship between the two.

[0103] PnP algorithm: Use the Perspective-n-Point (PnP) algorithm to estimate the camera pose The PnP algorithm solves the camera pose by minimizing the reprojection error:

[0104]

[0105] in, represents the 2D pixel measurement (i.e., the pixel coordinates of the feature point) at time step τ, which is a two-dimensional vector. The superscript i represents the i-th feature point. Π(·) represents the camera projection function, which projects the 3D point onto the 2D image plane. represents the camera pose at time step τ. Represents a 3D point in the world coordinate system The position coordinates of . represents the 2D pixel measurement at time step τ and the 3D points by the camera pose The difference between the positions projected onto the 2D image plane.

[0106] RANSAC Verification: Use the RANSAC (Random Sample Consensus) algorithm to verify the PnP algorithm's estimation results. RANSAC estimates by randomly selecting a subset of data points and verifies that these estimates are consistent with the majority of data points. This step effectively removes outliers and improves the robustness of the estimation.

[0107] Optical flow joint optimization: Combine optical flow information with the estimation results of the PnP algorithm to further improve the camera pose and dynamic object motion estimation through nonlinear least squares optimization. Specifically, the following errors are optimized:

[0108]

[0109] Among them, f τ-1,τ Represents the optical flow vector of the feature point between two frames, represents the pixel coordinates of the 2D feature points at time step τ. Π(·) represents the camera projection function. Indicates the camera pose The inverse matrix of . Indicates the position coordinates of the i-th 3D point in the world coordinate system. After the PnP algorithm, RANSAC verification and optical flow joint optimization, the optical flow optimization problem {θ,f τ-1,τ}, in order to get the initial camera pose of the current time step τ

[0110] "Object Motion Estimation" + "Optical Flow Optimization": responsible for estimating the motion of dynamic objects at each time step τ

[0111] Establish the correspondence between 2D points and 3D points: use the 2D feature points in the current time step τ and the 3D feature points in the previous time step τ-1 Establish a corresponding relationship between the two.

[0112] PnP algorithm: Use the Perspective-n-Point (PnP) algorithm to estimate the motion of dynamic objects The PnP algorithm solves the object's motion by minimizing the reprojection error:

[0113]

[0114] in, It represents the 2D pixel measurement at time step τ (i.e., the pixel coordinates of the 2D feature point), which is a two-dimensional vector. The superscript i represents the i-th feature point. represents the object motion from time step τ-1 to τ in the world coordinate system. Π(·) represents the camera projection function, which projects the 3D point onto the 2D image plane. represents the camera pose at time step τ. express The inverse matrix of . represents a 3D point in the world coordinate system at time step τ-1 The position coordinates of . represents the 2D pixel measurement at time step τ The difference between the 2D projected position calculated from the camera pose, object motion, and 3D point position. The PnP algorithm solves the object motion by minimizing the reprojection error.

[0115] RANSAC Verification: Similarly, the RANSAC (Random Sample Consensus) algorithm is used to verify the PnP algorithm's estimation results. RANSAC estimates by randomly selecting a subset of data points and verifies that these estimates are consistent with the majority of data points. This step can effectively eliminate outliers and improve the robustness of the estimation.

[0116] Optical flow optimization: Combine optical flow to optimize camera posture and dynamic object motion

[0117] Residual calculation: combining optical flow information with the estimation results of the PnP algorithm

[0118]

[0119] Among them, f τ-1,τ Represents the optical flow vector of the feature point between two frames, represents the pixel coordinates of the 2D feature points at time step τ. Π(·) represents the camera projection function. Indicates the camera pose The inverse matrix of . represents a 3D point in the world coordinate system at time step τ-1 After the PnP algorithm, RANSAC verification and optical flow joint optimization, the optical flow optimization problem {θ,f τ-1,τ}, the initial object motion at the current time step τ can be obtained

[0120] Optimization problem: Further improve the camera pose and dynamic object motion estimation through nonlinear least squares optimization. That is, by the following {θ,f τ-1,τ}Formula, optimization and Thus we get and

[0121]

[0122] Where θ represents the system state, including camera pose and object motion. τ-1,τ Represents the optical flow from time step τ-1 to τ. ρ h Represents the Huber function, which is used for robust estimation. represents the optical flow residual, i.e., the difference between the optical flow measurement and the optical flow calculated from the system state θ. Represents the prior residual of optical flow, that is, the difference between the optical flow measurement and the prior optical flow. f represents the covariance matrix of the optical flow residual, ∑0 represents the covariance matrix of the optical flow prior residual, and N represents the number of optical flow measurements.

[0123] {θ,f τ-1,τ} defines an optimization problem, the goal is to find the system state θ and optical flow f τ-1,τ , so that the weighted sum of squares of all optical flow residuals is minimized. Optical flow residual Robust estimation is performed through the Huber function to reduce the influence of outliers. Optical flow prior residual Used to compare optical flow measurements with prior optical flow to improve the accuracy of the estimation.

[0124] "Motion Optimization": responsible for further optimizing the motion estimation of dynamic objects

[0125] Calculate 3D motion residuals: Calculate the motion residuals of 3D points on dynamic objects between consecutive frames, that is, for each 3D point on a dynamic object Calculate its motion residual between time steps τ-1 and τ:

[0126]

[0127] in, Represents the position coordinates of the 3D point at time step τ in the world coordinate system. Represents the object motion from time step τ-1 to τ, which is a 4×4 homogeneous transformation matrix. It is used to construct an optimization problem to estimate the position of 3D points by minimizing the sum of squares of all residuals, and to measure the position consistency of 3D points under object motion.

[0128] Optimization problem: Define an optimization problem whose goal is to minimize the weighted sum of squares of all 3D motion residuals and estimate the object motion by achieving this goal The objective function is:

[0129]

[0130] in, and denotes the set of dynamic 3D points on the j-th object at time steps τ-1 and τ, respectively. Represents the residual of the 3D point position under object motion. Represents the covariance matrix of 3D point motion residuals.

[0131] Use a nonlinear least squares optimization algorithm (such as the Levenberg-Marquardt algorithm) to solve the above optimization problem The Levenberg-Marquardt algorithm is an efficient nonlinear least squares optimization algorithm that can avoid falling into local minima while ensuring convergence speed.

[0132] constructing a factor graph based on the initial camera pose, the initial object motion, static 3D feature points, and dynamic 3D feature points; performing a residual operation based on the factor graph, and obtaining the pose and motion trajectory of each dynamic object based on the residual operation result, the initial camera pose, and the initial object motion;

[0133] Furthermore, the process of constructing the factor graph includes:

[0134] The camera pose, object motion, static 3D feature points and dynamic 3D feature points are used as nodes, and the optical flow information and reprojection error are used as edge constraints to construct a factor graph.

[0135] Furthermore, the residual operation includes 3D point measurement residual, camera motion residual, 3D point position residual, and object motion smoothness residual.

[0136] Furthermore, an objective function is constructed based on the residual operation result, and the posture and motion trajectory of each dynamic object are obtained based on the objective function; the objective function is as follows:

[0137]

[0138] in, represents the residual of the initial camera pose, represents the camera motion residual, Represents the 3D point measurement residual, represents the 3D point position residual under object motion, Represents the residual error of object motion smoothness; ∑0, ∑ 3D,R , Respectively represent the covariance matrix of the corresponding residual; ρ h represents the Huber function, represents the set of static 3D points at time step τ, represents the set of dynamic 3D points at time step τ, represents the set of dynamic 3D points on the j-th object at time step τ, represents the set of objects observed at time step τ.

[0139] Furthermore, it is characterized in that it also includes: the current VR device performs contour rendering according to the posture and motion trajectory of each dynamic object, and after cross-verification of data transmitted by multiple VR devices, the precise posture and motion trajectory of all moving objects in the scene are transmitted back to the current VR device to update the contour of the participant.

[0140] Multi-person collision detection is performed based on the posture and motion trajectory of each dynamic object.

[0141] A Factor Graph is a graphical representation method used to describe the relationship between system state variables and measurement data.

[0142] Factor graphs can provide a structured and efficient way to represent and solve complex optimization problems, which can achieve structured representation, efficient solution, modular design, robustness, flexibility and real-time performance.

[0143] In this system, the system needs to simultaneously estimate the camera pose, object motion, and environment map. The relationships between these variables are complex and interdependent. Factor graphs make the optimization problem clearer and easier to solve by representing the relationships between these variables and measurement data as nodes and edges in a graph.

[0144] In this system, the factor graph consists of nodes and edges: nodes represent system state variables Like camera posture Object motion and position of 3D points

[0145] Residual calculation and joint optimization algorithm: The main purpose is to globally optimize the initial estimation and measurement data to generate consistent camera trajectories, object motions, and environment maps, and optimize the system state by minimizing the weighted sum of squares of all residuals to ensure the accuracy and robustness of the estimation results.

[0146] 3D point measurement residuals Used to measure the difference between the 3D measurement point and the 3D point position calculated by the camera pose.

[0147]

[0148] in, Represents the 3D measurement point at time step τ, which is a three-dimensional vector. The inverse matrix of the camera pose at time step τ is a 4×4 inverse homogeneous transformation matrix. Represents a 3D point at time step τ, which is a three-dimensional vector. For a static point, when the point is first observed, Initialized to For dynamic points, the same initialization method is used for the dynamic points at time step τ in each frame.

[0149] Camera Motion Residual It is used to measure the difference between the camera motion from time step τ-1 to τ and the camera motion calculated by the camera pose.

[0150]

[0151] in, represents the camera motion from time step τ-1 to τ, which is a 4×4 homogeneous transformation matrix that can be estimated by the inertial measurement unit (IMU). yes The inverse matrix of . represents the camera pose at time step τ, which is a 4×4 homogeneous transformation matrix, yes The inverse matrix of . log(·) represents the matrix logarithm, which is used to convert the transformation matrix into Lie algebra form, [·] V Indicates converting a vector in algebraic form into a normal vector form, log([·] V ) is used to convert the matrix form of SE(·) into The vector form of the residual It is used to formulate an optimization problem to estimate camera pose and camera motion by minimizing the sum of squares of all residuals.

[0152] "3D point position residual" Used to measure the 3D point at time step τ The position of the object is the 3D point calculated by the object motion The difference between the positions.

[0153]

[0154] in, represents the homogeneous space coordinates of the 3D point at time step τ in the world coordinate system. Represents the object motion from step τ-1 to τ, which is a 4×4 homogeneous transformation matrix. It is used to construct an optimization problem to estimate the position of 3D points by minimizing the sum of squares of all residuals, and is used to measure the position consistency of 3D points under object motion.

[0155] Object Motion Smoothness Residual It is used to measure the difference between the object motion from time step τ-2 to τ-1 and the object motion from time step τ-1 to τ.

[0156]

[0157] in, Represents the object motion from time step τ-2 to τ-1, which is a 4×4 homogeneous transformation matrix. represents the object motion from time step τ-1 to τ, and is also a 4×4 homogeneous transformation matrix. log(·) represents the matrix logarithm, which is used to convert the transformation matrix into Lie algebra form, [·] VIndicates converting a vector in algebraic form into a normal vector form, log([·] V ) is used to convert the matrix form of SE(·) into in vector form. It is used to formulate an optimization problem to estimate object motion by minimizing the sum of squares of all residuals. Its main function is to enforce constant motion in the world coordinate system, preventing sudden, drastic and unrealistic changes in object motion between consecutive frames.

[0158] “Joint Optimization Algorithm”: Constructing the objective function θ MAP , by minimizing the weighted sum of squares of all residuals, a globally consistent camera trajectory is obtained Optimized object pose Optimized object motion Optimized positions of static and dynamic 3D points (i.e., environment map).

[0159]

[0160] in, represents the residual of the initial camera pose, represents the camera motion residual, represents the residual error of 3D point measurement, Represents the residual of the 3D point position under object motion, Represents the smoothness residual of the object motion. ∑0, ∑ 3D,R , Respectively represent the covariance matrix of the corresponding residual. h Represents the Huber function, which is used for robust estimation. represents the set of static 3D points at time step τ, represents the set of dynamic 3D points at time step τ, represents the set of dynamic 3D points on the j-th object at time step τ. represents the set of objects observed at time step τ.

[0161] Sliding window optimization method: Optimize only within the most recent few time steps to improve efficiency. The window size and overlap can be adjusted according to the specific application to balance computational efficiency and optimization accuracy.

[0162] The core idea of ​​the sliding window optimization method is to limit the optimization process to a fixed-size time window, rather than performing global optimization on the entire data sequence. In this way, the system can significantly reduce the amount of computation while maintaining high accuracy, thereby improving real-time performance.

[0163] In sliding window optimization, the time window typically encompasses the most recent few time steps, such as 10 to 20 frames. Data within the window is used to formulate the optimization problem, while data outside the window is ignored or used only to provide prior information. The key to this approach lies in choosing the window size and update strategy. The window size is typically determined based on the specific application requirements and computational resources. A larger window can provide a more global optimization effect, but the computational effort also increases accordingly; a smaller window, on the other hand, prioritizes real-time performance but may sacrifice some global consistency.

[0164] To further improve efficiency, sliding window optimization can be combined with incremental optimization. When new measurement data arrives, the system adds the new data to the current window, removes the oldest data, and then performs incremental optimization on the data within the window. This incremental update method avoids re-optimizing the entire window data, saving significant computation time.

[0165] This embodiment also provides a multi-person collision detection system for VR large-space immersive tours, including:

[0166] The "client" consists of two stages: the "feature extraction" stage and the "algorithm optimization" stage. The former generates feature points and optical flow images, and the latter uses various optimization algorithms to obtain the final object posture and motion trajectory.

[0167] The "server" is used to store static scene data and update dynamic objects in real time through information such as "camera trajectory", "object posture" and "object motion" transmitted back by each VR device. The updated dynamic object posture and motion trajectory are distributed to different VR devices. Finally, the VR devices render the outlines of these dynamic objects, thereby reminding current participants to avoid collisions with other participants.

[0168] The "server" is connected to each VR device at high speed via LAN Wifi.

[0169] 3D Point Positions are used to build the "environment map," or static scene. This data is typically only used when the scene is first built. Once the server build is complete, 3D Point Positions are no longer transmitted to the server in real time. The server only rebuilds the static scene if it changes, such as due to renovations or repositioning of props. Otherwise, this data is not transmitted.

[0170] Information such as camera trajectory, object pose, and object motion must be transmitted to the server in real time for each frame. To minimize latency, the current participant's VR device renders the outlines of other participants within its field of view based on the current camera trajectory, object pose, object motion, and 3D point position. The server then cross-validates the data transmitted by multiple VR devices and transmits the precise pose and motion trajectory of all moving objects in the scene back to the current participant's VR device, updating the participant's outline.

[0171] The "server" will also send information about participants inside and outside the group to each VR device, allowing each VR device to identify participants inside and outside the group, and use different outline colors to draw different groups of participants, so as to more effectively remind the current scene participants of their situation.

[0172] A VR device is used to acquire an RGB image and a depth image, and obtain an instance segmentation image based on the RGB image;

[0173] A feature extraction module is used to extract features based on the instance segmentation image and the depth image to obtain a static optical flow image, static 3D feature points, a dynamic optical flow image and dynamic 3D feature points;

[0174] a pose and motion estimation module, configured to obtain an initial camera pose based on the static optical flow image and the static 3D feature points; calculate motion residuals between consecutive frames on each dynamic object based on the dynamic 3D feature points; and obtain initial object motion based on the dynamic optical flow image, the dynamic 3D feature points, and the motion residuals;

[0175] A factor graph construction module, configured to construct a factor graph based on the initial camera pose, the initial object motion, the static 3D feature points, and the dynamic 3D feature points;

[0176] an optimization module, configured to perform a residual operation based on the factor graph, and obtain a posture and motion trajectory of each dynamic object based on a residual operation result, the initial camera posture, and the initial object motion;

[0177] The collision detection module is used to perform multi-person collision detection based on the posture and motion trajectory of each dynamic object.

[0178] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multi-person collision detection method for VR large-space immersive tour, characterized in that: The following steps are involved: Obtain RGB images and depth images through VR devices, and obtain instance segmentation images based on the RGB images; Perform feature extraction based on the instance segmentation image and the depth image to obtain a static optical flow image, static 3D feature points, a dynamic optical flow image, and dynamic 3D feature points; Obtaining an initial camera pose based on the static optical flow image and the static 3D feature points; calculating motion residuals between consecutive frames on each dynamic object based on the dynamic 3D feature points; and obtaining an initial object motion based on the dynamic optical flow image, the dynamic 3D feature points, and the motion residuals; constructing a factor graph based on the initial camera pose, the initial object motion, static 3D feature points, and dynamic 3D feature points; performing a residual operation based on the factor graph, and obtaining the pose and motion trajectory of each dynamic object based on the residual operation result, the initial camera pose, and the initial object motion; Multi-person collision detection based on the posture and motion trajectory of each dynamic object; The process of constructing a factor graph includes: The camera pose, object motion, static 3D feature points, and dynamic 3D feature points are used as nodes, and the optical flow information and reprojection error are used as edge constraints to construct a factor graph. An objective function is constructed based on the residual operation result, and the posture and motion trajectory of each dynamic object are obtained based on the objective function; the objective function is as follows: ； in, represents the residual of the initial camera pose, represents the camera motion residual, Represents the 3D point measurement residual, represents the 3D point position residual under object motion, Represents the object motion smoothness residual; Respectively represent the covariance matrix of the corresponding residuals; represents the Huber function, Indicates that at time step A static 3D point set, Indicates that at time step A dynamic 3D point set, Indicates that at time step The dynamic 3D point set on the j-th object, Indicates that at time step The collection of observed objects.

2. The multi-person collision detection method for VR large-space immersive tour according to claim 1 is characterized in that: The process of extracting features based on the instance segmentation image and the depth image includes: Selecting a feature detection algorithm based on the environment map, performing sparse feature point detection on the instance segmentation image based on the feature detection algorithm, screening the feature points using an adaptive non-maximum suppression algorithm, and then using an optical flow tracking algorithm to match and track the screened feature points between consecutive frames to generate a static optical flow image, and reversely projecting the 2D feature points on the static optical flow image into 3D feature points to obtain static 3D feature points; An instance segmentation mask image is used to distinguish dynamic objects from static backgrounds, unique instance labels are assigned to dynamic objects, dense feature points are detected and screened, and a dynamic optical flow image is generated using an optical flow tracking algorithm. Feature points are then reversely projected to obtain dynamic 3D feature points.

3. The multi-person collision detection method for VR large-space immersive tour according to claim 1 is characterized in that: The process of obtaining the initial camera pose based on the static optical flow image and the static 3D feature points includes: Construct the correspondence between the 2D static feature points of the current time step and the static 3D feature points of the previous time step, perform a preliminary estimate of the camera pose by minimizing the reprojection error through the PnP algorithm, and verify the estimation result; combine the static optical flow information with the verified estimation result, and obtain the initial camera pose through nonlinear least squares optimization.

4. The multi-person collision detection method for VR large-space immersive tour according to claim 1 is characterized in that: Motion residuals between consecutive frames for each dynamic object The calculation is as follows: ； Where, Indicates the time step in the world coordinate system The position coordinates of the 3D feature points, Indicates that at time step -1 to object movement.

5. The multi-person collision detection method for VR large-space immersive tour according to claim 1 is characterized in that: The process of obtaining the initial object motion based on the dynamic optical flow image, the dynamic 3D feature points and the motion residual includes: The correspondence between the 2D dynamic feature points of the current time step and the dynamic 3D feature points of the previous time step is constructed, and a preliminary estimate of the dynamic object motion is made by minimizing the reprojection error through the PnP algorithm, and the estimation result is verified; the dynamic optical flow information is combined with the verified estimation result, and the preliminary estimated dynamic object motion is optimized through nonlinear least squares. The optimized dynamic object motion is secondary optimized to obtain the initial object motion by minimizing the weighted square sum of the motion residuals between consecutive frames on each dynamic object.

6. The multi-person collision detection method for VR large-space immersive tour according to claim 1 is characterized in that: The residual operation includes 3D point measurement residual, camera motion residual, 3D point position residual, and object motion smoothness residual.

7. The multi-person collision detection method for VR large-space immersive tour according to claim 1 is characterized in that: Also includes: The current VR device renders the outline based on the posture and motion trajectory of each dynamic object. After cross-verification of data transmitted by multiple VR devices, the precise posture and motion trajectory of all moving objects in the scene are transmitted back to the current VR device to update the outline of the participant.

8. A multi-person collision detection system for VR large-space immersive tour, used to implement the multi-person collision detection method for VR large-space immersive tour according to any one of claims 1 to 7, characterized in that: include: A VR device is used to acquire an RGB image and a depth image, and obtain an instance segmentation image based on the RGB image; A feature extraction module is used to extract features based on the instance segmentation image and the depth image to obtain a static optical flow image, static 3D feature points, a dynamic optical flow image and dynamic 3D feature points; a pose and motion estimation module, configured to obtain an initial camera pose based on the static optical flow image and the static 3D feature points; calculate motion residuals between consecutive frames on each dynamic object based on the dynamic 3D feature points; and obtain initial object motion based on the dynamic optical flow image, the dynamic 3D feature points, and the motion residuals; A factor graph construction module, configured to construct a factor graph based on the initial camera pose, the initial object motion, the static 3D feature points, and the dynamic 3D feature points; an optimization module, configured to perform a residual operation based on the factor graph, and obtain a posture and motion trajectory of each dynamic object based on a residual operation result, the initial camera posture, and the initial object motion; The collision detection module is used to perform multi-person collision detection based on the posture and motion trajectory of each dynamic object.

Images