A three-dimensional space trajectory recognition method and system based on multi-view stereo reconstruction

Abstract

The application provides a three-dimensional space trajectory recognition method based on multi-view stereoscopic reconstruction, which comprises the following steps: using multiple cameras to shoot the same scene from different angles, determining the picture intersection part of the multiple cameras as the research scene; performing frame extraction and pedestrian detection on the video of each camera, recognizing the pedestrians and their position information in each frame of video, and assigning a uniform identity number to the same pedestrian; registering the video frame images of different cameras to obtain the pixel position of each pedestrian in each camera; solving the three-dimensional point coordinates of each pedestrian in the camera coordinate system by a multi-view triangulation method combined with the internal and external parameters of the camera; converting the three-dimensional point coordinates into the coordinates in the world coordinate system to obtain the world coordinate system coordinates of each numbered pedestrian point position in each frame; connecting the lines of each numbered pedestrian point position in each frame to form a trajectory according to the time information, and completing the reconstruction of the three-dimensional space trajectory of the pedestrian in the world coordinate system. The application can reduce the labor and calculation cost.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of pedestrian travel characteristics research, and specifically relates to a three-dimensional space trajectory recognition method and system based on multi-view stereo reconstruction. BACKGROUND

[0002] With the rapid development of intelligent monitoring, security and urban management, the research and analysis of pedestrian travel behavior have gradually become an important application demand. Traditional pedestrian trajectory recognition technology usually detects and tracks targets based on two-dimensional images from a single camera, but these methods have some limitations. The main problems include the lack of depth information, the difficulty in handling occlusions and changes in viewing angles, and other factors. In order to overcome these problems, in recent years, three-dimensional space trajectory recognition methods based on multi-view stereo reconstruction technology have gradually become a research hotspot.

[0003] Currently, many pedestrian trajectory recognition systems rely on single camera planar image data, and use target detection and tracking algorithms (such as YOLO and SORT) to extract pedestrian trajectories. However, this method can only provide two-dimensional trajectories, lacks accurate capture of object depth information, and is difficult to accurately reflect the position and motion trajectory of objects in actual three-dimensional space, especially in complex environments and high-density scenes, two-dimensional trajectories may be affected by occlusions and changes in viewing angles, resulting in tracking errors and trajectory interruptions.

[0004] In order to solve these problems, in recent years, stereo reconstruction methods based on multi-camera perspectives have received extensive attention. By using multiple cameras to capture the same scene from different angles, rich parallax data can be obtained, thereby achieving more accurate three-dimensional space reconstruction. In particular, through the internal and external parameters of the camera and multi-perspective triangulation technology, the precise position of the object in three-dimensional space can be recovered, and a more realistic three-dimensional trajectory can be constructed.

[0005] However, existing multi-view trajectory recognition methods still have some technical challenges, such as how to effectively handle image registration between different perspectives, the accuracy of feature matching, and how to convert two-dimensional feature points into three-dimensional coordinates. With the development of computer vision technology and deep learning algorithms, stereo reconstruction and three-dimensional trajectory recognition methods based on multi-perspective are gradually overcoming these technical difficulties and showing broad application prospects in multiple fields. SUMMARY

[0006] The purpose of the present application is to provide a three-dimensional space trajectory recognition method based on multi-view stereo reconstruction, which realizes accurate recognition and reconstruction of pedestrian three-dimensional trajectories through the perspective information of multiple cameras. This method can reduce the cost of manpower and computation, and at the same time avoids the dependence on objects with known dimensions or known three-dimensional structures, and has strong universality and adaptability.

[0007] The technical solutions of the present application are as follows:

[0008] A three-dimensional space trajectory recognition method based on multi-view stereoscopic reconstruction, the method comprising the following steps:

[0009] Using multiple cameras to shoot the same scene from different angles, determining the intersection part of the multiple cameras as the research scene;

[0010] Frame extraction and pedestrian detection are performed on the video of each camera, the pedestrians and their position information in each frame of video are recognized, and a uniform identity number is assigned to the same pedestrian;

[0011] The video frame images of different cameras are registered to obtain the pixel position of each pedestrian in each camera, and the three-dimensional point coordinates of each pedestrian in the camera coordinate system are solved by a multi-view triangulation method combined with the internal and external parameters of the camera;

[0012] The three-dimensional point coordinates are converted into coordinates in the world coordinate system to obtain the world coordinate system coordinates of each numbered pedestrian point position in each frame;

[0013] According to the time information, the lines connecting each numbered pedestrian point position in each frame form a trajectory, and the three-dimensional space trajectory reconstruction of the pedestrian in the world coordinate system is completed.

[0014] Further, the frame extraction and pedestrian detection on the video of each camera, the recognition of the pedestrians and their position information in each frame of video, and the assignment of a uniform identity number to the same pedestrian are specifically:

[0015] First, the YOLOv8 model is used to detect and instance segment the pedestrians in each frame of video to obtain the bounding box and segmentation mask of each pedestrian, and then the detected bounding box is input into the BoT-SORT model for pedestrian tracking. The BoT-SORT model assigns a uniform identity number to each pedestrian in the entire video sequence according to the position and motion trajectory of each pedestrian.

[0016] Further, the assignment of a uniform identity number to the same pedestrian further includes matching the features of pedestrians in different cameras.

[0017] Further, the registration of the video frame images of different cameras to obtain the pixel position of each pedestrian in each camera, and the solving of the three-dimensional point coordinates by a multi-view triangulation method combined with the internal and external parameters of the camera are specifically:

[0018] First, the two cameras are calibrated by a camera calibration software and a calibration board to obtain the internal and external parameters of the cameras, thereby determining the position and attitude of the cameras in the world coordinate system;

[0019] Then, feature points are extracted in the human body bounding box in the two view images by a feature extraction algorithm, and the same feature points in different view images are matched by using a feature matching algorithm.

[0020] Then, according to the position of the feature point in each camera image and the camera internal and external parameters, a space ray passing through the feature point is constructed; the intersection of the space ray in the three-dimensional space is the three-dimensional point coordinates of the object.

[0021] Further, the internal parameters of the camera include focal length, principal point coordinates, pixel scale factor and distortion coefficient.

[0022] Further, the camera external parameters include a rotation matrix and a translation vector.

[0023] Further, the three-dimensional point is converted into coordinates in the world coordinate system, and the world coordinate system coordinates of each numbered pedestrian point in each frame are obtained.

[0024] The rotation matrix and the translation vector of the camera are used to transform the three-dimensional point coordinates in the camera coordinate system, so as to obtain the world coordinate system coordinates of each numbered pedestrian point in each frame.

[0025] The application also provides a three-dimensional space trajectory recognition system based on multi-view stereoscopic reconstruction.

[0026] The multi-camera acquisition module is used for shooting the same scene from different angles by multiple cameras and determining the picture intersection part of the multiple cameras as the research scene.

[0027] The video processing and pedestrian detection module is used for frame extraction of the video of each camera, pedestrian detection algorithm is adopted to identify the pedestrian and its position information in each frame of video, and a uniform identity number is allocated to each pedestrian.

[0028] The video registration and feature point extraction module is used for registering the video frame images of different cameras and extracting the pixel position of each pedestrian in each camera.

[0029] The three-dimensional reconstruction and depth calculation module is used for calculating the three-dimensional point coordinates of each pedestrian in the camera coordinate system based on the multi-view triangulation method and the internal and external parameters of the camera.

[0030] The coordinate conversion module is used for converting the three-dimensional point coordinates from the camera coordinate system to the coordinates in the world coordinate system, so as to obtain the world coordinate system coordinates of each pedestrian in each frame.

[0031] The trajectory reconstruction module is used for connecting the numbered pedestrian points in each frame according to the time information, forming the three-dimensional space trajectory of the pedestrian, and completing the reconstruction of the three-dimensional space trajectory of the pedestrian in the world coordinate system.

[0032] Compared with the prior art, the present application has the following advantages:

[0033] Compared with a single camera view, the present application can cover a larger area, reduce occlusion, provide more parallax data, and improve the accuracy of object detection and depth estimation in complex scenes.

[0034] The present application can identify pedestrian travel behavior and reconstruct trajectories in a world coordinate system, making the trajectories have real spatial information. Subsequent modeling with a three-dimensional model can intuitively analyze pedestrian travel trajectory characteristics and influencing factors. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings generally illustrate various embodiments of the present application and, together with the description, serve to explain the principles of the present application. Wherever possible, the same reference numbers are used in all the drawings to refer to the same or similar parts. Such embodiments are illustrative rather than restrictive and are not intended to exhaustively enumerate all possible implementations of the present device or method.

[0036] Fig. 1 A flowchart of the present application is shown;

[0037] Fig. 2 A three-dimensional reconstruction diagram of the present application is shown. DETAILED DESCRIPTION

[0038] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0039] As shown in Figs. 1-2 The present application provides a three-dimensional space trajectory recognition method based on multi-view stereo reconstruction, which comprises the following steps:

[0040] Multiple cameras are used to shoot the same scene from different angles, and the intersection part of the multiple cameras is determined as the research scene;

[0041] Frame extraction and pedestrian detection are performed on the video of each camera, the pedestrians and their position information in each frame of video are identified, and a uniform identity number is assigned to the same pedestrian;

[0042] The video frame images of different cameras are registered to obtain the pixel position of each pedestrian in each camera, and the three-dimensional point coordinates of each pedestrian in the camera coordinate system are solved by a multi-view triangulation method combined with the internal and external parameters of the camera;

[0043] The three-dimensional point coordinates are converted into coordinates in the world coordinate system to obtain the world coordinate system coordinates of each numbered pedestrian point in each frame;

[0044] According to the time information, the track is formed by connecting each numbered pedestrian point position of each frame, and the three-dimensional space track reconstruction of the pedestrian in the world coordinate system is completed.

[0045] The technology adopted by the pedestrian detection is: adopting a preset tracking model to perform multi-target detection and instance segmentation on the video collected by the camera, and obtaining the identity number, position information, bounding box and segmentation mask of each pedestrian in each frame of picture through frame-by-frame identification and tracking;

[0046] The tracking model is a YOLOv8+BoT-SORT model, which first performs target detection and instance segmentation on the human body in the video frame by the YOLOv8 model, obtains the bounding box and segmentation mask of each human body, and then sends the human body bounding box into the BoT-SORT type, the BoT-SORT model tracks the detected human body, and assigns a consistent human body identity number to the same human body, thereby ensuring continuous and stable tracking of the target in the entire video sequence and accurately depicting the motion trajectory thereof.

[0047] The technology for unifying the identity numbers of pedestrians in the two cameras is: multi-camera pedestrian re-identification; extracting pedestrian visual features through a deep learning model; using metric learning technology to train the network to pull the feature vectors of the same object closer and distinguish different object feature vectors;

[0048] The process of calculating the world coordinate system of the human body point position is:

[0049] The camera calibration software and the calibration board are used to calibrate the two cameras respectively to obtain the camera intrinsic parameters and extrinsic parameters; wherein the camera intrinsic parameters include focal length, principal point coordinates, pixel scale factor and distortion coefficient; the camera extrinsic parameters include rotation matrix and translation vector. The coordinates of the two cameras in the world coordinate system are determined.

[0050] The feature points of the target object are found in the two view images, and matching is performed in different images; specifically including: using a feature extraction algorithm to extract obvious features such as corner points and edges within the human body bounding box; the same feature matching algorithm is used to correspond the same feature points in the multi-view images;

[0051] The fundamental matrix and the essential matrix of the two cameras are calculated through the feature points in the two view images; the feature points in the image are restored to the three-dimensional space through multi-view triangulation to obtain the three-dimensional point coordinates of each human body; specifically including: according to the position of the feature points in the image of each camera and the camera intrinsic and extrinsic parameters, a space ray passing through the feature points is constructed; these rays should intersect at a point in the three-dimensional space, and the point is the three-dimensional coordinates of the object.

[0052] The rotation and translation matrix of the camera is used to convert the three-dimensional point coordinates to a unified world coordinate system to obtain the world coordinate system coordinates of each human body in the video frame.

[0053] The above merely illustrates the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical solution and the inventive concept of the present application, can make equivalent replacements or changes within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A three-dimensional spatial trajectory recognition method based on multi-view stereo reconstruction, characterized in that, The method includes the following steps: The same scene was photographed from different angles using multiple cameras, and the overlapping part of the images from the multiple cameras was determined as the research scene. Frame extraction and pedestrian detection are performed on the video from each camera to identify pedestrians and their location information in each frame of video, and a unique identification number is assigned to the same pedestrian. By registering video frame images from different cameras, the pixel position of each pedestrian in each camera is obtained. Then, by using multi-view triangulation and combining camera intrinsic and extrinsic parameters, the three-dimensional point coordinates of each pedestrian in the camera coordinate system are solved. The three-dimensional point coordinates are converted into coordinates in the world coordinate system to obtain the world coordinate system coordinates of each numbered pedestrian point in each frame. Based on the time information, the pedestrian locations with different numbers in each frame are connected to form a trajectory, thus completing the reconstruction of the pedestrian's three-dimensional spatial trajectory in the world coordinate system; The process of registering video frame images from different cameras to obtain the pixel position of each pedestrian in each camera, and then solving for the 3D point coordinates using a multi-view triangulation method combined with camera intrinsic and extrinsic parameters, is as follows: First, the two cameras are calibrated using camera calibration software and calibration board to obtain the intrinsic and extrinsic parameters of the cameras, thereby determining the position and attitude of the cameras in the world coordinate system. The intrinsic parameters of the cameras include focal length, principal point coordinates, pixel scale factor, and distortion factor; the extrinsic parameters of the cameras include rotation matrix and translation vector. Then, feature points are extracted within the human bounding boxes in the two viewpoint images using a feature extraction algorithm, and the same feature points in the different viewpoint images are matched using a feature matching algorithm. Next, based on the position of the feature point in each camera image and the camera's intrinsic and extrinsic parameters, a spatial ray passing through the feature point is constructed; the intersection of the spatial ray in three-dimensional space represents the three-dimensional coordinates of the object. The process of extracting frames and detecting pedestrians from the video footage of each camera, identifying pedestrians and their location information in each frame, and assigning a unique identification number to the same pedestrian specifically involves: First, the YOLOv8 model is used to perform object detection and instance segmentation on pedestrians in each frame of the video to obtain the bounding box and segmentation mask for each person. Then, the detected bounding boxes are input into the BoT-SORT model for pedestrian tracking. The BoT-SORT model assigns a unique identification number to each pedestrian throughout the video sequence based on their position and motion trajectory. Assigning a unique identification number to the same pedestrian also includes matching pedestrian features from different cameras; Transforming the three-dimensional points into coordinates in the world coordinate system, the specific world coordinate system coordinates of each numbered pedestrian point in each frame are as follows: By using the camera's rotation matrix and translation vector, the coordinates of three-dimensional points in the camera coordinate system are transformed to obtain the world coordinate system coordinates of each numbered pedestrian point in each frame.

2. A three-dimensional spatial trajectory recognition system based on multi-view stereo reconstruction, characterized in that, To implement the three-dimensional spatial trajectory recognition method based on multi-view stereo reconstruction as described in claim 1, the method includes: Multi-camera acquisition module: used to capture the same scene from different angles using multiple cameras, and to determine the intersection of the images from multiple cameras as the research scene; Video processing and pedestrian detection module: used to extract frames from the video of each camera, use pedestrian detection algorithms to identify pedestrians and their location information in each frame of video, and assign a unique identity number to each pedestrian; Video registration and feature point extraction module: used to register video frame images from different cameras and extract the pixel position of each pedestrian in each camera; 3D Reconstruction and Depth Calculation Module: Based on multi-view triangulation, combined with the camera's intrinsic and extrinsic parameters, calculates the 3D point coordinates of each pedestrian in the camera coordinate system; Coordinate transformation module: used to convert the coordinates of the three-dimensional points from the camera coordinate system to coordinates in the world coordinate system, so as to obtain the world coordinate system coordinates of each pedestrian in each frame; Trajectory Reconstruction Module: Based on time information, connect the numbered pedestrian points in each frame to form the pedestrian's three-dimensional spatial trajectory, thus completing the reconstruction of the pedestrian's three-dimensional spatial trajectory in the world coordinate system.

Images