A trajectory analysis method and system based on SFM combined with PDR based on BIM model

Abstract

The present application provides a trajectory analysis method and system combining SFM and PDR based on BIM model. The method includes the following steps: connecting a panoramic camera and an imu sensor module through terminal wifi and Bluetooth; using a panoramic camera to shoot a panoramic video and process it into a panoramic photo; using the SFM method to generate an SFM trajectory; based on the data obtained by the imu sensor, using the PDR algorithm to calculate and generate a complete PDR motion trajectory; fusing and correcting the SFM trajectory and the PDR motion trajectory and matching them with the BIM model; uploading the matched position and posture as a data to the cloud database. The method can realize the precise positioning of the trajectory and posture of the collected panoramic photos in a building or indoor space, and analyze the construction process; at the same time, through matching with the BIM model, it can provide more intuitive and accurate trajectory visualization results and comparison with the BIM model.

Description

Technical Field

[0001] The present invention relates to the field of intelligent positioning technology, and in particular to a method and system for matching and analyzing panoramic photos taken by a mobile terminal and PDR motion trajectory with a BIM model by combining SFM (Structure from Motion) and PDR (Pedestrian Dead Reckoning) technologies. Background Art

[0002] In the field of architecture and engineering, BIM (Building Information Modeling) models have become a common tool for designing, constructing and maintaining buildings. Panoramic photos can provide more intuitive visual information to record and display the actual situation of buildings. However, how to match and analyze panoramic photos and BIM models to achieve more accurate trajectory analysis remains a challenge.

[0003] In the prior art, only panoramic video is used as an input medium to achieve spatial relationship matching with the BIM model. However, due to the purely visual solution, the effect will be poor in scenes with poor lighting or large lighting changes, and there will be mismatches and matching failures. Summary of the invention

[0004] In order to solve the above problems, the present invention introduces a PDR sensor, uses PDR as a reference for the trajectory, can make up for the defects of the visual solution in a poor environment, and uses PDR as the leading correction trajectory to assist the image SFM method to obtain more accurate positioning information. The present invention specifically provides the following technical solutions:

[0005] A trajectory analysis method based on SFM combined with PDR of a BIM model, the method comprising the following steps:

[0006] S1. Turn on the Bluetooth and WiFi function buttons in the mobile terminal, open the corresponding app for data processing, connect the panoramic camera via the WiFi, and connect the imu sensor module via the Bluetooth; manually select the camera model to be used in the app, adapt the imaging of different cameras in the app, and ensure that the generated original panoramic photo is facing the front of the photographer;

[0007] S2. Calling a panoramic camera to shoot and obtain a panoramic video; processing the panoramic video into a panoramic photo and generating an SFM trajectory using the SFM method;

[0008] S3. Obtaining raw motion data files through the imu sensor, the raw motion data files including timestamps, quaternions, linear acceleration and angular acceleration; generating a complete PDR motion trajectory by calculating the data obtained by the imu sensor using the PDR algorithm;

[0009] S4. The SFM trajectory and the PDR motion trajectory are fused and corrected and matched with the BIM model;

[0010] S5. Upload each piece of data to the cloud database for storage, and generate a corresponding data information set corresponding to each collection task; the data information set includes panoramic photo information, location information, posture information, and information on the BIM model number.

[0011] Optionally, in S2, the process of processing the panoramic video is:

[0012] S21. Frame the panoramic video into the panoramic photo;

[0013] S22. Expanding the panoramic photo into a flat photo according to the set FOV (Field of View) and the set orientation;

[0014] S23. The SFM method is used to calculate and generate a motion trajectory for the unfolded planar photo; and the camera pose and position of each frame of the image are obtained by performing feature extraction, matching and camera motion estimation on the planar photo.

[0015] Optionally, in S3, the specific steps of generating the PDR motion trajectory include:

[0016] S31. Obtain linear acceleration, angular acceleration, timestamp, quaternion data from the imu sensor, define the position as the posture at the origin as the initial posture, and set the initial velocity to 0;

[0017] S32. Filtering and denoising the raw sensor data;

[0018] S33. The start and end of the step are detected by the local maximum value of the linear acceleration, and the data segment during the step is identified as walking motion data according to the set threshold value;

[0019] S34. For each step identified, the starting speed is set to 0, the linear acceleration in the three-axis direction is integrated to obtain a speed increment; the speed increment is added to the current speed to obtain an updated speed; the speed in the three-axis direction is integrated to obtain a position increment; the position increment is added to the current position to obtain an updated position;

[0020] S35. Repeat steps S33 to S34 until all motion data are processed and the complete PDR motion trajectory is generated.

[0021] Optionally, in step S4, the matching includes:

[0022] Synchronize the PDR motion trajectory and the time of the positioning point selected on the app, synchronize the time of both to count from 1 through different time offsets, and match the starting point and the second point corresponding to the time axis in the two data; after calculating the rotation, translation and scaling relationship of the corresponding two sets of points on the plane, correct the data of the PDR motion trajectory to the BIM model.

[0023] Optionally, the matching further includes:

[0024] After the data of the PDR motion trajectory is integrated with the BIM model, each point of the PDR result can find the corresponding position on the BIM model, and each point has a corresponding timestamp;

[0025] The position of the PDR is selected as a base, and the SFM trajectory data obtained by the SFM is used to correct its position and angle information;

[0026] After obtaining the position basis of the PDR, each segment of the generated SFM trajectory data is processed to obtain a corrected SFM trajectory; the processing includes:

[0027] Copy the original PDR position as the trajectory data that will be corrected in real time;

[0028] Time-sorting each section of the SFM trajectory data, and using the start and end points of the time sorting as the start and end points of the corresponding SFM trajectory, and matching them with the data of the PDR motion trajectory, matching the timestamps of the start and end points of the SFM trajectory with the position information of the corresponding timestamps of the PDR, and using these two sets of points to achieve the transformation of rotation, translation and scaling; after completing the transformation of rotation, translation and scaling, using the photo angle calculated by the SFM to update the orientation information of the photo at each position, and applying it to the corrected SFM trajectory;

[0029] After matching each SFM trajectory to the PDR trajectory and correcting the position and angle of the data, the corrected complete trajectory is retrieved again. For the data of the position and angle orientation that have not been corrected by the SFM trajectory, the original data of the PDR as the basis is still used, and an angle information calculated instead is assigned.

[0030] Optionally, the correction further comprises:

[0031] For photos that are not corrected using the SFM trajectory, each photo is placed in the corresponding position, and then the orientation of each photo is found; the orientation vector of each position is obtained by subtracting the front and back positions of the motion trajectory, and the orientation of the photo is determined by the angle between the orientation vector and the horizontal axis of the BIM model to correct the photo angle.

[0032] Optionally, the S5 further includes:

[0033] When viewing and analyzing the information set, the information set is loaded, and a sky ball with a rotation angle is generated at a corresponding spatial position of the BIM model. The sky ball is a container for loading the panoramic photo. By viewing various viewing angles of the sky ball, a standard plane picture viewing angle can be maintained.

[0034] Optionally, the panoramic camera has a direction locking function, which can keep the output video direction at a fixed angle relative to the camera lens when synthesizing the video.

[0035] The present invention also provides a trajectory analysis system combining SFM with PDR based on a BIM model, the system comprising:

[0036] Terminal app, used to connect the panoramic camera and imu sensor module, and manually select the camera type;

[0037] Data processing module: the cloud server receives and processes the data uploaded by the terminal app and generates corresponding processing tasks;

[0038] The data fusion module is used to upload the matching position and posture of each panoramic photo and BIM model to the cloud database as an information set.

[0039] Optionally, the system further comprises:

[0040] The information viewing module is used to view and analyze the information set; load the information set, and generate a sky ball with a rotation angle at a corresponding spatial position of the BIM model. The sky ball is a container for loading the panoramic photo. By viewing various viewing angles of the sky ball, a standard plane picture viewing angle can be maintained.

[0041] The present invention has the following beneficial technical effects: The present invention provides a trajectory analysis method and system combining SFM with PDR based on a BIM model. The present invention can realize trajectory analysis of buildings or indoor spaces and provide more accurate positioning and navigation services. In the method of the present invention, a panoramic camera and an IMU sensor module are first connected through a mobile terminal, and the camera type is selected for adaptation, and then a panoramic video is taken with a panoramic camera and processed into a panoramic photo; then an SFM method is used to generate an SFM trajectory, and the camera position and motion trajectory of each frame image are obtained by extracting features, matching and estimating camera motion on a plane photo; at the same time, the original motion data file is obtained through an IMU sensor, including timestamp, quaternion, linear acceleration and angular acceleration; a complete PDR motion trajectory is calculated and generated using a PDR algorithm, including step detection, posture correction and position update; next, the SFM trajectory and the PDR motion trajectory are fused and corrected, and matched with a BIM model; by removing the time offset, the rotation, translation and scaling relationship is calculated, and the data of the PDR motion trajectory is corrected to the BIM model; finally, the position and posture of each panoramic photo and the BIM model are uploaded to a cloud database as a data, and a corresponding information set is generated. By loading the information set and generating a sky ball with a rotation angle at the corresponding spatial position of the BIM model, the information set can be easily viewed and analyzed. Compared with the existing technology, the present invention combines SFM and PDR technologies to accurately locate the trajectory and posture of the collected panoramic photos in buildings or indoor spaces, and analyze the construction process; at the same time, through matching with the BIM model, it can provide more intuitive and accurate trajectory visualization results and comparison with the BIM model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flow chart of a trajectory analysis method combining SFM and PDR based on a BIM model provided in an embodiment of the present invention.

[0044] Figure 2 A framework diagram of a trajectory analysis system combining SFM and PDR based on a BIM model provided in an embodiment of the present invention.

[0045] Figure 3 This is a motion trajectory diagram generated by the PDR algorithm of the present invention.

[0046] Figure 4This is the PDR motion trajectory diagram after scaling processing of the present invention. DETAILED DESCRIPTION

[0047] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0048] The following specific examples are used to illustrate the embodiments of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0049] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any labeled structures and / or functions described herein are merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number and aspect set forth herein may be used to implement an apparatus and / or practice a method.

[0050] It should also be noted that the illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present application.

[0051] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these marked details.

[0052] The purpose of the present invention is to provide a trajectory analysis method combining SFM with PDR based on a BIM model, aiming to solve the problem of low accuracy of motion trajectory positioning analysis under a BIM model.

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] Reference Figure 1 , showing a trajectory analysis method of SFM combined with PDR based on a BIM model according to an embodiment of the present application, the method comprising the following steps:

[0055] S1. Turn on the Bluetooth and WiFi function buttons in the mobile terminal, open the corresponding app for data processing, connect the panoramic camera via the WiFi, and connect the imu sensor module via the Bluetooth; manually select the camera model to be used in the app, and adapt the imaging of different cameras in the app to ensure that the generated original panoramic photo is facing directly in front of the photographer.

[0056] S2. Invoke the insta360 panoramic camera to shoot and obtain a panoramic video; process the panoramic video into a panoramic photo, and use the SFM method to generate an SFM trajectory.

[0057] Optionally, in S2, the panoramic video is framed into the panoramic photo by processing each frame of the panoramic video and converting it into a panoramic photo. The video shot by the fisheye lens has a wide-angle field of view and can capture a wider scene, but due to the special shape of the lens, the image will appear curved. Therefore, it is necessary to perform dedistortion processing on each frame to convert the curved image into a flat panoramic photo. The specific processing steps are as follows: First, it is necessary to perform dedistortion processing on each frame of the panoramic video. This can be calculated and corrected by the distortion model of the fisheye lens to eliminate the curvature effect in the image. Next, each frame of the dedistorted image is spliced ​​to form a panoramic photo. Splicing can be achieved by aligning the pixels of each frame and fusing them. In this way, each frame of the panoramic video can be converted into a panoramic photo. Then, the panoramic photo is expanded and converted into a flat photo. The expansion can be achieved by mapping the pixels of the panoramic photo to a plane according to certain rules. The expanded flat photo can be more convenient for subsequent processing and analysis. Finally, the SFM method is used to generate a motion trajectory for the expanded flat photo. SFM is a method to reconstruct scenes by analyzing feature points and camera motion in images. By extracting features, matching, and estimating camera motion on flat photos, the camera position and motion trajectory of each frame can be obtained. This allows for better observation and analysis of scenes and motion in panoramic videos. The above panoramic video processing can be achieved through the camera SDK.

[0058] Exemplarily, the SFM trajectory drawing analysis can be implemented based on but not limited to colmap software.

[0059] Optionally, the SFM method further comprises:

[0060] Feature extraction: extract feature points from each frame in the image sequence. Common feature point algorithms include SIFT, SURF, ORB, etc. Feature matching: match feature points between adjacent frames, and determine the matching relationship by calculating the similarity or distance between feature points Motion estimation: use matched feature point pairs to estimate the camera motion between adjacent frames. Common methods include the RANSAC algorithm based on the essential matrix or the basic matrix 3D reconstruction: estimate the 3D point cloud in the scene through triangulation or optimization methods based on the camera motion and the depth information of the feature points Trajectory optimization: optimize the camera trajectory through optimization algorithms (such as Bundle Adjustment) to improve the accuracy of the reconstruction results. Finally, the SFM method can generate the camera motion trajectory and the 3D point cloud of the scene, namely the SFM trajectory.

[0061] S3. Obtaining a raw motion data file through the imu sensor, wherein the raw motion data file includes a timestamp, a quaternion, a linear acceleration, and an angular acceleration; and generating a complete PDR motion trajectory by calculating the data obtained by the imu sensor using a PDR algorithm.

[0062] Optionally, the implementation process of the PDR algorithm in step S3 to generate the motion trajectory is as follows:

[0063] Initialization includes defining the initial position and posture: the position is the origin, and the posture is the initial posture; the initial speed is set to 0; data preprocessing includes filtering and denoising the original motion data to reduce the impact of noise on the calculation; gait detection includes detecting the start and end of the step through the local maximum of the linear acceleration; according to the set threshold, the data segment during the step is identified as walking motion data; posture correction includes using quaternions to correct the posture of the sensor to eliminate the impact of posture changes on the calculation; velocity integration includes integrating the linear acceleration in the three-axis direction to obtain the velocity increment; accumulating the velocity increment to the current speed to obtain the updated speed; position integration includes integrating the velocity in the three-axis direction to obtain the position increment; accumulating the position increment to the current position to obtain the updated position. Updating the timestamp includes updating the timestamp according to the sampling frequency and the step detection result. Repeat the above steps until all motion data are processed. Figure 3 The motion trajectory diagram generated by the PDR algorithm is given.

[0064] Optionally, in the PDR algorithm:

[0065] Gait detection is to detect the local maximum value of linear acceleration and determine the start and end of the step. Posture correction uses quaternions to correct the posture of the sensor to the reference posture. The calculation of velocity integral includes:

[0066] Velocity increment = linear acceleration * time interval

[0067] Updated speed = current speed + speed increment

[0068] Position points include:

[0069] Position increment = velocity * time interval

[0070] Updated position = current position + position increment

[0071] Among them, data such as linear acceleration, time interval, and quaternion are obtained from the IMU sensor.

[0072] Optionally, in the present invention, the following filtering and denoising methods may be used for the linear acceleration, time interval and quaternion data obtained from the IMU sensor:

[0073] Linear acceleration filtering: Common linear acceleration filtering methods include low-pass filtering and Kalman filtering. Low-pass filtering can smooth the linear acceleration data by filtering out high-frequency noise, while Kalman filtering can filter the linear acceleration through state estimation and observation update.

[0074] Time interval filtering: Time interval data is usually generated by the sensor sampling frequency, so outliers can be removed by simple threshold filtering. For example, a reasonable time interval range can be set, and time interval data outside this range can be regarded as outliers and filtered out.

[0075] Quaternion filtering: Quaternion data usually needs to be subjected to attitude estimation and filtering. Common filtering methods include Kalman filtering, extended Kalman filtering, and unscented Kalman filtering. These filtering methods can filter and smooth quaternions through state estimation and observation update.

[0076] Optionally, for the PDR algorithm, the orientation of the image is estimated using geomagnetic and gyroscope data and expressed by Euler angles E (i.e., yaw, pitch, and roll); the orientation information of the image is expressed by quaternions (a, b, c, d); the quaternions (a, b, c, d) are used to correct the attitude of the imu sensor, where the orientation error σ is calculated as follows:

[0077] ;in, is the estimated image orientation, is the true value of the image orientation.

[0078] S4. After fusing and correcting the SFM trajectory and the PDR motion trajectory, the SFM trajectory and the PDR motion trajectory are matched with the BIM model.

[0079] Optionally, after steps S2 and S3, two types of trajectory data can be obtained, one of which is data generated by using a visual method through panoramic photo frames, and the other is a trajectory obtained by integrating sensor data using an imu sensor.

[0080] The relationship and usage of these two types of trajectories in matching with BIM models are as follows:

[0081] 1. The SFM trajectory is generated based on the plane expanded by the FOV of the image, so after the result is generated, there are both the trajectory and the direction of the photo. However, due to the relationship between the visual scheme, in some places with poor lighting or similar scenes, the generated trajectory model will be divided into multiple segments or errors will occur;

[0082] Second, since the PDR trajectory does not involve images but only sensors, its trajectory is pure position information and does not carry the orientation of the photo. Therefore, the position is relatively correct, but the orientation of the photo needs to be determined by other methods.

[0083] 3. Both of them have a common disadvantage, scale uncertainty. Since the scale of colmap is generated from the photos of the monocular camera, its scale will have errors with reality. The linear acceleration data used in PDR is normalized, that is, divided by the value of gravity acceleration g. This is done to eliminate the impact of changes in gravity acceleration in different environments on trajectory analysis. However, since the gravity acceleration calibrated in the sensor chip may be different from the gravity acceleration in the current environment, the generated trajectory has a scale scaling ratio on the whole. In other words, the trajectory shape generated by PDR is accurate, but there may be a scaling deviation on the whole. This is caused by the normalization of linear acceleration data, because the normalized data will be affected by gravity acceleration. Therefore, when using PDR for trajectory analysis, it is necessary to take into account the scaling ratio of this scale and make corresponding corrections or adjustments to ensure the accuracy and consistency of the trajectory. Figure 4 The motion trajectory of PDR after scaling is given.

[0084] The first one is to use only PDR data and BIM model fusion.

[0085] The main difference between this method and the method of integrating colmap with BIM model is that colmap requires multiple positioning points. Since colmap itself may fail to generate, we split the SFM tracks in two adjacent positioning points and put them in. This can be considered as a method of using multiple groups of positioning points to reduce the risk of error due to the inability to find positioning points due to generation failure. In the end, the entire track is actually segmented, not complete and continuous.

[0086] The PDR data is obtained by integrating continuous sensor data. There will be no intermediate breaks caused by reconstruction, so there is no need for multiple positioning points for disaster recovery. Therefore, only one set of positioning points is needed to determine the starting point, direction and scaling of the overall trajectory. Finally, the entire trajectory is complete. It has the same shape as when it was generated and will not be split.

[0087] Therefore, the final implementation logic of this method is as follows: when collecting data, we set the starting point, and during the collection process, we select the correct second point that can be found on the model. Record the setting time of these two points and the position coordinates in the BIM model.

[0088] In the matching process, the time offset of the PDR and the positioning point is first removed so that the time of both starts counting from 1, and the starting point and the second point corresponding to the time in the two data are matched. Since the matching is only performed on the plane relationship, the relative rotation, translation and scaling relationship of the two sets of points on the plane is calculated, and then the PDR data is corrected to the BIM model.

[0089] After getting the location, you need to first put each photo in the corresponding position, and then find the direction of each photo.

[0090] The panoramic camera used has a processing function called direction lock. When this function is turned on during video synthesis, the output video can always face the direction with a fixed angle relative to the camera lens (however, due to the SDK, it can only be guaranteed to always face the direction relatively fixed to the camera, and there will be offsets, which will be corrected during movement). In this mode, in most movements, the direction of the photo can be basically based on the front and back positions of the movement trajectory. By making a difference between the front and back positions, the direction vector of each position is obtained. The angle between this vector and the horizontal axis of the BIM model is used to determine the direction of the photo and correct the photo angle.

[0091] Since it is assumed by default that the vectors obtained from the front and back points of each position represent the direction of the photo, and the direction lock effect of the synthesized panoramic photo is perfect when using the SDK, there will be no offset. Therefore, due to the instability of the direction lock function, it is actually impossible to perfectly guarantee that the direction and the camera's direction remain completely consistent, and during the movement, the vector of each position may be different from the actual movement. For example, variable speed movement will cause the vector to have an angle difference with the direction during the actual movement.

[0092] Since it is pure trajectory prediction, there is no feature point between images for further correction. Therefore, if there is an error in the final generated angle, there is no possibility of correction.

[0093] The second method is to use the data fusion method of SFM and PDR to integrate with the BIM model.

[0094] As with the first method, we first merge the PDR data with the BIM model, so that each point in the PDR result can be found in the corresponding position and time on the BIM model.

[0095] After that, the position of the PDR will be used as the basis, and the data obtained by SFM will be used to correct the position and angle information.

[0096] In this method, we do not require all data to be completely matched as we do when using only SFM. Because there is a base position data, we can allow the data reconstructed by colmap to be fragmented and multi-segmented. Therefore, in the feature point matching stage of the photo, we can use sequential matching to improve processing efficiency.

[0097] After obtaining the positional basis of the PDR in the previous step, each segment of the generated SFM data will be processed. For each segment of data, it will be sorted according to the SFM output data, and its time-sorted start and end points will be used as the corresponding start and end points, and matched with the PDR trajectory data. These two sets of points will be used to implement the transformation of rotation, translation and scaling, and the photo angle calculated by SFM will be used to assist this rotation transformation to obtain more accurate photo angle information.

[0098] After matching each SFM trajectory to the PDR trajectory, the complete PDR trajectory will be retrieved again. For photos without SFM and positions with angle correction, the first method will be used to assign an alternative angle information.

[0099] S5. Upload the matching position and posture of each panoramic photo and the BIM model as a data to the cloud database, and generate a corresponding information set.

[0100] Optionally, when viewing and analyzing the information set, the information set is loaded, and a sky ball with a rotation angle is generated at a corresponding spatial position of the BIM model. The sky ball is a container for loading the panoramic photo. By viewing the various perspectives of the sky ball, a standard plane picture perspective can be maintained.

[0101] Reference Figure 2 , shows a trajectory analysis system combining SFM and PDR based on a BIM model according to an embodiment of the present application, the system comprising:

[0102] Terminal app, used to connect the panoramic camera and imu sensor module, and manually select the camera type;

[0103] Data processing module: the cloud server receives and processes the data uploaded by the terminal app and generates corresponding processing tasks;

[0104] The data fusion module is used to upload the matching position and posture of each panoramic photo and BIM model to the cloud database as an information set.

[0105] Optionally, the system further comprises:

[0106] The information viewing module is used to view and analyze the information set; load the information set, and generate a sky ball with a rotation angle at a corresponding spatial position of the BIM model. The sky ball is a container for loading the panoramic photo. By viewing various viewing angles of the sky ball, a standard plane picture viewing angle can be maintained.

[0107] The above are exemplary embodiments disclosed in the present invention, but it should be noted that various changes and modifications may be made without departing from the scope disclosed in the embodiments of the present invention as defined in the claims. The functions, steps and / or actions of the method claims according to the disclosed embodiments described herein do not need to be performed in any marked order. In addition, although the elements disclosed in the embodiments of the present invention may be described or required in individual form, they may also be understood as multiple unless explicitly limited to the singular.

[0108] In this specification, the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments described later, the description is relatively simple, and the relevant parts can be referred to the partial description of the previous embodiments.

[0109] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A trajectory analysis method based on SFM combined with PDR based on BIM model, characterized in that: The method comprises the following steps: S1. Turn on the Bluetooth and WiFi function buttons in the mobile terminal, open the corresponding app for data processing, connect the panoramic camera via the WiFi, and connect the imu sensor module via the Bluetooth; manually select the camera model to be used in the app, adapt the imaging of different cameras in the app, and ensure that the generated original panoramic photo is facing the front of the photographer; S2. Calling a panoramic camera to shoot and obtain a panoramic video; processing the panoramic video into a panoramic photo and generating an SFM trajectory using the SFM method; S3. Obtaining raw motion data files through the imu sensor, the raw motion data files including timestamps, quaternions, linear acceleration and angular acceleration; generating a complete PDR motion trajectory by calculating the data obtained by the imu sensor using the PDR algorithm; S4. The SFM trajectory and the PDR motion trajectory are fused and corrected and matched with the BIM model; S5. Upload each piece of data to the cloud database for storage, and generate a corresponding data information set corresponding to each collection task; the data information set includes panoramic photo information, location information, posture information and information on the BIM model number; In step S4, the matching includes: Synchronize the time of the PDR motion trajectory and the positioning point selected on the app, and synchronize the time of both to count from 1 through different time offsets, matching the starting point and the second point corresponding to the time axis in the two data; after calculating the rotation, translation and scaling relationship of the corresponding two sets of points on the plane, correct the data of the PDR motion trajectory to the BIM model; The matching also includes: After the data of the PDR motion trajectory is integrated with the BIM model, each point of the PDR result can find the corresponding position on the BIM model, and each point has a corresponding timestamp; The position of the PDR is selected as a base, and the SFM trajectory data obtained by the SFM is used to correct its position and angle information; After obtaining the position basis of the PDR, each segment of SFM trajectory data generated is processed to obtain a corrected SFM trajectory.

2. The trajectory analysis method of SFM combined with PDR based on BIM model according to claim 1 is characterized in that: In S2, the process of processing the panoramic video is: S21. Frame the panoramic video into the panoramic photo; S22. Expand the panoramic photo into a flat photo according to the set FOV and the set orientation; S23. The SFM method is used to calculate and generate a motion trajectory for the unfolded planar photo; and the camera pose and position of each frame image are obtained by performing feature extraction, matching and camera motion estimation on the planar photo.

3. The trajectory analysis method of combining SFM and PDR based on BIM model according to claim 1 is characterized in that: In S3, the specific steps of generating the PDR motion trajectory include: S31. Obtain linear acceleration, angular acceleration, timestamp and quaternion data from the imu sensor, and define the position as the posture at the origin as the initial posture, and set the initial velocity to 0; S32. Filtering and denoising the raw sensor data; S33. The start and end of the step are detected by the local maximum value of the linear acceleration, and the data segment during the step is identified as walking motion data according to the set threshold value; S34. For each step identified, the starting speed is set to 0, the linear acceleration in the three-axis direction is integrated to obtain a speed increment; the speed increment is added to the current speed to obtain an updated speed; the speed in the three-axis direction is integrated to obtain a position increment; the position increment is added to the current position to obtain an updated position; S35. Repeat steps S33 to S34 until all motion data are processed and the complete PDR motion trajectory is generated.

4. The trajectory analysis method of combining SFM and PDR based on BIM model according to claim 1 is characterized in that: The processing of each generated SFM trajectory data segment includes: Copy the original PDR position as the trajectory data that will be corrected in real time; Time-sorting each section of the SFM trajectory data, and using the start and end points of the time sorting as the start and end points of the corresponding SFM trajectory, and matching them with the data of the PDR motion trajectory, matching the timestamps of the start and end points of the SFM trajectory with the position information of the corresponding timestamps of the PDR, and using these two sets of points to achieve the transformation of rotation, translation and scaling; after completing the transformation of rotation, translation and scaling, using the photo angle calculated by the SFM to update the orientation information of the photo at each position, and applying it to the corrected SFM trajectory; After matching each SFM trajectory to the PDR trajectory and correcting the position and angle of the data, the corrected complete trajectory is retrieved again. For the data of the position and angle orientation that have not been corrected by the SFM trajectory, the original data of the PDR as the basis is still used, and an angle information calculated instead is assigned.

5. The trajectory analysis method of combining SFM and PDR based on BIM model according to claim 1 is characterized in that: The correction also includes: For photos that are not corrected using the SFM trajectory, each photo is placed in the corresponding position, and then the orientation of each photo is found; the orientation vector of each position is obtained by subtracting the front and back positions of the motion trajectory, and the orientation of the photo is determined by the angle between the orientation vector and the horizontal axis of the BIM model to correct the photo angle.

6. The trajectory analysis method of combining SFM and PDR based on BIM model according to claim 1 is characterized in that: The S5 further includes: When viewing and analyzing the information set, the information set is loaded, and a sky ball with a rotation angle is generated at a corresponding spatial position of the BIM model. The sky ball is a container for loading the panoramic photo. By viewing various viewing angles of the sky ball, a standard plane picture viewing angle can be maintained.

7. The trajectory analysis method of combining SFM and PDR based on BIM model according to claim 1 is characterized in that: The panoramic camera has a direction locking function, which can keep the output video direction and the camera lens in a fixed relative angle when synthesizing the video.

8. A trajectory analysis system combining SFM and PDR based on a BIM model, used to implement a trajectory analysis method combining SFM and PDR based on a BIM model as claimed in any one of claims 1 to 7, characterized in that: The system comprises: The terminal app is used to connect the panoramic camera and the imu sensor module and manually select the camera type; the data processing module, the cloud server receives and processes the data uploaded by the terminal app and generates corresponding processing tasks; the data fusion module is used to upload the matching position and posture of each panoramic photo and BIM model as an information set to the cloud database.

9. The trajectory analysis system of SFM combined with PDR based on BIM model according to claim 8 is characterized in that: The system further comprises: The information viewing module is used to view and analyze the information set; load the information set, and generate a sky ball with a rotation angle at a corresponding spatial position of the BIM model. The sky ball is a container for loading the panoramic photo. By viewing various viewing angles of the sky ball, a standard plane picture viewing angle can be maintained.

Images