Method for fast fusion of mobile video and real scene three dimensions

Abstract

The application discloses a kind of mobile video and real scene three-dimensional fast fusion method, comprising the following steps, key frame is selected according to predetermined frequency, and its rotation matrix is calculated based on the pose information corresponding to key frame;Determine whether the module length of angular velocity vector corresponding to non-key frame is greater than angular velocity threshold value;If not, the rotation matrix of adjacent key frame is directly selected for the non-key frame, and the position is updated according to the moving speed;If yes, the rotation compensation matrix corresponding to the non-key frame is calculated according to the angular velocity vector and integrated with the rotation matrix of adjacent key frame to obtain the rotation matrix of the non-key frame, while updating the position according to the moving speed.The application introduces non-key frame rotation matrix dynamic reuse and micro compensation mechanism, significantly reduces the average calculation complexity of system, greatly improves the response speed and frame rate of overall fusion processing, meets the demand of high real-time requirement application scene such as remote command.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a mobile video and real scene three-dimensional rapid fusion method. BACKGROUND

[0002] The existing real scene three-dimensional video fusion technology usually relies on fixed position cameras (such as monitoring probes), and the core is to project the video picture to the surface of the three-dimensional model through the pre-accurately calibrated coordinates. The three-dimensional visualization and virtual reality technology realizes the functions of map operation, venue information positioning query, security and protection, and all-around display of information on the three-dimensional electronic map. The system is very fast in browsing the map on the B / S client side, and meets the practical needs of the system. At present, the construction of the three-dimensional geographic information platform is still in the initial stage. There are still many aspects to be further developed and improved, such as keeping the browsing of massive three-dimensional models smooth, integrating the data in the existing two-dimensional system to the three-dimensional scene, integrating the existing two-dimensional geographic information system to the three-dimensional geographic information platform, and providing corresponding analysis and decision support modules for specific businesses (such as realizing population information query, simulation flight, etc.).

[0003] CN106874436A discloses a multi-source image fusion imaging system of a three-dimensional police geographic information platform. It solves the technical problems of low real degree of the simulated objective world in the prior art. It comprises an inclined remote sensing photogrammetry device, a near-ground mobile photogrammetry device, a real scene three-dimensional modeling subsystem, a building information model subsystem, a two-dimensional GIS subsystem and a police information subsystem. The inclined remote sensing photogrammetry device and the near-ground mobile photogrammetry device are connected with the real scene three-dimensional modeling subsystem. The real scene three-dimensional modeling subsystem and the building information model subsystem are connected with a virtual reality city information model construction system. The virtual reality city information model construction system, the two-dimensional GIS subsystem and the police information subsystem are connected with a multi-source fusion subsystem. The advantage is that it realizes technical breakthroughs in consistent global positioning, ground-air integration, indoor-outdoor integration, interface and standard, etc. However, for video data collected by mobile terminals such as mobile phones and mobile recorders, this technology faces a bottleneck and is difficult to realize fusion. First, the position and orientation of the mobile device are changing, which makes the traditional calibration fusion method relying on static preset invalid. Second, the motion state of the mobile video is mostly compound motion, and the traditional fusion method is difficult to realize effective matching. These defects limit the application of mobile terminal video in the existing real scene three-dimensional fusion system. SUMMARY

[0004] The present application aims to provide a mobile video and real scene three-dimensional rapid fusion method to solve the problems raised in the background.

[0005] To achieve the above technical effects, the present application adopts the following solutions:

[0006] A mobile video and real scene three-dimensional fast fusion method, comprising the following steps,

[0007] selecting a key frame according to a predetermined frequency,

[0008] calculating a rotation matrix of the key frame based on pose information of the key frame;

[0009] judging whether a module of an angular velocity vector corresponding to a non-key frame is greater than an angular velocity threshold value;

[0010] if not, directly selecting a rotation matrix of an adjacent key frame for the non-key frame and updating a position according to a moving speed;

[0011] if yes, calculating a rotation compensation matrix corresponding to the non-key frame according to the angular velocity vector and integrating the rotation compensation matrix and the rotation matrix of the adjacent key frame to obtain a rotation matrix of the non-key frame, and simultaneously updating the position according to the moving speed.

[0012] In the above technical solution, a composite motion metric index of each non-key frame relative to a previous key frame is calculated, the composite motion metric index reflecting a position change rate, an attitude change rate and a coupling degree thereof, and if the composite motion metric index exceeds a predetermined threshold value, the non-key frame is selected as a key frame.

[0013] In the above technical solution, a video acquisition terminal continuously and real-timely acquires geographical position and three-dimensional pose information and video information of a device, generates pose information corresponding to a key frame, the pose information including position information, a speed , an acceleration , a pitch angle , a yaw angle , a roll angle and an angular velocity vector ω.

[0014] In the above technical solution, a calculation method of the composite motion metric index is as follows, ,

[0015] is a composite motion metric index, a is an acceleration corresponding to a non-key frame, ω is an angular velocity vector corresponding to the non-key frame, and α, β and γ are weight coefficients.

[0016] In the above technical solution, the rotation compensation matrix is as follows:

[0017] ;

[0018] ,

[0019] is a rotation matrix of an adjacent key frame, The rotation matrix obtained by integration for the current non-keyframe, where ω is the angular velocity vector of the adjacent keyframe, ω x ω y ω z These represent the angular velocities about the x, y, and z axes, respectively. It is the difference between the current non-keyframe timestamp and the most recent keyframe timestamp.

[0020] By adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are as follows: The present invention introduces a dynamic reuse and micro-compensation mechanism for non-critical frame rotation matrices, concentrating core computing resources on processing a small amount of high-value critical frame data. For a large number of non-critical frames, projection parameters are quickly obtained through dynamic reuse and micro-compensation mechanisms. This processing strategy significantly reduces the average computational complexity of the system, greatly improves the overall fusion processing response speed and frame rate, and meets the needs of command and other application scenarios with high real-time requirements. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] The method for rapid fusion of mobile video and real-world 3D scene according to the present invention includes the following steps:

[0023] Keyframes are selected at a predetermined frequency, meaning that the selection of keyframes uses a fixed time interval extraction mechanism, such as setting a time window baseline value. =0.5s, based on the start time of the video. Based on this, keyframe time series are generated according to the following formula:

[0024] ,

[0025] Extract timestamp The corresponding video frames serve as keyframes, and the extraction time interval for these keyframes can be adjusted according to actual needs. The video acquisition terminal is equipped with an integrated multi-source sensor assembly, including a built-in satellite positioning module, inertial measurement unit, orientation sensor, and video acquisition module, which continuously acquires the device's geographical location and 3D attitude information in real time. The pose information of the keyframes is generated by fusing data from GNSS, IMU, and other multi-source sensors. This pose information includes position information P(x,y,z), where P(x,y,z) represents spatial coordinates based on an independent coordinate system, such as the 3D coordinates in the Tianjin 2000 coordinate system, in meters (m), and velocity... acceleration , pitch angle , yaw angle , roll angle and angular velocity vector ω. To achieve the pose information reading of the key frame, first, make the video acquisition device (such as a camera) and all attitude sensors (speed, acceleration, angular velocity, etc.) work based on the same time reference (such as system timestamp or hardware synchronization signal), ensure that the acquisition time of all devices can be accurately aligned, and then solve the data acquisition triggered by the timestamp of the key frame. The acquisition data can be obtained by using the corresponding nearest point value or the average value in the predetermined window range to obtain the pose information corresponding to the key frame. It lays a key foundation for subsequent accurate registration of video frames shot in a moving state and a three-dimensional scene, solves the problem of real scene video fusion in the mobile video acquisition process, and aligns the device clock with the central server through the time synchronization protocol, establishing a foundation for subsequent space-time data fusion.

[0026] To avoid the loss of key information caused by collecting key frames at a fixed frequency when the motion amplitude is too large, further, a composite motion metric index of each non-key frame relative to the previous key frame is calculated, the composite motion metric index reflects the position change rate, attitude change rate and coupling degree, and if it exceeds a predetermined threshold, the candidate non-key frame is selected as a key frame. The key frames obtained in this way are processed in the same way as the key frames selected at a fixed time. The present application adopts a dynamic key frame discrimination mechanism, breaking through the traditional fixed time interval key frame selection method.

[0027] The specific rules are as follows: by default, two frames per second are extracted as candidate key frames to ensure the minimum space-time resolution; then a composite motion metric index M reflecting the position change rate, attitude change rate and coupling degree is constructed, and when M is greater than the threshold, it means that the motion state has changed greatly, and the current frame is forced to be marked as a key frame. The calculation method of the composite motion metric index is, wherein M is the composite motion metric index, is the acceleration vector of the current calculated non-key frame, and ω is the angular velocity vector of the calculated non-key frame, , , is a weight coefficient, used for the contribution of acceleration and rotational motion to "complexity". When M is greater than the threshold, it is judged that the motion state has changed greatly, and the frame is forced to be selected as a key frame, and its position information and attitude parameters are recorded as key position parameters.

[0028] In the specific parameter design, first, analyze the physical order of magnitude and distribution: calculate the order of magnitude and distribution of (typically 0~2g), (typically 0~2rad / s), in a large number of samples. It is found that In general motion, the range of change is relatively small and smooth, and In rotation and compound motion, the change is dramatic and the range is larger. Then the independent component affects the test: fixed =0, =0, adjust : It is found that it is difficult to effectively capture the rotation and compound motion change only by relying on acceleration, and a smaller weight is required. Fixed =0, =0, adjust : Angular velocity is sensitive to rotation, and a higher weight is given to better reflect the attitude change. Fixed =0, =0, adjust : The coupling term significantly increases when acceleration and rotation occur in the non-gravity direction at the same time (such as braking / acceleration when turning), and is the key to identifying strong compound motion, requiring a higher weight. Then iterative adjustment: gradually increase each parameter, gradually lower the threshold, and observe the change in projection accuracy to determine the parameters and threshold. Finally, the adjustment of the parameters in different scenarios: high dynamic / strong motion scene: slightly increase the threshold, increase the trigger threshold, and reduce the key frame generated by ordinary strong motion; gentle motion scene: slightly decrease the threshold: increase the sensitivity, capture more subtle motion changes, strong rotation dominant scene: reduce , increase . Through laboratory testing, take =1, =5, =5, threshold=0.25, and if the accuracy is improved, it can be appropriately reduced. In specific implementation, the video acquisition module continuously outputs, and the attitude sensor continuously collects and caches attitude data, and the data processing unit calculates the above compound motion metric indicators in real time. When the compound motion metric indicator of a certain ordinary frame reaches or exceeds the threshold, the data saving mechanism is triggered, the timestamp of the non-key frame (ordinary frame) and the pose data at the corresponding time are extracted, and are stored with the non-key frame. Marked as a key frame.

[0029] Then calculate the rotation matrix of the key frame based on the pose information corresponding to the key frame , only the key frame is used to calculate the rotation matrix, which greatly reduces the processing amount and is convenient for mobile terminal implementation; specifically, for the rotation matrix of any key frame Calculate where the required parameters are: pitch angle , yaw angle , roll angle , and the rotation matrix of the key frame is calculated as follows:

[0030]

[0031]

[0032]

[0033]

[0034] Rotation matrix for rotation around z-axis by angle , Rotation angle around z-axis, i.e. yaw angle; z Rotation matrix for rotation around y-axis by angle , Rotation angle around y-axis, i.e. pitch angle; Rotation matrix for rotation around x-axis by angle , x Rotation angle around x-axis, i.e. roll angle. , Rotation matrix for rotation around z-axis by angle x , Rotation angle around z-axis, i.e. roll angle.

[0035] Then construct the projection matrix ,

[0036]

[0037] is the position information corresponding to the key frame, i.e. the spatial position information of the sensor at the time point corresponding to the time stamp of the key frame.

[0038] Then the key frame is transformed by perspective projection to map the video pixel point (x, y, z) to the three-dimensional model coordinates (X, Y, Z), i.e. to realize the mapping of pixel coordinates → world coordinates:

[0039]

[0040] is the homogeneous coordinate (4-dimensional vector) in the world coordinate system, which represents the position of a point in space, (X, Y, Z) is the three-dimensional coordinate of the point in the world coordinate system; the last dimension is fixed as 1 (which is the specification of the homogeneous coordinate, which is convenient for matrix multiplication to realize translation), is the camera intrinsic matrix, is the column and row index of the pixel on the image.

[0041] For non-key frames, first determine whether the length of the angular velocity vector corresponding to the non-key frame is greater than the angular velocity threshold, if not, the rotation matrix of the adjacent key frame is directly selected for the non-key frame, and the position is updated according to the moving speed corresponding to the adjacent key frame and the time difference; if yes, the rotation compensation matrix of the non-key frame is calculated according to the angular velocity vector of the adjacent key frame, and the rotation matrix of the non-key frame is obtained by integrating the rotation compensation matrix and the rotation matrix of the adjacent key frame, and the position is updated according to the moving speed corresponding to the adjacent key frame and the time difference, after the rotation matrix and the position of the non-key frame are calculated, the projection matrix corresponding to the non-key frame is calculated in the same way as the key frame, and then the video pixel points are mapped to the three-dimensional model coordinates through perspective projection transformation. The adjacent key frame referred to in the application is the nearest previous key frame.

[0042] Specifically, for each non-key frame: when ω≤threshold value (the threshold value is set to 0.03 rad / s through laboratory calibration), due to the selection of the key frame, the time interval is small, the posture changes little, and the position changes little, the rotation matrix of the nearest adjacent key frame is directly reused, and only the position is updated; the dynamic extraction mechanism of the key frame has ensured that the posture changes little between adjacent key frames, and the selection of the key frame also makes the position change little, and the error introduced by reusing can be ignored.

[0043] The position information of the non-key frame is updated by using the second kinematics equation, for acceleration and speed, due to the short time and small change, the position information , acceleration and speed of the adjacent key frame are directly selected to reduce the calculation amount on the premise of ensuring a certain accuracy, that is,

[0044]

[0045] When ω>threshold value, the application adopts a rotation compensation matrix, that is, a micro-rotation compensation mechanism, enables a micro-rotation compensation algorithm, calculates a rotation compensation matrix, that is, a micro-rotation matrix, and fits the rotation matrix of the nearest previous key frame, while ensuring to improve efficiency and reduce error, the rotation compensation matrix is:

[0046]

[0047] ,

[0048] is the rotation matrix of the current non-key frame obtained by integration, is the rotation matrix of the adjacent key frame, and ω is the angular velocity vector of the adjacent key frame, which is usually the three-axis angular velocity measured by a sensor, ω x , ω y , ωz respectively corresponding to the angular velocity around the x, y, z axis, is the difference between the current non-key frame timestamp and the nearest adjacent key frame timestamp, and the position information of the non-key frame is updated in the same manner as described above.

[0049] For non-key frames, after obtaining the respective rotation matrix and position information, the video pixel points (x, y, z) are mapped to the three-dimensional model coordinates (X, Y, Z) through perspective projection transformation in the same manner as the key frames, that is, the mapping of pixel coordinates to world coordinates is realized, which will not be described here.

[0050] The lightweight rotation matrix dynamic reuse mechanism proposed in the application fully utilizes the position parameters carried by the key frames through the hierarchical processing strategy and the dynamic micro compensation algorithm, effectively reduces the calculation amount in the projection process, significantly reduces the calculation complexity and ensures the projection accuracy of the non-key frames while ensuring the projection accuracy of the non-key frames, effectively solves the real-time fusion calculation bottleneck caused by frequent changes of device posture in a mobile scene.

[0051] The application continuously and real-timely obtains the geographical position and three-dimensional posture information of the device through the multi-source sensors (GNSS, IMU, gyroscope) integrated on the mobile video device; for a video with compound motion, the key frames are added according to the corresponding rules, the system preferentially processes the projection matrix acquisition of the received key frames, calculates the accurate fusion parameters under the key view angle, accurately projects and maps the key frames to the corresponding observation plane of the three-dimensional real scene model through the perspective projection transformation model, and constructs a non-key frame rotation matrix dynamic reuse and micro compensation mechanism; through the dynamic reuse and micro compensation mechanism, the calculation amount of the mobile video and real scene three-dimensional fusion is significantly reduced under the premise of ensuring the projection accuracy of the non-key frames.

Claims

1. A method for mobile video and real scene three-dimensional fast fusion, characterized in that, The method comprises the following steps of: selecting key frames at a predetermined frequency, calculating a rotation matrix of the key frame based on the pose information of the key frame; judging whether the length of the angular velocity vector corresponding to the non-key frame is greater than an angular velocity threshold value; if not, directly selecting the rotation matrix of the adjacent key frame for the non-key frame, and updating the position according to the moving speed; If yes, a rotation compensation matrix corresponding to the non-key frame is calculated according to the angular velocity vector, and the rotation matrix of the non-key frame is obtained by integrating the rotation matrix of the adjacent key frame and the rotation compensation matrix, and the position is updated according to the moving speed; the rotation compensation matrix is: ； ， Rk is the rotation matrix of the adjacent key frame, Rk is the rotation matrix of the adjacent key frame, x ωx, ωy, ωz are the angular velocities around x, y, z axes respectively, y ωx, ωy, ωz are the angular velocities around x, y, z axes respectively, z ωx, ωy, ωz are the angular velocities around x, y, z axes respectively, is the difference between the timestamp of the current non-key frame and the timestamp of the nearest key frame.

2. The mobile video and real scene three-dimensional fast fusion method of claim 1, wherein, further comprising calculating a composite motion metric index of each non-key frame relative to the previous key frame, wherein the composite motion metric index reflects the position change rate, the attitude change rate and the coupling degree thereof, and if the composite motion metric index exceeds a predetermined threshold value, the non-key frame is selected as a key frame.

3. The mobile video and real scene three-dimensional fast fusion method according to claim 1 or 2, characterized in that, The video acquisition terminal continuously and real-timely acquires the geographical position and three-dimensional attitude information of the device and video information, generates the attitude information corresponding to the key frame, and the attitude information includes position information, speed , acceleration, pitch angle, yaw angle, roll angle and angular velocity vector ω.

4. The mobile video and real scene three-dimensional fast fusion method of claim 2, wherein, The composite motion metric index calculation method is, , M is the composite motion metric index, a is the acceleration corresponding to the non-key frame, ω is the angular velocity vector corresponding to the non-key frame, and α, β and γ are weight coefficients.

Images