A fast method for acquiring three-dimensional images

Abstract

The present invention is applicable to the field of image processing technology, and provides a fast three-dimensional image acquisition method: obtaining the distance between the VR display terminal and the ground; adjusting the virtual camera interactive viewing angle according to the distance; determining the scene element data in the virtual scene and the first viewing angle weight corresponding to the scene element data according to the interactive viewing angle and the distance; displaying the determined three-dimensional image according to the scene element data and the first viewing angle weight. The present invention can effectively determine the corresponding virtual scene and its viewing angle weight by monitoring and adjusting the VR display terminal, so as to quickly and effectively acquire the corresponding three-dimensional image; by setting the second viewing angle weight, the three-dimensional image can be adaptive to the VR display terminal.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a fast three-dimensional image acquisition method. Background Art

[0002] 3D reconstruction technology is a branch of computer vision, and is a research direction that combines computer vision with computer graphics and image processing. It is widely used in industrial automation, reverse engineering, cultural relics protection, computer-aided medicine, virtual reality, augmented reality, and robotics applications.

[0003] Virtual reality (VR) is an emerging "professional application" that has the potential to revolutionize the way we currently handle a wide range of tasks. A 360-degree panoramic video is captured and used to create a computer-modeled 3D space. The user then wears special goggles such as a head-mounted display (HMD), which actively selects and changes the user's perspective to obtain an immersive experience.

[0004] However, in the prior art, the steps for determining images from the perspective of a VR display are complex and computationally intensive, resulting in slow system response and low user experience fluency. In addition, the displayed three-dimensional image cannot adapt to the VR display, especially when performing three-dimensional reconstruction of real-time target objects. This problem is more prominent.

[0005] Therefore, it is necessary to provide a technical solution to solve the above technical problems. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a fast three-dimensional image acquisition method, which greatly reduces the configuration data and time required for the virtual scene by reasonably setting the virtual scene. At the same time, by adjusting the viewing angle weight, it can adapt to the VR display end, which can greatly improve the system fluency and enhance the user experience.

[0007] A first aspect of an embodiment of the present invention provides a fast three-dimensional image acquisition method, which is applied to a VR display terminal. The acquisition method includes:

[0008] Obtaining the distance between the VR display terminal and the ground;

[0009] According to the distance, adjusting the interactive viewing angle of the virtual camera;

[0010] Determining scene element data in the virtual scene and a first perspective weight corresponding to the scene element data according to the interactive perspective and the distance;

[0011] The determined three-dimensional image is displayed according to the scene element data and the first viewing angle weight.

[0012] Preferably, in the present invention, adjusting the virtual camera interactive viewing angle according to the distance includes:

[0013] Adjusting the first viewing angle information of the VR display terminal and the virtual camera;

[0014] Adjust the second viewing angle information of the virtual camera and the central scene.

[0015] Preferably, the present invention further includes: determining a second viewing angle weight according to the configuration information of the VR display terminal and the matching information of the three-dimensional image.

[0016] Preferably, in the present invention, the determined three-dimensional image or two-dimensional image is displayed according to the scene element data and the second viewing angle weight.

[0017] Preferably, in the present invention, the virtual scene acquisition includes:

[0018] S1: Obtain the video captured by each camera of the multi-camera during movement;

[0019] S2: extracting key frames according to the video to be processed, and performing distortion correction on each of the key frames.

[0020] S3: extracting the encoding data and positioning data of the target object in the video to be processed;

[0021] S4: making a first three-dimensional reconstruction model according to the key frames corresponding to all the videos to be processed, and generating a point cloud and a camera posture of a camera;

[0022] S5: Performing a second three-dimensional reconstruction model of the target object in real time according to the coding data and the positioning data of the target object;

[0023] S6: Fusing the first three-dimensional reconstructed model with the second three-dimensional reconstructed model to form the virtual scene.

[0024] Preferably, in the present invention, a common point cloud generated by adjacent cameras is obtained, and the scales of all point clouds are determined according to the common point cloud generated by the adjacent cameras.

[0025] Preferably, in the present invention, making a first three-dimensional reconstruction model according to the key frames corresponding to all the videos to be processed to generate a point cloud and a camera posture of the camera includes:

[0026] According to the key frames corresponding to the to-be-processed videos shot by all the cameras, a first three-dimensional reconstruction model is made using the SFM algorithm to generate a point cloud and the camera posture of the camera.

[0027] Preferably, in the present invention, the real-time second three-dimensional reconstruction model of the target object according to the coded data and the positioning data of the target object includes:

[0028] The second processor decodes and classifies the coded data according to the coded data and the positioning data, performs three-dimensional reconstruction using the binocular stereoscopic perspective principle, obtains real-time three-dimensional data of the target object, and reconstructs the second three-dimensional reconstruction model of the target.

[0029] Preferably, in the present invention, the fusing of the first three-dimensional reconstructed model and the second three-dimensional reconstructed model comprises:

[0030] The second data in the second three-dimensional reconstruction model and the first data in the first three-dimensional reconstruction model are mapped to a global coordinate system, the first data and the second data are matched, and the first three-dimensional reconstruction model is updated.

[0031] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: on the one hand, by monitoring and adjusting the VR display end, the corresponding virtual scene and its perspective weight can be effectively determined, so that the corresponding three-dimensional image can be quickly and effectively obtained; by setting the second perspective weight, the three-dimensional image can be adaptive to the VR display end. On the other hand, the reasonable setting of the virtual scene greatly reduces the configuration data and time required for the virtual scene, and at the same time, it can greatly improve the system fluency and enhance the user experience. It realizes providing a wider field of view, covering a more complete angle, providing more common points for determining the scale of the point cloud, and increasing robustness. At the same time, the key frames of the to-be-processed frequency captured by all cameras are extracted to make the first three-dimensional reconstruction model, so the amount of data processed is greatly reduced. At the same time, the second three-dimensional model is reconstructed and fused using the three-dimensional data of the target object, so that high-density and high-precision measurement data can be obtained, especially for the three-dimensional measurement of large-sized objects, which has more advantages, thereby solving the problem of poor authenticity in the virtual scene and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 This is a schematic diagram of the implementation process of a fast three-dimensional image acquisition method provided in the first embodiment of the present invention;

[0034] Figure 2It is a schematic diagram of the implementation process of a virtual scene corresponding to a fast three-dimensional image acquisition method provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0035] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0038] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0039] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] In order to illustrate the technical solution of the present invention, a specific embodiment is provided below for illustration.

[0041] like Figure 1 As shown, a first aspect of an embodiment of the present invention provides a fast three-dimensional image acquisition method, which is applied to a VR display terminal. The acquisition method includes:

[0042] Obtaining the distance between the VR display terminal and the ground;

[0043] According to the distance, adjusting the interactive viewing angle of the virtual camera;

[0044] Determining scene element data in the virtual scene and a first perspective weight corresponding to the scene element data according to the interactive perspective and the distance;

[0045] The determined three-dimensional image is displayed according to the scene element data and the first viewing angle weight.

[0046] Among them, obtaining the distance between the VR display terminal and the ground is used to determine the user's optimal viewing angle height; the optimal viewing angle height is also in line with the visual effect of the human eye to improve the user's experience. Adjusting the interactive viewing angle is used to determine the user's line of sight angle and the optimal scene angle for virtual scene adaptation. While improving the user's experience, it also simulates the virtual scene in the best and most optimized way. The acquired three-dimensional image can be adjusted according to the first-view weight and the VR display configuration information to achieve the best display effect that meets the VR display end, that is, to achieve the three-dimensional image adaptive VR display end, which also further improves the user's experience.

[0047] Preferably, in the present invention, adjusting the virtual camera interactive viewing angle according to the distance includes:

[0048] Adjusting the first viewing angle information of the VR display terminal and the virtual camera;

[0049] Adjust the second viewing angle information of the virtual camera and the central scene.

[0050] Among them, adjusting the first perspective information and the second perspective information is used to determine the three-dimensional angle information according to the user in the virtual 1080 scene, so as to accurately obtain the user's focus perspective. At the same time, other perspective adjustments can also be made to meet the perspective required by the user.

[0051] Preferably, the present invention further includes: determining a second viewing angle weight according to the configuration information of the VR display terminal and the matching information of the three-dimensional image.

[0052] Among them, when the first perspective weight is determined according to the distance of the VR display terminal and the interactive perspective, the second perspective weight is determined again according to the configuration information of the VR display terminal and the matching information of the three-dimensional image. This is to further or at least have one perspective weight that can be used to obtain a three-dimensional image, and the first perspective weight and the second perspective weight can interact with each other to provide more accurate three-dimensional images, or even two-dimensional images and other images.

[0053] Preferably, in the present invention, the determined three-dimensional image or two-dimensional image is displayed according to the scene element data and the second viewing angle weight.

[0054] like Figure 2As shown, a virtual scene generation method corresponding to a fast three-dimensional image acquisition method provided by the present invention is shown. Specifically, in the present invention, the virtual scene acquisition includes:

[0055] S1: Obtain the video captured by each camera of the multi-camera during movement;

[0056] S2: extracting key frames according to the video to be processed, and performing distortion correction on each of the key frames.

[0057] S3: extracting the encoding data and positioning data of the target object in the video to be processed;

[0058] S4: making a first three-dimensional reconstruction model according to the key frames corresponding to all the videos to be processed, and generating a point cloud and a camera posture of a camera;

[0059] S5: Performing a second three-dimensional reconstruction model of the target object in real time according to the coding data and the positioning data of the target object;

[0060] S6: Fusing the first three-dimensional reconstructed model with the second three-dimensional reconstructed model to form the virtual scene.

[0061] Preferably, in the present invention, a common point cloud generated by adjacent cameras is obtained, and the scales of all point clouds are determined according to the common point cloud generated by the adjacent cameras.

[0062] The first processor is also used to pre-calibrate the camera internal parameters and camera external parameters of each camera to obtain the camera matrix and distortion coefficient of each camera.

[0063] Specifically, the distortion correction for each key frame includes:

[0064] For the video captured by each camera, use the internal parameters of each camera to perform distortion correction on each frame of the video;

[0065] Extract key frames from all videos taken by all cameras after distortion correction at the same time, and all key frames correspond to the same video time;

[0066] or,

[0067] The step of extracting key frames from videos shot by all cameras at the same time and performing distortion correction on each key frame specifically includes:

[0068] Extract key frames from videos shot by all cameras at the same time, and all key frames correspond to the same video time;

[0069] For the key frames corresponding to each camera, the distortion correction is performed on the key frames using the internal parameters of each camera.

[0070] Specifically, extracting key frames according to the video to be processed includes:

[0071] The first frame of the video captured by each camera is used as the key frame;

[0072] Match the current frame with the nearest key frame. If the match meets any of the following conditions, the current frame is also considered a key frame:

[0073] The average error of the matching points fitted with the homography matrix is ​​greater than a preset number of pixels;

[0074] Calculating the translation distance between two frames of images using the matching points, where the translation exceeds a preset percentage of the image diagonal; and

[0075] The scaling factor of the two frames of images is calculated using the matching points, and the scaling factor is greater than a first preset value or less than a second preset value.

[0076] Preferably, in the present invention, extracting key frames according to the video to be processed includes:

[0077] The first frame of the video captured by each camera is used as the key frame;

[0078] Match the current frame with the nearest key frame. If the match meets any of the following conditions, the current frame is also considered a key frame:

[0079] The average error of the matching points fitted with the homography matrix is ​​greater than a preset number of pixels;

[0080] Calculating the translation distance between two frames of images using the matching points, where the translation exceeds a preset percentage of the image diagonal; and

[0081] The scaling factor of the two frames of images is calculated using the matching points, and the scaling factor is greater than a first preset value or less than a second preset value.

[0082] Preferably, in the present invention, making a first three-dimensional reconstruction model according to the key frames corresponding to all the videos to be processed to generate a point cloud and a camera posture of the camera includes:

[0083] According to the key frames corresponding to the to-be-processed videos shot by all the cameras, a first three-dimensional reconstruction model is made using the SFM algorithm to generate a point cloud and the camera posture of the camera.

[0084] Among them, the key frames corresponding to the videos captured by all cameras are combined into a set of key frames;

[0085] Extract the corner points of all key frames and use the ORB (Oriented FAST and Rotated BRIEF, fast feature point extraction and description algorithm) descriptor to describe the corner points;

[0086] Match any two key frames, use homography matrix or basic matrix to constrain matching points, and remove wrong matching points (for example, use RANSAC method to remove wrong matching points); the basis for selecting homography matrix or basic matrix is ​​to select the matrix that satisfies the larger number of matching points;

[0087] The basic matrix calculated by the matching points is preferentially selected to restore the camera pose (R, t) of the camera. The essential matrix is ​​calculated using the basic matrix and the camera matrix of the camera. Four pairs of camera poses (R, t) are decomposed from the essential matrix, where R represents the rotation matrix and t represents the translation vector. The three-dimensional coordinates of the matching points are solved using the four pairs of camera poses (R, t) to obtain four sets of point clouds.

[0088] The camera pose (R, t) corresponding to a set of point clouds in front of the two cameras is taken as the correct camera pose (R, t);

[0089] Project the correct camera pose (R, t) and point cloud to the zero frame, which refers to the camera coordinate system corresponding to the first keyframe;

[0090] The bundle adjustment method is used for optimization. All 3D points in the zero system are projected onto each key frame according to the camera posture (R, t) of each key frame. The projection error of each 3D point is calculated, and the camera posture (R, t) of each key frame is adjusted (for example, using the LM algorithm). Multiple iterations (for example, 100 iterations are sufficient to reach convergence, usually a dozen or dozens of iterations are enough to converge) are performed to minimize the projection error.

[0091] Preferably, in the present invention, the real-time second three-dimensional reconstruction model of the target object according to the coded data and the positioning data of the target object includes:

[0092] The second processor decodes and classifies the coded data according to the coded data and the positioning data, performs three-dimensional reconstruction using the binocular stereoscopic perspective principle, obtains real-time three-dimensional data of the target object, and reconstructs the second three-dimensional reconstruction model of the target.

[0093] Preferably, in the present invention, the fusing of the first three-dimensional reconstructed model and the second three-dimensional reconstructed model comprises:

[0094] The second data in the second three-dimensional reconstruction model and the first data in the first three-dimensional reconstruction model are mapped to a global coordinate system, the first data and the second data are matched, and the first three-dimensional reconstruction model is updated.

[0095] When the surface of the target object is three-dimensionally measured, a first speckle image and a second speckle image of a local surface of the target object are obtained.

[0096] Specifically, a left camera and a right camera are used to simultaneously photograph the surface of the local area at different viewing angles to obtain a left speckle image (a first speckle image) and a right speckle image (a second speckle image).

[0097] According to the integer-pixel corresponding points and the coordinates of each pixel point in the first speckle image, a sub-pixel corresponding point search is performed on the second speckle image to obtain the sub-pixel corresponding points in the second speckle image.

[0098] Three-dimensional reconstruction is performed by utilizing the binocular stereo vision principle and combining the sub-pixel correspondence of the second speckle image to obtain second three-dimensional reconstruction model data of the target object surface.

[0099] The second three-dimensional reconstructed model is transformed into a global coordinate system, the second three-dimensional reconstructed model data is registered with the first three-dimensional reconstructed model data according to the overlapping area, and the first three-dimensional reconstructed model data is updated.

[0100] Thus, the technical solution provided by the present invention, on the one hand, can effectively determine the corresponding virtual scene and its perspective weight by monitoring and adjusting the VR display end, so that the corresponding three-dimensional image can be quickly and effectively obtained; by setting the second perspective weight, the three-dimensional image can be adaptive to the VR display end. On the other hand, the reasonable setting of the virtual scene greatly reduces the configuration data and time required for the virtual scene. At the same time, this can greatly improve the system fluency and enhance the user experience. It realizes providing a wider field of view, covering a more complete angle, providing more common points for determining the scale of the point cloud, and increasing robustness. At the same time, the key frames of the to-be-processed frequency captured by all cameras are extracted to make the first three-dimensional reconstruction model, so the amount of data processed is greatly reduced. At the same time, the second three-dimensional model is reconstructed and fused using the three-dimensional data of the target object, so that high-density and high-precision measurement data can be obtained, especially for the three-dimensional measurement of large-sized objects, which has more advantages, thereby solving the problem of poor authenticity in the virtual scene and improving the user experience.

[0101] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A fast three-dimensional image acquisition method, characterized in that: Applied to a VR display terminal, the acquisition method includes: Obtaining the distance between the VR display terminal and the ground; According to the distance, adjusting the interactive viewing angle of the virtual camera; Determining scene element data in a virtual scene and a first perspective weight corresponding to the scene element data according to the interactive perspective and the distance; Displaying the determined three-dimensional image according to the scene element data and the first viewing angle weight; The adjusting the interactive viewing angle of the virtual camera according to the distance comprises: Adjusting the first viewing angle information of the VR display terminal and the virtual camera; Adjusting the second viewing angle information of the virtual camera and the central scene; The virtual scene acquisition includes: S1: Obtain the video captured by each camera of the multi-camera during movement; S2: extracting key frames according to the video to be processed, and performing distortion correction on each of the key frames; S3: Extracting encoding data and positioning data of the target object in the video to be processed; S4: making a first three-dimensional reconstruction model according to the key frames corresponding to all the videos to be processed, and generating a point cloud and a camera posture of a camera; S5: Performing a second three-dimensional reconstruction model of the target object in real time according to the coding data and the positioning data of the target object; S6: fusing the first three-dimensional reconstruction model with the second three-dimensional reconstruction model to form the virtual scene; The fusing the first three-dimensional reconstruction model with the second three-dimensional reconstruction model includes: mapping the second data in the second 3D reconstruction model and the first data in the first 3D reconstruction model to a global coordinate system, matching the first data and the second data, and updating the first 3D reconstruction model, wherein when the surface of the target object is three-dimensionally measured, a first speckle image and a second speckle image of a local surface of the target object are obtained; Specifically, a left camera and a right camera are used to simultaneously shoot the surface of a local area at different viewing angles to obtain a first speckle image and a second speckle image; sub-pixel corresponding points are searched for the second speckle image according to the corresponding points at the whole pixel level and the coordinates of each pixel point in the first speckle image to obtain the sub-pixel corresponding points in the second speckle image; three-dimensional reconstruction is performed by using the binocular stereo vision principle and combining the sub-pixel corresponding relationship of the second speckle image to obtain second three-dimensional reconstruction model data of the surface of the target object; the second three-dimensional reconstruction model is transformed into a global coordinate system; the second three-dimensional reconstruction model data is registered with the first three-dimensional reconstruction model data according to the overlapping area, and the first three-dimensional reconstruction model data is updated.

2. A fast three-dimensional image acquisition method according to claim 1, characterized in that: Also includes: The second viewing angle weight is determined according to the configuration information of the VR display terminal and the matching information of the three-dimensional image.

3. A fast three-dimensional image acquisition method according to claim 2, characterized in that: The determined three-dimensional image or two-dimensional image is displayed according to the scene element data and the second viewing angle weight.

4. A fast three-dimensional image acquisition method according to claim 1, characterized in that: The virtual scene acquisition includes: S7: Acquire a common point cloud generated by adjacent cameras, and determine the scales of all point clouds according to the common point cloud generated by adjacent cameras.

5. The fast three-dimensional image acquisition method according to claim 1, characterized in that: The step of making a first three-dimensional reconstruction model according to the key frames corresponding to all the videos to be processed to generate a point cloud and a camera posture of a camera includes: According to the key frames corresponding to the video to be processed shot by all the cameras, a first three-dimensional reconstruction model is made using the SFM algorithm to generate a point cloud and the camera posture of the camera.

Images