Method and system for node vectorization
By identifying and calculating the three-dimensional coordinates of the points of interest in a 360-degree image, the accuracy and blur problems in the prior art under outdoor or strong light conditions are solved, and a more accurate and stable three-dimensional grid generation is achieved.
Patent Information
- Application Number
- CN201880056719.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2017-07-07
- Filing Date
- 2018-07-08
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2038-07-08
AI Technical Summary
The existing 360-degree image processing technology has accuracy problems and blurred deformation outdoors or under strong light conditions, and cannot effectively generate an accurate three-dimensional grid.
By identifying the points of interest in multiple 360-degree images and using triangulation to calculate their three-dimensional coordinates, nodes are generated, and dependence on sensor capture distance data is avoided, and pixel-enclosed points with high local contrast is detected by image processing method to realize nodes vectorization.
It improves node detection accuracy and image stability under different environmental conditions, and can generate clear and accurate three-dimensional grids, suitable for more scenarios and environments.
Smart Images

Figure CN111213181B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing and more particularly, but not exclusively, to processing 360 degree images. Background Art
[0002] 360° cameras are a fairly new commodity product that enable an individual to capture a scene along a camera path and display the corresponding 360° video on his phone, computer, or virtual reality (VR) headset.
[0003] It is desirable to enhance these 360° images or videos by using nodes to generate a three-dimensional mesh or identify points of interest.
[0004] Some augmented reality technologies (such as Tango platform or Hololens) are able to create nodes during the capture of a physical scene using a two-dimensional camera. These technologies use specific hardware devices that emit wave signals (usually infrared rays) and receive their reflections to collect the distance and / or color of all or some points (i.e., corners) in the scene. These technologies create a three-dimensional grid from these nodes. Unfortunately, this three-dimensional grid is often flawed because the technology has significant limitations when used outdoors or in strong light.
[0005] There are also technologies that combine a set of nodes with a photo of a physical scene. Exemplary, Matterport develops all of their own cameras. The camera has an optical lens and multiple sensors (infrared signals and gyroscopes) that enable continuous (non-360°) photos, distance and color measurements to be captured by rotating the pod around its longitudinal axis at one location in the scene. Repeating this operation every meter in the scene, the Matterport software builds a stitched 360° photo of every place and corresponding voxel in the entire scene. Users can attach content to the voxels, but this depends on the accuracy of the voxels, which is again a problem for sensors when the light is too strong and cannot be used outdoors. In addition, since the images are spaced every meter, blurriness and distortion between images will be added when the scene is displayed. The images are stabilized using gyroscope data.
[0006] The object of the present invention is to provide a node vectorization method and system, which can overcome the shortcomings of the prior art, or at least provide a useful choice. Summary of the invention
[0007] According to a first aspect of the present invention, there is provided a method for vectorizing nodes within a plurality of 360-degree images, comprising:
[0008] a) identifying a set of points of interest in a first 360 degree image of the plurality of 360 degree images;
[0009] b) identifying a corresponding set of points of interest in a second 360 degree image of the plurality of 360 degree images; and
[0010] c) Using triangulation of the two sets of points of interest to calculate the three-dimensional coordinates of the points of interest.
[0011] Further aspects of the invention are described in the claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Embodiments of the present invention will now be described by way of example with reference to the following drawings, in which:
[0013] Figure 1 shows a block diagram illustrating a system according to an embodiment of the present invention;
[0014] Figure 2 A flow chart illustrating a method according to an embodiment of the present invention is shown;
[0015] Figure 3 A flow chart illustrating a method according to an embodiment of the present invention is shown;
[0016] Figure 4 shows a diagram illustrating identification of points of interest in a plurality of 360 degree images according to an embodiment of the present invention;
[0017] Figure 5 shows a diagram illustrating identification of potential points of interest in a 360 degree image sample according to an embodiment of the present invention;
[0018] Figure 6 A graph illustrating the triangulation of points of interest to generate nodes according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0019] The present invention provides a node vectorization method and system for 360-degree images.
[0020] The inventors have discovered that continuous 360-degree images can be processed to identify points of interest without using distance data captured by the sensor.
[0021] exist Figure 1 , a system 100 according to an embodiment of the present invention is shown.
[0022] One or more processors 101 are shown. The one or more processors 101 are configured to identify a set of points of interest within the 360-degree image and calculate the three-dimensional coordinates of the points of interest using triangulation between two or more sets of points of interest. The processor (one or more) 101 can be configured to use an image processing method to identify the points of interest. The image processing method can use one or more visual features to identify the points of interest, for example, the method can detect points surrounded by pixels with high local contrast.
[0023] One or more memories 102 are shown. The memory 102 is configured to store a plurality of 360 degree images and points of interest.
[0024] refer to Figure 2 , a method for vectorizing nodes within multiple 360-degree images according to an embodiment of the present invention will be described.
[0025] A 360 degree camera may be used to capture a 360 degree image from a real world scene. It will be appreciated that the 360 degree camera may include a series of general spherical mirrors and an image sensor configured to capture a series of images from a scene that are later stitched together to generate a 360 degree image of the scene. In an alternative embodiment, the 360 degree camera may include a single lens and an image sensor configured to capture multiple images in different directions and stitch the captured images together to form a 360 degree image. In another embodiment, the 360 degree camera may include two fisheye lenses that capture two 180 degree images that are stitched together to form a 360 degree image.
[0026] It will be appreciated that when the term 360 degree image or 180 degree image is used, the image may not capture a full 360 or 180 degree view around the horizontal plane of the camera, and may not capture a full 360 or 180 degree view around the vertical plane of the camera. For example, in one embodiment, a 360 degree image may capture a sphere around the camera, except for the bottom portion of the sphere.
[0027] In step 201 , a set of points of interest in a first 360 degree image of a plurality of 360 degree images are identified (eg, by processor(s) 101 ).
[0028] Each point of interest within the group may be identified using an image processing method associated with the 360-degree image. The image processing method may use one or more visual features to identify the point of interest. The visual feature may be associated with a group of pixels near the point of interest. In one embodiment, the image processing method uses high contrast in a group of pixels within the 360-degree image to identify the point of interest (e.g., as described with respect to FIG. 1 ). Figure 5 described above).
[0029] In step 202, a corresponding set of points of interest in a second 360 degree image in the plurality of 360 degree images is identified (e.g., also by processor(s) 100). The first 360 degree image and the second 360 degree image may be vertically aligned before identifying the corresponding set of points of interest.
[0030] Each of the points of interest in the corresponding set may be identified by using their similarity in visual features and two-dimensional coordinates to the points of interest identified from the first 360 degree image.
[0031] The 360 degree images may be sequential and may be captured from a 360 degree camera moving along a path in a real world location such that the 360 degree images are captured at different locations from each other. The relative distances between these 360 degree images or their locations may be calculated and may be stored in association with each 360 degree image.
[0032] In step 203, triangulation (eg, as described with respect to Figure 6 ) calculate the three-dimensional coordinates of the point of interest.
[0033] In one embodiment, points of interest and their associated three-dimensional coordinates may be used to generate a three-dimensional mesh.
[0034] refer to Figure 3 , a method 300 for vectorizing nodes in a plurality of 360-degree images according to an embodiment of the present invention will be described.
[0035] In one embodiment of the invention, successive 360° images of a location are analyzed to obtain a number of strong points of interest in those images, and then the stabilized images and points of interest are displayed.
[0036] In this embodiment, a camera with a small parallax between its lenses can be used to obtain a stitched 360-degree image, that is, any area of the real scene at a distance of 70 cm from the camera can always be well captured, and the pixels of each 360° stitched image are clear and accurately represent the scene.
[0037] The present invention may include the following four steps:
[0038] 1) Capture using a 360° camera in step 301 - low parallax, preferably 4K minimum - typically using a Samsung Gear 360, another 360° camera, or a proprietary 360° camera;
[0039] 2) Parameterization of the scene in step 302 (determination of vertical lines in the first image, selection of image frequency based on the steps captured by the video) - can be performed sliding or automatically due to internal quality of service definition;
[0040] 3) the internal program execution method in step 303 to identify points of interest and determine their three-dimensional coordinates - as described below, and
[0041] 4) In step 304, the stabilized image and points of interest and the 3D mesh are displayed, for example, in a Unity application.
[0042] Methods for identifying points of interest and determining their 3D coordinates
[0043] A vectorization algorithm is used to define the coordinates (X / Y / Z) of a certain number of points of interest in a set of 360° images for a given location / scene.
[0044] The algorithm consists of a certain number of steps, starting with the first two 360° images:
[0045] 1) Load two images at a sufficiently far distance, such as 8 cm-10 cm - this may be the best distance to provide high accuracy (however, images closer than 8 cm may be used in some embodiments) - typically load one third of the image from a 30 fps video;
[0046] 2) Select points of interest in these two images;
[0047] 3) Using the azimuth angle of the first image to set the camera's azimuth reference;
[0048] 4) Fix the distance between the two images to a unit equal to 1 - to set the scale for the image group;
[0049] 5) Image analysis: Calculate the 2D coordinates of a selected (automatic) number of points of interest in each image, as well as their 3D coordinates in the 3D grid, and pair the 2D and 3D coordinates;
[0050] 6) Evaluate the orientation of the camera of the second image;
[0051] 7) Use triangulation algorithm to calculate the 3D coordinates of the points of interest in both images.
[0052] Then start analyzing the third image through the same process:
[0053] 1) Load the third image;
[0054] 2) selecting a point of interest of the third image;
[0055] 3) Pair the 2D coordinates of the points of interest with their 3D coordinates in the 3D grid;
[0056] 4) evaluating the orientation and position of the camera in the third image;
[0057] 5) Re-run the coordinate pairing algorithm on the third image - it will be more accurate because the position and orientation of the third image will now be better known;
[0058] 6) Re-run the camera orientation and position estimation - to improve accuracy;
[0059] 7) Calculate the three-dimensional coordinates of the points of interest of the three-dimensional grid using a triangulation algorithm;
[0060] 8) Check that the third image is not too close to the second image - the criterion is the number of points of interest that the second and third images have in common, if too close, the algorithm will not keep the third image and will start analyzing the fourth image.
[0061] The same process is repeated for each image one by one until the last image of the scene.
[0062] Once all images of the scene have been fully analyzed, an algorithm selects the image with the most equal distance as the image to be displayed - distance parameters may be provided.
[0063] Orient the images using the vertical lines defined on the first image.
[0064] The image can be oriented in a fixed direction or in the direction of the camera.
[0065] Some images can be generated by interpolating two consecutive images, making the display smooth by construction.
[0066] Afterwards, the images are clustered into a video or a series of relatively stable images of equal distance.
[0067] The points of interest can be used to construct a 3D grid. This 3D grid can be used to give a 3D map of the location and can also be tied to any other supplementary 3D grid (e.g. Google Earth coordinates, or any other coordinate reference).
[0068] Further details of the subprocess are as described above:
[0069] Image vertical orientation
[0070] Matrix and vector (eigen) operations can be used to find a rotation matrix that corrects for the non-vertical position of the camera operator by using the two gradients to find the slopes on the two axes of the camera operator (using a vertical line defined manually or automatically on the first image).
[0071] Select a point of interest
[0072] Point Detector – ORB from OpenCV can be used to select points of interest in a 360 degree image.
[0073] Detectors that are generally invariant to geometric transformations and have affordable computation time can be used. The number of points can be configured to maximize the accuracy of SLAM. Good results can be obtained between 5000 and 10000 points per image. The ORB detector is used to detect points of interest in each image and calculate their 2D coordinates. Points of interest can be defined by a batch of pixels using a contrast criterion by the ORB detector.
[0074] pair
[0075] For example, for each 2D point detected in the image, pair the 2D coordinate of the point in the image with its 3D coordinate in the 3D grid as follows:
[0076] 1) Load the points in the 3D grid that have the closest ORB descriptor to the 2D points in the last five images;
[0077] 2) limit the search to a distance of 150 pixels; and
[0078] 3) Parallel search on several CPU cores.
[0079] Evaluate the camera's orientation and position
[0080] The OpenGV library is used to calculate the position and orientation of each 360° image in the following way:
[0081] 1) Use the paired results - the 2D and 3D coordinates of each point of interest;
[0082] 2) Use the OpenGV RANSAC algorithm to limit the impact of pairing errors; and
[0083] 3) Use OpenGV PNP nonlinear optimization to perform several iterations to find the most accurate position and orientation for each image.
[0084] Secondary pairing
[0085] Pair the 2D coordinates of a point in the image with its 3D coordinates in the 3D grid with better accuracy than the first pairing, for example:
[0086] 1) An accurate estimate of the camera position is used to project the 3D mesh onto the 2D image;
[0087] 2) identifying the point in the 3D grid that has the closest ORB descriptor to the 2D point in the last five images; and
[0088] 3) Limit the search to a distance of 20 pixels.
[0089] Secondary evaluation of camera position and orientation
[0090] The position and orientation of each 360° image is calculated using the OpenGV library in the following way:
[0091] 1) Use the second pairing results - the 2D and 3D coordinates of each point of interest;
[0092] 2) Use the OpenGV RANSAC algorithm to limit the impact of pairing errors; and
[0093] 3) Perform several iterations using Open GV PNP nonlinear optimization to find the most accurate position and orientation for each image.
[0094] This iterative process (double pairing and evaluation of the cameras) may be provided to improve the accuracy of the location of the point of interest.
[0095] Triangulation of points of interest
[0096] The library VPP can be used to calculate the 3D coordinates of each point of interest in the following way:
[0097] 1) using two 2D coordinates, two positions and two viewpoints for each point of interest that forms a pair with a point of the 3D grid; and
[0098] 2) Triangulate each point of the 3D grid that forms a pair with a 2D point in the current image.
[0099] Add points of interest that do not yet have a corresponding relationship with the 3D grid
[0100] Add new points of interest to the current image by:
[0101] 1) Pairing and triangulating points of interest detected in the last two images but not yet triangulated; and
[0102] 2) Add points of interest to the 3D grid only if the reprojection error is below 10 pixels.
[0103] Image deformation
[0104] New images are added between known images to further smooth the playback of the series of images.
[0105] Correcting the misalignment of nodes by identifying identical locations in both images of a cycle
[0106] When a loop is captured (i.e., the camera passes through the exact same location twice in the same 360° video), all data points along the camera path (including the camera position and the coordinates of the points of interest) are rectified to eliminate the coordinate differences between the two images of the same location.
[0107] Select Key Image
[0108] Only 360° images that are sharp enough are retained to improve the results and reduce the computation time by:
[0109] If less than 70% of the points of interest detected in the current image are used to estimate the camera position, the current image is marked as a critical image.
[0110] Sampling of images for display
[0111] After all vectorization processes, the captured images are sampled to reduce camera operator step variations so that the images are as equidistant as possible by:
[0112] 1) Given a distance parameter, for example, 8 cm;
[0113] 2) from the sample of key images: select the first image, then the image that is closer to the first image by a given distance parameter, and so on; and
[0114] 3) From all the images taken: select the first image, then the images that are closer to the given distance parameter from the first image, and so on.
[0115] Generate 3D mesh
[0116] After all the vectorization processes, all points of interest are used to generate a 3D grid of locations.
[0117] Figure 4 Identification of a sample of points of interest within two 360 degree images 401 and 402 is shown in accordance with an embodiment of the present invention.
[0118] For illustration purposes, these two 360 degree images do not reflect the curvature of the straight lines expected in a 360 degree image. Furthermore, only a sample of the identified points of interest are shown.
[0119] In image 401, the following set of points of interest are identified: 401a, 401b, and 401c (eg, using Figure 5 method).
[0120] In image 402, the following corresponding set of points of interest are identified: 402a, 402b, and 402c (eg, partially using Figure 5 method).
[0121] Figure 5 The use of a contrast algorithm to determine points of interest within portion 501 of a 360 degree image is shown.
[0122] This portion includes a block of 7 pixels by 6 pixels. The sub-block within the block is a darker shade than the rest of the block. The point at 502 is identified as a point with high contrast based on the group of pixels near the point.
[0123] Figure 6 The use of triangulation to calculate the three-dimensional coordinates of the nodes for points of interest is shown.
[0124] Each of a set of points of interest 601a, 601b, 601c in the first 360-degree image is mapped to a corresponding point of interest 602a, 602b, 602c in the second 360-degree image. Each mapped point of interest (e.g., 601a and 602a) has two-dimensional coordinates, and using the distance between the first 360-degree image and the second 360-degree image, triangulation can be used to determine the three-dimensional coordinates of the point of interest.
[0125] A potential benefit of some embodiments of the invention is that nodes for points of interest can be added to 360 degree images with cheaper, simpler hardware and faster capture modes, and more accurate nodes can be detected and added in more different environmental conditions.
[0126] Although the present invention is illustrated by the description of the embodiments of the present invention, although the embodiments are described in considerable detail, it is not the intention of the applicant to limit or in any way restrict the scope of the appended claims in such detail. Additional benefits and modifications will be apparent to those skilled in the art. Therefore, the present invention in its broader aspects is not limited by the specific details, representative apparatus and methods, and illustrative examples shown and described. Therefore, variations from such specific details are possible without exceeding the scope or spirit of the applicant's general inventive concept.
Claims
1. A method for vectorizing nodes within a plurality of 360 degree images sequentially captured by the same 360 degree camera, comprising: selecting a first 360 degree image and a second 360 degree image from the plurality of sequentially captured 360 degree images, wherein each of the plurality of sequentially captured 360 degree images is associated with a location corresponding to a path along which the 360 degree camera moves, and the plurality of sequentially captured 360 degree images are captured by the same 360 degree camera, and selecting the second 360 degree image based on a location associated with the second 360 degree image that differs from a location associated with the first 360 degree image by more than a minimum threshold; identifying a first set of points of interest in the first of the plurality of sequentially captured 360-degree images, wherein each point of interest in the first set of points of interest is identified based on a plurality of pixel values near the point of interest; identifying a second set of points of interest corresponding to the first set of points of interest in a second 360 degree image of the plurality of sequentially captured 360 degree images, wherein the second 360 degree image is captured after the first 360 degree image, and the second set of points of interest are identified based on similarity in visual features to the first set of points of interest; and Three-dimensional coordinates of the points of interest are calculated using triangulation of both the first set of points of interest and the second set of points of interest.
2. The method according to claim 1, wherein: The first set of points of interest are identified within the first 360 degree image based on at least one visual feature exceeding a threshold.
3. The method according to claim 2, wherein: The visual characteristic is contrast.
4. The method according to claim 2, wherein: The visual features are associated with a set of pixels near each point of interest in the 360-degree image.
5. The method according to claim 1, wherein: Prior to identifying the second set of points of interest, the second 360 degree image is vertically aligned with the first 360 degree image.
6. The method of claim 1, further comprising the step of generating features for the first set of points of interest.
7. The method according to claim 6, wherein: The second set of points of interest are identified using the features for the first set of points of interest.
8. The method according to claim 1, wherein: The plurality of sequentially captured 360-degree images are captured by the 360-degree camera moving along a path in the real world.
9. The method according to claim 8, wherein: Each of the plurality of sequentially captured 360-degree images is captured by the 360-degree camera at a different position.
10. The method according to claim 9, wherein: The 360-degree camera is a video camera.
11. The method according to claim 8, wherein: The plurality of sequentially captured 360-degree images are at different distances from each other.
12. The method according to claim 1, wherein: Each of the plurality of sequentially captured 360 degree images is associated with a location in the real world.
13. The method of claim 1, further comprising the step of generating a three-dimensional mesh using the three-dimensional coordinates.
14. The method of claim 1, further comprising matching each of the three-dimensional coordinates to an existing three-dimensional mesh of the same real-world location.
15. A system for vectorizing nodes within a plurality of 360 degree images sequentially captured by the same 360 degree camera, comprising: One or more processors configured to: selecting a first 360 degree image and a second 360 degree image from the plurality of sequentially captured 360 degree images, wherein each of the plurality of sequentially captured 360 degree images is associated with a location corresponding to a path along which the 360 degree camera moves, and the plurality of sequentially captured 360 degree images are captured by the same 360 degree camera, and selecting the second 360 degree image based on a location associated with the second 360 degree image that differs from a location associated with the first 360 degree image by more than a minimum threshold; identifying a first set of points of interest in the first of the plurality of sequentially captured 360-degree images, wherein each point of interest in the first set of points of interest is identified based on a plurality of pixel values near the point of interest; identifying a second set of points of interest corresponding to the first set of points of interest in a second 360 degree image of the plurality of sequentially captured 360 degree images, wherein the second 360 degree image is captured after the first 360 degree image, and the second set of points of interest are identified based on similarity of visual features to the first set of points of interest; and calculating three-dimensional coordinates of the points of interest using triangulation of both the first set of points of interest and the second set of points of interest; and A memory is configured to store the plurality of sequentially captured 360-degree images and the three-dimensional coordinates of each of the points of interest.
16. A computer storage medium storing a computer program, wherein the computer program implements the method according to any one of claims 1 to 14 when executed by at least one processor.
