Image matching pixel retrieval method, device and storage medium
Through spherical quad-tree construction and antipole constraint search, the problem of large amount of matching point search operations in the panoramic imaging model is solved, real-time three-dimensional reconstruction is realized, and efficiency and customer experience are improved.
Patent Information
- Application Number
- CN202111613958.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-27
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-12-27
AI Technical Summary
The existing two-view matching point search method has a large computing scale in panoramic photos and cannot be applied to real-time scenes, resulting in inefficient three-dimensional reconstruction process.
The spherical quadtree construction method is used to process the panoramic image pixels and the spherical quadtree nodes according to the counterpole constraints, and search for intersecting circles in the spherical quadtree based on the opposing constraints to obtain candidate nodes to realize matching pixel retrieval.
It greatly shortens the matching point query time, improves computing efficiency, enables the three-dimensional reconstruction work to run in real time, and improves customer experience.
Smart Images

Figure CN114329032B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to an image matching pixel retrieval method, device and storage medium. Background Art
[0002] In the process of VR acquisition and presentation, generating 3D models from photos is a core technology. At present, the 3D reconstruction technology widely used in the industry is based on the pinhole imaging model. It generates 3D models from photos through multi-view geometry methods, and two-view matching point retrieval is the core step. To retrieve two-view matching points in two panoramic photos, it is necessary to cut the panoramic photos into pieces of pinhole model photos, and then apply the corresponding epipolar constraint matching query method. The calculation scale is large and cannot be applied to real-time scenes. Summary of the invention
[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide an image matching pixel retrieval method, device, electronic device, and storage medium.
[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image matching pixel retrieval method, comprising: triangulating a sphere to construct a spherical quadtree; numbering the nodes of the spherical quadtree, correspondingly processing first image pixels in a first panoramic image to the nodes and storing the spherical quadtree; in response to a request to retrieve matching pixel points of a second panoramic image and the first panoramic image, determining an intersection circle where an epipolar plane corresponding to a second image pixel of the second panoramic image intersects the spherical surface based on epipolar constraints between the second panoramic image and the first panoramic image; searching the spherical quadtree based on the intersection circle to obtain candidate nodes corresponding to the second image pixels; and in response to coordinate information of the candidate nodes satisfying a first search accuracy, using the corresponding first image pixels of the candidate nodes as matching pixels.
[0005] Optionally, the triangulation of the sphere includes: segmenting the unit sphere to generate multiple spherical triangles; recursively triangulating each of the spherical triangles according to tree depth information to construct the spherical quadtree; wherein the recursive triangulation includes: connecting the midpoints of each edge of the spherical triangle in pairs using geodesics so that the spherical triangle is segmented into four secondary spherical triangles.
[0006] Optionally, constructing the spherical quadtree includes: constructing the spherical quadtree based on all spherical triangles; wherein each spherical triangle is a node of the spherical quadtree; determining the circumscribed circle corresponding to the spherical triangle of each node, and determining the center vector and circumscribed fillet radius corresponding to the center of the circumscribed circle.
[0007] Optionally, the numbering of the nodes of the spherical quadtree includes: setting a corresponding first serial number for the nodes located at the top level of the spherical quadtree, and setting a second serial number corresponding to the parent node of the remaining nodes; numbering all nodes according to the numbering rules and based on the first serial number and / or the second serial number.
[0008] Optionally, the corresponding processing of the image pixels in the first panoramic image and the nodes includes: converting the pixel coordinates of the first image pixels into spherical coordinates; starting from the root node, judging whether the spherical coordinates are inside or outside the node in the spherical quadtree, and in response to the spherical coordinates being inside the node, determining whether the spherical coordinates are inside or outside the child nodes of this node, until it is determined that the spherical coordinates are inside the child node of the lowest level, and matching this child node with the pixel node corresponding to the first image pixel.
[0009] Optionally, the determining whether the spherical coordinate is inside or outside a node in the spherical quadtree comprises: determining a spherical triangle corresponding to the node; and if the spherical coordinate is within the spherical triangle, determining that the spherical coordinate is inside the node.
[0010] Optionally, a mask corresponding to the first image pixel is constructed; and a corresponding relationship between the number of the pixel node, the first image pixel and the mask is established and stored.
[0011] Optionally, the search processing in the spherical quadtree based on the intersection circle to obtain the candidate node corresponding to the second image pixel includes: determining the normal vector of the intersection circle; obtaining the center vector and the circumscribed fillet radius corresponding to the spherical triangle of the node; calculating the complementary angle of the angle between the center vector and the normal vector; obtaining a set of nodes intersecting with the intersection circle, and selecting the node at the lowest level in the node set as the candidate node based on the search rule; wherein the search rule includes: the complementary angle is greater than or equal to the circumscribed fillet radius, and the difference between the complementary angle and the circumscribed circle radius is less than a second search precision.
[0012] Optionally, the first search accuracy includes: an angle between the spherical coordinates of the first image pixel corresponding to the candidate node and the intersecting circular plane is less than an angle threshold.
[0013] According to a second aspect of an embodiment of the present disclosure, there is provided an image matching pixel retrieval device, comprising: a quadtree construction module, configured to perform triangulation processing on a sphere to construct a spherical quadtree; a pixel correspondence module, configured to perform numbering processing on the nodes of the spherical quadtree, perform correspondence processing between the first image pixels in a first panoramic image and the nodes, and store the spherical quadtree; an epipolar constraint processing module, configured to determine, in response to a request for retrieving matching pixel points between a second panoramic image and the first panoramic image, an intersection circle where an epipolar plane corresponding to a second image pixel of the second panoramic image intersects the spherical surface based on the epipolar constraint between the second panoramic image and the first panoramic image; a candidate node determination module, configured to perform search processing in the spherical quadtree based on the intersection circle to obtain a candidate node corresponding to the second image pixel; and a pixel matching determination module, configured to use the first image pixel corresponding to the candidate node as a matching pixel in response to the coordinate information of the candidate node satisfying the search accuracy.
[0014] Optionally, the quadtree construction module is used to segment the unit sphere to generate multiple spherical triangles; based on the tree depth information, each of the spherical triangles is recursively subdivided to construct the spherical quadtree; wherein the recursive subdivision includes: connecting the midpoints of each edge of the spherical triangle in pairs using geodesics so that the spherical triangle is divided into four secondary spherical triangles.
[0015] Optionally, the quadtree construction module is specifically used to construct the spherical quadtree based on all spherical triangles; wherein each spherical triangle is a node of the spherical quadtree; determine the circumscribed circle corresponding to the spherical triangle of each node, and determine the center vector and circumscribed fillet radius corresponding to the center of the circumscribed circle.
[0016] Optionally, the pixel corresponding module is specifically used to set a corresponding first serial number for the node located at the top layer of the spherical quadtree, and set a second serial number corresponding to its own parent node for the remaining nodes; and number all nodes according to the numbering rules and based on the first serial number and / or the second serial number.
[0017] Optionally, the pixel correspondence module is specifically used to convert the pixel coordinates of the first image pixel into spherical coordinates; starting from the root node, determine whether the spherical coordinates are inside or outside the node in the spherical quadtree, and in response to the spherical coordinates being inside the node, determine whether the spherical coordinates are inside or outside the child node of this node, until it is determined that the spherical coordinates are inside the child node of the lowest level, and correspond this child node to the pixel node corresponding to the first image pixel.
[0018] Optionally, the pixel correspondence module is used to determine a spherical triangle corresponding to the node; if the spherical coordinate is located in the spherical triangle, it is determined that the spherical coordinate is inside the node.
[0019] Optionally, the pixel correspondence module is further used to construct a mask corresponding to the first image pixel; establish and store the correspondence between the number of the child node, the first image pixel and the mask.
[0020] Optionally, the candidate node determination module is used to determine the normal vector of the intersection circle; obtain the center vector and the circumscribed fillet radius corresponding to the spherical triangle of the node; calculate the complementary angle of the angle between the center vector and the normal vector; obtain a set of nodes intersecting the intersection circle, and select the node at the lowest level in the node set as the candidate node based on a search rule; wherein the search rule includes: the complementary angle is greater than or equal to the circumscribed fillet radius, and the difference between the complementary angle and the circumscribed circle radius is less than a first search accuracy.
[0021] Optionally, the search accuracy includes: an angle between the spherical coordinates of the first image pixel corresponding to the candidate node and the intersecting circular plane is less than an angle threshold.
[0022] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the above method.
[0023] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for executing the above method.
[0024] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer program, comprising a computer-readable code, characterized in that when the computer-readable code is run on a device, a processor in the device executes an operation for implementing the above-mentioned method.
[0025] Based on the image matching pixel retrieval method, device, electronic device, and storage medium provided by the above-mentioned embodiments of the present disclosure, the problem of large amount of computational complexity in searching for epipolar constraints of panoramic imaging models is solved, and the search for epipolar constraints of spherical imaging models is converted into a query based on spherical quadtrees, which greatly shortens the query time of matching points, improves computational efficiency, enables three-dimensional reconstruction to be performed in real time, and effectively improves customer experience.
[0026] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] By describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present disclosure will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.
[0028] Figure 1 A flowchart of an embodiment of the image matching pixel retrieval method disclosed in the present invention;
[0029] Figure 2A Schematic diagram of triangulating the unit sphere; Figure 2B Schematic diagram of triangulating the unit sphere;
[0030] Figure 3 A flowchart of candidate nodes in one embodiment of the image matching pixel retrieval method disclosed in the present invention;
[0031] Figure 4 A schematic diagram of filling in node numbers in one embodiment of the image matching pixel retrieval method disclosed in the present invention;
[0032] Figure 5 A schematic diagram of the structure of an embodiment of the image matching pixel retrieval device disclosed in the present invention;
[0033] Figure 6 is a structural diagram of an embodiment of an electronic device disclosed herein. DETAILED DESCRIPTION
[0034] The exemplary embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.
[0035] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0036] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0037] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.
[0038] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0039] In addition, the term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.
[0040] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0041] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0042] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.
[0043] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.
[0044] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0045] The embodiments of the present disclosure can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0046] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, target programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks can be performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.
[0047] Application Overview
[0048] In the process of implementing the present disclosure, the inventors found that the existing retrieval method for searching for two-view matching points in two panoramic photos has a large computational scale and cannot be applied to real-time scenes. Therefore, a new solution for retrieval of matching pixels of two images is needed.
[0049] The image matching pixel retrieval method provided by the present invention performs triangulation processing on a sphere to construct a spherical quadtree, performs correspondence processing on first image pixels and nodes in a first panoramic image and stores the spherical quadtree; based on the epipolar constraints of the second panoramic image and the first panoramic image, determines the intersection circle where the epipolar plane corresponding to the second image pixel of the second panoramic image intersects with the spherical surface; performs search processing in the spherical quadtree based on the intersection circle to obtain candidate nodes corresponding to the second image pixels; in response to the coordinate information of the candidate node satisfying the first search accuracy, uses the corresponding first image pixel of the candidate node as the matching pixel; solves the problem of large amount of computational complexity in searching the epipolar constraints of the panoramic imaging model, converts the search of the epipolar constraints of the spherical imaging model into a query based on the spherical quadtree, improves computational efficiency, enables the three-dimensional reconstruction work to run in real time, and effectively improves the customer experience.
[0050] Exemplary Methods
[0051] The step numbers in the present disclosure, such as “step one”, “step two”, “S101”, “S102”, etc., are only for distinguishing different steps and do not represent the sequence between the steps. The order of steps with different numbers can be adjusted during execution.
[0052] The internal parameters of a panoramic camera, such as a binocular camera, are the parameters obtained through calibration and their own characteristics, such as focal length, pixel size, etc. The external parameters to be tested are the parameters of the binocular camera obtained through calibration in the world coordinate system, such as position, rotation direction, etc. Use two cameras to shoot the same object at different positions. If the scenes in the two photos overlap, there is a certain correspondence between the two photos. The most direct way to find the correspondence between two images is to match them point by point. If certain constraints, namely epipolar constraints, are imposed, the search range can be greatly reduced and reduced to a line.
[0053] For the problem of epipolar constraint retrieval of matching points in two views (panoramic images), the current retrieval process using the pinhole model is as follows: a certain imaging coordinate on the first frame image corresponds to a spatial point, and the geometric relationship constrains the possible projection position of the spatial point on the second frame image to a straight line. The retrieval process is to perform uniform iterative coordinate enumeration based on the parameters of this line. Unlike the pinhole model, the epipolar constraint of the spherical imaging model is no longer a straight line. If the epipolar constraint is mapped to the panorama, it is mapped like a geodesic unfolded in the panorama, which is generally a curve and cannot be obtained through coordinate iteration like the straight line epipolar constraint of the pinhole model.
[0054] Figure 1 FIG. 1 is a flowchart of an embodiment of the image matching pixel retrieval method disclosed in the present invention. Figure 1 The method shown includes steps: S101-S105. Each step is described below.
[0055] S101, triangulate the sphere to construct a spherical quadtree.
[0056] S102, numbering the nodes of the spherical quadtree, and making correspondence between the first image pixels in the first panoramic image and the nodes and storing the spherical quadtree.
[0057] S103, in response to a request to retrieve matching pixel points of the second panoramic image and the first panoramic image, determine an intersection circle where an epipolar plane corresponding to a second image pixel of the second panoramic image intersects the spherical surface based on an epipolar constraint between the second panoramic image and the first panoramic image.
[0058] S104, performing a search process in the spherical quadtree based on the intersection circle to obtain a candidate node corresponding to the second image pixel.
[0059] S105 , in response to the coordinate information of the candidate node satisfying the first search accuracy, taking the first image pixel corresponding to the candidate node as a matching pixel.
[0060] There are many methods for triangulating a sphere. For example, a unit sphere is segmented to generate multiple spherical triangles. According to the tree depth information, each spherical triangle is recursively triangulated to construct a spherical quadtree. The recursive triangulation includes connecting the midpoints of each side of the spherical triangle with geodesics so that the spherical triangle is segmented into four secondary spherical triangles.
[0061] A spherical quadtree is constructed based on all spherical triangles, each of which is a node of the spherical quadtree. The circumscribed circle corresponding to the spherical triangle of each node is determined, and the center vector and the circumscribed fillet radius corresponding to the center of the circumscribed circle are determined. The corresponding first serial number is set for the nodes at the top level of the spherical quadtree, and the second serial number corresponding to the parent node is set for the remaining nodes, and all nodes are numbered according to the numbering rule and based on the first serial number and / or the second serial number.
[0062] In one embodiment, Figure 2A As shown, the unit sphere is divided into 20 congruent spherical triangles from a regular icosahedron. Let the edge of a certain face f on the polyhedron surface be the set {e}, the two endpoints of e are the vertices of the polyhedron, and the three points of the endpoints and the center of the polyhedron o constitute the projective plane p; assuming that the normal vector n of the projective plane always points to the inside of f, then p can represent a half-space h (half-space); the intersection of all half-spaces h corresponding to f constitutes a cone r, and the spherical triangle corresponding to f can be obtained by the intersection of r and the unit sphere S2.
[0063] The triangulation is recursively refined to construct a tree structure. Given each spherical triangle t, the midpoints P, S, and U of each side are connected by geodesics. At this time, the triangle is divided into four spherical triangles. According to the specified tree depth (number of levels) d, d recursive operations are performed to obtain a quadtree representation of the unit sphere. The four spherical triangles obtained from each division are not of equal area. The area of the triangle enclosed by the midpoint connection is larger than that of the other three triangles. However, as the refinement level increases, this difference gradually decreases. When the tree level is deep enough, the impact of this difference on the calculation process can be ignored.
[0064] All spherical triangles constitute the nodes of the quadtree QuadTree. The circumscribed circle of the spherical triangle of each node is calculated, and the center and angular radius (Constain Radius) r of the circumscribed circle are determined. The center vector of the circumscribed circle is vector b, and b is normalized to a unit vector. All nodes of the quadtree QuadTree are numbered and represented by a 32-bit integer data: the top 20 spherical triangle nodes are numbered 1-20 (the first number), and the other intermediate nodes and child nodes are numbered by their own numbers under the parent node (the second number is 0-3).
[0065] like Figure 2B As shown, the memory corresponding to the 32-bit integer data is divided into multiple segments, and the initial values of all bits are set to 0 by default. For the first-level node numbers (1-20), the node numbers are converted into 5-bit binary codes and filled into the first 5 bits. For the nodes at the lower level, the numbers (0-3) corresponding to the node generation are converted into 2-bit binary codes, and filled into the subsequent positions step by step until all the node numbers are filled.
[0066] The pixel coordinates of the first image pixel are converted into spherical coordinates, starting from the root node, determining whether the spherical coordinates are inside or outside the node in the spherical quadtree, and in response to the spherical coordinates being inside the node, determining whether the spherical coordinates are inside or outside the child node of this node, until it is determined that the spherical coordinates are inside the child node of the lowest level, and the child node is matched with the pixel node corresponding to the first image pixel.
[0067] The method for determining whether the spherical coordinate is inside or outside the node in the spherical quadtree may be: determining a spherical triangle corresponding to the node, and if the spherical coordinate is inside the spherical triangle, determining that the spherical coordinate is inside the node. Constructing a mask corresponding to the first image pixel, establishing and storing the corresponding relationship between the number of the pixel node, the first image pixel, and the mask.
[0068] In one embodiment, the quadtree structure corresponds to the first panoramic image, the maximum supported photo resolution is 8192x4096, and the maximum tree depth is 13. For the first panoramic image of the specified resolution, all pixels are mapped to the nodes of the quadtree, specifically: given a pixel coordinate, it is converted into a spherical coordinate (bearing vector) p, which is a three-dimensional vector with a length of 1. Starting from the root node, it is determined whether the coordinate is inside or outside the node on the quadtree QuadTree. If the vector is inside the node, it continues to recursively process and search in the child nodes of the node until the vector establishes a corresponding relationship with the child nodes of the quadtree QuadTree.
[0069] To determine whether a vector is inside a node, we need to determine whether the vector satisfies the condition of being inside the three half-spaces at the same time, that is, whether the directed distances from point p to the half-space set {h} are all positive.
[0070] Serialize the QuadTree to disk, specifically: represent the image by a pixel matrix, assuming its width is w and height is h; construct a mask of the same size according to the resolution of the image, with the data type being a 32-bit integer; determine the corresponding child node according to the above method, determine the child node number, and assign it to the corresponding pixel on the mask. At this time, any pixel on the mask records all relationship information from the top layer to the child node; finally, save the QuadTree and mask to disk in the form of a yaml file.
[0071] The image matching pixel retrieval method disclosed in the present invention proposes a serialization and deserialization method for a spherical quadtree, which can build an offline tree structure in advance when the image resolution is fixed, read the structure through deserialization when online, and accelerate the tree construction process through encoding mapping; the search for epipolar constraints is converted into a query based on the spherical quadtree, which greatly shortens the matching point query time.
[0072] Figure 3 A flowchart of candidate nodes in an embodiment of the image matching pixel retrieval method disclosed in the present invention is shown in FIG. Figure 3 The method shown includes steps: S301-S304. Each step is described below.
[0073] S301, determine the normal vector of the intersecting circle.
[0074] S302, obtain the center vector and circumscribed fillet radius corresponding to the spherical triangle of the node.
[0075] S303, calculate the complementary angle of the angle between the center vector and the normal vector.
[0076] S304, obtaining a node set intersecting with the intersecting circle, and selecting a node at the lowest level in the node set as a candidate node based on a search rule; the search rule includes: the complementary angle is greater than or equal to the circumscribed fillet radius, and the difference between the complementary angle and the circumscribed circle radius is less than a second search precision, etc.
[0077] Determine whether the coordinate information of the candidate node meets the first search accuracy. If so, use the corresponding first image pixel of the candidate node as the matching pixel. The first search accuracy includes that the angle between the spherical coordinates of the first image pixel corresponding to the candidate node and the intersecting circular plane is less than the angle threshold, etc.
[0078] In one embodiment, a pre-constructed QuadTree file is read into memory. Given the epipolar constraint between the second panoramic image and the first panoramic image, which can be expressed as a coplanarity condition for the spherical imaging model, that is, the three-dimensional coordinates of the matching pixel points of the second image pixels of the second panoramic image under the first panoramic image should fall on the intersection of the epipolar plane and the sphere (the sphere where the spherical quad-tree is located) of the first panoramic image, which is a great circle (intersecting circle) of the sphere, as Figure 4 shown.
[0079] The process of epipolar constraint query is to find the pixel coordinates corresponding to all points falling on the great circle. The epipolar constraint gives the normal vector n of the great circle. The unit vector v corresponding to any point on the great circle is perpendicular to the normal vector n, that is, <n, v> = 0. According to this determination rule, all pixels that meet the constraints can be found step by step on the QuadTree. The specific search algorithm is as follows: Calculate the circumscribed corner radius (Constain Radius) r of the spherical triangle of the node, and the central vector b (center vector) corresponding to the center of the circumscribed circle; calculate the complementary angle a of the angle between the vector b and the normal vector n, and determine whether there is an intersection of the node with the great circle. If a >= r and a - r is less than the specified search accuracy, or a < r, then this node meets the conditions, and all child nodes of the node are recursively executed for this determination until the child node is searched, and this child node is used as a candidate node.
[0080] Obtain the pixel coordinates included in all the searched child nodes, convert the pixel coordinates to spherical coordinates (bearing vector) p, and determine whether the angle between the pixel coordinates and the great circle plane meets the specified first search accuracy, and remove the pixels that do not meet the specified first search accuracy to obtain the final pixel coordinates that meet the epipolar constraint, which are the coordinates of the matching pixels.
[0081] Exemplary Devices
[0082] In one embodiment, as Figure 5 shown, the present disclosure provides an image matching pixel retrieval device, including: a quad-tree construction module 51, a pixel correspondence module 52, an epipolar constraint processing module 53, a candidate node determination module 54, and a pixel matching determination module 55.
[0083] The quad-tree construction module 51 performs triangulation processing on the sphere to construct a spherical quad-tree. The pixel correspondence module 52 numbers the nodes of the spherical quad-tree, and corresponds the first image pixels in the first panoramic image to the nodes and stores the spherical quad-tree. The epipolar constraint processing module 53 responds to a request for retrieving the matching pixel points between the second panoramic image and the first panoramic image, and determines the intersecting circle of the intersection of the epipolar plane corresponding to the second image pixels of the second panoramic image and the sphere based on the epipolar constraint between the second panoramic image and the first panoramic image.
[0084] The candidate node determination module 54 performs a search process in the spherical quadtree based on the intersection circle to obtain the candidate node corresponding to the second image pixel. If the coordinate information of the candidate node meets the search accuracy, the pixel matching determination module 55 uses the corresponding first image pixel of the candidate node as the matching pixel.
[0085] In one embodiment, the quadtree construction module 51 performs segmentation processing on the unit sphere to generate multiple spherical triangles. The quadtree construction module 51 performs recursive segmentation processing on each spherical triangle according to the tree depth information to construct a spherical quadtree; wherein the recursive segmentation processing includes connecting the midpoints of each side of the spherical triangle with geodesics so that the spherical triangle is segmented into four secondary spherical triangles, etc.
[0086] The quadtree construction module 51 constructs a spherical quadtree based on all spherical triangles, each of which is a node of the spherical quadtree. The quadtree construction module 51 determines the circumscribed circle corresponding to the spherical triangle of each node, and determines the center vector corresponding to the center of the circumscribed circle and the circumscribed fillet radius.
[0087] In one embodiment, the pixel correspondence module 52 sets a corresponding first serial number for the node at the top level of the spherical quadtree, and sets a second serial number corresponding to its parent node for the remaining nodes. The pixel correspondence module 52 numbers all nodes according to the numbering rule and based on the first serial number and / or the second serial number.
[0088] The pixel correspondence module 52 converts the pixel coordinates of the first image pixel into spherical coordinates. The pixel correspondence module 52 starts from the root node, determines whether the spherical coordinates are inside or outside the node in the spherical quadtree, and in response to the spherical coordinates being inside the node, determines whether the spherical coordinates are inside or outside the child node of the node, until it is determined that the spherical coordinates are inside the child node of the lowest level, and matches the child node with the pixel node corresponding to the first image pixel.
[0089] The pixel correspondence module 52 determines the spherical triangle corresponding to the node, and if the spherical coordinate is located within the spherical triangle, the spherical coordinate is determined to be inside the node. The pixel correspondence module 52 constructs a mask corresponding to the first image pixel, establishes and stores the correspondence between the number of the child node, the first image pixel, and the mask.
[0090] In one embodiment, the candidate node determination module 54 determines the normal vector of the intersecting circle. The candidate node determination module 54 obtains the center vector and the circumscribed fillet radius corresponding to the spherical triangle of the node, and calculates the complementary angle of the angle between the center vector and the normal vector. The candidate node determination module 54 obtains a set of nodes intersecting the intersecting circle, and selects the node at the lowest level in the node set as a candidate node based on a search rule; wherein the search rule includes that the complementary angle is greater than or equal to the circumscribed fillet radius, and the difference between the complementary angle and the circumscribed circle radius is less than the first search precision, etc.
[0091] Figure 6 is a structural diagram of an embodiment of the electronic device disclosed in the present invention, such as Figure 6 As shown, the electronic device 61 includes one or more processors 611 and a memory 612 .
[0092] The processor 611 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 61 to perform desired functions.
[0093] The memory 612 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory, for example, may include: random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory, for example, may include: read-only memory (ROM), hard disk, and flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 611 may execute the program instructions to implement the image matching pixel retrieval method of each embodiment of the present disclosure above and / or other desired functions. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.
[0094] In one example, the electronic device 61 may also include: an input device 613 and an output device 614, etc., which are interconnected through a bus system and / or other forms of connection mechanisms (not shown). In addition, the input device 613 may also include, for example, a keyboard, a mouse, etc. The output device 614 may output various information to the outside. The output device 614 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.
[0095] Of course, to simplify, Figure 6 Only some of the components related to the present disclosure in the electronic device 61 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 61 may also include any other appropriate components.
[0096] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the image matching pixel retrieval method according to various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification.
[0097] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0098] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enables the processor to execute the steps of the image matching pixel retrieval method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.
[0099] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive enumeration) of readable storage media can include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0100] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.
[0101] The image matching pixel retrieval method, device, electronic device, and storage medium in the above-mentioned embodiments solve the problem of large amount of computational complexity in searching for epipolar constraints of panoramic imaging models, and convert the search for epipolar constraints of spherical imaging models into a query based on a spherical quadtree, so that the matching process does not need to project and cut panoramic photos into slices, greatly shortening the matching point query time, improving computational efficiency, and allowing three-dimensional reconstruction to run in real time, effectively improving customer experience.
[0102] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0103] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present disclosure are intended only as illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, and configured in any manner. Words such as "include," "comprise," "have," and the like are open words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.
[0104] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
[0105] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.
[0106] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, etc., will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
[0107] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. An image matching pixel retrieval method, comprising: Triangulate the sphere to construct a spherical quadtree; Numbering the nodes of the spherical quadtree, correspondingly processing the first image pixels in the first panoramic image to the nodes and storing the spherical quadtree; In response to a request to retrieve matching pixels of a second panoramic image and the first panoramic image, determining an intersection circle where an epipolar plane corresponding to a second image pixel of the second panoramic image intersects the spherical surface based on an epipolar constraint between the second panoramic image and the first panoramic image; Performing a search process in the spherical quadtree based on the intersection circle to obtain a candidate node corresponding to the second image pixel; In response to the coordinate information of the candidate node satisfying a first search accuracy, a first image pixel corresponding to the candidate node is used as a matching pixel.
2. The method according to claim 1, wherein the triangulation process on the spherical surface comprises: The unit sphere is segmented to generate multiple spherical triangles; Recursively subdividing each of the spherical triangles according to the tree depth information to construct the spherical quadtree; The recursive subdivision process includes: connecting the midpoints of each side of the spherical triangle in pairs using geodesics, so that the spherical triangle is divided into four secondary spherical triangles.
3. The method of claim 2, wherein constructing the spherical quadtree comprises: Constructing the spherical quadtree based on all spherical triangles; wherein each spherical triangle is a node of the spherical quadtree; The circumscribed circle corresponding to the spherical triangle of each node is determined, and the center vector and the circumscribed fillet radius corresponding to the center of the circumscribed circle are determined.
4. The method according to claim 3, wherein the step of numbering the nodes of the spherical quadtree comprises: The nodes at the top level of the spherical quadtree are assigned a first serial number, and the remaining nodes are assigned a second serial number corresponding to their parent nodes; All nodes are numbered according to a numbering rule and based on the first serial number and / or the second serial number.
5. The method according to claim 3, wherein the processing of corresponding image pixels in the first panoramic image to the nodes comprises: converting pixel coordinates of pixels of the first image into spherical coordinates; Starting from the root node, determine whether the spherical coordinate is inside or outside the node in the spherical quadtree. In response to the spherical coordinate being inside the node, determine whether the spherical coordinate is inside or outside the child node of this node, until it is determined that the spherical coordinate is inside the child node of the lowest level, and then match this child node to the pixel node corresponding to the first image pixel.
6. The method of claim 5, wherein determining whether the spherical coordinate is inside or outside a node in the spherical quadtree comprises: Determining a spherical triangle corresponding to the node; If the spherical coordinate is within the spherical triangle, the spherical coordinate is determined to be inside the node.
7. The method of claim 5, further comprising: constructing a mask corresponding to pixels of the first image; A corresponding relationship between the number of the pixel node, the first image pixel and the mask is established and stored.
8. The method according to claim 3, wherein the searching process in the spherical quadtree based on the intersection circle to obtain the candidate node corresponding to the second image pixel comprises: Determining a normal vector of the intersecting circle; Obtain the center vector and the circumscribed fillet radius corresponding to the spherical triangle of the node; Calculate the complementary angle of the angle between the center vector and the normal vector; Acquire a node set intersecting with the intersection circle, and select a node at the lowest level in the node set as the candidate node based on a search rule; The search rule includes: the complementary angle is greater than or equal to the circumscribed fillet radius, and the difference between the complementary angle and the circumscribed circle radius is less than a second search accuracy.
9. An image matching pixel retrieval device, comprising: A quadtree construction module is used to triangulate the sphere to construct a spherical quadtree; A pixel correspondence module, used for numbering the nodes of the spherical quadtree, correspondingly processing the first image pixels in the first panoramic image with the nodes and storing the spherical quadtree; an epipolar constraint processing module, configured to, in response to a request to retrieve matching pixel points of a second panoramic image and the first panoramic image, determine an intersection circle where an epipolar plane corresponding to a second image pixel of the second panoramic image intersects the spherical surface based on the epipolar constraint between the second panoramic image and the first panoramic image; A candidate node determination module, configured to perform a search process in the spherical quadtree based on the intersection circle to obtain a candidate node corresponding to a pixel of the second image; The pixel matching determination module is used to take the first image pixel corresponding to the candidate node as the matching pixel in response to the coordinate information of the candidate node satisfying the first search accuracy.
10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 8.
