Method for aligning 2D images and floor plans based on 3D geometry and image alignment device using the same

Abstract

The present invention provides a method for aligning a 2D image and a floor plan based on 3D geometry, comprising: (a) acquiring a 2D image taken through a camera installed in a specific space; and an image alignment device aligning the 2D image and the floor plan based on a neural network geometry estimation model. (b) generating an image 3D coordinate system by extending the 2D image into a 3D space with reference to 3D coordinate values ​​for each pixel of the 2D image inferred through a 3D model, and generating a floor plan 3D coordinate system by extending the 2D image into 3D space with a plane of the floor plan for the specific space as a floor plane; (b) generating alignment information between the image 3D coordinate system and the floor plan 3D coordinate system with reference to plane correspondence information between a first plane recognized in the image 3D coordinate system and a second plane in the floor plan 3D coordinate system corresponding to the first plane, and line correspondence information between 3D points in the image 3D coordinate system and 2D points in the floor plan 3D coordinate system corresponding to the 3D points; and (c) performing alignment between the 2D image and the floor plan by projecting each of the pixels of the 2D image onto the floor plan with reference to the alignment information.

Description

[Technical Field]

[0001] The present invention relates to a method for aligning a 2D image and a floor plan based on 3D geometry, and more particularly to a technique for alignment between an image 3D coordinate system and a floor plan 3D coordinate system. [Background technology]

[0002] Image registration is a technique for aligning two or more images captured from different viewpoints, sensors, or observation conditions onto the same coordinate system. It plays an important role in various fields, including medical imaging, satellite imagery, augmented reality, and robot vision. While there are several methods for achieving image registration, the most commonly used method is homography, which describes the relationship between points on a single plane. Initially, feature-based methods such as the Harris corner detector, SIFT, and SURF were mainstream, and direct registration methods utilizing correlation and optical flow were also being researched. However, these conventional methods had problems with their vulnerability to changes in lighting, viewpoint differences, and subject deformation.

[0003] Among these, the most commonly used homography method uses a matrix that represents the correspondence between points on a single plane in space between two different images. This method can explain how structures on the same plane move, rotate, and distort even when the camera's viewpoint changes, and has been widely used in various computer vision fields, including image matching, panorama generation, and image distortion correction. Homography can be estimated if sufficient corresponding points between images are secured, but it is limited in that it cannot model general 3D changes and is only effective for planar structures. This leads to the problem of inaccurate matching results when dealing with non-planar scenes or when viewpoint differences are large.

[0004] In addition, because homography methods perform 2D transformations in projection space, they are unable to reflect depth differences or structural information in the scene at all, and in non-planar scenes, only partial matching is possible or distortions may occur. Furthermore, in complex scenes with various elements such as multi-layered structures, curved surfaces, and dynamic objects, it is impossible to simultaneously match all areas using homography, and in such cases, multiple local matching methods are usually required. However, there is a problem that inconsistencies can occur at boundary areas, making it impossible to apply and manage local optimization that takes such areas into account in practical applications. Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to solve all of the problems of the prior art described above.

[0006] Another object of the present invention is to overcome the limitations of planarity by representing the matching between two spaces based on three-dimensional geometry, thereby enabling effective utilization of matching between images even in scenes with more complex and diverse structures.

[0007] Another object of the present invention is to utilize a three-dimensional space to measure the size of an arbitrary object on a two-dimensional image in the three-dimensional space and minimize errors.

[0008] Another object of the present invention is to determine the identity of an object by utilizing an object tracking algorithm, etc., when a specific object is partially out of the field of view of the camera corresponding to the image or is hidden behind an object, and to accurately estimate its position by utilizing size information in three-dimensional space. [Means for solving the problem]

[0009] According to one embodiment of the present invention, a method for aligning a 2D image and a floor plan based on 3D geometry includes: (a) acquiring a 2D image captured through a camera installed in a specific space; and (b) aligning the 2D image and the floor plan based on 3D geometry using a neural network-based geometry estimation model. (b) generating an image 3D coordinate system by extending the 2D image into a 3D space with reference to 3D coordinate values ​​for each pixel of the 2D image inferred through a 3D model, and generating a floor plan 3D coordinate system by extending the 2D image into 3D space with a plane of the floor plan for the specific space as a floor plane; (b) generating alignment information between the image 3D coordinate system and the floor plan 3D coordinate system with reference to plane correspondence information between a first plane recognized in the image 3D coordinate system and a second plane in the floor plan 3D coordinate system corresponding to the first plane, and line correspondence information between 3D points in the image 3D coordinate system and 2D points in the floor plan 3D coordinate system corresponding to the 3D points; and (c) performing alignment between the 2D image and the floor plan by projecting each of the pixels of the 2D image onto the floor plan with reference to the alignment information.

[0010] In one example, in step (b), the image alignment device acquires, as the plane correspondence information, the 1_1 plane normal vector and 1_1 plane constant corresponding to the 1_1 plane included in the first plane, the 1_n plane normal vector and 1_n plane constant corresponding to the 1_1 plane through the 1_n plane, and the 2_1 plane normal vector and 2_1 plane constant corresponding to the 2_1 plane included in the second plane, the 2_n plane normal vector and 2_n plane constant corresponding to the 2_1 plane through the 2_n plane, as the line correspondence information, and acquires, as the line correspondence information, each of the perpendiculars of the 2D points in the plan view 3D coordinate system (the perpendiculars are lines perpendicular to the floor plane in the plan view 3D coordinate system) and each of the corresponding 3D points in the image 3D coordinate system.

[0011] In one example, in step (b), when generating the matching information between the image 3D coordinate system and the plan view 3D coordinate system, the image matching device refers to the plane correspondence information and the line correspondence information to estimate a rotation matrix between the image 3D coordinate system and the plan view 3D coordinate system, then refers to the rotation matrix to estimate a size and a movement, and refers to the size and the movement to generate a transformation matrix, thereby generating the matching information between the image 3D coordinate system and the plan view 3D coordinate system, and is characterized in that the accuracy of the matching information (the accuracy refers to the degree of agreement between the transformation result obtained by transforming the 3D points in the image 3D coordinate system into the plan view 3D coordinate system with reference to the matching information) is confirmed by calculating a residual of the matching information.

[0012] In one example, in the step (b), while referring to the number of plane pairs of corresponding planes for each of the 1_1 plane to the 1_n plane included in the first plane and each of the 2_1 plane to the 2_n plane included in the second plane, the image alignment device (i) calculates a 1_1 plane normal vector to a 1_n plane normal vector as a first plane normal vector for each of the 1_1 plane to the 1_n plane, calculates a 2_1 plane normal vector to a 2_n plane normal vector as a second plane normal vector for each of the 2_1 plane to the 2_n plane, and calculates the 1_1 plane normal vector to the 1_n plane normal vector. and (ii) if the number of plane pairs is less than the predetermined threshold, a subprocess of calculating a first floor normal vector of a first floor plane that is a floor plane in the image 3D coordinate system and a second floor normal vector of a second floor plane that is a floor plane in the plan view 3D coordinate system corresponding thereto, calculating a second degree of agreement for the first floor normal vector and the corresponding second floor normal vector, and estimating a second rotation matrix as the rotation matrix that maximizes the second degree of agreement.

[0013] In one example, in step (i) of step (b), the image matching device defines a correlation matrix between pairs of each of the 1_1 plane normal vectors through the 1_n plane normal vectors and their corresponding pairs of the 2_1 plane normal vectors through the 2_n plane normal vectors, performs SVD (Singular Value Decomposition) by referring to the correlation matrix, and calculates the first rotation matrix as an optimal solution for which the first degree of agreement satisfies a predetermined condition, and at that time, it repeatedly performs the SVD by brute force to extract the optimal solution for the combinations of signs of each of the 1_1 plane normal vectors through the 1_n plane normal vectors and their corresponding pairs of the 2_1 plane normal vectors through the 2_n plane normal vectors.

[0014] In one example, in (ii) of step (b), the image alignment device refers to the first floor normal vector and the second floor normal vector, and calculates an initial rotation matrix as the rotation matrix so that the directional difference between the first floor normal vector and the second floor normal vector is minimized, generates projected coordinates by projecting the 3D coordinate values ​​of the 3D point in the image 3D coordinate system rotated by the initial rotation matrix onto 2D coordinate values ​​of the second floor plane in the plan view 3D coordinate system, calculates a 2D rotation matrix as a rotation matrix within the second floor plane by referring to the line correspondence information and the 2D UMEYAMA model between the projected coordinates and the 2D coordinate values ​​of the 2D point in the plan view 3D coordinate system, expands the 2D rotation matrix to 3D to calculate a 3D rotation matrix, and calculates the second rotation matrix as a final rotation matrix by combining the initial rotation matrix and the 3D rotation matrix.

[0015] In one example, in step (b), after estimating the rotation matrix, the image alignment device generates the transformation matrix by referring to the plane correspondence information and the line correspondence information. Since a plane equation for making the first plane correspond to the second plane must be maintained even after performing the alignment between the 2D image and the floor plan, the image alignment device generates a plane constraint equation and a line constraint equation for matching the image 3D coordinate system with the 2D points of the floor plan 3D coordinate system through the rotation matrix. The size and the movement are estimated by a linear least squares method by referring to the plane constraint equation and the line constraint equation, and the transformation matrix between the image 3D coordinate system and the floor plan 3D coordinate system is generated.

[0016] In one example, in step (c), when projecting each of the pixels of the 2D image onto the floor plan with reference to the alignment information, the image alignment device calculates each of the normalized unit direction vectors corresponding to each of the pixels of the 2D image, and converts the camera center coordinates of the camera and each of the unit direction vectors of each of the pixels into the floor plan 3D coordinate system with reference to the alignment information, and projects them onto the floor plan, and at this time, calculates each of the tangent points with each of the unit direction vectors with reference to height information of each of the pixels of the 2D image, and determines a position on the floor plan 3D coordinate system to project onto the floor plan.

[0017] In one example, when a predetermined height cannot be calculated for a predetermined object on the plan view 3D coordinate system, the image alignment device acquires the height of the predetermined object on the plan view 3D coordinate system by referring to each of the other pixels of other objects other than the predetermined object on the 2D image, and at that time, acquires the predetermined height of the predetermined object on the plan view 3D coordinate system by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to the floor plane of the other object among the other pixels on the 2D image and a unit direction vector of a camera ray of a top pixel located at the top of the other object, or when the height of the predetermined object is higher than that of the other object, acquires the predetermined height by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to the floor plane of the other object and a unit direction vector of a camera ray of a top pixel located at the top of the predetermined object.

[0018] In one example, when calculating each of the normalized unit direction vectors corresponding to each pixel of the 2D image, the image alignment device refers to the ratio between the resolution of the camera and the resolution of the 2D image to correct the scale of the image 3D coordinate system and the principal point of the camera, and calculates each of the unit direction vectors.

[0019] According to another embodiment of the present invention, an image alignment device for aligning a 2D image and a plan view based on 3D geometry includes at least one memory for storing instructions; and at least one processor configured to execute the instructions, wherein the processor (I) calculates a 2D image based on a neural network geometry estimation model when the 2D image is acquired through a camera installed in a specific space. and (III) a process of performing alignment between the 2D image and the floor plan by projecting each of the pixels of the 2D image onto the floor plan with reference to the alignment information.

[0020] In one example, in the (II) process, the processor acquires, as the plane correspondence information, the 1_1 plane normal vector and the 1_1 plane constant corresponding to the 1_1 plane included in the first plane, the 1_n plane normal vector and the 1_n plane constant corresponding to the 1_1 plane through the 1_n plane, and the 2_1 plane normal vector and the 2_1 plane constant corresponding to the 2_1 plane included in the second plane, and the 2_n plane normal vector and the 2_n plane constant corresponding to the 2_1 plane through the 2_n plane, as the line correspondence information, and acquires, as the line correspondence information, each of the perpendicular lines (the perpendicular lines are lines perpendicular to the floor plane in the plan view 3D coordinate system) of the 2D points and each of the corresponding 3D points in the image 3D coordinate system.

[0021] In one example, in the process (II), when generating matching information between the image 3D coordinate system and the plan view 3D coordinate system, the processor refers to the plane correspondence information and the line correspondence information to estimate a rotation matrix between the image 3D coordinate system and the plan view 3D coordinate system, then estimates a size and a movement by referring to the rotation matrix, and generates a transformation matrix by referring to the size and the movement, thereby generating the matching information between the image 3D coordinate system and the plan view 3D coordinate system, and is characterized in that the accuracy of the matching information is confirmed by calculating a residual of the matching information (the accuracy refers to the degree of agreement between the transformation result obtained by transforming the 3D points in the image 3D coordinate system into the plan view 3D coordinate system by referring to the matching information and the corresponding 2D points in the plan view 3D coordinate system).

[0022] In one example, in the process (II), while referring to the number of plane pairs of corresponding planes for each of the 1_1 plane to the 1_n plane included in the first plane and each of the 2_1 plane to the 2_n plane included in the second plane, the processor (i) calculates a 1_1 plane normal vector to a 1_n plane normal vector as a first plane normal vector for each of the 1_1 plane to the 1_n plane, and calculates a 2_1 plane normal vector to a 2_n plane normal vector as a second plane normal vector for each of the 2_1 plane to the 2_n plane, and calculates the 1_1 plane normal vector to the 1_n plane normal vector and (ii) if the number of plane pairs is less than the predetermined threshold, a subprocess of calculating a first floor normal vector of a first floor plane which is a floor plane in the image 3D coordinate system and a second floor normal vector of a second floor plane which is a floor plane in the plan view 3D coordinate system corresponding thereto, calculating a second degree of agreement for the first floor normal vector and the corresponding second floor normal vector, and estimating a second rotation matrix as the rotation matrix which maximizes the second degree of agreement.

[0023] In one example, in (i) of the (II) process, the processor defines a correlation matrix between each pair of the 1_1 plane normal vector through the 1_n plane normal vector and each pair of the corresponding 2_1 plane normal vector through the 2_n plane normal vector, and performs SVD (Singular Value Decomposition) by referring to the correlation matrix to calculate the first rotation matrix as an optimal solution for which the first degree of similarity satisfies a predetermined condition, and at that time, the SVD is repeatedly performed by brute force to extract the optimal solution for the combination of signs of each of the 1_1 plane normal vector through the 1_n plane normal vector and each of the corresponding 2_1 plane normal vector through the 2_n plane normal vector.

[0024] In one example, in (ii) of the (II) process, the processor refers to the first floor normal vector and the second floor normal vector, and calculates an initial rotation matrix as the rotation matrix so that the directional difference between the first floor normal vector and the second floor normal vector is minimized, generates projected coordinates by projecting the 3D coordinate values ​​of the 3D point in the image 3D coordinate system rotated by the initial rotation matrix onto 2D coordinate values ​​of the second floor plane in the plan view 3D coordinate system, calculates a 2D rotation matrix as a rotation matrix within the second floor plane between the projected coordinates and the 2D coordinate values ​​of the 2D point in the plan view 3D coordinate system by referring to the line correspondence information and the 2D UMEYAMA model, expands the 2D rotation matrix to 3D to calculate a 3D rotation matrix, and calculates the second rotation matrix as a final rotation matrix by combining the initial rotation matrix and the 3D rotation matrix.

[0025] In one example, in the process (II), after estimating the rotation matrix, when generating the transformation matrix by referring to the plane correspondence information and the line correspondence information, the processor generates a plane constraint equation because a plane equation for making the first plane correspond to the second plane must be maintained even after performing the alignment between the 2D image and the floor plan, and generates a line constraint equation for matching the image 3D coordinate system with the 2D points of the floor plan 3D coordinate system through the rotation matrix, and estimates the size and the movement by a linear least squares method by referring to the plane constraint equation and the line constraint equation, thereby generating the transformation matrix between the image 3D coordinate system and the floor plan 3D coordinate system.

[0026] In one example, in the process (III), when projecting each of the pixels of the 2D image onto the floor plan by referring to the alignment information, the processor calculates each of the normalized unit direction vectors corresponding to each of the pixels of the 2D image, and converts the camera center coordinates of the camera and each of the unit direction vectors of each of the pixels into the floor plan 3D coordinate system by referring to the alignment information, and projects them onto the floor plan, and at that time, calculates each of the tangent points with each of the unit direction vectors by referring to each height information of each of the pixels of the 2D image, and determines a position on the floor plan 3D coordinate system to project onto the floor plan.

[0027] In one example, when a predetermined height cannot be calculated for a predetermined object on the plan view 3D coordinate system, the processor acquires the height of the predetermined object on the plan view 3D coordinate system by referring to each of the other pixels of other objects other than the predetermined object on the 2D image, and at that time, acquires the predetermined height of the predetermined object on the plan view 3D coordinate system by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to the floor plane of the other object among the other pixels on the 2D image and a unit direction vector of a camera ray of a top pixel located at the top of the other object, or when the height of the predetermined object is higher than that of the other objects, acquires the predetermined height by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to the floor plane of the other object and a unit direction vector of a camera ray of a top pixel located at the top of the predetermined object.

[0028] In one example, when calculating each of the normalized unit direction vectors corresponding to each pixel of the 2D image, the processor refers to the ratio between the resolution of the camera and the resolution of the 2D image to correct the scale of the image 3D coordinate system and the principal point of the camera, and calculates each of the unit direction vectors. [Effects of the Invention]

[0029] According to the present invention, by expressing the matching between two spaces based on three-dimensional geometry, the limitations of planarity can be overcome, and it is possible to effectively utilize the matching between images even in scenes with more complex and diverse structures.

[0030] According to the present invention, there is another effect that the size of an arbitrary object on a 2D image is measured in three-dimensional space by utilizing three-dimensional space, thereby minimizing errors.

[0031] In addition, if a specific object is partially out of the field of view of the camera corresponding to the image or is hidden behind an object, the object's identity can be determined using an object tracking algorithm, and its position can be accurately estimated using its size information in three-dimensional space. [Brief explanation of the drawings]

[0032] The following drawings attached for use in explaining embodiments of the present invention are only a part of the embodiments of the present invention, and a person having ordinary knowledge in the technical field to which the present invention pertains (hereinafter referred to as "ordinary engineer") can derive other drawings based on these drawings without performing any inventive work.

[0033] [Figure 1] 1 is a schematic diagram illustrating an image alignment apparatus for aligning a 2D image and a plan view based on 3D geometry according to an embodiment of the present invention; [Figure 2] FIG. 1 is a diagram illustrating a method for performing camera calibration through AI-based geometric estimation according to an embodiment of the present invention. [Figure 3] 10A and 10B are diagrams illustrating a process of aligning a 2D image and a floor plan based on 3D geometry according to an embodiment of the present invention. [Figure 4a] 10A to 10D are diagrams illustrating an example of a process for generating an image 3D coordinate system by extending a 2D image into a 3D space according to an embodiment of the present invention. [Figure 4b] 10A to 10D are diagrams illustrating an example of a process for generating a plan view 3D coordinate system by expanding a plan view into 3D according to an embodiment of the present invention. [Figure 5] 10A and 10B are diagrams illustrating an example of a process for confirming a correspondence relationship between a first plane recognized in an image 3D coordinate system and a second plane recognized in a plan view 3D coordinate system according to an embodiment of the present invention. [Figure 6] 1A to 1C are diagrams illustrating an example of a process for projecting each pixel of a 2D image onto a plan view according to an embodiment of the present invention. [Figure 7]FIG. 10 is a diagram illustrating an example of a process for calculating each of the contact points intersecting with each of the unit direction vectors by referring to each of the height information of each of the pixels of a 2D image according to one embodiment of the present invention, determining a position on a plan view 3D coordinate system, and projecting the result onto the plan view. DETAILED DESCRIPTION OF THE INVENTION

[0034] The following detailed description of the present invention refers to the accompanying drawings, which show, by way of example, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable one of ordinary skill in the art to practice the invention. It should be understood that although the various embodiments of the present invention are different from one another, they are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be varied and implemented from one embodiment to another without departing from the spirit and scope of the invention. It should also be understood that the location or arrangement of individual components within each embodiment may be changed without departing from the spirit and scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention should be understood to encompass the scope of the appended claims and all equivalents thereto. In the drawings, like reference numerals indicate the same or similar components throughout the various aspects.

[0035] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following, a number of preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily practice the present invention.

[0036] FIG. 1 is a schematic diagram illustrating an image alignment apparatus for aligning a 2D image and a plan view based on 3D geometry according to one embodiment.

[0037] As shown in FIG. 1, the image matching device 100 of the present invention may include a memory 110 and a processor 120, and may further include a neural network-based geometric estimation model 130, a planar region segment model (not shown), etc. Alternatively, the neural network-based geometric estimation model 130, the planar region segment model (not shown), etc. may be provided separately from the image matching device 100.

[0038] Here, memory 110 may store instructions, specifically code generated to cause image registration device 100 to function in a particular manner, for execution by processor 120. The instructions may be stored in computer-usable or computer-readable memory for a computer or other programmable data processing device. The instructions may cause processes to perform the functions described in this specification of the present invention.

[0039] The processor 120 may include hardware components such as an MPU (Micro Processing Unit) or a CPU (Central Processing Unit), cache memory, a data bus, etc. The image registration device 100 may also include software components such as an operating system and applications that perform specific purposes.

[0040] However, this does not exclude the case where processor 120 of image registration device 100 includes an integrated processor in which a medium, processor, and memory for implementing the present invention are integrated.

[0041] Here, the geometric estimation model 130 based on a neural network, the planar region segment model (not shown), and the like may be in a pre-trained state.

[0042] Furthermore, the geometric estimation model 130 based on a neural network and the planar region segment model (not shown) may be any conventionally known model, but are not limited to this.

[0043] The image matching device 100 can also receive a 2D image 10, which is an image of a specific space captured by a single camera.

[0044] In this case, when the image matching device 100 acquires a 2D image 10 captured by a camera installed in a specific space, it can generate an image 3D coordinate system that extends the 2D image 10 into a 3D space by referring to the 3D coordinate values ​​for each pixel of the 2D image 10 inferred through a neural network-based geometry estimation model 130.

[0045] Hereinafter, a method and an execution process using the image registration device 100 according to an embodiment of the present invention will be described with reference to FIGS. 2 and 3. FIG.

[0046] FIG. 2 is a diagram illustrating a method for performing camera calibration through AI-based geometric estimation according to one embodiment of the present invention, and FIG. 3 is a diagram illustrating a process for aligning a 2D image and a plan view based on 3D geometry according to one embodiment of the present invention.

[0047] 2 and 3, when a 2D image 10 is acquired by a camera installed in a specific space, the image alignment device 100 can generate an image 3D coordinate system 11 by extending the 2D image 10 into 3D space by referring to the 3D coordinate values ​​for each pixel of the 2D image 10 inferred through the neural network-based geometry estimation model 130, and can also generate a plan view 3D coordinate system 21 by extending the plane of a floor plan 20 for the specific space into 3D as a floor plane (S100). For more detailed explanation, a description will be given with reference to FIGS. 4a and 4b.

[0048] First, FIG. 4a is a diagram illustrating an example of a process for generating an image 3D coordinate system by extending a 2D image into a 3D space according to an embodiment of the present invention.

[0049] Referring to FIG. 4a, a 2D image 10 can be input into a neural network-based geometric estimation model 130 to generate an image 3D coordinate system 11 that extends the 2D image 10 into 3D space as a camera coordinate system, thereby obtaining camera position information of the camera that captured a specific space.

[0050] FIG. 4b is a diagram illustrating an example of a process for generating a plan view 3D coordinate system by expanding a plan view into 3D according to an embodiment of the present invention.

[0051] Referring to Figure 4b, a plan view 3D coordinate system 21 can be generated by extending the plane of the plan view 20 for a particular space into 3D as the floor plane, which is a diagram illustrating the extension into geometric space by using a right-handed coordinate system with the plane of the 2D plan view 20 as the floor plane (Z = 0 plane).

[0052] 2 and 3, the process of aligning a 2D image and a floor plan based on 3D geometry will be described. With reference to the generated image 3D coordinate system 11 and floor plan 3D coordinate system 21, the image alignment device 100 can generate (S200) alignment information 160 between the image 3D coordinate system 11 and the floor plan 3D coordinate system 21 by referring to plane correspondence information 140 between a first plane recognized in the image 3D coordinate system 11 and a second plane in the floor plan 3D coordinate system 21 corresponding to the first plane, and line correspondence information 150 between 3D points in the image 3D coordinate system 11 and corresponding 2D points in the floor plan 3D coordinate system 21. Here, the plane correspondence information 140 and the line correspondence information 150 will be described below with reference to FIG. 5.

[0053] FIG. 5 is a diagram illustrating an example of a process for confirming a correspondence relationship between a first plane recognized in an image 3D coordinate system and a second plane recognized in a plan view 3D coordinate system according to an embodiment of the present invention.

[0054] 5, the image alignment device 100 can segment the pixels in the image 3D coordinate system 11 to calculate multiple planes, and generate a plane perpendicular to the vertical plane as a first plane. For example, the image 3D coordinate system 11 can be segmented to divide the pixels into a floor plane, a small table area, a large table area, etc. in FIG. 5 as the first plane.

[0055] Each of the first planes thus generated can be represented by a 1_1 plane normal vector and a 1_1 plane constant corresponding to the 1_1 plane through a 1_n plane normal vector and a 1_n plane constant corresponding to the 1_n plane, where n is an integer equal to or greater than 1.

[0056] Furthermore, in the plan view 3D coordinate system 21, second planes can be generated as planes perpendicular to the vertical plane, and each of the second planes thus generated can be represented by a 2_1 plane normal vector corresponding to the 2_1 plane and a 2_1 plane constant through a 2_n plane normal vector and a 2_n plane constant corresponding to the 2_n plane.

[0057] The 1_1 plane normal vector and 1_1 plane constant corresponding to the 1_1 plane, the 1_n plane normal vector and 1_n plane constant corresponding to the 1_1 through 1_n planes, and the 2_1 plane normal vector and 2_1 plane constant corresponding to the 2_1 plane, the 2_n plane normal vector and 2_n plane constant corresponding to the 2_n planes, can be acquired as plane correspondence information 150. At this time, information related to plane correspondence, such as the number of plane correspondences between the image 3D coordinate system 11 and the plan view 3D coordinate system 21, can be acquired through the plane correspondence information 150. Here, the number of plane correspondences may be the number of pairs of each first plane and each corresponding second plane.

[0058] Here, the 1_1 plane normal vector to the 1_n plane normal vector of the image 3D coordinate system 11 are expressed as follows: JPEG0007813503000002.jpg1018, and the 1_1th plane constant to the 1_nth plane constant are JPEG0007813503000003.jpg1220. In this case, C is the camera coordinate system and can represent the image 3D coordinate system 11. Similarly, the 2_1 plane normal vector to the 2_n plane normal vector of the plan view 3D coordinate system 21 can be expressed as follows: JPEG0007813503000004.jpg1022, and the 2_1 plane constants to the 2_n plane constants are JPEG0007813503000005.jpg922. In this case, B can represent the plan view 3D coordinate system 21 as the plan view coordinate system.

[0059] 5, line correspondence information 150 is not shown, but to explain, each perpendicular line of each 2D point in the plan view 3D coordinate system 21 and each corresponding 3D point in the image 3D coordinate system 11 can be acquired as line correspondence information 150. In this case, the perpendicular line can refer to a line perpendicular to the floor plane in the plan view 3D coordinate system 21.

[0060] To explain in more detail, the line correspondence information 150 may be information defined so that a correspondence relationship is established between each perpendicular line of each 2D point in the plan view 3D coordinate system 21 and each line connecting the 3D point in the image 3D coordinate system 11 perpendicularly to the floor plane.

[0061] Furthermore, the information may be defined by obtaining plan view 3D coordinates on each perpendicular line of each 2D point in the plan view 3D coordinate system 21, so that a correspondence relationship is established between each 3D point in the image 3D coordinate system 11 and each of the plan view 3D coordinates.

[0062] Therefore, in the present invention, the perpendicular lines of the 2D points in the plan view 3D coordinate system 21 and the corresponding 3D points in the image 3D coordinate system 11 are acquired as line correspondence information 150, which will be described below.

[0063] Here, the line correspondence information 150 is a perpendicular line on the plan view 3D coordinate system 21. JPEG0007813503000006.jpg1240 and a 3D point on the image 3D coordinate system 11 JPEG0007813503000007.jpg1017, where (u j ,v j ) may be the number of each of the coordinate values ​​of the 2D points in the plan view 3D coordinate system 21, and M may be the number of each of the perpendiculars of each of the 2D points.

[0064] Also, if the plane correspondence information 140 is insufficient and at least one line correspondence is provided, a stable matching result can be obtained with only two plane correspondences.

[0065] By referring to the plane correspondence information 140 and line correspondence information 150 obtained above, the image alignment device 100 generates alignment information 160 between the image 3D coordinate system 11 and the plan view 3D coordinate system 21 by estimating a rotation matrix between the image 3D coordinate system 11 and the plan view 3D coordinate system 21, estimating size and movement by referring to the rotation matrix, and generating a transformation matrix by referring to the size and movement, thereby generating alignment information 160 between the image 3D coordinate system 11 and the plan view 3D coordinate system 21.

[0066] Here, the rotation matrix R is a matrix that maximizes the degree of correspondence between each of the 1_1 plane normal vectors through the 1_n plane normal vectors of the first plane and each of the 2_1 plane normal vectors through the 2_n plane normal vectors of the corresponding second planes, It can be expressed as the formula JPEG0007813503000008.jpg1036, where N is the number of corresponding planes, JPEG0007813503000009.jpg79 is the normal vector of the plan view 3D coordinate system 21, JPEG0007813503000010.jpg77 is the normal vector of the image 3D coordinate system 11, and a rotation matrix R can be found that maximizes the sum of the dot products between pairs of normal vectors of the image 3D coordinate system 11 and the plan view 3D coordinate system 21.

[0067] At this time, in order to find a solution for the rotation matrix R, it may be necessary to calculate and confirm the accuracy of the matching information 160 by calculating the residual of the matching information 160. Here, the accuracy may refer to the degree of evaluation of the degree of agreement between the transformation result, in which 3D points in the image 3D coordinate system 11 are transformed into the plan view 3D coordinate system 21 with reference to the matching information 160, and the corresponding 2D points in the plan view 3D coordinate system 21.

[0068] As a result, when generating the matching information 160 between the image 3D coordinate system 11 and the plan view 3D coordinate system 21, the image matching device 100 refers to the number of plane pairs of corresponding planes, each of the 1_1 plane to the 1_n plane included in the first plane and each of the 2_1 plane to the 2_n plane included in the second plane, and when the number of plane pairs is equal to or greater than a preset threshold, the image matching device 100 uses the 1_1 plane normal vector to the 1_n plane normal vector as the normal vector of the first plane for each of the 1_1 plane to the 1_n plane. It is possible to calculate each of the vectors of the second planes for each of the 2_1 planes through 2_n planes, and it is possible to calculate each of the 2_1 plane normal vectors through 2_n plane normal vectors as the normal vectors of the second planes for each of the 2_1 planes through 2_n planes, and it is possible to calculate a first degree of agreement for each of the 1_1 plane normal vectors through 1_n plane normal vectors and for each of the 2_1 plane normal vectors through 2_n plane normal vectors corresponding thereto, and to estimate a first rotation matrix as a rotation matrix that maximizes the first degree of agreement.

[0069] For example, if the number of corresponding plane pairs between the first plane and the second plane is three or more, it is possible to calculate the degree of correspondence between the 1_1 plane normal vector through the 1_3 plane normal vector of each of the 1_1 plane through the 1_3 plane included in the first plane, and the 2_1 plane normal vector through the 2_3 plane normal vector of each of the 2_1 plane through the 2_3 plane included in the second plane, and calculate a rotation matrix by referring to the degree of correspondence for each of the three pairs of planes, thereby enabling a more accurate value of the rotation matrix to be estimated. Therefore, the greater the number of corresponding planes, the more accurate the rotation matrix can be estimated.

[0070] Next, the image matching device 100 calculates a correlation matrix between each pair of the 1_1 plane normal vector through the 1_n plane normal vector and each pair of the corresponding 2_1 plane normal vector through the 2_n plane normal vector as follows: JPEG0007813503000011.jpg941, and by repeatedly performing SVD (Singular Value Decomposition) with reference to the correlation matrix H, the first rotation matrix is ​​obtained as the optimal solution that satisfies the predetermined conditions. JPEG0007813503000012.jpg1198 can be calculated, where U and V may be orthogonal matrices calculated through the correlation matrix of the corresponding vector pair.

[0071] In addition, the combination of signs of each of the 1_1 plane normal vectors through the 1_n plane normal vectors and the corresponding 2_1 plane normal vectors through the 2_n plane normal vectors can be extracted by repeatedly performing SVD by brute force, and the optimal solution can be extracted by the following equation. It is possible to select the optimal solution for the rotation matrix that satisfies JPEG0007813503000013.jpg736.

[0072] In addition, when generating the matching information 160 between the image 3D coordinate system 11 and the plan view 3D coordinate system 21, the image matching device 100 refers to the number of plane pairs of corresponding planes for each of the 1_1 plane to 1_n plane included in the first plane and each of the 2_1 plane to 2_n plane included in the second plane, and if the number of plane pairs is less than a predetermined threshold, it calculates each of the first floor normal vector of the first floor plane, which is the floor plane in the image 3D coordinate system 11, and the second floor normal vector of the second floor plane, which is the floor plane corresponding to it in the plan view 3D coordinate system 21, calculates a second degree of matching for the first floor normal vector and the corresponding second floor normal vector, and estimates a second rotation matrix that maximizes the second degree of matching.

[0073] For example, if the number of plane pairs of the first plane and the second plane is less than three, the minimum rotation matrix can be calculated by referring to the degree of correspondence between the first floor normal vector and the second floor normal vector of each of the first floor plane and the second floor plane, which are one of the 1_1 plane and the 1_2 plane included in the first plane, and one of the 2_1 plane and the 2_2 plane included in the second plane.

[0074] Here, the image alignment device 100 refers to the first floor normal vector and the second floor normal vector, calculates an initial rotation matrix R0 as a rotation matrix so that the directional difference between the first floor normal vector and the second floor normal vector is minimized, and then refers to the line correspondence information 150 to generate projected coordinates by projecting 3D coordinate values ​​as 3D points in the image 3D coordinate system 11 rotated by the initial rotation matrix R0 onto 2D coordinate values ​​of the second floor plane in the plan view 3D coordinate system 21, and then refers to the line correspondence information and the 2D UMEYAMA model to generate a 2D rotation matrix as a rotation matrix within the second floor plane between the projected coordinates and the 2D coordinate values ​​that are 2D points in the plan view 3D coordinate system. JPEG0007813503000014.jpg966 can be calculated, where 3D point in the image 3D coordinate system 11 rotated by the minimum rotation matrix R JPEG0007813503000015.jpg711 as a line on the plan view 3D coordinate system 21 The sum of squares of the errors is closest to JPEG0007813503000016.jpg87 It is possible to find the rotation matrix that minimizes JPEG0007813503000017.jpg941.

[0075] Then we extend the 2D rotation matrix to 3D and write it as a 3D rotation matrix JPEG0007813503000018.jpg1028 is calculated, and the initial rotation matrix R0 is combined with the 3D rotation matrix to obtain the second rotation matrix R = R 3D R0 can be calculated.

[0076] Thereafter, after calculating the first rotation matrix or the second rotation matrix as the rotation matrix, a transformation matrix can be generated by referring to the plane correspondence information 140 and the line correspondence information 150.

[0077] First, even after the image alignment device 100 aligns the image 3D coordinate system 11 of the 2D image 10 with the plan view 3D coordinate system 21 of the plan view 20, the plane equation for making the first plane correspond to the second plane must be maintained. Therefore, the plane constraint equation JPEG0007813503000019.jpg841 can be generated, where t is the motion vector, s is the magnitude scale, JPEG0007813503000020.jpg810 is the normal vector of the plan view 3D coordinate system 21, JPEG0007813503000021.jpg88 is the plane constant of the image 3D coordinate system 11, JPEG0007813503000022.jpg812 may be a plane constant of the plan view 3D coordinate system 21.

[0078] In addition, as a line constraint equation for matching the image 3D coordinate system 11 with the 2D points of the plan view 3D coordinate system 21 through a rotation matrix, JPEG0007813503000023.jpg976 can be defined. Here, the position after rotation of the image 3D coordinate system 11 with respect to the line correspondence j between the image 3D coordinate system 11 and the plan view 3D coordinate system 21 can be defined as Since JPEG0007813503000024.jpg820 must match a point on the floor plan, a line constraint equation can be generated that states that the x coordinate of the original image 3D coordinate system 11 must match a point on the floor plan after size and translation transformation, and a line constraint equation can also be generated that states that the y coordinate of the original image 3D coordinate system 11 must match a point on the floor plan after size and translation transformation, similar to the x coordinate.

[0079] Therefore, A[t; s]=b can be generated by referring to the plane constraint equation and the line constraint equation, and its magnitude and translation can be calculated by the linear least squares method. JPEG0007813503000025.jpg943. In this case, A[t; s] is a matrix formed by combining N plane constraint equations and 2M line constraint equations, where A is a coefficient matrix combining the plane constraint equations and the line constraint equations, and [t; s] may be the translation and scale variables to be calculated. b may be the distance between the image 3D coordinate system 11 and the plan view 3D coordinate system.

[0080] With reference to the size and translation estimated above, the transformation matrix between the image 3D coordinate system 11 and the plan view 3D coordinate system 21 JPEG0007813503000026.jpg1125 can be generated. The accuracy of matching at this time is determined by the residual error This can be seen through JPEG0007813503000027.jpg855.

[0081] Next, the image alignment device 100 can perform alignment between the 2D image 10 and the plan view 20 (S300) by projecting each pixel of the 2D image 10 onto the plan view 20 with reference to alignment information 160 including the transformation matrix T. The following description will be given with reference to FIGS. 6 and 7.

[0082] First, FIG. 6 is a diagram schematically illustrating an example of a process for projecting each pixel of a 2D image onto a plan view according to an embodiment of the present invention.

[0083] 3 and 6 , when the coordinate values ​​of the image 3D coordinate system 11 for each pixel of the 2D image 10 are projected onto the plan view 3D coordinate system 21 by referring to the transformation matrix T as the alignment information 160 generated by the above process, the image 3D coordinate system 11 is first generated for each pixel of the 2D image 10. When the 3D coordinate values ​​of the image 3D coordinate system 11 are projected onto the plan view 3D coordinate system 21, the scale of the image 3D coordinate system 11 is corrected for each pixel of the 2D image 10 by referring to the ratio between the resolution of the camera that captured the 2D image 10 and the resolution of the 2D image 10, and the principal point of the camera is corrected, and then a unit direction vector can be calculated. By referring to this, alignment between the 2D image 10 and the plan view 20 can be performed. Here, the unit direction vector may be a vector indicating the direction of a camera ray when the camera ray is emitted from the center of the camera toward an arbitrary pixel, and the magnitude (length) of the vector may be normalized to 1. In this regard, the following description will refer to arrow 710 in FIG.

[0084] For example, the original pixel (u, v) of the 2D image 10 (a round point on the 2D image 10) can be displayed as a round point on the image 3D coordinate system 11. In this case, the original pixel (u, v) is converted to the camera center coordinate (c x ,c y ) standard, and the image resolution ratio (w ratio ,h ratio) can be applied to calculate the corrected coordinates (u', v'). Here, the corrected coordinates are calculated using the following formula: It could be JPEG0007813503000028.jpg877.

[0085] Also, the corrected coordinates (u', v') are calculated using the focal length (f x ,f y ) to obtain a normalized 3D direction vector, and then use this as a reference to find the unit direction vector JPEG0007813503000029.jpg2551 can be obtained.

[0086] The 3D coordinate values ​​of the image 3D coordinate system 11 can be converted into coordinates on the plan view 3D coordinate system 21 by referring to the unit direction vector as the camera ray calculated in this way and the transformation matrix T corresponding to the matching information 160. In this regard, as the z coordinate of the center of the camera on the plan view 3D coordinate system 21, JPEG0007813503000030.jpg730 can be calculated as a unit direction vector on the plan view of the plan view 3D coordinate system 21. JPEG0007813503000031.jpg1023 can be calculated. c may be the camera center in the image 3D coordinate system 11, R may be a rotation matrix, and d c may be a unit direction vector on the image 3D coordinate system 11.

[0087] Next, referring to FIG. 7 below, a process of projecting the image 3D coordinate system 11 into the plan view 3D coordinate system 21 in consideration of the height information will be described.

[0088] FIG. 7 is a diagram illustrating an example of a process of calculating each of the tangent points intersecting with each of the unit direction vectors by referring to each of the height information of each of the pixels of a 2D image according to one embodiment of the present invention, determining a position on the plan view 3D coordinate system, and projecting the position onto the plan view.

[0089] 3 and 7, when projecting each pixel of the 2D image 10 onto the plan view with reference to the alignment information 160, the image alignment device 100 calculates normalized unit direction vectors corresponding to the camera rays (arrows) 710 of each pixel of the 2D image 10, and converts the camera center coordinates 720 and each unit direction vector of each pixel into the plan view 3D coordinate system 21 with reference to the alignment information 160, so that the distance can be accurately calculated by taking into account the direction corresponding to the camera rays (arrows) 710 extended from the camera center coordinates 720 of the camera.

[0090] Next, with reference to each height information of the plan view 3D coordinate system stored in advance, each point of intersection with each of the previously calculated camera rays (arrows) 710 is calculated, and this is referenced to determine the position of the pixel on the plan view 3D coordinate system 21, thereby generating the 2D image-plan view matching result 170.

[0091] To explain in more detail in relation to the above, the camera center coordinates 720 of the camera on the image 3D coordinate system 11 calculated above and each unit direction vector of the pixel are transformed into the plan view 3D coordinate system 21, and the camera center coordinates on the plan view and the unit direction vector on the plan view are calculated, and the height h is referenced on the plan view 3D coordinate system 21 to find the intersecting point P b can be calculated by the following formula:

[0092] First, the coordinates of the camera center on the floor plan are calculated as o b and d, which is a unit direction vector on the plane. b As a distance ratio to the intersection point of the plan view 3D coordinate system 21, JPEG0007813503000032.jpg1021 can be calculated.

[0093] Also, referring to the distance ratio, the camera center coordinates on the plan view are used as the starting point, and the camera ray extended according to the unit direction vector on the plan view intersects with the By calculating JPEG0007813503000033.jpg735, the (x, y) value on the plan view 3D coordinate system 21 can be the final position of the object projected onto the plan view. At this time, the camera ray on the plan view is extended until it reaches the height h on the plan view 3D coordinate system 21, and P b can be calculated, b may be the position vector of the camera center transformed into the plan view 3D coordinate system 21, and d b may be a normalized unit direction vector transformed into the plan view 3D coordinate system 21, JPEG0007813503000034.jpg76 may be a scalar value representing the distance to the view point where the camera ray intersects with height h on the floor plan.

[0094] In other cases, if it is not possible to calculate a predetermined height for a given object on the plan view 3D coordinate system 21, the image alignment device 100 can also estimate the predetermined height of the given object on the plan view 3D coordinate system by referring to each of the other pixels of other objects on the 2D image 10 that are not the given object.

[0095] To explain this in more detail, the height can be estimated by referring to pre-stored height information of other objects, but it is also possible to obtain height information of other objects by referring to the intersection obtained from the camera ray of the floor pixel corresponding to the floor plane of the other object and the unit direction vector of the camera ray of the top pixel located at the topmost position of the other object, and then refer to this to estimate the specified height of a specified object on the plan view 3D coordinate system.

[0096] Also, since the intersection point based on the floor pixels has already been obtained from other objects, it is possible to further obtain the top pixel of a given object to estimate the given height.

[0097] To explain this in more detail, even if a predetermined object is hidden by another object other than the predetermined object or if a part of the predetermined object moves outside the camera, the height of the predetermined object can be estimated. First, through the process described above, the distance ratio, which is the position on the plane view relative to the object, is calculated. JPEG0007813503000035.jpg1255 can be calculated, and the intersection point (x t , y t ) and a given height of the given object on the plan view 3D coordinate system 21, with reference to a unit direction vector obtained from the camera ray of the top pixel located at the top of the given object. JPEG0007813503000036.jpg730 can be calculated on the assumption that the other objects are shorter in height than the given object.

[0098] The above-described embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer components and stored on a computer-readable storage medium. The computer-readable storage medium may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions stored on the computer-readable storage medium may be specially designed and constructed for the present invention, or may be well-known and available to those skilled in the art of computer software. Examples of computer-readable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine code, such as produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices may be configured to operate as one or more software modules to perform processes according to the present invention, or vice versa.

[0099] Although the present invention has been described above using specific details such as concrete components and limited examples and drawings, these are merely provided to facilitate a more comprehensive understanding of the present invention, and the present invention is not limited to the above examples. Those skilled in the art will be able to make various modifications and variations from such descriptions.

[0100] Therefore, the spirit of the present invention should not be limited to the embodiments described above, and not only the scope of the claims below, but also all modifications equivalent to or equivalent to the scope of these claims shall fall within the scope of the spirit of the present invention.

Claims

1. 1. A method for aligning a 2D image and a floor plan based on 3D geometry, comprising: (a) when a 2D image is acquired through a camera installed in a specific space, an image matching device generates an image 3D coordinate system by extending the 2D image into a 3D space with reference to 3D coordinate values ​​for each pixel of the 2D image inferred through a neural network-based geometry estimation model, and generates a plan view 3D coordinate system by extending the 2D image into 3D space with a plane of a plan view for the specific space as a floor plane; (b) the image alignment device generates alignment information between the image 3D coordinate system and the plan view 3D coordinate system by referring to plane correspondence information between a first plane recognized in the image 3D coordinate system and a second plane in the plan view 3D coordinate system corresponding to the first plane, and line correspondence information between a 3D point in the image 3D coordinate system and a 2D point in the plan view 3D coordinate system corresponding to the 3D point; (c) the image alignment device performs alignment between the 2D image and the plan view by projecting each of the pixels of the 2D image onto the plan view with reference to the alignment information; A method comprising:

2. In the step (b), 2. The method of claim 1, wherein the image matching device acquires, as the plane correspondence information, a 1_1 plane normal vector and a 1_1 plane constant corresponding to a 1_1 plane included in the first plane, a 1_n plane normal vector and a 1_n plane constant corresponding to a 1_1 plane through a 1_n plane, and a 2_1 plane normal vector and a 2_1 plane constant corresponding to a 2_1 plane included in the second plane, and a 2_n plane normal vector and a 2_n plane constant corresponding to a 2_1 plane through a 2_n plane, and

3. In the step (b), When generating the matching information between the image 3D coordinate system and the plan view 3D coordinate system, the image matching device refers to the plane correspondence information and the line correspondence information to estimate a rotation matrix between the image 3D coordinate system and the plan view 3D coordinate system, then refers to the rotation matrix to estimate a size and a movement, and generates a transformation matrix by referring to the size and the movement, thereby generating the matching information between the image 3D coordinate system and the plan view 3D coordinate system, 2. The method of claim 1, wherein the accuracy of the alignment information is confirmed by calculating a residual of the alignment information (the accuracy indicates a degree of agreement between a transformation result of the 3D points in the image 3D coordinate system into the plan view 3D coordinate system with reference to the alignment information and the corresponding 2D points in the plan view 3D coordinate system).

4. In the step (b), In a state where the image alignment device refers to the number of plane pairs of corresponding planes for each of the 1_1 plane through 1_n planes included in the first plane and each of the 2_1 plane through 2_n planes included in the second plane, (i) when the number of plane pairs is equal to or greater than a preset threshold, calculates a 1_1 plane normal vector through a 1_n plane normal vector as a first plane normal vector for each of the 1_1 plane through 1_n plane, calculates a 2_1 plane normal vector through a 2_n plane normal vector as a second plane normal vector for each of the 2_1 plane through 2_n plane, and calculates a 1_1 plane normal vector through a 1_n plane normal vector and a 2_n plane normal vector as a second plane normal vector for each of the 2_1 plane through 2_n plane.

4. The method of claim 3, characterized in that it executes one of the following sub-processes: (i) a sub-process of calculating a first degree of agreement for each of the corresponding 2_1 plane normal vectors to the 2_n plane normal vectors, and estimating a first rotation matrix as the rotation matrix that maximizes the first degree of agreement; and (ii) when the number of plane pairs is less than the predetermined threshold, a sub-process of calculating a first floor normal vector of a first floor plane that is a floor plane in the image 3D coordinate system, and a second floor normal vector of a second floor plane that is a floor plane in the plan view 3D coordinate system corresponding thereto, and calculating a second degree of agreement for the first floor normal vector and the corresponding second floor normal vector, and estimating a second rotation matrix that maximizes the second degree of agreement.

5. In the step (i) of the step (b), The image matching device defines a correlation matrix between each pair of the 1_1 plane normal vector through the 1_n plane normal vector and each pair of the 2_1 plane normal vector through the 2_n plane normal vector corresponding thereto, and performs Singular Value Decomposition (SVD) with reference to the correlation matrix to calculate the first rotation matrix as an optimal solution in which the first degree of agreement satisfies a preset condition, and at that time, 5. The method of claim 4, wherein the combination of signs of each of the 1_1 plane normal vectors through the 1_n plane normal vectors and the corresponding 2_1 plane normal vectors through the 2_n plane normal vectors is repeatedly performed by brute force to extract the optimal solution.

6. In the step (b), 5. The method of claim 4, wherein the image alignment device refers to the first floor normal vector and the second floor normal vector, calculates an initial rotation matrix as the rotation matrix so that a direction difference between the first floor normal vector and the second floor normal vector is minimized, generates projected coordinates by projecting the 3D coordinate values ​​of the 3D points in the image 3D coordinate system rotated by the initial rotation matrix onto 2D coordinate values ​​of the second floor plane in the plan view 3D coordinate system, calculates a 2D rotation matrix as a rotation matrix within the second floor plane by referring to the line correspondence information and a 2D UMEYAMA model between the projected coordinates and the 2D coordinate values ​​of the 2D points in the plan view 3D coordinate system, calculates a 3D rotation matrix by expanding the 2D rotation matrix to 3D, and calculates the second rotation matrix as a final rotation matrix by combining the initial rotation matrix and the 3D rotation matrix.

7. In the step (b), 4. The method of claim 3, wherein after estimating the rotation matrix, the image alignment device generates the transformation matrix by referring to the plane correspondence information and the line correspondence information, since a plane equation for making the first plane correspond to the second plane must be maintained even after performing the alignment between the 2D image and the floor plan, and generates a plane constraint equation, and a line constraint equation for matching the image 3D coordinate system with the 2D points of the floor plan 3D coordinate system through the rotation matrix, and estimates the magnitude and the movement by a linear least squares method by referring to the plane constraint equation and the line constraint equation, thereby generating the transformation matrix between the image 3D coordinate system and the floor plan 3D coordinate system.

8. In the step (c), When projecting each of the pixels of the 2D image onto the plan view with reference to the alignment information, the image alignment device calculates each of normalized unit direction vectors corresponding to each of the pixels of the 2D image, and converts the camera center coordinates of the camera and each of the unit direction vectors of the pixels into the plan view 3D coordinate system with reference to the alignment information, and projects them onto the plan view, 2. The method of claim 1, wherein height information of each of the pixels of the 2D image is referenced to calculate points of intersection with each of the unit direction vectors, and positions on the plan view 3D coordinate system are determined and projected onto the plan view.

9. When it is not possible to calculate a predetermined height of a predetermined object on the plan view 3D coordinate system, the image alignment device refers to each of other pixels of other objects other than the predetermined object on the 2D image to obtain a height of the predetermined object on the plan view 3D coordinate system, and 9. The method of claim 8, wherein the predetermined height of the predetermined object on the plan view 3D coordinate system is acquired by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to a floor plane of the other object among the other pixels on the 2D image and a unit direction vector of a camera ray of a top pixel located at the top of the other object, or, if the height of the predetermined object is higher than that of the other object, the predetermined height is acquired by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to a floor plane of the other object and a unit direction vector of a camera ray of a top pixel located at the top of the predetermined object.

10. 9. The method of claim 8, wherein, when calculating each of the normalized unit direction vectors corresponding to each of the pixels of the 2D image, the image alignment device corrects the scale of the image 3D coordinate system and the principal point of the camera by referring to the ratio between the resolution of the camera and the resolution of the 2D image, and calculates each of the unit direction vectors.

11. 1. An image alignment apparatus for aligning a 2D image and a plan view based on 3D geometry, comprising: at least one memory for storing instructions; at least one processor configured to execute the instructions; The processor (I) acquires a 2D image taken through a camera installed in a specific space, and then calculates the image by a neural network-based geometry estimation model. and (III) a process of performing alignment between the 2D image and the floor plan by projecting each of the pixels of the 2D image onto the floor plan with reference to the alignment information.

12. In the process (II), 12. The image alignment device of claim 11, wherein the processor acquires, as the plane correspondence information, a 1_1 plane normal vector and a 1_1 plane constant corresponding to a 1_1 plane included in the first plane, and a 1_n plane normal vector and a 1_n plane constant corresponding to a 1_1 plane through a 1_n plane included in the first plane, and a 2_1 plane normal vector and a 2_1 plane constant corresponding to a 2_1 plane included in the second plane, and a 2_n plane normal vector and a 2_n plane constant corresponding to a 2_1 plane through a 2_n plane included in the second plane, and acquires, as the line correspondence information, each of the perpendiculars of the 2D points in the plan view 3D coordinate system (the perpendiculars are lines perpendicular to a floor plane in the plan view 3D coordinate system) and each of the corresponding 3D points in the image 3D coordinate system.

13. In the process (II), In generating the matching information between the image 3D coordinate system and the plan view 3D coordinate system, the processor refers to the plane correspondence information and the line correspondence information to estimate a rotation matrix between the image 3D coordinate system and the plan view 3D coordinate system, then estimates a size and a movement by referring to the rotation matrix, and generates a transformation matrix by referring to the size and the movement, thereby generating the matching information between the image 3D coordinate system and the plan view 3D coordinate system; 12. The image alignment device of claim 11, wherein the accuracy of the alignment information is confirmed by calculating a residual of the alignment information (the accuracy indicates a degree of evaluation of a degree of agreement between a transformation result, obtained by transforming the 3D points in the image 3D coordinate system into the plan view 3D coordinate system with reference to the alignment information, and the corresponding 2D points in the plan view 3D coordinate system).

14. In the process (II), In a state where the processor refers to the number of plane pairs of corresponding planes for each of the 1_1 plane through 1_n planes included in the first plane and each of the 2_1 plane through 2_n planes included in the second plane, (i) when the number of plane pairs is equal to or greater than a preset threshold, calculates a 1_1 plane normal vector through a 1_n plane normal vector as a first plane normal vector for each of the 1_1 plane through 1_n plane, calculates a 2_1 plane normal vector through a 2_n plane normal vector as a second plane normal vector for each of the 2_1 plane through 2_n plane, and calculates each of the 1_1 plane normal vector through the 1_n plane normal vector and the corresponding and (ii) if the number of plane pairs is less than the predetermined threshold, a subprocess of calculating a first floor normal vector of a first floor plane that is a floor plane in the image 3D coordinate system and a second floor normal vector of a second floor plane that is a floor plane in the plan view 3D coordinate system corresponding thereto, calculating a second degree of agreement for the first floor normal vector and the corresponding second floor normal vector, and estimating a second rotation matrix that maximizes the second degree of agreement.

15. In the (i) of the process (II), The processor defines a correlation matrix between each pair of the 1_1 plane normal vector through the 1_n plane normal vector and each pair of the 2_1 plane normal vector through the 2_n plane normal vector corresponding thereto, and performs Singular Value Decomposition (SVD) with reference to the correlation matrix to calculate the first rotation matrix as an optimal solution in which the first degree of agreement satisfies a preset condition, and at that time, 15. The image matching device of claim 14, wherein the combinations of signs of the 1_1 plane normal vector through the 1_n plane normal vector and the corresponding 2_1 plane normal vector through the 2_n plane normal vector are extracted by repeatedly performing the SVD using brute force.

16. In the step (ii) of the process (II), 15. The image alignment device of claim 14, wherein the processor refers to the first floor normal vector and the second floor normal vector, calculates an initial rotation matrix as the rotation matrix so that a direction difference between the first floor normal vector and the second floor normal vector is minimized, generates projected coordinates by projecting the 3D coordinate values ​​of the 3D points in the image 3D coordinate system rotated by the initial rotation matrix onto 2D coordinate values ​​of the second floor plane in the plan view 3D coordinate system, calculates a 2D rotation matrix as a rotation matrix within the second floor plane by referring to the line correspondence information and a 2D UMEYAMA model between the projected coordinates and the 2D coordinate values ​​of the 2D points in the plan view 3D coordinate system, calculates a 3D rotation matrix by expanding the 2D rotation matrix to 3D, and calculates the second rotation matrix as a final rotation matrix by combining the initial rotation matrix and the 3D rotation matrix.

17. In the process (II), 14. The image alignment device of claim 13, wherein after estimating the rotation matrix, the processor generates the transformation matrix by referring to the plane correspondence information and the line correspondence information, since a plane equation for making the first plane correspond to the second plane must be maintained even after performing the alignment between the 2D image and the plan view, and generates a plane constraint equation, and a line constraint equation for matching the image 3D coordinate system with the 2D points of the plan view 3D coordinate system through the rotation matrix, and estimates the magnitude and the movement by a linear least squares method by referring to the plane constraint equation and the line constraint equation, thereby generating the transformation matrix between the image 3D coordinate system and the plan view 3D coordinate system.

18. In the process (III), When projecting each of the pixels of the 2D image onto the plan view with reference to the alignment information, the processor calculates each of normalized unit direction vectors corresponding to each of the pixels of the 2D image, and converts the camera center coordinates of the camera and each of the unit direction vectors of each of the pixels into the plan view 3D coordinate system with reference to the alignment information, and projects them onto the plan view, The image alignment device according to claim 11, characterized in that, by referring to the height information of each of the pixels of the 2D image, each point of contact with each of the unit direction vectors is calculated, and a position on the plan view 3D coordinate system is determined and projected onto the plan view.

19. When it is not possible to calculate a predetermined height on the plan view 3D coordinate system for a predetermined object, the processor refers to each of other pixels of other objects other than the predetermined object on the 2D image to obtain a height of the predetermined object on the plan view 3D coordinate system, and at that time, 19. The image matching device of claim 18, wherein the predetermined height of the predetermined object on the plan view 3D coordinate system is acquired by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to a floor plane of the other object among the other pixels on the 2D image and a unit direction vector of a camera ray of a top pixel located at the top of the other object, or, if the height of the predetermined object is higher than that of the other object, the predetermined height is acquired by referring to an intersection point obtained from a camera ray of a floor pixel corresponding to a floor plane of the other object and a unit direction vector of a camera ray of a top pixel located at the top of the predetermined object.

20. 19. The image alignment device of claim 18, wherein when calculating each of the normalized unit direction vectors corresponding to each pixel of the 2D image, the processor refers to a ratio between the resolution of the camera and the resolution of the 2D image to correct a scale of the image 3D coordinate system and a principal point of the camera, and calculates each of the unit direction vectors.

Images